Top 10 New Prefect Features of 2022
Python dataflow coordination enhancements you don’t want to miss!
In the five months since Prefect v2 was released to general availability, the Prefect team has published over 30 releases! While the release velocity is impressive, it can be hard to keep up with all this awesomeness, so I’m going to distill my ten favorite enhancements in this article. Let’s do this countdown style. 😎
10. Log prints 🪵
With Prefect you can bring your Python code, add a decorator to your main function, and that’s all you need to get started with orchestration. No need to rewrite large chunks of code as with legacy orchestrators. You also get to remove any boilerplate code that dealt with things such as retries and caching — because Prefect gives you those for free.
Prefect allows you to add logs to your code, but wouldn’t it be nice to just log print statements directly? Well, since 2.7.0 you can. Add the argument log_prints=True to your flow or task decorator and call print. Boom — now print statements also get logged to the Prefect backend and show up in the UI — where you and your collaborators can see them.
9. GitHub-based deployments 🫂
Version control is essential in modern software development. Since 2.3.0, you can leverage Prefect’s patented hybrid model to store your flow code on GitHub while executing your code in a different infrastructure.
Now there are GitLab and BitBucket integrations too!
8. Result configuration ⬇️
Results of flows and tasks were made nicely configurable in 2.6.0. Add persist_result=True to a flow or task and the results of a run are saved to disk. Results can be a big help with memory management for large datasets. You can specify the output type of the results — pickle or JSON — and you can even add compression.
7. Run flows from deployments 🏃
Release 2.5.0 introduced the run_deployment function to kick off deployment runs from flow runs. We call this a flow of deployments pattern, providing even more flexibility and looser coupling between flows. The ergonomics are 🔥!
6. CLI to UI login
Want to log in to Prefect Cloud from the command line? You no longer need to generate an API key explicitly and then use it to login (but you still can). 🔐 As of 2.6.9, prefect cloud login gives you options:
How would you like to authenticate? [Use arrows to move; enter to select]
> Log in with a web browser
Paste an API key
Opening browser…
Waiting for response…If you choose to use the browser option, you’ll see a new browser window that looks like this:
Ta-da, you’re logged in!
5. Recipes
Want a guide for setting up Prefect in your execution environment with ready-made ingredients such as Dockerfiles, Terraform files, and GitHub Actions? There’s a whole GitHub repo of popular recipes you can extend to fit your setup. Here are some examples from the catalog:
The recipes repo catalog was introduced into the docs in 2.4.0. Lots of potential Winter Jamboree (like Hacktoberfest, but in the winter) contributions are recipes. If you want to contribute, here is a board of recipes awaiting creation!
4. Python deployment definitions 🐍
At v2 general availability, you could create a deployment from the command line or directly through the API, but there wasn’t a nice interface for creating deployments in Python. Users asked for a straightforward way to use Python for deployments, and we delivered in version 2.3.0, with Deployment.build_from_flow. Check out the details in the docs or in my blog post.
3. The UI ✨
The front-end team has added numerous enhancements to the UI over the past five months. New features include manual flow runs retries, deployment pause and resume, and task run concurrency limits. Smaller UX improvements include improved movement options from one Prefect object to another, bulk deletion of flows and deployments, and many sort and filter options.
You can see important information at a glance and drill down to the specifics when needed.
There are many more UI enhancements headed your way in the new year!
2. Automations 🤖
Automations are actions in response to triggers. They were just released to public beta. Automations open up all kinds of observation and orchestration and observation possibilities.
For example, if a particular flow run crashes or stays in an undesirable state for too long, any of the following actions can be taken automatically:
- Cancel a flow run
- Pause a schedule
- Pause a work queue
- And soon — send a notification
Check out this PrefectLive video with Will Raphaelson to learn more. ➡️
1. Integrations 🔀
The modern data stack relies on a tapestry of different technologies. Prefect integrates with the major cloud providers, version control systems, notification providers, containerization systems, data ingestion tools, and parallel computing platforms. The Prefect Collections page features 39 integrations as of this writing — and most of which have been added since GA.
Community members can contribute collections — you can see Alessandro Lollo’s name next to several above. So if you would like to add a collection, check out the prefect-collection-template.
Wrap
It wasn’t easy to narrow this list down to the top ten. Here are a few honorable mentions:
- One click flow retries from the UI
- Apply and build a deployment in one step
- Colored log levels
- The ability to choose your color palette — great for accessibility
- Dark mode for the docs
- Many scheduling enhancements
See the complete release notes here. We can’t wait to see what new features 2023 brings! What other Prefect v2 enhancements floated your boat in 2022? ⛵️
Thanks to the community and the Prefect team for all the hard work to make v2 the powerful and enjoyable dataflow coordination platform it is today. And special thanks to Michael Adkins for the idea for this post (as well as lots of hard work 🙂).
If you want to stay updated on the latest action, join 23,000+ folks in the Prefect Community Slack.
Happy Engineering!
