- Flow entrypoint: path to your flow function
- Schedule or Trigger: optional schedule or triggering rules for this deployment
- Tags: optional text labels for organizing deployments
- trigger new runs, cancel active runs, pause scheduled runs, customize parameters, and more
- remotely configure schedules and automation rules for your deployments
- dynamically provision infrastructure using workers
Connect to Prefect Cloud or a self-hosted Prefect server instanceSome features in this tutorial, such as scheduling, require you to connect to a Prefect server.
If using a self-hosted setup, run
prefect server start
to run both the webserver and UI.
If using Prefect Cloud, make sure you have
successfully authenticated your local environment.Create a deployment
Here’s an example of creating a deployment by calling a single method on the flow object:flow.serve
.
repo_info.py
- create a deployment called
my-first-deployment
for your flow in the Prefect API - continue running to listen for flow runs for this deployment; when a run is found, it
asynchronously executes within a subprocess
You must define deployments in static filesYou can define and run flows interactively, within REPLs or Notebooks.
However, deployments require that your flow definition exist in a known file.
Additional options
Theserve
method on flows exposes many options for the deployment. For more information, see
Serve a flow
Here’s how to use some of those options:
cron
: a keyword that allows you to set a cron string schedule for the deployment; see schedules for more advanced scheduling optionstags
: a keyword that allows you to tag this deployment and its runs for bookkeeping and filtering purposesdescription
: a keyword that allows you to document what this deployment does; by default the description is set from the docstring of the flow function (if documented)version
: a keyword that allows you to track changes to your deployment; uses a hash of the file containing the flow by default; popular options include semver tags or git commit hashes
Stop the script in the CLI using
CTRL+C
and your schedule automatically pauses.
.serve
is a long-running processTo execute remotely triggered or scheduled runs, your script with flow.serve
must be actively running.Run multiple deployments at once
This method is useful for creating deployments for single flows. For two or more flows, you must provide a few additional method calls and imports:multi_flow_deployment.py
- the
flow.to_deployment
interface exposes the exact same options asflow.serve
; this method produces a deployment object - the deployments are only registered with the API once
serve(...)
is called - when serving multiple deployments, the only requirement is that they share a Python environment; they can be executed and scheduled independently of each other
- pause and unpause the schedule for the “sleeper” deployment
- use the UI to submit ad-hoc runs for the “sleeper” deployment with different values for
sleep
- cancel an active run for the “sleeper” deployment from the UI
Hybrid execution optionPrefect’s deployment interface allows you to choose a hybrid execution model.
Whether you use Prefect Cloud or host a Prefect server instance, you can run workflows in the
environments best suited to their execution.
This model enables efficient use of your infrastructure resources while maintaining the privacy
of your code and data.
There is no ingress required.
Read more about our hybrid model.
Deployment schema
Here’s the complete schema that defines a deployment:Required data
Deployments universally require aname
and a reference to an underlying Flow
.
The deployment name is not required to be unique across all deployments, but is required to be unique
for a given flow ID. This means you will often see references to the deployment’s unique identifying name
{FLOW_NAME}/{DEPLOYMENT_NAME}
.
For example, triggering a run of a deployment from the Prefect CLI can be done with:
path
: think of the path as the runtime working directory for the flow. For example, if a deployment references a workflow defined within a Docker image, thepath
is the
absolute path to the parent directory where that workflow will run anytime the deployment is triggered. This interpretation is more subtle in the case of flows defined in remote filesystems.entrypoint
: the entrypoint of a deployment is a relative reference to a function decorated as a flow that exists on some filesystem. It is always specified relative to thepath
. Entrypoints use Python’s standard path-to-object syntax (for example,path/to/file.py:function_name
or simplypath:object
).
Deployments do not contain code definitionsDeployment metadata references code that exists in potentially diverse locations within your environment.
This separation means that your flow code stays within your storage and execution
infrastructure, and never lives on the Prefect server or database.This is key to the Prefect hybrid model: there’s a boundary between your proprietary assets,
such as your flow code, and the Prefect backend (including Prefect Cloud).
Scheduling and parametrization
One of the primary motivations for creating deployments of flows is to remotely schedule and trigger them. Just as you can call flows as functions with different input values, deployments can be triggered or scheduled with different values through parameters. These are the fields to capture the required metadata for those actions:schedules
: a list of schedule objects. Most of the convenient interfaces for creating deployments allow users to avoid creating this object themselves. For example, when updating a deployment schedule in the UI basic information such as a cron string or interval is all that’s required.trigger
(Cloud-only): triggers allow you to define event-based rules for running a deployment. For more information see Automations.parameter_openapi_schema
: an OpenAPI compatible schema that defines the types and defaults for the flow’s parameters. This is used by the UI and the backend to expose options for creating manual runs as well as type validation.parameters
: default values of flow parameters that this deployment will pass on each run. These can be overwritten through a trigger or when manually creating a custom run.enforce_parameter_schema
: a boolean flag that determines whether the API should validate the parameters passed to a flow run against the schema defined byparameter_openapi_schema
.
Scheduling is asynchronous and decoupledPausing a schedule, updating your deployment, and other actions reset your auto-scheduled runs.
Run a deployed flow from within Python flow code
Prefect provides arun_deployment
function
to schedule the run of an existing deployment when your Python code executes.
Run a deployment without blockingBy default,
run_deployment
blocks until the scheduled flow run finishes
executing. Pass timeout=0
to return immediately and not block.run_deployment
from within a flow or task, the scheduled flow
run is linked to the calling flow run (or the calling task’s flow run)
as a subflow run by default.
Subflow runs have different behavior than regular flow runs. For example, you can’t suspend a
subflow run independently of its parent flow. To not link the scheduled flow run to the
calling flow or task run, pass as_subflow=False
to disable it:
run_deployment
is a FlowRun
object containing metadata about the scheduled run. Use this object to retrieve information
about the run after calling
run_deployment
:
Using the Prefect clientFor more information on using the Prefect client to interact with Prefect’s
REST API, see Interact with the API.
Versioning and bookkeeping
Here is important information on the versions, descriptions, and tags fields:version
: versions are always set by the client and can be any arbitrary string. We recommend tightly coupling this field on your deployments to your software development lifecycle. For example if you leveragegit
to manage code changes, use either a tag or commit hash in this field. If you don’t set a value for the version, Prefect will compute a hash.description
: provide reference material such as intended use and parameter documentation. Markdown is accepted. The docstring of your flow function is the default value.tags
: group related work together across a diverse set of objects. Tags set on a deployment are inherited by that deployment’s flow runs. Filter, customize views, and searching by tag.
Everything has a versionDeployments have a version attached; and flows and tasks also have
versions set through their respective decorators. These versions are sent to the API
anytime the flow or task runs, allowing you to audit changes.
Workers and work pools
Workers and work pools are an advanced deployment pattern that allow you to dynamically provision infrastructure for each flow run. The work pool job template interface allows users to create and govern opinionated interfaces to their workflow infrastructure. To do this, a deployment using workers needs the following fields:work_pool_name
: the name of the work pool this deployment is associated with. Work pool types mirror infrastructure types, which means this field impacts the options available for the other fields.work_queue_name
: if you are using work queues to either manage priority or concurrency, you can associate a deployment with a specific queue within a work pool using this field.infra_overrides
: often calledjob_variables
within various interfaces, this field allows deployment authors to customize whatever infrastructure options have been exposed on this work pool. This field is often used for Docker image names, Kubernetes annotations and limits, and environment variables.pull_steps
: a JSON description of steps that retrieves flow code or configuration, and prepares the runtime environment for workflow execution.
Serving flows on long-lived infrastructure
When you have several flows running regularly, theserve
method
of the Flow
object or the serve
utility
is a great option for managing multiple flows simultaneously.
Once you have authored your flow and decided on its deployment settings, run this long-running
process in a location of your choosing.
The process stays in communication with the Prefect API, monitoring for work and submitting each run
within an individual subprocess.
Because runs are submitted to subprocesses, any external infrastructure configuration
must be set up beforehand and kept associated with this process.
This approach has these benefits:
- Users are in complete control of their infrastructure, and anywhere the “serve” Python process can run is a suitable deployment environment.
- It is simple to reason about.
- Creating deployments requires a minimal set of decisions.
- Iteration speed is fast.
Dynamically provisioning infrastructure with work pools
Consider running flows on dynamically provisioned infrastructure with work pools under these circumstances:- Flows that have expensive infrastructure due to the long-running process.
- Flows with heterogeneous infrastructure needs across runs.
- Large volumes of deployments.
- If your internal team structure requires that deployment authors be members of a different team than the team managing infrastructure.
- You can configure and monitor infrastructure configuration within the Prefect UI.
- Infrastructure is ephemeral and dynamically provisioned.
- Prefect is more infrastructure-aware and collects more event data from your infrastructure by default.
- Highly decoupled setups are possible.
You don’t have to commit to one approachYou can mix and match approaches based on the needs of each flow. You can also change the
deployment approach for a particular flow as its needs evolve.
For example, you might use workers for your expensive machine learning pipelines,
but use the serve mechanics for smaller, more frequent file-processing pipelines.