Run Flows with Docker

Learn how to store your flow code in a Docker image and serve your flow on any docker-compatible infrastructure.

The Deployments guide showed how to serve a flow on a schedule.

With a Python script, you can build a Docker image for that script, allowing you to serve your flow This guide demonstrates running a flow in Kubernetes, but you can use any Docker-compatible infrastructure.

This guide covers how to:

• Write a Dockerfile to build an image that stores your Prefect flow code.
• Build a Docker image for your flow.
• Deploy and run your Docker image on a Kubernetes cluster.
• Look at the Prefect-maintained Docker images and discuss options for use.

In this guide you will create a Dockerfile from scratch. Alternatively, Prefect makes it convenient to build a Docker image as part of deployment creation. You can even include environment variables and specify additional Python packages to install at runtime.

If creating a deployment with a prefect.yaml file, the build step makes it easy to customize your Docker image and push it to the registry of your choice. See an example.

Deployment creation with a Python script that includes flow.deploy similarly allows you to customize your Docker image with keyword arguments as shown below.

...

if __name__ == "__main__":
    hello_world.deploy(
        name="my-first-deployment",
        work_pool_name="above-ground",
        image='my_registry/hello_world:demo',
        job_variables={"env": { "EXTRA_PIP_PACKAGES": "boto3" } }
    )

Prerequisites

• A Python script that defines and serves a flow. This guide uses the flow script and deployment from the Deployments guide.
• Access to a running Prefect API server. Sign up for a forever free Prefect Cloud account, or run a Prefect API server locally with prefect server start.
• Docker Desktop installed on your machine.

Write a Dockerfile

Make a clean directory to work from, called prefect-docker-guide.

mkdir prefect-docker-guide
cd prefect-docker-guide

In this directory, create a sub-directory named flows and put your flow script from the Deployments guide in it.

mkdir flows
cd flows
touch prefect-docker-guide-flow.py

Here’s the flow code for reference:

prefect-docker-guide-flow.py

import httpx
from prefect import flow


@flow(log_prints=True)
def get_repo_info(repo_name: str = "PrefectHQ/prefect"):
    url = f"https://api.github.com/repos/{repo_name}"
    response = httpx.get(url)
    response.raise_for_status()
    repo = response.json()
    print(f"{repo_name} repository statistics 🤓:")
    print(f"Stars 🌠 : {repo['stargazers_count']}")
    print(f"Forks 🍴 : {repo['forks_count']}")


if __name__ == "__main__":
    get_repo_info.serve(name="prefect-docker-guide")

Next, add a requirements.txt to the prefect-docker-guide directory. Include all dependencies required for your prefect-docker-guide-flow.py script in the Docker image to build.

# ensure you run this line from the top level of the `prefect-docker-guide` directory
touch requirements.txt

Put the following in your requirements.txt file:

requirements.txt

prefect>=3.0.0
httpx

Create a Dockerfile to create a Docker image that also stores the flow code:

touch Dockerfile

Add the following content to your Dockerfile:

Dockerfile

# Using the latest version of Prefect with Python 3.10
FROM prefecthq/prefect:2-python3.10

# Add your requirements.txt file to the image and install dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt --trusted-host pypi.python.org --no-cache-dir

# Add your flow code to the image
COPY flows /opt/prefect/flows

# Run your flow script when the container starts
CMD ["python", "flows/prefect-docker-guide-flow.py"]

Build a Docker image

Now that you have a Dockerfile, build your image by running:

docker build -t prefect-docker-guide-image .

Check that your build worked by running a container from your new image.

Cloud

Your container needs an API URL and an API key to communicate with Prefect Cloud.

• You can get an API key from the API Keys section of the user settings in the Prefect UI.

• You can get your API URL by running prefect config view and copying the PREFECT_API_URL value.

Provide both these values to your container by passing them as environment variables with the -e flag.

docker run -e PREFECT_API_URL=YOUR_PREFECT_API_URL -e PREFECT_API_KEY=YOUR_API_KEY prefect-docker-guide-image

After running the above command, the container should start up and serve the flow within the container.

Self-hosted

Your container needs an API URL and network access to communicate with the Prefect API.

For this guide, assume the Prefect API is running on the same machine that you will run your container on and the Prefect API was started with prefect server start. If you’re running a different setup, check out the Hosting a Prefect server guide for information on how to connect to your Prefect API instance.

To ensure that your flow container can communicate with the Prefect API, set your PREFECT_API_URL to http://host.docker.internal:4200/api. If you’re running Linux, set your PREFECT_API_URL to http://localhost:4200/api and use the --network="host" option instead.

docker run --network="host" -e PREFECT_API_URL=http://host.docker.internal:4200/api prefect-docker-guide-image

After running the above command, the container should start and the flow should run within the container.

Deploy to a remote environment

Now that you have a Docker image with your flow code embedded, you can deploy it to a remote environment.

This guide simulates a remote environment by using Kubernetes locally with Docker Desktop. Use the instructions provided by Docker to set up Kubernetes locally.

Create a Kubernetes deployment manifest

To ensure the process serving your flow is always running, create a Kubernetes deployment. If your flow’s container crashes, Kubernetes automatically restarts it, ensuring you don’t miss any scheduled runs.

First, create a deployment-manifest.yaml file in your prefect-docker-guide directory:

touch deployment-manifest.yaml

Add the following content to your deployment-manifest.yaml file:

Cloud

deployment-manifest.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-docker-guide
spec:
  replicas: 1
  selector:
    matchLabels:
      flow: get-repo-info
  template:
    metadata:
      labels:
        flow: get-repo-info
    spec:
      containers:
      - name: flow-container
        image: prefect-docker-guide-image:latest
        env:
        - name: PREFECT_API_URL
          value: YOUR_PREFECT_API_URL
        - name: PREFECT_API_KEY
          value: YOUR_API_KEY
        # Never pull the image because we're using a local image
        imagePullPolicy: Never

Keep your API key secret

In the above manifest, you are passing the Prefect API URL and API key as environment variables. This approach is simple, but not secure. If you are deploying your flow to a remote cluster, use a Kubernetes secret to store your API key.

Self-hosted

deployment-manifest.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-docker-guide
spec:
  replicas: 1
  selector:
    matchLabels:
      flow: get-repo-info
  template:
    metadata:
      labels:
        flow: get-repo-info
    spec:
      containers:
      - name: flow-container
        image: prefect-docker-guide-image:latest
        env:
        - name: PREFECT_API_URL
          value: <http://host.docker.internal:4200/api>
        # Never pull the image because we're using a local image
        imagePullPolicy: Never

Linux users

If you’re running Linux, set your PREFECT_API_URL to use the IP address of your machine instead of host.docker.internal.

This manifest defines how your image runs when deployed in your Kubernetes cluster. Note that you will run a single replica of your flow container. If you want to run multiple replicas of your flow container to keep up with an active schedule, or because your flow is resource-intensive, you can increase the replicas value.

Deploy your flow to the cluster

You now have a deployment manifest. To deploy your flow to the cluster, use:

kubectl apply -f deployment-manifest.yaml

To monitor the status of your Kubernetes deployment, use:

kubectl get deployments

Once the deployment successfully starts, check the logs of your flow container by running the following:

kubectl logs -l flow=get-repo-info

You’re now serving your flow in your cluster. To trigger a flow run, use:

prefect deployment run get-repo-info/prefect-docker-guide

Navigate to the URL provided by the prefect deployment run command and follow the flow run through the logs in the Prefect UI.

Prefect-maintained Docker images

Every release of Prefect results in several new Docker images. These images are all named prefecthq/prefect and their tags identify their differences.

Image tags

When a release is published, images are built for all of Prefect’s supported Python versions. These images are tagged to identify the combination of Prefect and Python versions contained. Additionally, we have “convenience” tags which are updated with each release to facilitate automatic updates.

For example, when release 3.1.1 is published:

1. Images with the release packaged are built for each supported Python version (3.9, 3.10, 3.11, 3.12) with both standard Python and Conda.
2. These images are tagged with the full description. For example, prefect:3.1.1-python3.10 and prefect:3.1.1-python3.10-conda.
3. For users that want more specific pins, these images are also tagged with the SHA of the git commit of the release. For example, sha-88a7ff17a3435ec33c95c0323b8f05d7b9f3f6d2-python3.10.
4. To be on the latest 3.1.x release and receiving patch updates, we update a tag without the patch version to this release. For example, prefect.3.1-python3.10.
5. To be on the latest 3.x.y release and receiving minor version updates, we update a tag without the minor or patch version to this release. For example, prefect.3-python3.10.
6. For users who want the latest 3.x.y release without specifying a Python version, we update 3-latest to the image for our highest supported Python version, which in this case is equivalent to prefect:3.1.1-python3.10.

Choose image versions carefully

It’s a good practice to use Docker images with specific Prefect versions in production.

Be careful when employing images that automatically update to new versions (such as prefecthq/prefect:2-python3.11 or prefecthq/prefect:3-latest).

Standard Python

Standard Python images are based on the official Python slim images, such as python:3.10-slim.

Tag	Prefect Version	Python Version
3-latest	most recent v2 PyPi version	3.10
3-python3.12	most recent v2 PyPi version	3.12
3-python3.11	most recent v2 PyPi version	3.11
3-python3.10	most recent v2 PyPi version	3.10
3-python3.9	most recent v2 PyPi version	3.9
3.X-python3.12	2.X	3.12
3.X-python3.11	2.X	3.11
3.X-python3.10	2.X	3.10
3.X-python3.9	2.X	3.9
sha-<hash>-python3.12	<hash>	3.12
sha-<hash>-python3.11	<hash>	3.11
sha-<hash>-python3.10	<hash>	3.10
sha-<hash>-python3.9	<hash>	3.9

Conda-flavored Python

Conda-flavored images are based on continuumio/miniconda3. Prefect is installed into a conda environment named prefect.

Tag	Prefect Version	Python Version
3-latest-conda	most recent v3 PyPi version	3.10
3-python3.12-conda	most recent v3 PyPi version	3.12
3-python3.11-conda	most recent v3 PyPi version	3.11
3-python3.10-conda	most recent v3 PyPi version	3.10
3-python3.9-conda	most recent v3 PyPi version	3.9
3.X-python3.12-conda	3.X	3.12
3.X-python3.11-conda	3.X	3.11
3.X-python3.10-conda	3.X	3.10
3.X-python3.9-conda	3.X	3.9
sha-<hash>-python3.12-conda	<hash>	3.12
sha-<hash>-python3.11-conda	<hash>	3.11
sha-<hash>-python3.10-conda	<hash>	3.10
sha-<hash>-python3.9-conda	<hash>	3.9

Build your own image

If your flow relies on dependencies not found in the default prefecthq/prefect images, you may want to build your own image. You can either base it off of one of the provided prefecthq/prefect images, or build your own image. See the Work pool deployment guide to build custom images with dependencies specified in a requirements.txt file.

By default, Prefect work pools that use containers refer to the 3-latest image. You can specify another image at work pool creation. You can override the work pool image choice in individual deployments.

Extend the prefecthq/prefect image manually

Here is an example Dockerfile for building an image based on prefecthq/prefect:2-latest, but with scikit-learn installed:

FROM prefecthq/prefect:3-latest

RUN pip install scikit-learn

Choose an image strategy

The options described above have different complexity and performance characteristics. We recommend the following when choosing a strategy:

• If your flow only makes use of tasks defined in the same file as the flow, or tasks that are part of prefect itself, you can rely on the default provided prefecthq/prefect image.

• If your flow requires a few extra dependencies found on PyPI, you can use the default prefecthq/prefect image and specify additional packages to install at runtime when you create a deployment or a work pool.

• If the installation process requires compiling code or other expensive operations, consider building a custom image instead.

• If your flow requires extra dependencies or shared libraries, we recommend building a shared custom image with all the extra dependencies and shared task definitions you need. Your flows can all rely on the same image, but have their source stored externally. This option eases development, since the shared image only needs to be rebuilt when dependencies change, not when a flow’s code changes.

Next steps

• You can extend this setup to serve multiple flows in a single Docker image by updating your Python script to use flow.to_deployment and serve to serve multiple flows or the same flow with different configuration.
• To learn more about deploying flows, check out the Deployments concept doc.
• For advanced infrastructure requirements, such as executing each flow run within its own dedicated Docker container, learn more in the Work pool deployment guide.