Store flow code

Learn about where to store your flow code.

When a deployment runs, the execution environment needs access to the flow code. Flow code is not stored in a Prefect server instance or in Prefect Cloud.

You have several flow code storage options.

This guide focuses on deployments created with the interactive CLI experience or a prefect.yaml file. To create deployments with Python code, see the work pools and workers guide.

Option 1: Git-based storage

Git-based version control platforms provide redundancy, version control, and easier collaboration. Prefect supports:

• GitHub
• GitLab
• Bitbucket

Create a deployment with git-based storage

1. Run prefect deploy from the root directory of the git repository
2. Create a new deployment
3. You will see a series of prompts. Select yes creating a new deployment.
4. Select the flow code entrypoint
5. Name your deployment

Prefect detects that you are in a git repository and asks if you want to store your flow code in a git repository. Select “y” to be prompted to confirm the URL of your git repository and the branch name, as in the example below:

? Your Prefect workers will need access to this flow's code in order to run it. 
Would you like your workers to pull your flow code from its remote repository when running this flow? [y/n] (y): 
? Is https://github.com/my_username/my_repo.git the correct URL to pull your flow code from? [y/n] (y): 
? Is main the correct branch to pull your flow code from? [y/n] (y): 
? Is this a private repository? [y/n]: y

In this example, the git repository is hosted on GitHub. If you are using Bitbucket or GitLab, the URL will match your provider. If the repository is public, enter “n”.

If the repository is private, you can enter a token to access your private repository. This token is saved to an encrypted Prefect Secret block.

? Please enter a token that can be used to access your private repository. 
This token will be saved as a Secret block via the Prefect API: "123_abc_this_is_my_token"

Verify that you have a new Secret block in your active workspace named in the format “deployment-my-deployment-my-flow-name-repo-token”.

Creating access tokens differs for each provider.

GitHub

We recommend using HTTPS with fine-grained Personal Access Tokens to limit access by repository. See the GitHub docs for Personal Access Tokens (PATs).

Under Your Profile -> Developer Settings -> Personal access tokens -> Fine-grained token choose Generate New Token and fill in the required fields. Under Repository access choose Only select repositories and grant the token permissions for Contents.

Bitbucket

We recommend using HTTPS with Repository, Project, or Workspace Access Tokens.

You can create a Repository Access Token with Scopes -> Repositories -> Read.

Bitbucket requires you prepend the token string with x-token-auth: The full string looks like x-token-auth:abc_123_this_is_my_token.

GitLab

We recommend using HTTPS with Project Access Tokens.

In your repository in the GitLab UI, select Settings -> Repository -> Project Access Tokens and check read_repository under Select scopes.

To configure a Secret block ahead of time, create the block in code or the Prefect UI and reference it in your prefect.yaml file.

pull:
    - prefect.deployments.steps.git_clone:
        repository: https://gitlab.com/org/my-private-repo.git
        access_token: "{{ prefect.blocks.secret.my-block-name }}"

Alternatively, you can create a Credentials block ahead of time and reference it in the prefect.yaml pull step.

GitHub

1. Install the Prefect-Github library with pip install -U prefect-github
2. Register the blocks in that library to make them available on the server with prefect block register -m prefect_github
3. Create a GitHub Credentials block in code or the Prefect UI and reference it as shown:

pull:
    - prefect.deployments.steps.git_clone:
        repository: https://github.com/org/my-private-repo.git
        credentials: "{{ prefect.blocks.github-credentials.my-block-name }}"

Bitbucket

1. Install the relevant library with pip install -U prefect-bitbucket
2. Register the blocks in that library with prefect block register -m prefect_bitbucket
3. Create a Bitbucket credentials block in code or the Prefect UI and reference it as shown:

pull:
    - prefect.deployments.steps.git_clone:
        repository: https://bitbucket.org/org/my-private-repo.git
        credentials: "{{ prefect.blocks.bitbucket-credentials.my-block-name }}"

GitLab

1. Install the relevant library with pip install -U prefect-gitlab
2. Register the blocks in that library with prefect block register -m prefect_gitlab
3. Create a GitLab credentials block in code or the Prefect UI and reference it as shown:

pull:
    - prefect.deployments.steps.git_clone:
        repository: https://gitlab.com/org/my-private-repo.git
        credentials: "{{ prefect.blocks.gitlab-credentials.my-block-name }}"

Push your code

When you make a change to your code, Prefect does not push your code to your git-based version control platform. You need to push your code manually or as part of your CI/CD pipeline. This is intentional to avoid confusion about the git history and push process.

Option 2: Docker-based storage

Another way to store your flow code is to include it in a Docker image. All work pool options except Process and Prefect Managed work pools allow you to bake your code into a Docker image.

1. Run prefect init in the root of your repository and choose docker for the project name. Answer the prompts to create a prefect.yaml file with a build step that creates a Docker image with the flow code built in. See the Workers and Work Pools page of the tutorial for more info.
2. Run prefect deploy from the root of your repository to create a deployment.
3. When a deployment runs, the worker pulls the Docker image and spins up a container.
4. The flow code baked into the image runs inside the container.

CI/CD may not require push or pull steps

You don’t need push or pull steps in the prefect.yaml file if using CI/CD to build a Docker image outside of Prefect. Instead, the work pool can reference the image directly.

Option 3: Cloud-provider storage

You can store your code in an AWS S3 bucket, Azure Blob Storage container, or GCP GCS bucket and specify the destination directly in the push and pull steps of your prefect.yaml file.

To create a templated prefect.yaml file run prefect init and select the recipe for the applicable cloud-provider storage. Below are the recipe options and the relevant portions of the prefect.yaml file.

AWS S3 bucket

Choose s3Bucket as the recipe and enter the bucket name when prompted.

# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_aws.deployments.steps.push_to_s3:
    id: push_code
    requires: prefect-aws>=0.3.4
    bucket: my-bucket
    folder: my-folder
    credentials: "{{ prefect.blocks.aws-credentials.my-credentials-block }}" # if private

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect_aws.deployments.steps.pull_from_s3:
    id: pull_code
    requires: prefect-aws>=0.3.4
    bucket: '{{ push_code.bucket }}'
    folder: '{{ push_code.folder }}'
    credentials: "{{ prefect.blocks.aws-credentials.my-credentials-block }}" # if private 

If the bucket requires authentication to access it, you can do the following:

1. Install the Prefect-AWS library with pip install -U prefect-aws
2. Register the blocks in Prefect-AWS with prefect block register -m prefect_aws
3. Create a user with a role with read and write permissions to access the bucket. If using the UI, create an access key pair with IAM -> Users -> Security credentials -> Access keys -> Create access key. Choose Use case -> Other and then copy the Access key and Secret access key values.
4. Create an AWS Credentials block in code or the Prefect UI. In addition to the block name, most users will fill in the AWS Access Key ID and AWS Access Key Secret fields.
5. Reference the block as shown in the push and pull steps

Azure Blob Storage container

Choose azure as the recipe and enter the container name when prompted.

# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_azure.deployments.steps.push_to_azure_blob_storage:
    id: push_code
    requires: prefect-azure>=0.2.8
    container: my-prefect-azure-container
    folder: my-folder
    credentials: "{{ prefect.blocks.azure-blob-storage-credentials.my-credentials-block }}" # if private

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect_azure.deployments.steps.pull_from_azure_blob_storage:
    id: pull_code
    requires: prefect-azure>=0.2.8
    container: '{{ push_code.container }}'
    folder: '{{ push_code.folder }}'
    credentials: "{{ prefect.blocks.azure-blob-storage-credentials.my-credentials-block }}" # if private

If the blob requires authentication to access it, follow these steps:

1. Install the Prefect-Azure library with pip install -U prefect-azure
2. Register the blocks in Prefect-Azure with prefect block register -m prefect_azure
3. Create an access key for a role with sufficient (read and write) permissions to access the blob. You can create a connection string containing all required information in the UI under Storage Account -> Access keys.
4. Create an Azure Blob Storage Credentials block in code or the Prefect UI. Enter a name for the block and paste the connection string into the Connection String field.
5. Reference the block as shown in the push and pull steps above.

GCP GCS bucket

Choose `gcs“ as the recipe and enter the bucket name when prompted.

# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_gcp.deployment.steps.push_to_gcs:
    id: push_code
    requires: prefect-gcp>=0.4.3
    bucket: my-bucket
    folder: my-folder
    credentials: "{{ prefect.blocks.gcp-credentials.my-credentials-block }}" # if private 

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect_gcp.deployment.steps.pull_from_gcs:
    id: pull_code
    requires: prefect-gcp>=0.4.3
    bucket: '{{ push_code.bucket }}'
    folder: '{{ pull_code.folder }}'
    credentials: "{{ prefect.blocks.gcp-credentials.my-credentials-block }}" # if private 

If the bucket requires authentication to access it, follow these steps:

1. Install the Prefect-GCP library with pip install -U prefect-gcp
2. Register the blocks in Prefect-GCP with prefect block register -m prefect_gcp
3. Create a service account in GCP for a role with read and write permissions to access the bucket contents. If using the GCP console, go to IAM & Admin -> Service accounts -> Create service account. After choosing a role with the required permissions, see your service account and click on the three dot menu in the Actions column. Select Manage Keys -> ADD KEY -> Create new key -> JSON. Download the JSON file.
4. Create a GCP Credentials block in code or the Prefect UI. Enter a name for the block and paste the entire contents of the JSON key file into the Service Account Info field.
5. Reference the block as shown in the push and pull steps above.

Another authentication option is to give the worker access to the storage location at runtime through SSH keys.

Alternatively, you can inject environment variables into your deployment. See this example that uses an environment variable named CUSTOM_FOLDER:

 push:
    - prefect_gcp.deployment.steps.push_to_gcs:
        id: push_code
        requires: prefect-gcp>=0.4.3
        bucket: my-bucket
        folder: '{{ $CUSTOM_FOLDER }}'

Include or exclude files from storage

By default, Prefect uploads all files in the current folder to the configured storage location when you create a deployment.

When using a git repository, Docker image, or cloud-provider storage location, you may want to exclude certain files or directories:

• If you are familiar with Docker you are likely familiar with the .dockerignore file.
• For cloud-provider storage the .prefectignore file serves the same purpose and follows a similar syntax as those files. So an entry of *.pyc will exclude all .pyc files from upload.