Option 1: Git-based storage
Git-based version control platforms provide redundancy, version control, and easier collaboration. Prefect supports:Create a deployment with git-based storage
- Run
prefect deploy
from the root directory of the git repository - Create a new deployment
- You will see a series of prompts. Select yes creating a new deployment.
- Select the flow code entrypoint
- Name your deployment
We recommend using HTTPS with fine-grained Personal Access Tokens
to limit access by repository.
See the GitHub docs for Personal Access Tokens (PATs).Under Your Profile -> Developer Settings -> Personal access tokens -> Fine-grained token choose Generate New Token and
fill in the required fields.
Under Repository access choose Only select repositories and grant the token permissions for Contents.
prefect.yaml
file.
prefect.yaml
pull step.
- Install the Prefect-Github library with
pip install -U prefect-github
- Register the blocks in that library to make them available on the server with
prefect block register -m prefect_github
- Create a GitHub Credentials block in code or the Prefect UI and reference it as shown:
Push your codeWhen you make a change to your code, Prefect does not push your code to your git-based version control platform.
You need to push your code manually or as part of your CI/CD pipeline.
This is intentional to avoid confusion about the git history and push process.
Option 2: Docker-based storage
Another way to store your flow code is to include it in a Docker image. All work pool options except Process and Prefect Managed work pools allow you to bake your code into a Docker image.- Run
prefect init
in the root of your repository and choosedocker
for the project name. Answer the prompts to create aprefect.yaml
file with a build step that creates a Docker image with the flow code built in. See the Workers and Work Pools page of the tutorial for more info. - Run
prefect deploy
from the root of your repository to create a deployment. - When a deployment runs, the worker pulls the Docker image and spins up a container.
- The flow code baked into the image runs inside the container.
CI/CD may not require push or pull stepsYou don’t need push or pull steps in the
prefect.yaml
file if using CI/CD to build a Docker image outside of Prefect.
Instead, the work pool can reference the image directly.Option 3: Cloud-provider storage
You can store your code in an AWS S3 bucket, Azure Blob Storage container, or GCP GCS bucket and specify the destination directly in thepush
and pull
steps of your prefect.yaml
file.
To create a templated prefect.yaml
file run prefect init
and select the recipe for the applicable cloud-provider storage.
Below are the recipe options and the relevant portions of the prefect.yaml
file.
Choose If the bucket requires authentication to access it, you can do the following:
s3Bucket
as the recipe and enter the bucket name when prompted.- Install the Prefect-AWS library with
pip install -U prefect-aws
- Register the blocks in Prefect-AWS with
prefect block register -m prefect_aws
- Create a user with a role with read and write permissions to access the bucket. If using the UI, create an access key pair with IAM -> Users -> Security credentials -> Access keys -> Create access key. Choose Use case -> Other and then copy the Access key and Secret access key values.
- Create an AWS Credentials block in code or the Prefect UI. In addition to the block name, most users will fill in the AWS Access Key ID and AWS Access Key Secret fields.
- Reference the block as shown in the push and pull steps
CUSTOM_FOLDER
:
Include or exclude files from storage
By default, Prefect uploads all files in the current folder to the configured storage location when you create a deployment. When using a git repository, Docker image, or cloud-provider storage location, you may want to exclude certain files or directories:- If you are familiar with Docker you are likely familiar with the
.dockerignore
file. - For cloud-provider storage the
.prefectignore
file serves the same purpose and follows a similar syntax as those files. So an entry of*.pyc
will exclude all.pyc
files from upload.