-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Co-authored-by: jakekaplan <[email protected]>
- Loading branch information
1 parent
67eadc1
commit 0c9cae1
Showing
1 changed file
with
135 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,135 @@ | ||
--- | ||
description: Prefect will run your deployment on our infrastructure. | ||
tags: | ||
- managed infrastructure | ||
search: | ||
exclude: true | ||
--- | ||
|
||
# Managed Execution | ||
|
||
Prefect Cloud provides a **prefect:managed** work pool that you can use for your workflow execution environment. | ||
Deployments run with this work pool do not require a worker and do not require a cloud provider account. | ||
Prefect handles the infrastructure for you. | ||
|
||
Managed execution is a great option for users who want to get started quickly, with no infrastructure setup. | ||
|
||
!!! warning "Managed Execution is in alpha" | ||
Managed Execution is currently in alpha. | ||
Features are likely to change without warning. | ||
|
||
## Usage guide | ||
|
||
Run a flow with managed infrastructure in three steps. | ||
|
||
### Step 1 | ||
|
||
Create a new work pool of type **prefect:managed**. | ||
|
||
!!! note | ||
In the alpha period, you must have the managed execution feature enabled for your account to create a work pool of this type. | ||
|
||
### Step 2 | ||
|
||
Create a deployment using the flow `deploy` method or `prefect.yaml`. | ||
|
||
Specify the name of your managed work pool, as shown in this example that uses the `deploy` method: | ||
|
||
```python hl_lines="9" title="managed-execution.py" | ||
from prefect import flow | ||
|
||
if __name__ == "__main__": | ||
flow.from_source( | ||
source="https://github.com/desertaxle/demo.git", | ||
entrypoint="flow.py:my_flow", | ||
).deploy( | ||
name="test-managed-flow", | ||
work_pool_name="my-managed-pool", | ||
) | ||
``` | ||
|
||
With your CLI authenticated to your Prefect Cloud workspace, run the script to create your deployment: | ||
|
||
<div class="terminal"> | ||
```bash | ||
python managed-execution.py | ||
``` | ||
</div> | ||
|
||
Note that this deployment uses flow code stored in a GitHub repository. | ||
If you would like to use your own flow code, push your code to your repository and specify your own repository URL and entrypoint. | ||
|
||
### Step 3 | ||
|
||
Schedule a deployment run from the UI or from the CLI. | ||
|
||
It may take a minute for the flow run to start and the logs to appear in the UI. | ||
|
||
That's it! You ran a flow on remote infrastructure without any infrastructure setup, worker, or cloud provider account. | ||
|
||
### Adding dependencies | ||
|
||
You can install Python package dependencies at runtime by passing `job_variables={"env": {"EXTRA_PIP_PACKAGES": ["pandas", "prefect-aws"] }}` like this: | ||
|
||
```python hl_lines="10" | ||
from prefect import flow | ||
|
||
if __name__ == "__main__": | ||
flow.from_source( | ||
source="https://github.com/desertaxle/demo.git", | ||
entrypoint="flow.py:my_flow", | ||
).deploy( | ||
name="test-managed-flow", | ||
work_pool_name="prefect-managed", | ||
job_variables={"env": {"EXTRA_PIP_PACKAGES": ["pandas", "prefect-aws"] }} | ||
) | ||
``` | ||
|
||
Alternatively, you can specify a `requirements.txt` file and reference it in your `pull_step`. | ||
|
||
## Limitations | ||
|
||
Managed execution requires Prefect 2.14.4 or newer. | ||
|
||
All limitations listed below may change without warning during the alpha period. | ||
We will update this page as we make changes. | ||
|
||
### Concurrency & work pools | ||
|
||
- Maximum of 10 concurrent flow runs per workspace across all `prefect:managed` pools. | ||
- Maximum of 5 managed execution work pools per workspace. | ||
|
||
### Images | ||
|
||
Managed execution requires that you run one of the offered Docker images. | ||
You may not use your own Docker image. | ||
One Docker image is supported at this time: `prefecthq/prefect:2-latest`. | ||
We plan to support additional images with common dependencies soon. | ||
|
||
However, as noted above, you can install Python package dependencies at runtime. | ||
|
||
If you need to use your own image, we recommend using another type of work pool. | ||
|
||
### Code storage | ||
|
||
Flow code must be stored in an accessible remote location. | ||
This means git-based cloud providers such as GitHub, Bitbucket, or GitLab are supported. | ||
Remote block-based storage is also supported, so S3, GCS, and Azure Blob are additional code storage options. | ||
|
||
### Resources | ||
|
||
Memory is limited to 2GB of RAM, which includes all operations such as dependency installation. | ||
|
||
Maximum run time is 1 hour. | ||
|
||
## Pricing | ||
|
||
Managed execution is free during the alpha period. | ||
We will announce pricing before the beta period begins. | ||
|
||
## Next steps | ||
|
||
Read more about creating deployments in the [deployment guide](/guides/prefect-deploy/). | ||
|
||
If you find that you need more control over your infrastructure, serverless push work pools might be a good option. | ||
Read more [here](/guides/deployment/push-work-pools/). |