Use PyPI constraints for PROD image in non-main branch #33789
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When we are building PROD image in CI for non main branch, we are installing providers from PyPI rather than building them locally from sources. Therefore we should use `PyPI` constraints for such builds not the "source" constraints (they might differ). This PR adds two steps: * In the CI build, when we do not build providers we generate PyPI constraints additionally to source constraints * In the PROD build we use the PyPI constraints in case we do not build providers locally
potiuk
commented
Aug 27, 2023
| @@ -34,12 +34,18 @@ runs: | |||
| - name: "Build & Push AMD64 CI images ${{ env.IMAGE_TAG }} ${{ env.PYTHON_VERSIONS }}" | |||
| shell: bash | |||
| run: breeze ci-image build --push --tag-as-latest --run-in-parallel --upgrade-on-failure | |||
| - name: "Show dependencies to be upgraded" | |||
| - name: "Generate source constraints" | |||
There was a problem hiding this comment.
Those constraints are useful regardless to see and they are very fast to generate, so we can generate them also in non-main branch. It's helpful to see them
potiuk
commented
Aug 27, 2023
| shell: bash | ||
| run: > | ||
| breeze release-management generate-constraints --run-in-parallel | ||
| --airflow-constraints-mode constraints-source-providers | ||
| if: env.UPGRADE_TO_NEWER_DEPENDENCIES != 'false' | ||
| - name: "Generate PyPI constraints" |
There was a problem hiding this comment.
Those are slower (we need check which provders are available in PyPI and reinstall them before gnerating the constraints so we should only run them in "non-main" build).
|
This is the ultimate one needed for the 2-7-tests build to succeed (due to the openlineage limitation I had to bump the version as well to 2.7.1). |
|
I already cherry-picked that one to v2-7-test - but will replace the cherry-pick including PR# once this one is merged |
ephraimbuddy
approved these changes
Aug 27, 2023
ephraimbuddy
added
the
changelog:skip
potiuk
added a commit
that referenced
this pull request
Aug 27, 2023
When we are building PROD image in CI for non main branch, we are installing providers from PyPI rather than building them locally from sources. Therefore we should use `PyPI` constraints for such builds not the "source" constraints (they might differ). This PR adds two steps: * In the CI build, when we do not build providers we generate PyPI constraints additionally to source constraints * In the PROD build we use the PyPI constraints in case we do not build providers locally (cherry picked from commit f9276f0)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When we are building PROD image in CI for non main branch, we are installing providers from PyPI rather than building them locally from sources. Therefore we should use
PyPIconstraints for such builds not the "source" constraints (they might differ).This PR adds two steps:
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.