Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve detection of when breeze CI image needs rebuilding #33603

Merged
merged 2 commits into from
Aug 22, 2023
Merged

Improve detection of when breeze CI image needs rebuilding #33603

merged 2 commits into from
Aug 22, 2023

Commits on Aug 22, 2023

  1. Improve detection of when breeze CI image needs rebuilding 

    Previously we have been using provider.yaml file modification as
a sign that the docker image needs rebuilding when starting image.
However just modification of provider.yaml file is not a sign
that the image needs rebuilding. The image needs rebuilding when
provider dependencies changed, but there are many more reasons why
provider.yaml file changed - especially recently provider.yaml
file contains much more information and dependencies are only part
of it. Provider.yaml files can also be modified by release manager
wnen documentation is prepared, but none of the documentation
change is a reason for rebuilding the image.

This PR optimize the check for image building introducing two
step process:

* first we check if provider.yaml files changed
* if they did, we regenerate provider dependencies by manully
  running the pre-commit script
* then provider_dependencies.json is used instead of all providers
  to determine if the image needs rebuilding

This has several nice side effects:

* the list of files that have been modified displayed to the
  user is potentially much smaller (no provider.yaml files)
* provider_dependencies.json is regenereated automatically when
  you run any breeze command, which means that you do not have
  to have pre-commit installed to regenerate it
* the notification "image needs rebuilding" will be printed less
  frequently to the user - only when it is really needed
* preparing provider documentation in CI will not trigger
  image rebuilding (which might occasionally fail in such case
  especially when we bring back a provider from long suspension
  like it happened in apache#33574
    @potiuk
    potiuk committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    8f9a476 View commit details
    Browse the repository at this point in the history

  2. Update dev/breeze/src/airflow_breeze/commands/developer_commands.py

    @potiuk
    potiuk authored Aug 22, 2023
    Configuration menu
    Copy the full SHA
    5dc9ae3 View commit details
    Browse the repository at this point in the history