Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stop using autobuilds of DockerHub #16555

Closed
4 tasks done
potiuk opened this issue Jun 21, 2021 · 4 comments
Closed
4 tasks done

Stop using autobuilds of DockerHub #16555

potiuk opened this issue Jun 21, 2021 · 4 comments
Assignees
@potiuk
Labels
kind:feature Feature Requests

Comments

@potiuk
Copy link
Member

potiuk commented Jun 21, 2021 •
edited
Loading

According to https://www.docker.com/blog/changes-to-docker-hub-autobuilds/ Docker is going to disable autobuilds for free tiers. We might be exempt from that via ASF, but docker autobuilds never worked well for us for multitude of reasons, so we should likely get rid of them anyway. This means:

  • switch to manual docker image release when we release new airflow version
  • switch to the GitHub container registry (ghcr.io) for breeze image caching - it has now support for GITHUB_TOKEN and allows for better retention of images
  • move images for ci to GitHub container registry
  • entirely remove airflow-ci for airflow imageafter the above is complete
@potiuk potiuk added the kind:feature Feature Requests label Jun 21, 2021
@potiuk potiuk self-assigned this Jun 21, 2021
@potiuk
Copy link
Member Author

potiuk commented Jun 21, 2021

cc: @ashb @kaxil - the most important change is that preparing images manually when we release Airflow will no longer be optional. It's likely best to do that on "release manager owned machine" rather than pushing it from CI, but I think we can discuss it. I will make a PR to cover this part.

@potiuk potiuk mentioned this issue Jun 21, 2021
Merged
potiuk added a commit to potiuk/airflow that referenced this issue Jun 21, 2021
@potiuk
Switches to manually building docker images
0287b3f 
According to
https://www.docker.com/blog/changes-to-docker-hub-autobuilds/
Docker is going to disable autobuilds for free tiers. We might be exempt
from that via ASF, but docker autobuilds never worked well for us for
multitude of reasons.

This PR turns manually preparing the image into obligatory, manual step
when releasing Airflow.

Part of apache#16555
potiuk added a commit that referenced this issue Jun 21, 2021
@potiuk
Switches to manually building docker images (#16570)
9f92786 
According to
https://www.docker.com/blog/changes-to-docker-hub-autobuilds/
Docker is going to disable autobuilds for free tiers. We might be exempt
from that via ASF, but docker autobuilds never worked well for us for
multitude of reasons.

This PR turns manually preparing the image into obligatory, manual step
when releasing Airflow.

Part of #16555
potiuk added a commit to potiuk/airflow that referenced this issue Jun 22, 2021
@potiuk
Switches to manually building docker images (apache#16570)
b1ef766 
According to
https://www.docker.com/blog/changes-to-docker-hub-autobuilds/
Docker is going to disable autobuilds for free tiers. We might be exempt
from that via ASF, but docker autobuilds never worked well for us for
multitude of reasons.

This PR turns manually preparing the image into obligatory, manual step
when releasing Airflow.

Part of apache#16555

(cherry picked from commit 9f92786)
Jorricks pushed a commit to Jorricks/airflow that referenced this issue Jun 24, 2021
@potiuk @Jorricks
Switches to manually building docker images (apache#16570)
544bbde 
According to
https://www.docker.com/blog/changes-to-docker-hub-autobuilds/
Docker is going to disable autobuilds for free tiers. We might be exempt
from that via ASF, but docker autobuilds never worked well for us for
multitude of reasons.

This PR turns manually preparing the image into obligatory, manual step
when releasing Airflow.

Part of apache#16555
This was referenced Jul 2, 2021
Merged
Merged
potiuk added a commit to potiuk/airflow that referenced this issue Jul 4, 2021
@potiuk
Switch Breeze/CI to ghcr.io excusively
76ee7e6 
Breeze used traditionally DockerHub to pull images, because
they were public and GitHub Packages were not. With GitHub Container
Regisry however, we can switch fully to using GitHub Container
Registry also for Breeze.

Thanks to moving to Github Container Registry we can remove
a lot of code responsible for maintaining different naming
and different versions of the images in DockerHub and
GitHub Container Registry. Also it streamlines and simplifies
the process of refreshing the images when new python versions
are released - the CI push builds will check if the new Python
image is released in DockerHub and it will rebuild the base
image automatically if needed (and push it as cache)

The CI documentation (including sequence diagrams) has been
refreshed to reflect those changes (and other changes done in
the meantime). The flows are now simplified as DockerHub is
largely moved out of the picture.

The only remaining DockerHub Images now are:

* images used during CI for integrations (airflow-ci)
* officially released Production Airflow images (airflow)

The integration images will be moved to GitHub Container Registry
in a subsequent PR and the only images remaining in DockerHub
will be the officially released Production Airflow images.

Part of apache#16555
@potiuk potiuk mentioned this issue Jul 4, 2021
Merged
potiuk added a commit to potiuk/airflow that referenced this issue Jul 11, 2021
@potiuk
Switch Breeze/CI to ghcr.io excusively
4e9d9c5 
Breeze used traditionally DockerHub to pull images, because
they were public and GitHub Packages were not. With GitHub Container
Regisry however, we can switch fully to using GitHub Container
Registry also for Breeze.

Thanks to moving to Github Container Registry we can remove
a lot of code responsible for maintaining different naming
and different versions of the images in DockerHub and
GitHub Container Registry. Also it streamlines and simplifies
the process of refreshing the images when new python versions
are released - the CI push builds will check if the new Python
image is released in DockerHub and it will rebuild the base
image automatically if needed (and push it as cache)

The CI documentation (including sequence diagrams) has been
refreshed to reflect those changes (and other changes done in
the meantime). The flows are now simplified as DockerHub is
largely moved out of the picture.

The only remaining DockerHub Images now are:

* images used during CI for integrations (airflow-ci)
* officially released Production Airflow images (airflow)

The integration images will be moved to GitHub Container Registry
in a subsequent PR and the only images remaining in DockerHub
will be the officially released Production Airflow images.

Part of apache#16555
potiuk added a commit that referenced this issue Jul 12, 2021
@potiuk
Switch Breeze/CI to ghcr.io excusively (#16780)
af0598f 
Breeze used traditionally DockerHub to pull images, because
they were public and GitHub Packages were not. With GitHub Container
Regisry however, we can switch fully to using GitHub Container
Registry also for Breeze.

Thanks to moving to Github Container Registry we can remove
a lot of code responsible for maintaining different naming
and different versions of the images in DockerHub and
GitHub Container Registry. Also it streamlines and simplifies
the process of refreshing the images when new python versions
are released - the CI push builds will check if the new Python
image is released in DockerHub and it will rebuild the base
image automatically if needed (and push it as cache)

The CI documentation (including sequence diagrams) has been
refreshed to reflect those changes (and other changes done in
the meantime). The flows are now simplified as DockerHub is
largely moved out of the picture.

The only remaining DockerHub Images now are:

* images used during CI for integrations (airflow-ci)
* officially released Production Airflow images (airflow)

The integration images will be moved to GitHub Container Registry
in a subsequent PR and the only images remaining in DockerHub
will be the officially released Production Airflow images.

Part of #16555
potiuk added a commit to potiuk/airflow that referenced this issue Jul 12, 2021
@potiuk
Move CI-integration images to ghcr.io
e168992 
This is the final step of moving the images used for CI integration
to `ghcr.io` from DockerHub. With Publicly available images
with self-management provided by GitHub, we can finally move to
keep the images "properly" - i.e. each image is separate and
tag is only image version.

Part of apache#16555
potiuk added a commit that referenced this issue Jul 14, 2021
@potiuk
Move CI-integration images to ghcr.io (#16797)
3143f1a 
This is the final step of moving the images used for CI integration
to `ghcr.io` from DockerHub. With Publicly available images
with self-management provided by GitHub, we can finally move to
keep the images "properly" - i.e. each image is separate and
tag is only image version.

Part of #16555
@potiuk
Copy link
Member Author

potiuk commented Jul 14, 2021

We should remove airflow-ci images after we cherry-pick build changes to v2-1-test line (will do it shortly)

potiuk added a commit that referenced this issue Jul 18, 2021
@potiuk
Switch Breeze/CI to ghcr.io excusively (#16780)
425f85b 
Breeze used traditionally DockerHub to pull images, because
they were public and GitHub Packages were not. With GitHub Container
Regisry however, we can switch fully to using GitHub Container
Registry also for Breeze.

Thanks to moving to Github Container Registry we can remove
a lot of code responsible for maintaining different naming
and different versions of the images in DockerHub and
GitHub Container Registry. Also it streamlines and simplifies
the process of refreshing the images when new python versions
are released - the CI push builds will check if the new Python
image is released in DockerHub and it will rebuild the base
image automatically if needed (and push it as cache)

The CI documentation (including sequence diagrams) has been
refreshed to reflect those changes (and other changes done in
the meantime). The flows are now simplified as DockerHub is
largely moved out of the picture.

The only remaining DockerHub Images now are:

* images used during CI for integrations (airflow-ci)
* officially released Production Airflow images (airflow)

The integration images will be moved to GitHub Container Registry
in a subsequent PR and the only images remaining in DockerHub
will be the officially released Production Airflow images.

Part of #16555

(cherry picked from commit af0598f)
potiuk added a commit that referenced this issue Jul 18, 2021
@potiuk
Move CI-integration images to ghcr.io (#16797)
e10acaa 
This is the final step of moving the images used for CI integration
to `ghcr.io` from DockerHub. With Publicly available images
with self-management provided by GitHub, we can finally move to
keep the images "properly" - i.e. each image is separate and
tag is only image version.

Part of #16555

(cherry picked from commit 3143f1a)
@potiuk
Copy link
Member Author

potiuk commented Jul 18, 2021

I cherry-picked the necessary commits to v2-1-test, but we need to merge #17065 in order to build and push the images.

josh-fell pushed a commit to josh-fell/airflow that referenced this issue Jul 19, 2021
@potiuk @josh-fell
Switch Breeze/CI to ghcr.io excusively (apache#16780)
1478a81 
Breeze used traditionally DockerHub to pull images, because
they were public and GitHub Packages were not. With GitHub Container
Regisry however, we can switch fully to using GitHub Container
Registry also for Breeze.

Thanks to moving to Github Container Registry we can remove
a lot of code responsible for maintaining different naming
and different versions of the images in DockerHub and
GitHub Container Registry. Also it streamlines and simplifies
the process of refreshing the images when new python versions
are released - the CI push builds will check if the new Python
image is released in DockerHub and it will rebuild the base
image automatically if needed (and push it as cache)

The CI documentation (including sequence diagrams) has been
refreshed to reflect those changes (and other changes done in
the meantime). The flows are now simplified as DockerHub is
largely moved out of the picture.

The only remaining DockerHub Images now are:

* images used during CI for integrations (airflow-ci)
* officially released Production Airflow images (airflow)

The integration images will be moved to GitHub Container Registry
in a subsequent PR and the only images remaining in DockerHub
will be the officially released Production Airflow images.

Part of apache#16555
josh-fell pushed a commit to josh-fell/airflow that referenced this issue Jul 19, 2021
@potiuk @josh-fell
Move CI-integration images to ghcr.io (apache#16797)
def9142 
This is the final step of moving the images used for CI integration
to `ghcr.io` from DockerHub. With Publicly available images
with self-management provided by GitHub, we can finally move to
keep the images "properly" - i.e. each image is separate and
tag is only image version.

Part of apache#16555
kaxil pushed a commit that referenced this issue Aug 17, 2021
@potiuk @kaxil
Switch Breeze/CI to ghcr.io excusively (#16780)
ceb4a8d 
Breeze used traditionally DockerHub to pull images, because
they were public and GitHub Packages were not. With GitHub Container
Regisry however, we can switch fully to using GitHub Container
Registry also for Breeze.

Thanks to moving to Github Container Registry we can remove
a lot of code responsible for maintaining different naming
and different versions of the images in DockerHub and
GitHub Container Registry. Also it streamlines and simplifies
the process of refreshing the images when new python versions
are released - the CI push builds will check if the new Python
image is released in DockerHub and it will rebuild the base
image automatically if needed (and push it as cache)

The CI documentation (including sequence diagrams) has been
refreshed to reflect those changes (and other changes done in
the meantime). The flows are now simplified as DockerHub is
largely moved out of the picture.

The only remaining DockerHub Images now are:

* images used during CI for integrations (airflow-ci)
* officially released Production Airflow images (airflow)

The integration images will be moved to GitHub Container Registry
in a subsequent PR and the only images remaining in DockerHub
will be the officially released Production Airflow images.

Part of #16555

(cherry picked from commit af0598f)
kaxil pushed a commit that referenced this issue Aug 17, 2021
@potiuk @kaxil
Move CI-integration images to ghcr.io (#16797)
d8bfaca 
This is the final step of moving the images used for CI integration
to `ghcr.io` from DockerHub. With Publicly available images
with self-management provided by GitHub, we can finally move to
keep the images "properly" - i.e. each image is separate and
tag is only image version.

Part of #16555

(cherry picked from commit 3143f1a)
jhtimmins pushed a commit that referenced this issue Aug 17, 2021
@potiuk @jhtimmins
Switch Breeze/CI to ghcr.io excusively (#16780)
8691ca0 
Breeze used traditionally DockerHub to pull images, because
they were public and GitHub Packages were not. With GitHub Container
Regisry however, we can switch fully to using GitHub Container
Registry also for Breeze.

Thanks to moving to Github Container Registry we can remove
a lot of code responsible for maintaining different naming
and different versions of the images in DockerHub and
GitHub Container Registry. Also it streamlines and simplifies
the process of refreshing the images when new python versions
are released - the CI push builds will check if the new Python
image is released in DockerHub and it will rebuild the base
image automatically if needed (and push it as cache)

The CI documentation (including sequence diagrams) has been
refreshed to reflect those changes (and other changes done in
the meantime). The flows are now simplified as DockerHub is
largely moved out of the picture.

The only remaining DockerHub Images now are:

* images used during CI for integrations (airflow-ci)
* officially released Production Airflow images (airflow)

The integration images will be moved to GitHub Container Registry
in a subsequent PR and the only images remaining in DockerHub
will be the officially released Production Airflow images.

Part of #16555

(cherry picked from commit af0598f)
jhtimmins pushed a commit that referenced this issue Aug 17, 2021
@potiuk @jhtimmins
Move CI-integration images to ghcr.io (#16797)
b0a006c 
This is the final step of moving the images used for CI integration
to `ghcr.io` from DockerHub. With Publicly available images
with self-management provided by GitHub, we can finally move to
keep the images "properly" - i.e. each image is separate and
tag is only image version.

Part of #16555

(cherry picked from commit 3143f1a)
@potiuk potiuk mentioned this issue Aug 25, 2021
Merged
@potiuk
Copy link
Member Author

potiuk commented Aug 26, 2021

airflow-ci DockerHub deleted.

@potiuk potiuk closed this as completed Aug 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@potiuk potiuk

Labels
kind:feature Feature Requests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@potiuk