Host Airflow-managed Helm repo

[scottrigby]

Description

@mik-laj Hi 👋 I was just coming to this repo to see if you're interested in help getting the Helm chart repo set up to replace the chart in stable.

Stable repo is nearing the end of it's deprecation period, and I'm glad to see a version of the Airflow chart is already here. See https://github.com/helm/charts/issues/21103 for the meta issue tracking moving all the stable repo charts to new homes.

To-do

• Decide on hosting options below (self-host/leverage Bitnami)
• If self-host, set up CI/CD for chart-testing and chart-releasing
• if self-host, list in Helm Hub (http://hub.helm.sh/) and Artifact Hub (https://artifacthub.io/)

Use case / motivation

Set up Helm repo hosting for your chart

Set up a Helm repo, either as a separate git repo in your org, or keeping the same setup you have now. We have created Helm chart repo actions for chart testing (CI) and releasing chart packages as your own GitHub-hosted Helm repo (CD).

Self-hosted options:

1. If we either move the chart to a separate git repo in the artifacthub gh org, or even move the hub github pages setting to a branch other than the main one, we can use the @helm/chart-releaser-action GitHub Action to automate the helm repo.
2. If we keep structure as-is, we can still use the helm/chart-releaser project, just with a custom script.

For either option we can also use the @helm/chart-testing-action to wrap the chart-testing project @mattfarina mentioned above. Here's an demo repo to see how they work together: https://github.com/helm/charts-repo-actions-demo

Whichever option you decide I'm make a PR if it helps.

If you do decide to host your own Helm repo, you will also want to list it in

Alternatively leverage existing Bitnami Helm repo

There is also a version of the chart maintained by Bitnami, who have been very involved in the stable repo for years, but : https://github.com/bitnami/charts/tree/master/bitnami/airflow. You could instead decide to leverage that chart as the canonical source, and not host your own. It is also fine to have multiple instances of a chart to install the same app.

Related Issues

• https://github.com/helm/charts/issues/21103
• helm pull stable/airflow pulls the [stable/airflow] chart from github.com/helm/charts repository and not this repository. #10486
• Chart README is incorrect #10379

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

cc: @dimberman

[mik-laj]

@scottrigby In our repository, we have a Chart based on the work of Astronomer. Our chart is not released so it is not available in the repository, but we hope to do so in the near future. If you want stable/airflow to be still available to users, you need to find another place for discussions.

[ryw]

@schnie what do we need to do here? is there an issue that clearly captures the steps we need to take to reduce community confusion?

[scottrigby]

@mik-laj I'm not sure I understand fully. Are you saying the chart at https://github.com/apache/airflow/tree/master/chart installs an enterprise edition of Airflow – or are you referring only to https://github.com/astronomer/astronomer?

And (either because of that or as an independent statement) are you saying you don't wish to host a chart for pure apache Airflow?

@ryw From an end user point of view, I would say all of the above is definitely not clear.

In order to know what (if anything) to do with the current stable/airflow chart, this should probably be clarified.

[dimberman]

Ohhh kay I think there's a bit of confusion so lemme try to patch this up.

@ryw @schnie @mik-laj It appears that Scott works on the helm project and is asking us (apache/airflow) to take over stable/airflow as helm is deprecating their stable charts overall. It would be a good idea for us to take over this chart immediately so we can make it easier for us to deploy OSS airflow to our community.

@mik-laj was referring to the fact that our chart at Astronomer is actually GA and enterprise ready (though we're not donating that as we already donated a snapshot of our chart for the airflow image). I think there was a minor mistranslation there :).

@mik-laj @ashb @potiuk would there be an issue with us taking the chart in it's current version and releasing it as a beta on the stable/airflow? If everyone is ok with it I can do this tomorrow.

@scottrigby apologies for the confusion here :).

[scottrigby]

@dimberman I see 🙂 Thanks for clearing that up.

One important thing I'd suggest is bringing in git history from the stable repo chart source (there have been a number of charts copied over, and git history is later requested for a few reasons: 1. Crediting past contributors, and 2. Seeing why the current state got the way it did, to help evaluate future changes).

And if you have any questions setting up the Helm GitHub Actions, give a shout to one of us in kubernetes.slack.com #chart-testing channel, or ping me here and I'll be happy to help.

Either way, once this is done, and the new airflow repo is listed in the Helm Hub and Artifact Hub, we can deprecate the stable chart.

[potiuk]

I do not have problems anymore with the licenses or sources :). This has been solved.

However, I am afraid it's not as easy as 'just create repo' and as PMCs, we have formal obligations that we have to fulfill to release the Helm Chart. We discussed it several times in the past with the original authors of the Helm chart

There are a couple of things here:

Taking over the chart

I think we already decided we are not "taking over" the stable/helm chart. Those charts are being deprecated (Nov 13 is officially when the project ends). The project was developed outside of the community and "taking over" means also taking responsibility towards the users of the "helm/stable" chart, taking care of migration from the old chart, and being generally responsible for it - including troubleshooting and support. While this is possible and maybe even advisable - this is something that we as a community have to approve, vote on making sure that we "receive" the software formally - IMHO basically the software (including its git history) will have to be donated to Apache Airflow, by whoever is the current owner. Software is not only an asset.  It might also be a liability and we have to make sure that we know what we agree to.

I honestly do not think moving history is at all possible and I think it is a bad idea. Those projects do not share the common history and we even cannot legally do that before the code is officially donated to Apache Software Foundation. What I suggest is to write a migration guide from the old Helm chart to the new one, maybe (if there are missing features in the "future one") it would be great if the original authors or users open issues in the Airflow project and we could consider adding them (@gsemet - as the original author, I would love to hear what you have to say here).

IMHO the "deprecation" notice should be "You have to migrate to the new chart from the old one but they are different" - because essentially they are. Now the question is - who should write and take care about such a migration guide, I think as a community we should help with that, but I consider that the work of the authors of the original chart to lead that effort. I am super happy to help (going for vacations for a week) and discuss/add missing features in the new chart or help to review and collaborate on the migration guide, but since the community was not involved at all in the old chart development  I think we should be very careful in stating that we are "taking it over". I think we should care about our community and people who are using the old chart and help them to migrate to the new one (and as a community to take on the task of maintaining it) so I am happy to help with that, but we simply cannot take it over. I am also super happy to refer to the past contributors of the old helm (and give them the credit) and point people to the migration guide (that might be initially part of the helm/stable repo), and once it is well tried by some people we might add it is a "contributed" migration guide from the "stable" chart

We discussed it several times for example here or here.

Of course, we could go the whole process - the original authors (and only them) might propose at the devlist that the community take over also the history and the community might vote on it. But this is something the authors have to propose officially at the devlist, not in a GitHub issue, make a formal vote on it and it has to pass the vote. None of us here individually can make that decision. This is how the Airflow community works.

Releasing our current chart

Since we are going to release the Helm Chart as sources to our users, we cannot "just create a repository" and release it. This is not a "convenience binary" as in the case of Docker image (which is built from the officially released sources) - those are "sources". We are obliged by  ASF policy ASF Release Policy to release it officially. This means that we have to do it formally - including preparing packages (storing, checksumming, signing on SVN), voting, releasing the software formally in Apache Software Foundation Subversion, etc. This can be connected with releasing the whole Airflow project or separately. I believe we agreed that we are going to release it separately, thus should follow a separate release process. Of course, the repository with master code synced from Apache Airflow can be done before. We can easily use GitHub Actions and tools like https://github.com/google/copybara to automatically move the Chart - related history from  https://github.com/apache/airflow (charts folder) to (for example) https://github.com/apache/airflow-chart. I am happy to help to set it up after my vacation. But we cannot advertise it to anyone else than members of the devlist. This is strictly forbidden by the ASF release policy. Relevant quote here:

Do not include any links on the project website that might encourage non-developers to download and use nightly builds, snapshots, release candidates, or any other similar package. The only people who are supposed to know about such packages are the people following the dev list (or searching its archives) and thus aware of the conditions placed on the package. If you find that the general public are downloading such test packages, then remove them.

So announcing it as "replacement" of the helm/stable chart without formally releasing it is against the rules of ASF. In order to announce it to the users, we have to follow the formal release process. There is no way around it.

As a PMC member, I simply cannot agree on a different approach as this is something we as PMCs are obliged to make sure happens and that we follow the right process. ASF indemnifies us - PMC members - from any legal obligations resulting from the release process - but only as long as we follow the rules of ASF. @dimberman - same with you, I am afraid :).  That's why any approach not following the ASF rules is a very strong -1 (veto) from my side.

@scottrigby -> sorry for being an obstacle here, I am going to be a bit more involved with the helm chart testing in the coming weeks (after my holidays) and I am happy to help in people having a smooth transition to the new chart, but it simply cannot be done in the form that one day we just take over your obligations towards the "stable helm" users. I am happy to help to make it easier, but Apache Airflow is a mature project already with a mature ASF organization and we have to follow the rules.

For the reference - here is the relevant chapter of the ASF policy:

Every ASF release must contain a source package, which must be sufficient for a user to build and test the release provided they have access to the appropriate platform and tools. The source package must be cryptographically signed by the Release Manager with a detached signature; and that package together with its signature must be tested prior to voting +1 for release. Folks who vote +1 for release may offer their own cryptographic signature to be concatenated with the detached signature file (at the Release Manager's discretion) prior to release.

Note that the PMC is responsible for all artifacts in their distribution directory, which is a subdirectory of downloads.apache.org ; and all artifacts placed in their directory must be signed by a committer, preferably by a PMC member. It is also necessary for the PMC to ensure that the source package is sufficient to build any binary artifacts associated with the release.

Every ASF release must comply with ASF licensing policy. This requirement is of utmost importance and an audit should be performed before any full release is created. In particular, every artifact distributed must contain only appropriately licensed code. More information can be found in the foundation website and in the release licensing FAQ.

[scottrigby]

@potiuk airflow voting processes make sense. Just a few quick replies:

• donation: the entire helm/charts project - this includes each chart there - is already licensed under an Apache License 2.0: https://github.com/helm/charts/blob/master/LICENSE, by The Helm Authors. We've already stated our interest in donating charts(s) sources to distributed repos, with a preference for maintainers of the software a chart is installing. So you're good on that side
• That said, we understand the app maintainers may not wish to take on maintenance and users for an existing chart, as you said. That part is up to the rules and preference of each community, and I respect your decision if I understood correctly. I just want to make sure a few points are clear from our side, like the ownership question above, so you have the best information in order to make the decision.
• The prometheus-community considered similar questions and decided to make a separate community repo and invite the contributors/maintainers of the stable prometheus charts continue to co-maintain them. Their process required a sponsor and vote - I understand your process is different, but in case this becomes the main sticking point, could that be a potential option for you?
• splicing in the git history: do you mean it doesn't make sense to do so with the current chart in this git repo? If so I agree. The question is do you plan to make versioned packages of that chart available to end users via a Helm repo? From experience in a last company, I can tell you end users are already unclear about that. But if so, I agree an easy depreciation notice for the stable chart is for users to use the one you have here.
• when a chart moves helm repos there must always be an upgrade path, but if the initial version in the new location changes only the Readme's install instructions, the upgrade path is usually as simple as adding the new repo and upgrading to the new chart path and version. However if the chart is structured differently, there's usually more burden on the end user for migrating. IIRC with Airflow what uses want to migrate are 1. DAGS and maybe 2. Run history? We can also simply say there is no migration path, but still point users here. Alternatively, as mentioned above, the Bitnami team hosts and maintains an Airflow chart, and the depreciation notice from stable could point end users there instead. Do you have a preference?

Thanks for your time and attention on this.

[potiuk]

• donation: the entire helm/charts project - this includes each chart there - is already licensed under an Apache License 2.0: https://github.com/helm/charts/blob/master/LICENSE, by The Helm Authors. We've already stated our interest in donating charts(s) sources to distributed repos, with a preference for maintainers of the software a chart is installing. So you're good on that side

Licence is fine, this is not about the licence but about the "ownership". So who is the "Licensor". From the Apache 2.0 Licence:

"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.

The donation process is about passing the ownership and it brings in certain responsibilities (including legal ones). We had a similar discussion with the CWL community and we decided not to accept a donation then. Here (since we have our own chart - donated by Astronomer) I personally think it makes very little sense to accept the donation.

• The prometheus-community considered similar questions and decided to make a separate community repo and invite the contributors/maintainers of the stable prometheus charts continue to co-maintain them. Their process required a sponsor and vote - I understand your process is different, but in case this becomes the main sticking point, could that be a potential option for you?

I think we have no such process/rules in ASF. We have very clear rules about software owned by the ASF. And I think - again - it makes little sense to support two different charts.

• splicing in the git history: do you mean it doesn't make sense to do so with the current chart in this git repo? If so I agree. The question is do you plan to make versioned packages of that chart available to end-users via a Helm repo?

Yes. That is the current plan I believe. @dimberman @schnie?

From experience in a last company, I can tell you end users are already unclear about that. But if so, I agree an easy depreciation notice for the stable chart is for users to use the one you have here.

• when a chart moves helm repos there must always be an upgrade path, but if the initial version in the new location changes only the Readme's install instructions, the upgrade path is usually as simple as adding the new repo and upgrading to the new chart path and version. However if the chart is structured differently, there's usually more burden on the end user for migrating.

I believe this is the case -the original chart is very different from the one we plan to release as a community one. It would be fantastic if someone reviews it and provides even a rough way to migrate. Just a starting point, a simple one might be a good idea - then we can point the users to it and ask then to update it. I think it is rather important to give something to the existing helm users, but the question is how to start it.

To be constructive - an idea I have for that - we could ask the users of the old chart to help to create it.

How about this: when we release our chart, you can make a note about it in the "stable" chart, clear deprecation timeline and we could together ask your users to migrate and provide some migration notes that can later turn into "migration guide" ?

This is something we successfully tried before - our community is fantastic and if asked, people will gladly spend their time and help us with it. Having it from the users will be also much more valuable as they will simply do it. Can you help with that @scottrigby ? We can possibly speed up our work on releasing the chart (even not fully tested). I might be able to help after I am back next week @dimberman @schnie - WDYT?

IIRC with Airflow what uses want to migrate are 1. DAGS and maybe 2. Run history?

It is not about DAGS/Run history. This is all stored in the DB and this is a matter of exporting/importing the DB (if at all). This should be independent from the chart changes IMHO - and should be case-by-case basis depending on the database used. Likely this might not even need a migration because, besides development setup, the DB should be external from the Kubernetes deployment (the Airflow deployment should be stateless) so in the vast majority of cases it's just a matter of connecting newly deployed Airflow to the existing database. It's more about moving the configuration of the helm chart itself - whether hand-deployed or via terraform or whatever. This might mean different user ids, volumes, sidecars etc. , different values in Chart, different capabilities, potentially some missing features that will need to be added or dropped by the users/replaced with other solutions.

We can also simply say there is no migration path, but still point users here. Alternatively, as mentioned above, the Bitnami team hosts and maintains an Airflow chart, and the depreciation notice from stable could point end users there instead. Do you have a preference?

If Bitnami can provide a clear migration path and support for the current 'stable' helm users - I would be all for it, actually.
This might be fantastic news for the current 'stable' helm users and great for the community! Then "new" users could focus on the chart from Airflow and "stable" helm users can go to Bitnami. I would be 100% for it. But If their chart is also much different than the previous "stable" one and they could not provide any migration/maintenance, then probably better to follow the "let the users help with building the migration path" approach to the soon-community managed chart that we might release soon.

@schnie @dimberman - what are your thoughts about it?

[scottrigby]

@potiuk I'm happy to help ask chart users and maintainers to contribute to a migration guide. The current stable/airflow maintainers are here: https://github.com/helm/charts/blob/master/stable/airflow/OWNERS. CCing @gsemet @maver1ck @thesuperzapper just so everyone is in the loop.

But ok yes that sounds like a good game plan, depending on what the Bitnami folks say. I'm pinging them about it now to see what they think.

[thesuperzapper]

Hi All,

I am the current maintainer of the stable/airflow chart.

A few quick observations:

1. I don't think the chart in apache/airflow is even close to feature parity with stable/airflow right now. (So a migration guide would NOT be possible)
2. The chart in apache/airflow is effectively undocumented right now, I expect it will need many pages of explanation and guides before it is usable by a normal person.
3. The Bitnami chart is very different from the current stable/airflow (it is effectively a very old version of stable/airflow)
4. [Important] the helm chart release versioning needs to be decoupled from airflow itself

[potiuk]

Let's talk about constructive approach then @thesuperzapper :).

I'd love to know how we can get the "airlfow" chart to a usable state by a normal person. The whole discussion is all about that I believe. We know that it misses a little to be completely ready for production, so I propose we all focus on making it happens as soon as possible:

1. Could you please point out what kind of documentation you think is missing in https://github.com/apache/airflow/blob/master/chart/README.md ? I am happy to help in improving the docs as a good community member, but I think your experience from the "stable" chart might be invaluable to point out what can be improved. Ideally a Github Issue would be great - then we can even ask our community members to improve it as we have an ongoing effort to improve documentation and our community members are super-helpful in improving it.

2. Could you please list the issues that are missing in the new chart and you think they are really useful and used in the 'stable chart'. I will be happy to bring some of those in (or help to decide that we don't migrate it). Ideally one issue per feature. Even if I am on vacations, I am happy to help in my free time to organize a bit the effort around it, label the issues and make some of them marked as "good-first-issue" so that our community members can possibly even help in implementing it. But we need to have those issues created to be able to act on them.

I think our common goal should be to make it as easy for the current 'stable' helm users to migrate to the new chart (that was the community decision rather than adopting the original help) and your help there might be super valuable.

Just a personal comment, I understand, and sorry for that if you feel a bit bitter that we have not adopted the chart of yours directly. I am afraid this was the community decision and voting on it passed a long time ago, and we made significant investments in making the current "airflow" helm chart part of the test/integration pipelines first - to make sure that we can support and maintain it in the long term. We often - as a community - make decisions that have no immediate impact but are made with the "long-term maintenance" point of view. The current chart will soon become a central point of "Kubernetes tests" - we are going to implement many more of those, we already unit-test part of it automatically and we have a commitment to make it rock solid and tested for 2.0 release (and also support latest 1.10.12).

I think the best course of action is now to work together on making it happen. I think your help in identifying what can be improved in the "airflow/chart" might be invaluable.

[gsemet]

Hello everybody,

I am the original maintainer of stable/airflow, along with @thesuperzapper. I just wanted to add some points, especially on how we would like to handle the transition.

1. stable/airflow Chart "donation"

There are two points heres:

• is it possible?
• is it wanted?

1.1 Is it possible to donate state/airflow?

About the chart donation of "stable/airflow" to the ASF, I of course do not have any problem and I guess most contributors won't neither, but on a pure legal point, I guess we would need approval of all contributors to the chart, of see with the Kubernetes legal team if this is even possible (there were no CLA signed, so the ownership is still "shared"). This is a community effort, and legal side might be tricky to deal with.

That said...

1.2 Is it even wanted?

I also understand the Airflow teams might not actually want the chart because it is (a little bit) opinionated, originally based on an another chart, it has "history" behind it, lot of features, etc. I can fully understand that, and of course if a migration guide can be written we will fully back it.

But I see some drawback here:

• migration might not be easy for users
• featureset might not be as close as wanted by most users
• I really hate when a project suddently disapear without a migration path, letting me control when I want to do the migration, I guess other users feel the same.
• we have the november deadline

2. Backward compatibility

My original thought was to have 2 charts, the 'stable/airflow' that would kind of be frozen or slowed down in it evolution and would die slowly in favor of the official airflow chart, and with the help of the community, the official chart will increase its featureset and replace naturally stable/airflow.

But with the november deadline, I would be sad to let users without any soft migration path (ie just change the repository and it continues to work until they are ready to invest time and effort into migrating to the official chart).

My opinion is that 'stable/airflow' should continue to live somewhere after the november deadline, ideally at Airflow, but I understand that might cause problem. We will probably create a dedidacated group on github and see if we can redo the CI in Github Actions... For the name, "airflow-chart-contrib"? "airflow-chart-community" ?

So there would be 2 charts:

• "stable/airflow" at a new location
• "airflow/chart"

Each one pointing to the other, with a clear explanation that which is the official chart and other is the community driven chart that is ultimately meant to be superseeded.

3. Review Process incompatibility

On the approval process itself, "stable/airflow" is really a product of the community. There is no roadmap. We strongly rely on the CI and the automatic checks and does an amazing job. I am actually surprised to see such a high level of contributions, most of the time I simply approve because the patch is great, the documentation is updated and even the migration section is well done. (Ex: helm/charts#23651, thanks @thesuperzapper !)

So I feel that the heavy/strong process Airflow project will not be compatible with the way this chart has been developed until now. I started it because I needed some features, and most of the features added later had been so by the community (I have not even tested all of these features!). But that's the role of well respecting the versioning contract (minor/major/bugfix) and having a powerful community (and the contributors to my chart has been very well educated and did very good code AND documentation), so now we have a feature-packed chart that is really stable and used in production, that evolved sometimes rapidly and things broke from time to time, but at the end, and it is very solid (that's my opinion :D )

I do not have any install statistics, would love to get some numbers.

4. Solutions?

Apart from the initialization scripts and some minor points, I do not see major blockers that would prevent users from switching from 'stable/airflow' to 'airflow/chart' when the migration guide will be ready and when the user is willing to. We are already using the airflow docker image.

Like I said, if Airflow would accept this "light" review process on this non-official charts but that would be still located under the Airflow umbrella for a time after november, that would be great for everyone. The other solution is we move to our own unofficial github group but that would still introduce ambiguity, I guess lot of forks will happen, users won't have a clear path to the new path of the "official-unofficial chart"...

Of course a migration guide to the official chart would be constructed over time, and with the help of the community, this should be feasible. I would like to avoid an "hard die" of 'stable/airflow', I would really like to see a "slow but controlled die" of the chart.

On the Airflow side, I would like to request its charts/documentation to clearly explain what it does differently from stable/airflow, and points to its new location after november 3rd so that existing user can still use it. Since we do not have any "homepage", I guess most users will try to find the Kubernetes chart for Airflow at Airflow... again, moving 'stable/airflow' under airflow (example: https://github.com/apache/airflow-legacy-chart) and let it die here would be ideal for us and the existing user base. My little finger tells me that is won't "died" instantaneously so there will be some contributions, even new features, so its versions will continue to increase but that would be still done like today, by the community.

Naturally, users will switch to the official chart if it is mature enough when they want to (that's very important for me, philosophically speaking) and then, later we can shut down 'stable/airflow' completely.

These are my opinions, I hope it will help this discussion.

[scottrigby]

@potiuk I agree the only way forward is to take a constructive approach.

Following up on the Bitnami discussion above, I've spoken with @carrodher who is also happy to collaborate with the existing stable/airflow chart maintainers to bring bitnami/airflow to feature parity, with a migration path from stable. Here's an issue to help track that effort: bitnami/charts#3580 @gsemet @maver1ck @thesuperzapper would you be willing to help with that?

@thesuperzapper if bitnami/airflow is from an older version of stable/airflow, we should be able to splice in the more recent commit history (with some thoughtful conflict resolution of course, given the Bitnami changes since).

@gsemet could this be effectively what you are looking for above - a community airflow chart with a clear migration path for users coming from stable?

And the Apache org airflow/airflow chart can develop at its own pace. Collaboration can still happen without being constrained by an existing userbase and specific features. @potiuk how does that sound?

For discoverability, both charts can be listed in CNCF Artifact Hub (the successor to Helm Hub). The Bitnami charts - including airflow - are already listed there. @potiuk if you'd like help listing the airflow helm repo charts once they're indexed, please feel free to reach out.

Does that cover all concerns for this issue? Are there any open thoughts?

[dimberman]

@scottrigby I think that sounds like a good idea. I agree that unfortunately the stable/airflow chart is far too diverged from our chart for any form of merging. We would love input from the stable/airflow maintainers on what documentation/features we are missing. Perhaps we can have a session to brainstorm issue tickets to port over some popular features? This would also be a great way to bring in the larger community as these documentation/feature tickets could be great "getting started" tickets.

WDYT @gsemet @potiuk ?

[potiuk]

Following up on the Bitnami discussion above, I've spoken with @carrodher who is also happy to collaborate with the existing stable/airflow chart maintainers to bring bitnami/airflow to feature parity, with a migration path from stable. Here's an issue to help track that effort: bitnami/charts#3580 @gsemet @maver1ck @thesuperzapper would you be willing to help with that?

That's perfect - as I explained before - if there is a clear migration path from the 'stable' chart to 'bitnami' and the bitnami is updated to the feature parity with it - that's the best solution. We are focusing on 2.0 efforts now and if we can focus on this rather than supporting users of the existing helm chart - that's perfect. - I have completely no problem if people choose another, well maintained Helm Chart for Airflow.

And some more context as well:

For us - Helm Chart is one of the "extra" artifacts which we might or might not provide and our users might, or might not use. The core of our project is releasing Apache Airflow Sources. We went a similar process with the Docker Image - replacing the old Puckel's one. The Dockerfile (sources) is already officially released together with Airflow Sources, well documented, supports OpenShift, and has a few more features that we will add soon. It's also very well tested (during the CI) and documented https://github.com/apache/airflow/blob/master/IMAGES.rst and the "stable" helm chart already uses it. And it has all the optimizations and process on how it can be extended or customized and even the whole Airflow Summit Talk that I gave about it. And I am going to apply for an official status at the DockerHub soon as well. We want, eventually, a similar level of docs/support/approach for the Helm Chart. We are not there yet.

First, we want to add a "Docker-Compose" configuration to it very soon. And Helm Chart will follow on its own pace. If we can have another solution in place that supports current users of 'stable' helm chart without bringing the extra effort to the team of Airflow Committers who work now on Airflow 2.0 - that's perfect. This will give us time to release it when it's ready and not earlier than that.

Just to reiterate - being able to bring in the issues with missing docs and features if we already know that we have them might be a great way to contribute to the Airflow Community - @gsemet @thesuperzapper @maver1ck. Your experience with the 'stable' chart might be invaluable to tell us what you think is important that we miss in the current chart and we can evaluate it and make it happen (either by the committers or users contributing those). That might even help us to build proper migration documentation.

Perhaps we can have a session to brainstorm issue tickets to port over some popular features?

Absolutely, I fully agree with @dimberman that session, where we could brainstorm about missing features, might be rather useful. We have weekly (as of next week) Monday meeting where we discuss 2.0 and Helm Chart is one of the topics. So maybe you would like to join one of those (Mondays 7.30 pm CEST). It's announced at our devlist and there is a calendar you can subscribe to @gsemet @thesuperzapper @maver1ck - let us know here if/when you would like to join so we can add it to the agenda.

Each one pointing to the other, with a clear explanation that which is the official chart and other is the community driven chart that is ultimately meant to be superseeded.

As far as the reference to either stable or Bitnami charts - sure, this is completely no problem. We are fully ready for that, and we have the very right place for all the things "Airflow" that are outside of the direct Apache Airflow Community scope/control and ASF ownership. We have some, rather clear, rules about referring to the information which is outside of the Airflow Community, and recently we introduced the Ecosystem page, where it is clear, that those links are outside of the main Airflow Community with a required disclaimer (ASF has some clear guidelines about that as well). The page is at the official "airlflow.apache.org" page, where we aim to have "All things Airflow" from the users (not contributors) perspective, so this seems like the perfect place for the link to the BitNami/Stable chart. The "chart" folder in GitHub is only for developers and people following the devlist. According to ASF rules we are not even supposed to advertise it or encourage the regular "users" before we release it.

Feel free to make PR to add those links even now! The process is super-simple - anyone can propose to add new entries there and open PR - which we will review and merge. @gsemet @thesuperzapper @maver1ck @carrodher: Feel free to make PR to that page - for both, stable one and Bitnami charts. I think they should be there already - but we copied the "tools and utils" section from JGhoman's "Awesome Apache Airflow" initially. When you scroll down the page you will get "Suggest a Change" on that page and it opens PR where you can add the right links. I will be glad to review it.

For discoverability, both charts can be listed in CNCF Artifact Hub (the successor to Helm Hub). The Bitnami charts - including airflow - are already listed there. @potiuk if you'd like help listing the airflow helm repo charts once they're indexed, please feel free to reach out.

Sure - thank you for your kind offer. I am sure we will reach out when we are ready (and not earlier than that :D ).

[thesuperzapper]

@potiuk, can you confirm if you plan to have the official helm chart releases tagged to airflow releases?

If so, I personally think that will cause big issues, as the helm chart and airflow itself are significantly different artifacts from a user perspective. (As an example, the helm chart that may not even need to change between airflow releases)

Almost every other project I am aware of has their helm chart releases completely independent of the app itself, and stored in a seperate repo. (Otherwise it would be impossible to push a critical fix for some helm specific issue without an app release)

Therefore, I think the discussion about apache/charts is important, because I expect we will have to move the offical chart out of this repo before we can properly release it.

If we did move to an apache/charts style repo, then we could easily have both the old stable/airflow and the new one in from this repo until they reach feature parity, after which we would drop support for stable/airflow, and help users with a migration guide. (This is important, because stable/airflow will have to go somewhere before the November deadline, as it is widely used, and there is not currently an alternative)

[gsemet]

There is this discussion about moving some charts (like... the Airflow one !) from helm/charts to apache/charts. That would be ideal for us. See helm/charts#23685

[potiuk]

@potiuk, can you confirm if you plan to have the official helm chart releases tagged to airflow releases?

Not exactly. We will use the same "repo" to release the helm chart but we can (and most likely will) release them with completely different cadence and separate tag scheme. During our CI process, we will make sure that the same helm chart will work with different versions of Airflow PR in progress for that is here: #9663. We discussed it in detail in the Separate Repo vs MonoRepo for Dockerfile & Helm Chart discussion and I think together with @dimberman and other committers we reached the conclusion that in our testing and development process keeping monorepo is the way to go. And - as discussed there - where the chart is developed has nothing to do wiht the cadence of releasing it. For sure the conclusion is that we do not want to keep a separate repo for helm chart. Even if many projects are doing it, I think this is not necessarily the best way of doing it for us, and we are trying to work-out the best process for that also on the ASF level. My proposal in the discussion started in ASF

Almost every other project I am aware of has their helm chart releases completely independent of the app itself, and stored in a seperate repo. (Otherwise it would be impossible to push a critical fix for some helm specific issue without an app release)

That is not necessarily correct (the "Otherwise" statement). We already release different packages (Backport packages) on a completely different cadence - from master. You can see it here: https://downloads.apache.org/airflow/ - as you will see there "Airflow" is released with "1.10.12" version and backport packages are released with different versioning scheme (CALVER based) - the last release is "2020.6.24". I am just going to release new set of backport packages (in a few days) and it is completely independent from the Airflow releases. They even use different branches (backports are released from master where Airflow is released from v1-10-stable. The release process produces different packages (both sources and binaries).

We have to follow the same process when we release the helm chart of ours. The package we are going to release will likely contain only helm chart sources packages in src.tar.gz that will need to be signed, checksummed, and voted on as required by the ASF Release Policy. And we will likely do it from master rather than from 1.10 branch.

Just to comment (again) - for Apache projects, those are the only official releases we can have and the only ones that we can advertise to our users. As a PMC member I am (together with others) legally responsible for doing it this way and ASF indemnifies me and other PMC members for any legal consequences but only if we are following the ASF policies. There is no way around it, really - we cannot violate those policies.

Therefore, I think the discussion about apache/charts is important, because I expect we will have to move the offical chart out of this repo before we can properly release it.

I agree it is important (and I already explained the currrent state of Airflow Chart and took part in the discussion by proposing how it could look like and I even offered a lot of my own time to make it happen in a consistent way for all ASF projects (I think it is super important to have good policies and mechanism for all ASF projects to release their charts).

But I totally disagree with "we will have to move the official chart out". There is nothing in this discussion that tells about moving where the charts are developed. The discussion is only how the charts are going to be released for use by the users. Those two things might be (and will likely be in my opninion) two different things if we want to follow the ASF rules. Similarly as the "airflow" source is developed in GitHub and released via Apache SVN.

In fact a proposal (not ny me - by Bertrand, another ASF member) is that we use the current SVN infrastructure (available via downloads.apache.org) to also expose the officially released helm charts. I very strongly agree with it. As long as we make the chart sources exposed by the webserver in the way that helm expects (including metadata) that sounds like the best solution. We'd only keep the "snapshots" there (we can do it automatically from the tags in GitHub) and users will only have acceess to the "officially released and voted"). We can even use existing mirroring infrastructure of the ASF to make it available to everone all over the world with unlimited scalability.

If we did move to an apache/charts style repo, then we could easily have both the old stable/airflow and the new one in from this repo until they reach feature parity, after which we would drop support for stable/airflow, and help users with a migration guide. (This is important, because stable/airflow will have to go somewhere before the November deadline, as it is widely used, and there is not currently an alternative)

If you think the discussion in ASF ends by November., then I think you do not realise how the ASF work. The discussion just started yesterday - it will take at least several weeks until everyone interested will have an opportunity to state their opinion, argue etc. then we will have to make a consistent proposal, then we will have to vote on it, and then it will take weeks by the ASF infrastructure to implement what's needed. This is a marathon, not a sprint and the discussion is not aimed to solve "current" problems but provide a long-term sustainable solution that will satisfy the needs of potentially 300+ ASF projects. Those decisions need time to discuss, weigh pros and cons and do very conscious and deliberate decision. If we have such a Chart repo by the end of the year, I would say this will be record-breaking speed.

And BTW. We might NEVER reach the parity of the current "helm/stable" chart. As a community we might simply decide that our chart is going to be simpler, and will not handle all the cases you implemented. All that at the expense of easier maintainability and handling say 90% of the cases, your chart handles. This has not yet been decided and discussed - and again - we are here for the marathon, not the sprint - we do not have to release the chart quickky and we will likely only release it around the 2.0 release timeline (which is by the enf of the year). We might even decide that we wait with the release until the ASF-wide solution is in place since the discussion started (that actually would be my vote now).

So, since as @scottrigby proposed and negotiated - going the "Bitnami route" for the current "stable" helm chart is the best course of action IMHO. As a member of the Airflow community, I have completely no problem if someone else maintains the chart of Airflow (as long as it is done well and it helps Airflow to grow - this is fantastic for the community). And since Bitnami already agreed to do it, I think you should focus your efforts on making sure this is happenning I think.

Just to reiterate that - for us, the Helm Chart is not the main focus. This is a bit of an aftertthought. Airflow Sources are the main "product" the community works on. It is important to have it at some point in time, but I think it is more important to have it released properly, according to ASF rules, tested well, including integration testing and long-term maintainable by the community - rather than to have it fast.

Especially that there is no the grreat route proposed by @scottrigby where Bitnami takes your chart over (which I am very happy about).

[gsemet]

We may be diverging actually here. ASF will handle their chart sources and releases the way they want.

My main interogation is still not answered: where do we put stable/airflow by November?

• ASF hosted project?
• Apache "Community" Project
• unofficial github group?

I think we (at least @thesuperzapper and I) can continue maintaining stable/airflow from some time, so the "Community Project" option is my preference.

[potiuk]

We may be diverging actually here. ASF will handle their chart sources and releases the way they want.

My main interogation is still not answered: where do we put stable/airflow by November?

@gsemet I believe it was answered in the discussion above #10523 (comment). @carrodher from Bitnami kindly offered that they adopt your changes in the Bitnami chart. Yoy have not responded to the proposal by @scottrigby, but I believe this is best solution.

[gsemet]

Following up on the Bitnami discussion above, I've spoken with @carrodher who is also happy to collaborate with the existing stable/airflow chart maintainers to bring bitnami/airflow to feature parity, with a migration path from stable. Here's an issue to help track that effort: bitnami/charts#3580 @gsemet @maver1ck @thesuperzapper would you be willing to help with that?

@thesuperzapper if bitnami/airflow is from an older version of stable/airflow, we should be able to splice in the more recent commit history (with some thoughtful conflict resolution of course, given the Bitnami changes since).

@gsemet could this be effectively what you are looking for above - a community airflow chart with a clear migration path for users coming from stable?

Ok actually I misread. The "rebase" should not be that hard to do with the bitnami chart if they share an history. From what I see they may have diverged a bit that's why I am a bit afraid not to have a solution by november...

[potiuk]

Ok actually I misread. The "rebase" should not be that hard to do with the bitnami chart if they share an history. From what I see they may have diverged a bit that's why I am a bit afraid not to have a solution by november...

Then I strongly suggest that you simply work with @carrodher asap then. I think we are discussing that for weeks if not months already and it's quite clear from the very beginning of the discussion that the Airflow Community is not going to take over directly or provide a good solution to the stable helm "deprecation" problem (one reason is the decision we made about adopting the Astronomer's chart and another about the way we want to follow the legalities and rules of the ASF).

Really sorry, we can't help with that, but I thought that was clear from the very beginning of the discussion.

[thesuperzapper]

@potiuk thanks heaps for your thorough and thoughtful replies, all-in-all I agree the direction you are going, and look forward to bringing some of the cool features of stable/airflow into the official chart.

In terms of stable/airflow itself, @gsemet and I will figure out were to migrate it, and will maintain it as long as is necessary untill an official alternative is released and adopted.

Currently the bitnami/charts repo looks promising, but we will have to discuss further with them.

Thanks again 👍

[Luttik]

Any news on this issue?

It is hard for users (like mysef) who need to deploy a new airflow infra to pick the right chart now.

[potiuk]

From Airflow side - everything goes according to the original plan.

We decided to release the community-managed Chart together with Airflow 2.0. Yesterday we published first Airflow 2.0 alfa and we are going to release it by the end of the year. It has already gone through a number of fixes and improvements as many people (including one of the projects we've done). I am going to resume discussion in ASF now about the policies (there was a person from OpenOffice who was very interested in the discussion and was on holidays for three weeks and I was busy with 2.0). But the discussion is already advanced and few iteration of the policy, now we will have to agree if some other interested parties (including Open Office) are happy with it, submit it to Legal, and get it approved by Board (but also any of this steps might fail - I will be relentlessly pushing to get it through though). So here - everything so far is according to our initial statements and plan.

I am not sure what happens with the idea of @thesuperzapper and @gsemet to host the "stable" one elsewhere - I will not speak for them :)

[gsemet]

We wanted to migrate https://github.com/helm/charts/tree/master/stable/airflow to https://github.com/airflow-helm before 1st november. We will maintain and evolve it until we think the airflow chart has completely replaced this chart.

Migration document is a must have, don't know who will do it :)

[sherifabdlnaby]

@potiuk
Is there are any updates on this?
For someone who is going to use Airflow on Kubernetes, should I go with the airflow/chart ? or with stable/chart? The only problem with airflow/chart that it is not released somewhere and I'll have to push it to my own registry and maintain syncing it with the mainstream. also, it will also use Airflow's release tags that are not necessary changes to the Chart itself.

[thesuperzapper]

I am still maintaining the stable/airflow chart, its new home is:
https://github.com/airflow-helm/charts/tree/main/charts/airflow

I will probably keep maintaining stable/airflow until most users migrate to the offical chart, (but that is unlikely to happen until the official chart releases, has feature parity, improves its docs, and has been around for a while).

BTW: I will be releasing Airflow 2.0.1 support in a few days for stable/airflow

[sherifabdlnaby]

@thesuperzapper

This issue raised a concern in my team that eventually airflow/chart is going to be the official and supported chart from now on, and that stable/airflow is eventually gonna be deprecated. So we wanted to start with airflow/chart initially to avoid having to migrate.

We initially experimented with stable/airflow but for some reason, it didn't work "out of the box" in our AWS EKS like apache/chart did. we didn't put much time yet as we're waiting on deciding on which chart to use for now.

[potiuk]

I think at the last dev call we discussed that we need to put some work to get it finished. There is a planning on what needs to be done and several issues created and I think very soon some real work will start to speed it up. Just follow the dev calls and announcements :)

[cheunhong]

@potiuk do you have an expected timeline for the official release?

[ashb]

@kaxil will be working on this "soon".

[kaxil]

Yes the Airflow Helm Chart is next. We were hands down on Airflow 2.0.0 and then getting 2.0.1 out so that the 2.0 series is stable.

[sryabkov]

@kaxil What's the best way to track progress on the chart work? Maybe contribute too... We spent a lot of time with the airflow stable chart and are looking to switch soon, so we have both experience and motivation to help out...

[sherifabdlnaby]

I am with @sryabkov , we already patched some bugs and added some features to the chart we forked (initially just to push it to a registry). I'd like to contribuite.

[kaxil]

@sryabkov @sherifabdlnaby I have created a Project Board to track the progress: https://github.com/apache/airflow/projects/7

Would love help with the following issues to start with

• Helm Chart: Add Integration tests & expand unit tests coverage #14301
• Helm Chart: Review Architecture #14302
• Review and Improve Current Documentation for Helm Chart #14303