-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use sql_alchemy_conn for celery result backend when result_backend is not set #24496
Use sql_alchemy_conn for celery result backend when result_backend is not set #24496
Conversation
db+
prefix if missing for celery result backend
A little unsure about this one. On one hand I get the pain, and like the idea of making it more convenient. On the other hand it's good to be explicit. Another way of approaching this... what if we just made it so that if celery result backend is not configured, then it would take the database as the default (with prefix)? So then, if it's configured, we use what we are given (even if it's wrong because missing db prefix, thus forcing the user to update the config to be correct). But if it's not configured, i.e. so that there is no value for |
@dstandish For me the issue is mostly related to airflow helm chart/kubernetes: I have the same problem as described in #18283 - I want to use the same pre-created secret for both metadata and resultBackendConnection in helm chart, see https://github.com/apache/airflow/blob/main/chart/values.yaml#L322 . Currently it's not possible because one of the secrets has to start with "db+". I suppose using sql_alchemy_conn when result_backend is not configured would work, but in this case some changes will be required in the helm chart too, will it be ok? For example currently with celery executor this secret for result_backend connection is created every time when custom result backend secret is not provided and the env for the connection is always defined. |
i see re https://github.com/apache/airflow/blob/main/chart/templates/secrets/result-backend-connection-secret.yaml#L21 we create the secret when none referenced in your config. good point. so yeah if you like this suggestion, i.e. to deal with this in airflow config, then i think in the chart we just have to look at airflow version and if greater than 2.4 (or whenever this hypothetical change implemented) then don't create the secret (because sql_alchemy_conn will be used by default anyway) |
Sure, I'll try implementing the change so sql_alchemy_conn is used when result_backend is not set. Looks like I'll have to change default value of result_backend to None (currently it is set to db+postgresql://postgres:airflow@postgres/airflow - hopefully it won't be too much of a problem. |
db+
prefix if missing for celery result backend498efd3
to
2a89d02
Compare
1f43dce
to
26cec60
Compare
The PR is ready for review - not sure why WIP check is still in progress :/ I can retrigger the checks if needed. As @potiuk suggested, I modified helm chart tests with |
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
26cec60
to
6426d27
Compare
Nice! You made it with parameterizzed tests :) |
087d0cb
to
6c3cfe0
Compare
6c3cfe0
to
890749c
Compare
All the tests have finally passed 😌 |
d8df6ba
to
ea893ac
Compare
ea893ac
to
c6e201e
Compare
c6e201e
to
f2d7d84
Compare
f2d7d84
to
e033b68
Compare
This PR adds possibility to use the same connection for celery result_backend and metadata by automatically adding
db+
for result_backend connection.closes #18283
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragement file, named
{pr_number}.significant.rst
, in newsfragments.@wip ready for review