Filter out default configs when overrides exist. #21539

xyu · 2022-02-12T18:30:58Z

When sending configs to Airflow workers we materialize a temp config file. In #18772 a feature was added so that _cmd generated secrets are not written to the files in some cases instead favoring maintaining the raw _cmd settings. Unfortunately during materializing of the configs via as_dict() Airflow defaults are generated and materialized as well including defaults for the non _cmd versions of some settings. And due to Airflow setting precedence stating bare versions of settings winning over _cmd versions it results in _cmd settings being discarded:
https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html

This change checks _cmd, env, and secrets when materializing configs via as_dict() so that if the bare versions of the values is exactly the same as Airflow defaults and we have "hidden" / special versions of these configs that are trying to be set we remove the bare versions so that the correct version can be used.

closes: #20092
related: #18772 #4050

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

When sending configs to Airflow workers we materialize a temp config file. In apache#18772 a feature was added so that `_cmd` generated secrets are not written to the files in some cases instead favoring maintaining the raw `_cmd` settings. Unfortunately during materializing of the configs via `as_dict()` Airflow defaults are generated and materialized as well including defaults for the non `_cmd` versions of some settings. And due to Airflow setting precedence stating bare versions of settings winning over `_cmd` versions it results in `_cmd` settings being discarded: https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html This change checks `_cmd`, env, and secrets when materializing configs via `as_dict()` so that if the bare versions of the values is exactly the same as Airflow defaults and we have "hidden" / special versions of these configs that are trying to be set we remove the bare versions so that the correct version can be used. Fixes: apache#20092 Related to: apache#18772 apache#4050

xyu · 2022-02-19T00:07:42Z

This should be a more general solution for #21604 as well

xyu · 2022-02-21T00:16:36Z

Merged trunk in because #21664 from @potiuk was needed to fix broken tests.

potiuk

LGTM - but I think description about this behaviour should be added to "as_dict" docstring/

github-actions · 2022-02-26T19:01:39Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

potiuk · 2022-02-28T00:26:08Z

And we need another committer chiming in.

uranusjr · 2022-02-28T23:06:35Z

It’d be nice if a docstring could be added to explain what filter_by_source does and why it is needed.

xyu · 2022-03-04T03:26:19Z

@potiuk / @uranusjr can you take a look at the docstrings I've added to see if they make sense? Also if one of you can trigger the full test run after the latest merge that would be awesome and let me know if there's anything else I should patch up with this PR to get it ready.

potiuk · 2022-03-14T01:06:12Z

After looking agian - I think what is also missing here is a bit "user" description in the https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html

I think we might have a lot of questions from users about the behaviour of config values as this is a "user" thing. I doubt our users will look in detail in "docstring" - they will mostly look at the documentation to understand how configuration precedence works, and especially - there should be a documentation explaining that it has changed in 2.3.0.

Would it be possible tha tyou add such "user" facing documentation @xyu ?

xyu · 2022-03-14T20:16:03Z

there should be a documentation explaining that it has changed in 2.3.0.

I'm not quite sure what you mean @potiuk, as you pointed out in the original issue (#20092 (comment)) the precedence for configuration options is bare setting over _cmd or secret versions. This PR filters out the bare configs if they are set to Airflow’s built in defaults which then allows the user set _cmd overrides to work again. So this is really fixing a bug not adding or changing config behavior. (I mean I guess it is changing the config behavior but it's changing the behavior to an unbroken as documented state.)

EDIT: Just to clarify with Airflow >= 2.2.1 when as_dict() was called if a "sensitive_config_values" had the bare value (e.g. sql_alchemy_conn) unset but a cmd or secret version of that set (e.g. sql_alchemy_conn_cmd) the dict produced by the as_dict() function would contain the Airflow default for the bare setting set. This in turn causes anything downstream, e.g. Airflow workers to ignore the cmd or secret version of that config and always use Airflow defaults.

potiuk · 2022-03-14T20:38:55Z

I'm not quite sure what you mean @potiuk

I mean that we should mention it was NOT working as expected before.

xyu · 2022-03-14T20:50:58Z

Oh I see as in for a changelog? Or do you mean add another "note" box to this page?

potiuk · 2022-03-14T20:55:53Z

Oh I see as in for a changelog? Or do you mean add another "note" box to this page?

I think would be nice to mentiont it on this page that it was working differently before 2.3.0

potiuk · 2022-03-15T18:06:38Z

Random environmental problems. Merging.

boring-cyborg · 2022-03-15T18:06:53Z

Awesome work, congrats on your first merged pull request!

* Filter out default configs when overrides exist. When sending configs to Airflow workers we materialize a temp config file. In #18772 a feature was added so that `_cmd` generated secrets are not written to the files in some cases instead favoring maintaining the raw `_cmd` settings. Unfortunately during materializing of the configs via `as_dict()` Airflow defaults are generated and materialized as well including defaults for the non `_cmd` versions of some settings. And due to Airflow setting precedence stating bare versions of settings winning over `_cmd` versions it results in `_cmd` settings being discarded: https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html This change checks `_cmd`, env, and secrets when materializing configs via `as_dict()` so that if the bare versions of the values is exactly the same as Airflow defaults and we have "hidden" / special versions of these configs that are trying to be set we remove the bare versions so that the correct version can be used. Fixes: #20092 Related to: #18772 #4050 (cherry picked from commit e07bc63)

xyu added 9 commits February 12, 2022 18:29

Black Reformat

5de0978

Merge branch 'main' into fix-20092

d45846c

Update tests

3170396

More tests

5f4f40b

Less double neg in test assertions

0064158

More Black

905fefd

Don't mess with global state

027a758

Wrap getter in a try/catch

d85ac8e

xyu force-pushed the fix-20092 branch from d4bb317 to d85ac8e Compare February 17, 2022 21:16

Catch errors, avoid leakage

bf8bf03

Merge branch 'main' into fix-20092

5da7ca2

kaxil added this to the Airflow 2.2.5 milestone Feb 21, 2022

potiuk approved these changes Feb 26, 2022

View reviewed changes

github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Feb 26, 2022

xyu added 2 commits February 28, 2022 15:23

Merge branch 'main' into fix-20092

8bf87c9

Merge branch 'main' into fix-20092

fbad1b1

xyu added 2 commits March 3, 2022 18:30

Add docstrings to explain config filtering, props @potiuk, @uranusjr

2fa8198

Merge branch 'main' into fix-20092

2a5ce39

xyu requested a review from potiuk March 3, 2022 18:33

Merge branch 'main' into fix-20092

1cae766

xyu added 3 commits March 4, 2022 20:05

Merge branch 'main' into fix-20092

56c9c63

Merge branch 'main' into fix-20092

6a7719a

Merge branch 'main' into fix-20092

3b7c51e

Merge branch 'main' into fix-20092

848f862

Add note about differing config precedence in some versions

46ece2d

xyu requested a review from kaxil as a code owner March 14, 2022 21:35

Merge branch 'main' into fix-20092

10c9174

potiuk approved these changes Mar 14, 2022

View reviewed changes

potiuk merged commit e07bc63 into apache:main Mar 15, 2022

ephraimbuddy added the type:bug-fix Changelog: Bug Fixes label Mar 16, 2022

ephraimbuddy mentioned this pull request Mar 27, 2022

Status of testing of Apache Airflow 2.2.5rc3 #22549

Closed

36 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filter out default configs when overrides exist. #21539

Filter out default configs when overrides exist. #21539

xyu commented Feb 12, 2022

xyu commented Feb 19, 2022

xyu commented Feb 21, 2022 •

edited

Loading

potiuk left a comment

github-actions bot commented Feb 26, 2022

potiuk commented Feb 28, 2022

uranusjr commented Feb 28, 2022

xyu commented Mar 4, 2022

potiuk commented Mar 14, 2022 •

edited

Loading

xyu commented Mar 14, 2022 •

edited

Loading

potiuk commented Mar 14, 2022

xyu commented Mar 14, 2022

potiuk commented Mar 14, 2022

potiuk commented Mar 15, 2022

boring-cyborg bot commented Mar 15, 2022

Filter out default configs when overrides exist. #21539

Filter out default configs when overrides exist. #21539

Conversation

xyu commented Feb 12, 2022

xyu commented Feb 19, 2022

xyu commented Feb 21, 2022 • edited Loading

potiuk left a comment

Choose a reason for hiding this comment

github-actions bot commented Feb 26, 2022

potiuk commented Feb 28, 2022

uranusjr commented Feb 28, 2022

xyu commented Mar 4, 2022

potiuk commented Mar 14, 2022 • edited Loading

xyu commented Mar 14, 2022 • edited Loading

potiuk commented Mar 14, 2022

xyu commented Mar 14, 2022

potiuk commented Mar 14, 2022

potiuk commented Mar 15, 2022

boring-cyborg bot commented Mar 15, 2022

xyu commented Feb 21, 2022 •

edited

Loading

potiuk commented Mar 14, 2022 •

edited

Loading

xyu commented Mar 14, 2022 •

edited

Loading