Add conn to jinja template context #16686

ecerulm · 2021-06-28T08:24:26Z

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

ecerulm · 2021-06-28T08:28:19Z

pinging @uranusjr , @potiuk , and @turbaszek as they were involved in the discussion of #14597

uranusjr · 2021-06-28T08:34:51Z

Logic looks good to me, but this won’t pass static checks (formatting issues). Please try to run those locally and fix them first.

airflow/models/taskinstance.py

ecerulm · 2021-06-28T09:01:54Z

Logic looks good to me, but this won’t pass static checks (formatting issues). Please try to run those locally and fix them first.

I did run those locally prior to submitting the PR. pre-commit run did not complain for me:

git fetch origin
pre-commit run --from-ref $(git merge-base origin/main jinjaconn) --to-ref jinjaconn

identity.................................................................................Passed
- hook id: identity
- duration: 0.11s

airflow/models/taskinstance.py
tests/models/test_taskinstance.py

Check hooks apply to the repository..................................(no files to check)Skipped
No-tabs checker..........................................................................Passed
Add license for all SQL files........................................(no files to check)Skipped
Add license for all other files......................................(no files to check)Skipped
Add license for all rst files........................................(no files to check)Skipped
Add license for all JS/TS/TSX/CSS/PUML files.........................(no files to check)Skipped
Add license for all JINJA template files.............................(no files to check)Skipped
Add license for all shell files......................................(no files to check)Skipped
Add license for all Python files.........................................................Passed
Add license for all XML files........................................(no files to check)Skipped
Add license for all YAML files.......................................(no files to check)Skipped
Add license for all md files.........................................(no files to check)Skipped
Add license for all mermaid files....................................(no files to check)Skipped
Add TOC for md files.................................................(no files to check)Skipped
black....................................................................................Passed
blacken-docs.............................................................................Passed
Check for merge conflicts................................................................Passed
Debug Statements (Python)................................................................Passed
Check builtin type constructor use.......................................................Passed
Detect Private Key.......................................................................Passed
Fix End of Files.........................................................................Passed
Mixed line ending........................................................................Passed
Check that executables have shebangs.................................(no files to check)Skipped
Check Xml............................................................(no files to check)Skipped
Trim Trailing Whitespace.................................................................Passed
Fix python encoding pragma...............................................................Passed
Pretty format JSON...................................................(no files to check)Skipped
pyupgrade................................................................................Passed
rst ``code`` is two backticks........................................(no files to check)Skipped
use logger.warning(......................................................................Passed
Check YAML files with yamllint.......................................(no files to check)Skipped
Run isort to sort imports in Python files................................................Passed
Run pydocstyle...........................................................................Passed
Lint OpenAPI using spectral..........................................(no files to check)Skipped
Lint OpenAPI using openapi-spec-validator............................(no files to check)Skipped
Lint dockerfile......................................................(no files to check)Skipped
Check order of dependencies in setup.cfg and setup.py................(no files to check)Skipped
Checks setup extra packages..........................................(no files to check)Skipped
Update output of breeze command in BREEZE.rst........................(no files to check)Skipped
Update mounts in the local yml file..................................(no files to check)Skipped
Update setup.cfg file with all licenses..............................(no files to check)Skipped
Build cross-dependencies for providers packages......................(no files to check)Skipped
Update extras in documentation.......................................(no files to check)Skipped
Check for pydevd debug statements accidentally left......................................Passed
Don't use safe in templates..........................................(no files to check)Skipped
No providers imports in core example DAGs............................(no files to check)Skipped
No relative imports......................................................................Passed
Check for language that we do not accept as community....................................Passed
Check for inconsistent pylint disable/enable without space...............................Passed
Check BaseOperator[Link] core imports....................................................Passed
Check BaseOperator[Link] other imports...............................(no files to check)Skipped
Check provide_session and create_session imports.........................................Passed
Make sure LoggingMixin is not used alone.................................................Passed
Make sure days_ago is imported from airflow.utils.dates..................................Passed
'start_date' should not be defined in default_args in example_dags...(no files to check)Skipped
Check if integration list is aligned.................................(no files to check)Skipped
Check if licenses are OK for Apache..................................(no files to check)Skipped
Checks for consistency between config.yml and default_config.cfg.....(no files to check)Skipped
Verify that DB Migrations have been documented.......................(no files to check)Skipped
Checks for Boring Cyborg configuration consistency.......................................Passed
Sort INTHEWILD.md alphabetically.....................................(no files to check)Skipped
Sort alphabetically and uniquify spelling_wordlist.txt...............(no files to check)Skipped
Lint Helm Chart......................................................(no files to check)Skipped
Check Shell scripts syntax correctness...............................(no files to check)Skipped
Run BATS bash tests for changed Breeze bash files....................(no files to check)Skipped
Run BATS bash tests for changed bash files...........................(no files to check)Skipped
stylelint............................................................(no files to check)Skipped
Provider init file.......................................................................Passed
Validate providers.yaml files........................................(no files to check)Skipped
Generate mermaid images..............................................(no files to check)Skipped
Check if pre-commits are described...................................(no files to check)Skipped
Ensure hook ids are not overly long..................................(no files to check)Skipped
Checks providers available when declared by extras in setup.py.......(no files to check)Skipped
Run markdownlint.....................................................(no files to check)Skipped
Lint JSON Schema files with JSON Schema..............................(no files to check)Skipped
Lint NodePort Service with JSON Schema...............................(no files to check)Skipped
Lint Docker compose files with JSON Schema...........................(no files to check)Skipped
Lint chart/values.schema.json file with JSON Schema..................(no files to check)Skipped
Lint chart/values.yaml file with JSON Schema.........................(no files to check)Skipped
Lint airflow/config_templates/config.yml file with JSON Schema.......(no files to check)Skipped
Convert to f-strings with flynt..........................................................Passed
ESLint against airflow/ui............................................(no files to check)Skipped
ESLint against current UI js files...................................(no files to check)Skipped
Run in container bats tests..........................................(no files to check)Skipped
Check if image build is needed...........................................................Passed
Run mypy.................................................................................Passed
Run mypy for helm chart tests........................................(no files to check)Skipped
Run mypy for /docs/ folder...........................................(no files to check)Skipped
Run pylint for main code.................................................................Passed
Run pylint for tests.....................................................................Passed
Run pylint for helm chart tests......................................(no files to check)Skipped
Run flake8...............................................................................Passed

Also it seems to be passing the static checks in CI too https://github.com/apache/airflow/pull/16686/checks?check_run_id=2929988055

ecerulm · 2021-06-28T09:35:08Z

@potiuk The CI job is failing due to pylint in a file I didn't touch:

tests/jobs/test_scheduler_job.py:3663:15: E1101: Instance of 'str' has no 'task_id' member (no-member)

I'm rebased to the latest origin/branch commit 2833c2b

potiuk · 2021-06-28T09:40:04Z

Yeah. We are just about to remove pylint. Don't worry about that

potiuk · 2021-06-28T09:51:34Z

See #16682

github-actions · 2021-06-28T12:52:47Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

ecerulm · 2021-06-28T12:57:40Z

I did change the connection accessor syntax from conn.value.my_connection.host to conn.my_connection.host as per request of @ashb.

Also, I guess I need to do a PR towards airflow-site too? https://github.com/apache/airflow/edit/main/docs/apache-airflow/macros-ref.rst , right? Do I do that already now or we shall wait until this PR is merged.

potiuk · 2021-06-28T13:02:05Z

Also, I guess I need to do a PR towards airflow-site too? https://github.com/apache/airflow/edit/main/docs/apache-airflow/macros-ref.rst , right? Do I do that already now or we shall wait until this PR is merged.

Good point. Not really PR to airflow-site - you need to update the https://github.com/apache/airflow/blob/main/docs/apache-airflow/macros-ref.rst in the current PR. The docs at "airflow-site" are generated from there when we release.

eladkal · 2021-06-28T13:07:33Z

Question: how is it going to work for connection custom fields? for example: Asana has workspace in a separate connection field

kaxil · 2021-06-28T13:10:44Z

Question: how is it going to work for connection custom fields? for example: Asana has workspace in a separate connection field

That should be {{ conn.a_connection.extra_dejson['extra__asana__workspace'] }} but good point, we should add test for such case probably

kaxil

We should add docs in https://airflow.apache.org/docs/apache-airflow/stable/concepts/connections.html (similar to https://airflow.apache.org/docs/apache-airflow/stable/concepts/variables.html) about getting connections in template.

And in https://airflow.apache.org/docs/apache-airflow/stable/macros-ref.html

The source code for those files are:

ecerulm · 2021-06-28T14:07:02Z

Ok, I added testcase for extra_dejson and update documentation in macros-ref.rst and connnections.rst

potiuk

Really nice now!

potiuk · 2021-06-28T18:42:58Z

Would you please rebase @ecerulm ? We have merged two stability improvements for CI (pylint + helm race condition). It should help a a bit :)

tests/models/test_taskinstance.py

docs/apache-airflow/concepts/connections.rst

docs/apache-airflow/macros-ref.rst

ashb · 2021-06-29T07:57:29Z

{{ conn.a_connection.extra_dejson['extra__asana__workspace'] }}

Because jinja is flexible this can also be done as

{{ conn.a_connection.extra_dejson.extra__asana__workspace }}

ashb · 2021-06-29T07:59:09Z

airflow/models/taskinstance.py

+            """
+            Wrapper around Connection. This way you can get connections in
+            templates by using ``{{ conn.conn_id }}`` or
+            ``{{ conn.get('conn_id', 'fallback') }}``.


Does a fallback make sense here? For variables is was easier as it is a "simple" object, but connection is much more complex, so I can't think what we would put in the fallback value to make the template be useful?

(Not saying we should remove it, just that maybe the 'fallback' isn't the best example here.

Ah, you've got a useful example in the macros doc. Maybe just put a "see macros" here to avoid duplication.

docs/apache-airflow/concepts/connections.rst

airflow/models/taskinstance.py

ashb · 2021-06-29T08:04:31Z

(None of my comments are blocking -- if you think they are useful go for it)

ecerulm · 2021-06-29T08:07:31Z

@potiuk the CI still fails for reason I believe unrelated to the content of this PR. I can rebase again but judging from the last PR merged https://github.com/apache/airflow/pull/15397/checks which had the same checks failing as this one, I don't think that will help.

potiuk · 2021-06-29T08:10:49Z

Yeah. It looks like a problem with the GitHub (public) runners this time. There seem to be no relation between Python/DB or any other consistent pattern with those failures. Maybe you could apply some of the comments from @ashb and when you do - we merge it finally.

closes: apache#14597

potiuk · 2021-06-29T10:49:48Z

Something fishy going on with helm installation on public runners randomly:

  Error: failed post-install: timed out waiting for the condition

potiuk · 2021-06-29T10:49:58Z

But unrelated. Merging.

ecerulm requested review from ashb, kaxil, turbaszek and XD-DENG as code owners June 28, 2021 08:24

ashb reviewed Jun 28, 2021

View reviewed changes

airflow/models/taskinstance.py Outdated Show resolved Hide resolved

ashb reviewed Jun 28, 2021

View reviewed changes

airflow/models/taskinstance.py Outdated Show resolved Hide resolved

ecerulm force-pushed the jinjaconn branch 2 times, most recently from f4dca29 to 57283d4 Compare June 28, 2021 12:49

ecerulm requested a review from ashb June 28, 2021 12:50

potiuk approved these changes Jun 28, 2021

View reviewed changes

github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Jun 28, 2021

kaxil reviewed Jun 28, 2021

View reviewed changes

ecerulm requested a review from kaxil June 28, 2021 14:07

potiuk approved these changes Jun 28, 2021

View reviewed changes

kurtqq reviewed Jun 28, 2021

View reviewed changes

tests/models/test_taskinstance.py Show resolved Hide resolved

ecerulm force-pushed the jinjaconn branch from 585f944 to 313bdc4 Compare June 28, 2021 19:27

eladkal approved these changes Jun 28, 2021

View reviewed changes

kaxil approved these changes Jun 28, 2021

View reviewed changes

uranusjr reviewed Jun 29, 2021

View reviewed changes

docs/apache-airflow/concepts/connections.rst Outdated Show resolved Hide resolved

docs/apache-airflow/macros-ref.rst Outdated Show resolved Hide resolved

ashb reviewed Jun 29, 2021

View reviewed changes

ecerulm added 5 commits June 29, 2021 10:33

Add conn to jinja template context

998d78a

closes: apache#14597

Change conn.value.{host,...} to just conn.{host,...}

97006e9

Add conn extras testcase and documentation

1390100

Update doc

5ae647f

Add test case

a535dd0

ecerulm force-pushed the jinjaconn branch from 79711a4 to a535dd0 Compare June 29, 2021 08:34

ashb approved these changes Jun 29, 2021

View reviewed changes

uranusjr approved these changes Jun 29, 2021

View reviewed changes

potiuk merged commit d3ba80a into apache:main Jun 29, 2021

ashb added this to the Airflow 2.2 milestone Jun 29, 2021

ecerulm deleted the jinjaconn branch June 29, 2021 13:52

josh-fell mentioned this pull request Oct 20, 2021

Add dag to extract data from zendesk to snowflake astronomer/success-registry-dags#1

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add conn to jinja template context #16686

Add conn to jinja template context #16686

ecerulm commented Jun 28, 2021

ecerulm commented Jun 28, 2021

uranusjr commented Jun 28, 2021

ecerulm commented Jun 28, 2021 •

edited

Loading

ecerulm commented Jun 28, 2021

potiuk commented Jun 28, 2021

potiuk commented Jun 28, 2021

github-actions bot commented Jun 28, 2021

ecerulm commented Jun 28, 2021

potiuk commented Jun 28, 2021

eladkal commented Jun 28, 2021

kaxil commented Jun 28, 2021 •

edited

Loading

kaxil left a comment

ecerulm commented Jun 28, 2021

potiuk left a comment

potiuk commented Jun 28, 2021

ashb commented Jun 29, 2021

ashb Jun 29, 2021

ashb Jun 29, 2021

ashb commented Jun 29, 2021

ecerulm commented Jun 29, 2021

potiuk commented Jun 29, 2021

potiuk commented Jun 29, 2021

potiuk commented Jun 29, 2021

Add conn to jinja template context #16686

Add conn to jinja template context #16686

Conversation

ecerulm commented Jun 28, 2021

ecerulm commented Jun 28, 2021

uranusjr commented Jun 28, 2021

ecerulm commented Jun 28, 2021 • edited Loading

ecerulm commented Jun 28, 2021

potiuk commented Jun 28, 2021

potiuk commented Jun 28, 2021

github-actions bot commented Jun 28, 2021

ecerulm commented Jun 28, 2021

potiuk commented Jun 28, 2021

eladkal commented Jun 28, 2021

kaxil commented Jun 28, 2021 • edited Loading

kaxil left a comment

Choose a reason for hiding this comment

ecerulm commented Jun 28, 2021

potiuk left a comment

Choose a reason for hiding this comment

potiuk commented Jun 28, 2021

ashb commented Jun 29, 2021

ashb Jun 29, 2021

Choose a reason for hiding this comment

ashb Jun 29, 2021

Choose a reason for hiding this comment

ashb commented Jun 29, 2021

ecerulm commented Jun 29, 2021

potiuk commented Jun 29, 2021

potiuk commented Jun 29, 2021

potiuk commented Jun 29, 2021

ecerulm commented Jun 28, 2021 •

edited

Loading

kaxil commented Jun 28, 2021 •

edited

Loading