Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve handling of string type and non-attribute template_fields #21054

Merged

Conversation

josh-fell
Copy link
Contributor

@josh-fell josh-fell commented Jan 24, 2022

Related: #19047, #19052
Closes: #12876

This PR intends to improve both the handling of template_fields values declared as strings (most typically seen as template_fields = ("some_field") as well as the error message for declaring template_fields that are not attributes of the operator. In the first case, a warning is logged and the template_fields is "automagically" converted to a list for the task execution. For the second case, the error message is updated from the default AttributeError to something more specific to what is happening when attempting to render any template fields.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

Comment on lines 747 to 755
f"The `template_fields` value for {self.task_type} is a string "
"but should be a Sequence type. Converting to a Sequence type for execution. "
f"Please update {self.task_type} accordingly.",
Copy link
Member

@uranusjr uranusjr Jan 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
f"The `template_fields` value for {self.task_type} is a string "
"but should be a Sequence type. Converting to a Sequence type for execution. "
f"Please update {self.task_type} accordingly.",
f"The `template_fields` value for {self.task_type} is a string "
"but should be a list of string. Wrapping it in a list for execution. "
f"Please update {self.task_type} accordingly.",

str is technically a sequence, so sequence of string is a better term. But I feel the term sequence itself might not be the easiest to understand for less experienced developers, so let’s just recommend using a list instead, which is the most fool-proof syntax.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent point.

Comment on lines 1098 to 1073
if hasattr(parent, attr_name):
content = getattr(parent, attr_name)
if content:
rendered_content = self.render_template(content, context, jinja_env, seen_oids)
setattr(parent, attr_name, rendered_content)
else:
raise AttributeError(
f"{attr_name!r} is configured as a template field "
f"but {parent.task_type} does not have this attribute."
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if hasattr(parent, attr_name):
content = getattr(parent, attr_name)
if content:
rendered_content = self.render_template(content, context, jinja_env, seen_oids)
setattr(parent, attr_name, rendered_content)
else:
raise AttributeError(
f"{attr_name!r} is configured as a template field "
f"but {parent.task_type} does not have this attribute."
)
try:
content = getattr(parent, attr_name)
except AttributeError:
raise AttributeError(
f"{attr_name!r} is configured as a template field "
f"but {parent.task_type} does not have this attribute."
)
else:
if content:
rendered_content = self.render_template(content, context, jinja_env, seen_oids)
setattr(parent, attr_name, rendered_content)

EAFP over LBYL.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

except ... else not needed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes. More succinct as well.

Comment on lines 1370 to 1286
if hasattr(self, field):
content = getattr(self, field)
if content and isinstance(content, str):
self.log.info('Rendering template for %s', field)
self.log.info(content)
else:
raise AttributeError(
f"{field!r} is configured as a template field "
f"but {self.task_type} does not have this attribute."
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

@josh-fell josh-fell force-pushed the operator-template-fields-issue-handling branch 2 times, most recently from e745924 to abd1b85 Compare January 24, 2022 14:43
@josh-fell josh-fell force-pushed the operator-template-fields-issue-handling branch from abd1b85 to 69a0d62 Compare January 24, 2022 16:45
@josh-fell
Copy link
Contributor Author

Pushed some fixes for static check errors...come on Josh. 🤦

@potiuk
Copy link
Member

potiuk commented Jan 24, 2022

Pushed some fixes for static check errors...come on Josh. facepalm

Hint:

pre-commit run --from-ref=HEAD^ --to-ref=HEAD

And a sneaky preview of new Breeze2 equivalent:

./Breeze2 static-checks --last-commit

@potiuk
Copy link
Member

potiuk commented Jan 24, 2022

cc: @Bowrna ^^

@josh-fell
Copy link
Contributor Author

Pushed some fixes for static check errors...come on Josh. facepalm

Hint:

pre-commit run --from-ref=HEAD^ --to-ref=HEAD

And a sneaky preview of new Breeze2 equivalent:

./Breeze2 static-checks --last-commit

Yeah that's usually my go-to move but I completely spaced on it this time. That new command is lovely. I'll have to try it out.

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Jan 26, 2022
@github-actions
Copy link

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice. Love the explicit communication!

@potiuk potiuk merged commit beb2a20 into apache:main Jan 26, 2022
@josh-fell josh-fell deleted the operator-template-fields-issue-handling branch January 26, 2022 19:48
@jedcunningham jedcunningham added the type:improvement Changelog: Improvements label Mar 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
full tests needed We need to run full set of tests for this PR to merge type:improvement Changelog: Improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve error message when template fields contain invalid entries
5 participants