feat(jobs/triggerer_job_runner): add triggerer canceled log #31757
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
the
area:Scheduler
Lee-W
force-pushed
the
add-a-log-message-when-a-trigger-is-canceled
branch
2 times, most recently
from
e7ed504 to
fd96e7d
Compare
1 task
o-nikolas
reviewed
Jun 7, 2023
Lee-W
force-pushed
the
add-a-log-message-when-a-trigger-is-canceled
branch
2 times, most recently
from
92a4772 to
efd4e0c
Compare
Lee-W
force-pushed
the
add-a-log-message-when-a-trigger-is-canceled
branch
from
69471d7 to
f98d0fe
Compare
phanikumv
reviewed
Jun 13, 2023
Lee-W
force-pushed
the
add-a-log-message-when-a-trigger-is-canceled
branch
from
f98d0fe to
9e65ed4
Compare
|
@dstandish I tried to dig into the code a bit and found out the error message mentioned in #30853 (comment) is from check_trigger_timeouts. Thus, I tried to add a similar timeout check here. I've not yet had time to test it today but would like to see what you think about it. Thanks! |
dstandish
reviewed
Jun 13, 2023
dstandish
reviewed
Jun 13, 2023
Lee-W
force-pushed
the
add-a-log-message-when-a-trigger-is-canceled
branch
3 times, most recently
from
1679327 to
d5580b6
Compare
uranusjr
reviewed
Jun 14, 2023
…ror when logging triggerer timeout
Lee-W
force-pushed
the
add-a-log-message-when-a-trigger-is-canceled
branch
from
0fe0cd4 to
c38635b
Compare
uranusjr
reviewed
Jun 16, 2023
…o error when encountering asyncio.CancelledError
jedcunningham
approved these changes
Jun 16, 2023
dstandish
reviewed
Jun 16, 2023
airflow/jobs/triggerer_job_runner.py
Outdated
| trigger_timeout = trigger.task_instance.trigger_timeout | ||
| if trigger_timeout: | ||
| if not trigger_timeout.tzinfo: | ||
| trigger_timeout = trigger_timeout.replace(tzinfo=timezone.utc) |
There was a problem hiding this comment.
curious why the the tzinfo line is necessary. it seems that it should already be UTC. are you just being defensive or did you encounter an issue?
There was a problem hiding this comment.
Yep, I encountered the timezone issue after I change the fallback here to 1.0 for testing.
dstandish
reviewed
Jun 16, 2023
… operator Co-authored-by: Daniel Standish <[email protected]>
syedahsn
reviewed
Jun 16, 2023
| if timeout := trigger.task_instance.trigger_timeout: | ||
| timeout = timeout.replace(tzinfo=timezone.utc) if not timeout.tzinfo else timeout | ||
| if timeout < timezone.utcnow(): | ||
| self.log.error("Trigger cancelled due to timeout") |
There was a problem hiding this comment.
In the timeout case, does it log the error twice?
There was a problem hiding this comment.
Yes, it logs the timeout error and the original asyncio.CancelledError message
dstandish
approved these changes
Jun 16, 2023
ephraimbuddy
pushed a commit
that referenced
this pull request
Jul 6, 2023
Emit log message when trigger is cancelled Co-authored-by: Daniel Standish <[email protected]> Co-authored-by: Jed Cunningham <[email protected]> (cherry picked from commit a60429e)
ephraimbuddy
pushed a commit
that referenced
this pull request
Jul 7, 2023
53 tasks
pankajkoti
added a commit
to astronomer/airflow
that referenced
this pull request
Aug 29, 2023
We tried to address apache#31720 in PR apache#31757. The issue talked about adding a trigger cancelled log when trigger timesout, but we also added a generic Trigger canceled log. This log appears even in successful runs of the triggers when they finish. This is confusing some users as the log level is Error and there are sometimes quite a few log lines saying "Trigger cancelled; err=" with giving no clue as to what is happening. So, I am removing this generic error log line and we can add specific cancel scenarios with detailed reasons when we implement those.
This was referenced Aug 29, 2023
potiuk
pushed a commit
that referenced
this pull request
Aug 29, 2023
We tried to address #31720 in PR #31757. The issue talked about adding a trigger cancelled log when trigger timesout, but we also added a generic Trigger canceled log. This log appears even in successful runs of the triggers when they finish. This is confusing some users as the log level is Error and there are sometimes quite a few log lines saying "Trigger cancelled; err=" with giving no clue as to what is happening. So, I am removing this generic error log line and we can add specific cancel scenarios with detailed reasons when we implement those.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes: #31720
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.