Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ship zombie info #32693

Merged
merged 2 commits into from
Nov 17, 2023
Merged

Ship zombie info #32693

merged 2 commits into from
Nov 17, 2023

Conversation

Lee-W
Copy link
Member

@Lee-W Lee-W commented Jul 19, 2023

This PR depends on #32646 and adds scheduler log to task log if the task is detected as a zombie task.

How to test it

  1. Create a long-running task (e.g., BashOperator with bash_command="sleep 600")
  2. Set scheduler_zombie_task_threshold as 1
  3. Run airflow with breeze
  4. Trigger the DAG
  5. Kill work

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@RNHTTR
Copy link
Contributor

RNHTTR commented Jul 28, 2023

Thoughts on somehow linking to docs on zombies in the message? Or even just explaining in the message some things that might cause zombies? An Airflow end-user might not know what a zombie is.

@pankajkoti
Copy link
Member

Thoughts on somehow linking to docs on zombies in the message? Or even just explaining in the message some things that might cause zombies? An Airflow end-user might not know what a zombie is.

yes makes sense.

We can add "Refer: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#zombie-undead-tasks"

@pankajkoti
Copy link
Member

I am also thinking perhaps we need to enhance the documentation to include possible scenarios when tasks can become zombies based on all information we have observed so far.

@github-actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Sep 18, 2023
@github-actions github-actions bot closed this Sep 24, 2023
@pankajkoti pankajkoti reopened this Nov 14, 2023
@github-actions github-actions bot removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Nov 16, 2023
@Lee-W Lee-W force-pushed the ship-zombie-info branch 2 times, most recently from ecdad8c to 9f0d4cc Compare November 17, 2023 06:05
@Lee-W Lee-W marked this pull request as ready for review November 17, 2023 07:55
@pankajkoti
Copy link
Member

image

@RNHTTR this is how the message appears upon zombie detections. Please check if you would like to open a PR to modify the message or have something additional to add in the message. I will merge this one for now

cc: @dstandish @phanikumv @kaxil

@pankajkoti pankajkoti merged commit 69cea85 into apache:main Nov 17, 2023
46 checks passed
@pankajkoti pankajkoti deleted the ship-zombie-info branch November 17, 2023 08:39
@phanikumv
Copy link
Contributor

Have we improved doumentation as mentioned in earlier comments?

#32693 (comment)

@pankajkoti
Copy link
Member

Have we improved doumentation as mentioned in earlier comments?

#32693 (comment)

Yes, this can come up in a separate PR. I have created a good-first issue for this #35698

@ephraimbuddy ephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Nov 20, 2023
@ephraimbuddy ephraimbuddy added this to the Airflow 2.8.0 milestone Nov 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:logging area:providers area:Scheduler including HA (high availability) scheduler changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) provider:elasticsearch provider:microsoft-azure Azure-related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants