Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use joint context to cancel prestart hooks #6395

Merged
merged 2 commits into from
Oct 1, 2019

Conversation

drewbailey
Copy link
Contributor

The prestart hook for templates blocks while it resolves vault secrets.
If the secret is not found it continues to retry. If a task is shutdown
during this time, the prestart hook currently does not receive
shutdownCtxCancel, causing it to hang.

This PR joins the two contexts so either killCtx or shutdownCtx cancel
and stop the task.

fixes #6382

@hashicorp-cla
Copy link

hashicorp-cla commented Sep 27, 2019

CLA assistant check
All committers have signed the CLA.

Copy link
Contributor

@notnoop notnoop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a good improvement.

Though I'd be curious what the state of alloc/task when template rendering fails. Ideally, the alloc gets marked as failed, and then nomad client may free/gc up alloc as necessary. Doing so may require us to fix template failure handling but i'm not 100% sure.

client/allocrunner/taskrunner/task_runner_test.go Outdated Show resolved Hide resolved
Copy link
Member

@nickethier nickethier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

Copy link
Member

@schmichael schmichael left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Please update the comment here as well:

// The context is cancelled if the task is killed.

fixes #6382

The prestart hook for templates blocks while it resolves vault secrets.
If the secret is not found it continues to retry. If a task is shutdown
during this time, the prestart hook currently does not receive
shutdownCtxCancel, causing it to hang.

This PR joins the two contexts so either killCtx or shutdownCtx cancel
and stop the task.
@drewbailey drewbailey force-pushed the b-missing-vault-sercret branch from a8aeae0 to 6cf5573 Compare September 30, 2019 17:49
defer shutdown to cleanup after failed run

Co-Authored-By: Michael Schurter <[email protected]>

update comment to include ctx note for shutdown
@drewbailey drewbailey force-pushed the b-missing-vault-sercret branch from 6cf5573 to 12be120 Compare September 30, 2019 18:02
@drewbailey drewbailey merged commit e779d5b into master Oct 1, 2019
@drewbailey drewbailey deleted the b-missing-vault-sercret branch October 1, 2019 14:07
@schmichael schmichael added this to the 0.10.1 milestone Oct 8, 2019
@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 30, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Missing Vault secrets causes Nomad Client to ignore shutdown request
5 participants