Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bug where Atlantis hangs on terraform crash. #421

Merged
merged 3 commits into from
Jan 11, 2019
Merged

Conversation

lkysow
Copy link
Member

@lkysow lkysow commented Jan 10, 2019

Because of some weirdness with how terraform executes (it forks itself
and assigns the parent's stderr and stdout file descriptors to the
child) when terraform panics, Go's exec.CombinedOutput() blocks
indefinitely even though the terraform process has exited.

To handle this, we switch to using os.Pipe for the cmd's stderr/out
which for some reason allows Go to know when the process has exited.
Unfortunately, this brings up a new problem when reading the output from
that pipe. For some reason, after a terraform crash, we never get an EOF
on the pipe and so as we again get in a situation where we're blocking
indefinitely. To handle this, I've added an edge case where we look at
the log lines and if its the end of a terraform panic, we break out of
the loop.

Thanks Terraform 0.12-alpha for discovering this bug!

Because of some weirdness with how terraform executes (it forks itself
and assigns the parent's stderr and stdout file descriptors to the
child) when terraform panics, Go's exec.CombinedOutput() blocks
indefinitely even though the terraform process has exited.

To handle this, we switch to using os.Pipe for the cmd's stderr/out
which for some reason allows Go to know when the process has exited.
Unfortunately, this brings up a new problem when reading the output from
that pipe. For some reason, after a terraform crash, we never get an EOF
on the pipe and so as we again get in a situation where we're blocking
indefinitely. To handle this, I've added an edge case where we look at
the log lines and if its the end of a terraform panic, we break out of
the loop.

Thanks Terraform 0.12-alpha for discovering this bug!
@codecov
Copy link

codecov bot commented Jan 11, 2019

Codecov Report

Merging #421 into master will increase coverage by 0.04%.
The diff coverage is 86.66%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #421      +/-   ##
==========================================
+ Coverage   70.23%   70.27%   +0.04%     
==========================================
  Files          63       63              
  Lines        3924     3943      +19     
==========================================
+ Hits         2756     2771      +15     
- Misses        972      974       +2     
- Partials      196      198       +2
Impacted Files Coverage Δ
server/events/terraform/terraform_client.go 75.26% <86.66%> (+0.94%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update acf46ba...1d7bb42. Read the comment docs.

@lkysow lkysow merged commit 173fdde into master Jan 11, 2019
@lkysow lkysow deleted the handle-panics branch January 11, 2019 00:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant