Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/thanos/receive: fix close chan panic #1595

Merged
merged 2 commits into from
Oct 2, 2019

Conversation

squat
Copy link
Member

@squat squat commented Oct 1, 2019

This commit fixes a panic caused by sending on the chan which is closed
too early. This chan was closed when the setup func returned, rather
than when the group's goroutine returned.

Fixes: #1594
Signed-off-by: Lucas Servén Marín [email protected]

@dtrejod
Copy link
Contributor

dtrejod commented Oct 1, 2019

You also close the uploadC channel twice.

defer close(uploadC)

defer close(uploadC)

@squat
Copy link
Member Author

squat commented Oct 1, 2019

Yes you’re totally right. This is an unrelated issue but a bug nonetheless. I’ll address this in the same pr

This commit fixes a panic caused by sending on the chan which is closed
too early. This chan was closed when the setup func returned, rather
than when the group's goroutine returned.

Fixes: thanos-io#1594
Signed-off-by: Lucas Servén Marín <[email protected]>
@dtrejod
Copy link
Contributor

dtrejod commented Oct 1, 2019

Confirmed these changes resolved the panics. However hit another issue where the TSDB doesn't start back up after a flush event. Looks to me this cancel call closes the upload on demand function prematurely --

cancel()

@squat
Copy link
Member Author

squat commented Oct 1, 2019

Yeah you’re right. That cancel is leftover from a previous iteration where the upload used a cancellable context and cancel() needed to be called after the upload to clean up.

This commit fixes how the uploader is cleaned up. Currently, it was
cleaned up in a defer block that was run before the run groups. This
ensures the cleanup occurs only when the run groups are returning.

Signed-off-by: Lucas Servén Marín <[email protected]>
@brancz brancz merged commit 4c7ecd3 into thanos-io:master Oct 2, 2019
@squat squat deleted the fixreceivepanic branch October 22, 2019 09:42
GiedriusS pushed a commit that referenced this pull request Oct 28, 2019
* cmd/thanos/receive: fix close chan panic

This commit fixes a panic caused by sending on the chan which is closed
too early. This chan was closed when the setup func returned, rather
than when the group's goroutine returned.

Fixes: #1594
Signed-off-by: Lucas Servén Marín <[email protected]>

* cmd/thanos/receive: correctly clean up uploader

This commit fixes how the uploader is cleaned up. Currently, it was
cleaned up in a defer block that was run before the run groups. This
ensures the cleanup occurs only when the run groups are returning.

Signed-off-by: Lucas Servén Marín <[email protected]>
Signed-off-by: Giedrius Statkevičius <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

recieve: Panics
3 participants