spanconfig/spanconfigkvsubscriber: TestBlockedKVSubscriberDisablesQueues failed #128453
Assignees
Labels
branch-release-24.2
Used to mark GA and release blockers, technical advisories, and bugs for 24.2
branch-release-24.2.0-rc
C-bug
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
C-test-failure
Broken test (automatically or manually discovered).
O-robot
Originated from a bot.
release-blocker
Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
T-observability
Comments
cockroach-teamcity
added
branch-release-24.2.0-rc
C-test-failure
|
Looks similar to #122608 |
|
Seems stuck on stopping the test cluster, when the defer is executed in last line of the test: |
|
Unclear why the test reports timing out at 41 minutes. From logs, the test cluster has been running between |
pav-kv
added
C-bug
|
2 goroutines in |
|
Presumably, |
|
The goroutine dump is the last thing printed in the logs, but it's not a fatal message. So I think the test has really been stuck in this state for 40-ish minutes, but everything was nearly shut down, except there is some deadlock preventing all the tasks being shut down. |
pav-kv
added
release-blocker
|
Removing the |
|
The only task in the stacks dump is the following, and it appears to be stuck: |
|
This would be the observability team cc @dhartunian |
dhartunian
added a commit
to dhartunian/cockroach
that referenced
this issue
Aug 9, 2024
The buffer ingester could get stuck during closing because it would independently evaluate a `running` bool and then try to write to a channel with a buffer of 1. This channel could have accumulated a payload prior to the ingester shutting down and would then block forever. Instead of using the boolean, the flush operation now selects on a channel that we close on shutdown. This ensure that no matter how many extra flushes we have blocked, they will all get cancelled. Resolves: cockroachdb#128453 Epic: None Release note: None
craig bot
pushed a commit
that referenced
this issue
Aug 13, 2024
128713: sqlstats: add close chan to buffer ingester r=xinhaoz a=dhartunian The buffer ingester could get stuck during closing because it would independently evaluate a `running` bool and then try to write to a channel with a buffer of 1. This channel could have accumulated a payload prior to the ingester shutting down and would then block forever. Instead of using the boolean, the flush operation now selects on a channel that we close on shutdown. This ensure that no matter how many extra flushes we have blocked, they will all get cancelled. Resolves: #128453 Epic: None Release note: None Co-authored-by: David Hartunian <[email protected]>
![blathers[crl]](https://avatars.githubusercontent.com/in/59700?s=40&u=373d9101a12edc3847d2fb6ae78e3adcfb4ca41e&v=4){kind=avatar}
|
Based on the specified backports for linked PR #128713, I applied the following new label(s) to this issue: branch-release-24.2. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
blathers-crl
bot
added
the
branch-release-24.2
blathers-crl bot
pushed a commit
that referenced
this issue
Aug 13, 2024
The buffer ingester could get stuck during closing because it would independently evaluate a `running` bool and then try to write to a channel with a buffer of 1. This channel could have accumulated a payload prior to the ingester shutting down and would then block forever. Instead of using the boolean, the flush operation now selects on a channel that we close on shutdown. This ensure that no matter how many extra flushes we have blocked, they will all get cancelled. Resolves: #128453 Epic: None Release note: None
dhartunian
added a commit
to dhartunian/cockroach
that referenced
this issue
Oct 15, 2024
The buffer ingester could get stuck during closing because it would independently evaluate a `running` bool and then try to write to a channel with a buffer of 1. This channel could have accumulated a payload prior to the ingester shutting down and would then block forever. Instead of using the boolean, the flush operation now selects on a channel that we close on shutdown. This ensure that no matter how many extra flushes we have blocked, they will all get cancelled. Resolves: cockroachdb#128453 Epic: None Release note: None
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
spanconfig/spanconfigkvsubscriber.TestBlockedKVSubscriberDisablesQueues failed on release-24.2.0-rc @ 3d1bcc1df630c10c4065a22fc846daa911b436d6:
Fatal error:
Stack:
Log preceding fatal error
Parameters:
attempt=1race=truerun=1shard=1Help
See also: How To Investigate a Go Test Failure (internal)
This test on roachdash | Improve this report!
Jira issue: CRDB-41025
The text was updated successfully, but these errors were encountered: