Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: acceptance/decommission failed #35307

Closed
tbg opened this issue Mar 1, 2019 · 1 comment
Closed

roachtest: acceptance/decommission failed #35307

tbg opened this issue Mar 1, 2019 · 1 comment
Labels
C-test-failure Broken test (automatically or manually discovered).
Milestone
19.1

Comments

@tbg
Copy link
Member

tbg commented Mar 1, 2019 •
edited
Loading

Failed in CI, debug zip attached. The logs on two nodes show spam of this sort

E190228 22:05:49.768007 3015 storage/queue.go:829  [n2,replicate,s2,r2/2:/System/NodeLiveness{-Max}] 1 matching stores are currently throttled
E190228 22:05:49.779112 2974 storage/queue.go:829  [n2,replicate,s2,r15/3:/Table/{19-20}] 1 matching stores are currently throttled
E190228 22:05:49.795309 2907 storage/queue.go:829  [n2,replicate,s2,r17/3:/Table/2{1-2}] 1 matching stores are currently throttled
E190228 22:05:49.811383 3004 storage/queue.go:829  [n2,replicate,s2,r16/2:/Table/2{0-1}] 1 matching stores are currently throttled
E190228 22:05:49.822201 2908 storage/queue.go:829  [n2,replicate,s2,r10/2:/Table/1{4-5}] 1 matching stores are currently throttled
E190228 22:05:49.827774 3016 storage/queue.go:829  [n2,replicate,s2,r9/2:/Table/1{3-4}] 1 matching stores are currently throttled
E190228 22:05:49.838734 3017 storage/queue.go:829  [n2,replicate,s2,r2/2:/System/NodeLiveness{-Max}] 1 matching stores are currently throttled
E190228 22:05:49.844351 3005 storage/queue.go:829  [n2,replicate,s2,r15/3:/Table/{19-20}] 1 matching stores are currently throttled

debug.zip
@tbg tbg added the C-test-failure Broken test (automatically or manually discovered). label Mar 1, 2019
@tbg tbg added this to the 2.2 milestone Mar 1, 2019
@tbg tbg mentioned this issue Mar 1, 2019
Merged
@tbg tbg mentioned this issue Mar 1, 2019
Merged
tbg added a commit to tbg/cockroach that referenced this issue Mar 1, 2019
@tbg
storage: better error when stores are throttled
44a9481 
This is an annoying blocker during replication, and has also played a
role in test flakes such as cockroachdb#35307.

Release note: None
tbg added a commit to tbg/cockroach that referenced this issue Mar 5, 2019
@tbg
storage: better error when stores are throttled
2beec4e 
This is an annoying blocker during replication, and has also played a
role in test flakes such as cockroachdb#35307.

Release note: None
craig bot pushed a commit that referenced this issue Mar 5, 2019
@tbg
Merge #35308
d8f7e85 
35308: storage: better error when stores are throttled r=petermattis a=tbg

This is an annoying blocker during replication, and has also played a
role in test flakes such as #35307.

Release note: None

Co-authored-by: Tobias Schottdorf <[email protected]>
@tbg
Copy link
Member Author

tbg commented May 8, 2019

It's been two months, and we have better logging if it comes back. Chances are something was fixed.

@tbg tbg closed this as completed May 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered).
Projects
None yet
Milestone
19.1
Development

No branches or pull requests
1 participant
@tbg