Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: backup/nodeShutdown/coordinator/n4cpu4 failed #75866

Closed
cockroach-teamcity opened this issue Feb 2, 2022 · 1 comment
Closed

roachtest: backup/nodeShutdown/coordinator/n4cpu4 failed #75866

cockroach-teamcity opened this issue Feb 2, 2022 · 1 comment
Assignees
@adityamaru
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery

Comments

@cockroach-teamcity
Copy link
Member

roachtest.backup/nodeShutdown/coordinator/n4cpu4 failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Feb 2, 2022
@rhu713
Copy link
Contributor

rhu713 commented Feb 7, 2022 •
edited
Loading

It looks like very early on in the test n1 and n3 had trouble reporting liveness. Most of the logs are errors like this

teamcity-4275964-1643782799-03-n4cpu4-0001> W220202 06:29:10.650473 224 kv/kvserver/liveness/liveness.go:771 ⋮ [n1,liveness-hb] 1289 failed node liveness heartbeat: ‹×›: context deadline exceeded
(1) ‹×›
Wraps: (2) context deadline exceeded
Error types: (1) *contextutil.TimeoutError (2) context.deadlineExceededError

An inability to maintain liveness will prevent a node from participating in a
cluster. If this problem persists, it may be a sign of resource starvation or
of network connectivity problems. For help troubleshooting, visit:

    https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#node-liveness-issues```

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adityamaru adityamaru

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
No milestone
Development

No branches or pull requests
3 participants
@adityamaru @rhu713 @cockroach-teamcity