Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: disk-full failed #78349

Closed
cockroach-teamcity opened this issue Mar 23, 2022 · 8 comments
Closed

roachtest: disk-full failed #78349

cockroach-teamcity opened this issue Mar 23, 2022 · 8 comments
Assignees
@nicktrav
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-storage Storage Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 23, 2022 •
edited by cockroach-jira-scripts
Loading

roachtest.disk-full failed with artifacts on release-21.2 @ 2c0a06e8f24f7d84d5fbf30c7f2977d4dadd6a8a:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-14085
@cockroach-teamcity cockroach-teamcity added branch-release-21.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Mar 23, 2022
@blathers-crl blathers-crl bot added the T-storage Storage Team label Mar 23, 2022
@nicktrav
Copy link
Collaborator

Same as #78337 and #78270. Blocked on a query that never returns.

@nicktrav nicktrav mentioned this issue Mar 23, 2022
Closed
nicktrav added a commit to nicktrav/cockroach that referenced this issue Mar 24, 2022
@nicktrav
roachtest: retry failed node liveness queries in disk-full test
2163de6 
Currently, when querying for node liveness in `disk-full`, a query may
hang indefinitely, failing the test only after it has timed out
(currently after 10 hours).

Add retry logic for the node liveness query. The query is cancelled
after a second, and exponentially retries for up to (approximately) one
minute.

Release note: None.

Touches cockroachdb#78337, cockroachdb#78349.
@cockroach-teamcity
Copy link
Member Author

roachtest.disk-full failed with artifacts on release-21.2 @ 1831ab7dc579879476b4161caccc1b17d8780acd:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@nicktrav nicktrav mentioned this issue Mar 24, 2022
Closed
@nicktrav nicktrav removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Mar 25, 2022
@nicktrav nicktrav self-assigned this Mar 25, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.disk-full failed with artifacts on release-21.2 @ 0404196bd03e7a972022f9f7d2153c45eb0431b4:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@nicktrav
Copy link
Collaborator

#78456 should fix this, once backported.

@cockroach-teamcity
Copy link
Member Author

roachtest.disk-full failed with artifacts on release-21.2 @ 791fd46847e4db8b989fd19974d9c79e917a6826:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.disk-full failed with artifacts on release-21.2 @ 3cfa6e69d75020fd3470bfbdf56b44529f9fb492:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@nicktrav nicktrav mentioned this issue Mar 28, 2022
Merged
@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 29, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.disk-full failed with artifacts on release-21.2 @ ecb6d59ee32b042827b30d40523534c1d3cb23e2:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@nicktrav
Copy link
Collaborator

The fix in #78537 didn't make it in time for this run. Expecting subsequent runs to be less flaky.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nicktrav nicktrav

Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-storage Storage Team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nicktrav @cockroach-teamcity