Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: backup/mixed-version failed #78399

Closed
cockroach-teamcity opened this issue Mar 24, 2022 · 6 comments
Closed

roachtest: backup/mixed-version failed #78399

cockroach-teamcity opened this issue Mar 24, 2022 · 6 comments
Assignees
Labels
branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-disaster-recovery

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 24, 2022

roachtest.backup/mixed-version failed with artifacts on release-22.1 @ 9613c48ab9723f10ffb57f6c507b65bf819fbc95:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/backup/mixed-version/run_1
	assertions.go:262,assertions.go:1332,require.go:1231,backup.go:200,versionupgrade.go:208,backup.go:219,test_runner.go:866: 
			Error Trace:	backup.go:200
			            				versionupgrade.go:208
			            				backup.go:219
			            				test_runner.go:866
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	pq: failed to run backup: exporting 3 ranges: failed to create blob client: connecting to node 1: unable to dial n1: breaker open
			Test:       	backup/mixed-version

	test_impl.go:259,require.go:1234,backup.go:200,versionupgrade.go:208,backup.go:219,test_runner.go:866: 
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

Jira issue: CRDB-14107

@cockroach-teamcity cockroach-teamcity added branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Mar 24, 2022
@celiala
Copy link
Collaborator

celiala commented Mar 30, 2022

Hi @tbg - I saw that #78721 was recently closed -- is this also a transient failure & can be closed? thanks!

@tbg
Copy link
Member

tbg commented Mar 31, 2022

Hi @celiala, #78721 was an unrelated failure mode. The failure mode here seems benign (not worthy of a blocker) but it's bulk's test so they should have the last word.

@adityamaru
Copy link
Contributor

adityamaru commented Mar 31, 2022

I think @stevendanna was poking around at this, but yes I'm inclined to agree that this shouldn't block anything. Removing the labels but leaving it for Steven to close.

@adityamaru adityamaru removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. blocks-22.1.0-beta.1 branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 labels Mar 31, 2022
@stevendanna
Copy link
Collaborator

Agree with removing the release blocker. I'm inclined to close this for now and investigate anew if it crops up again. I haven't had much luck reproducing this unfortunately.

@exalate-issue-sync exalate-issue-sync bot reopened this Apr 1, 2022
@exalate-issue-sync exalate-issue-sync bot added branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 1, 2022
@stevendanna stevendanna removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Apr 5, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.backup/mixed-version failed with artifacts on release-22.1 @ fd435e6d411a79425eba7ecdf82f0efb66025550:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/backup/mixed-version/run_1
	assertions.go:262,assertions.go:1332,require.go:1231,backup.go:201,versionupgrade.go:206,backup.go:220,test_runner.go:875: 
			Error Trace:	backup.go:201
			            				versionupgrade.go:206
			            				backup.go:220
			            				test_runner.go:875
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	pq: failed to run backup: exporting 3 ranges: failed to create blob client: connecting to node 1: unable to dial n1: breaker open
			Test:       	backup/mixed-version

	test_impl.go:259,require.go:1234,backup.go:201,versionupgrade.go:206,backup.go:220,test_runner.go:875: 
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@stevendanna
Copy link
Collaborator

I think we got the underlying cause here: #90432 but didn't include this one in the fix list for some reason.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-disaster-recovery
Projects
No open projects
Archived in project
Development

No branches or pull requests

6 participants