roachtest: backup/mixed-version failed #73327

cockroach-teamcity · 2021-12-01T07:12:24Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ 3525f2a5ce36c312ee7447aaa62c511f19bdc421:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:246,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:246
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 35.238.192.232:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:89,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3824371-1638342455-07-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 3: 10427
		4: 10350
		1: 12591
		2: dead (exit status 7)
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!

Jira issue: CRDB-11543}

The text was updated successfully, but these errors were encountered:

stevendanna · 2021-12-02T09:38:12Z

I'm going to remove release blocker from this for now. The error:

F211201 07:11:54.973435 5748 sql/catalog/descpb/descriptor.go:191 ⋮ [-] 147  read table descriptor ‹"eventlog"› (12) which has a ModificationTime after its MVCC timestamp: has 1585824876.057272000,0, expected 1585824876.057140000,0

Is exactly the thing we were trying to workaround in this code here:

https://github.com/cockroachdb/cockroach/pull/73110/files#diff-b7fbbccbe8906f088b04816ce14ac051be2de78ed05d4e21984a6278c628f942R190-R214

That we currently believe is an artifact of the test setup. It looks like that workaround might be a bit insufficient.

stevendanna · 2021-12-02T11:12:23Z

Perhaps something like #73377 would help us see exactly what is happening here.

cockroach-teamcity · 2021-12-06T07:12:07Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ 39f8d3219c4d8a15f75884947f059fcff93051f2:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:246,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:246
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 34.135.46.97:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:89,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3851968-1638774424-16-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 2: dead (exit status 7)
		3: 12056
		4: 11826
		1: 13195
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

adityamaru · 2021-12-06T23:24:29Z

@stevendanna since you've been looking into this I'm reassigning to better reflect reality. Feel free to throw it back my way.

cockroach-teamcity · 2021-12-14T07:17:08Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ 219f20d617f3ba585186d86ac7ce2fe40474bfda:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:249,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:249
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 35.238.162.139:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:89,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3904091-1639465914-06-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: 10037
		3: 10259
		1: 11087
		2: dead (exit status 7)
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

Same failure on other branches

roachtest: backup/mixed-version failed #73708 roachtest: backup/mixed-version failed [C-test-failure O-roachtest O-robot T-bulkio branch-master release-blocker]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

stevendanna · 2021-12-14T11:19:27Z

It might be hard to make this test reliable on this branch without cleaning up this problematic data more completely. We could retry the garbage collection a few times I suppose.

07:15:35 backup.go:229: EnqueueRangeResponse: {
[node_id:4 error:"n4 has no replica for r5"
 node_id:1 error:"replica [n1,s1,r5/1:/Table/{SystemCon\342\200\246-11}] does not have the range lease"
 node_id:3 error:"replica [n3,s3,r5/3:/Table/{SystemCon\342\200\246-11}] does not have the range lease"
 node_id:2 events:<time:<seconds:1639466135 nanos:878872219 > message:"kv/kvserver/store.go:2775 [n2,s2,r5/2:/Table/{SystemCon\342\200\246-11}] running gc.process" >
           events:<time:<seconds:1639466135 nanos:878926049 > message:"kv/kvserver/replica_protected_timestamp.go:267 [n2,s2,r5/2:/Table/{SystemCon\342\200\246-11}] not gc'ing replica [n2,s2,r5/2:/Table/{SystemCon\342\200\246-11}] due to new lease repl=(n2,s2):2 seq=12 start=1639466127.994887474,0 epo=19 pro=1639466127.994896843,0 started after 1639466127.993857719,0" >
           events:<time:<seconds:1639466135 nanos:878939067 > message:"kv/kvserver/store.go:2777 [n2,s2,r5/2:/Table/{SystemCon\342\200\246-11}] processed: false" > ]}

cockroach-teamcity · 2021-12-18T07:11:24Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ 827dc79373dbe47aa5ef1d9cfbe11be9214246f7:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:249,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:249
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 130.211.239.178:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:89,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3944385-1639811188-13-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 1: 13544
		4: 12517
		3: 12671
		2: dead (exit status 7)
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

Same failure on other branches

roachtest: backup/mixed-version failed #73897 roachtest: backup/mixed-version failed [C-test-failure O-roachtest O-robot T-server-and-security branch-release-21.2 release-blocker]
roachtest: backup/mixed-version failed #73708 roachtest: backup/mixed-version failed [C-test-failure O-roachtest O-robot T-bulkio branch-master release-blocker]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-01-16T07:24:17Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ 42e7e51f596fc9955ea47a2d477c7d0a6e18ad97:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:249,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:249
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 35.202.221.8:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:89,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-4131618-1642317553-14-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: 10734
		1: 12378
		3: 10980
		2: dead (exit status 7)
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-01-20T07:15:20Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ 4d2b93943c4333cfeabbe240465926f3b1f9e724:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:249,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:249
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 34.66.184.207:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:89,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-4159968-1642662636-01-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 3: 9784
		1: 11727
		4: 9848
		2: dead (exit status 7)
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-01-28T07:16:18Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ 0fd6eead6c6eb7b2529deb39197cc3c95e93ded8:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:249,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:249
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 35.238.20.219:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:89,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-4236957-1643353874-11-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 3: 10171
		1: 11920
		4: 10668
		2: dead (exit status 7)
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-02-05T07:08:43Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ 1689a1f59df7c0cd1483c1d3f92a66b6e8f5db35:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:249,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:249
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 34.66.153.213:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:89,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-4307498-1644044632-02-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: 11801
		3: 11815
		1: 13488
		2: dead (exit status 7)
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

Same failure on other branches

roachtest: backup/mixed-version failed #76101 roachtest: backup/mixed-version failed [C-test-failure O-roachtest O-robot T-bulkio branch-master release-blocker]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-02-18T07:11:37Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ 9930ef174e01cb992b1996f94040b9b6f2bcbfcb:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:249,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:249
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 34.70.101.112:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:89,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-4407713-1645167990-01-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 3: 9640
		4: 9456
		2: dead (exit status 7)
		1: 11152
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-03-10T07:12:04Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ 6ef65c3ff4c145a0d85387f6f1e2b61590c46a8d:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:249,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:249
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 35.222.248.162:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:89,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-4539176-1646896021-02-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: 11235
		3: 11375
		1: 12776
		2: dead (exit status 7)
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-04-08T06:08:17Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ c1cc2b0955b77dc242ab4f47435dc5118f71dfa5:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:249,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:249
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 34.121.181.18:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:89,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-4831853-1649397803-01-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: 11081
		3: 11054
		1: 13131
		2: dead (exit status 7)
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

Same failure on other branches

roachtest: backup/mixed-version failed #78399 roachtest: backup/mixed-version failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-04-12T05:35:22Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ ee99a26485d34dfd73688cd7c4b2be52f7bac6c8:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:249,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:249
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 35.223.252.104:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:89,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-4867658-1649741422-05-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: 12162
		1: 13958
		3: 11908
		2: dead (exit status 7)
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

Same failure on other branches

roachtest: backup/mixed-version failed #78399 roachtest: backup/mixed-version failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-04-16T05:48:06Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ b0897408a939f45c338bef8f9b126cc559d2c2c6:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:249,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:249
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 35.184.191.136:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:91,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-4907599-1650087769-10-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 2: dead (exit status 7)
		3: 11124
		4: 10611
		1: 11412
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

Same failure on other branches

roachtest: backup/mixed-version failed #78399 roachtest: backup/mixed-version failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-04-19T05:48:43Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ 9a83455c514d2f29c86ec9dbb3cc3b26a3e8ace1:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:249,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:249
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 104.154.128.165:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:91,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-4924084-1650347032-01-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 3: 11118
		4: 11358
		1: 13360
		2: dead (exit status 7)
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

Same failure on other branches

roachtest: backup/mixed-version failed #78399 roachtest: backup/mixed-version failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-04-20T05:48:51Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ 0cb819a3ab2a5797970eb50570dfa71d885dc2ad:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:249,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:249
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 35.193.186.88:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:91,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-4935073-1650433386-01-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 2: dead (exit status 7)
		4: 9966
		3: 10006
		1: 11435
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

Same failure on other branches

roachtest: backup/mixed-version failed #78399 roachtest: backup/mixed-version failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-03T05:45:57Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ 15cce99eb8c35828e7f3052a9e25eb66405027df:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:249,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:249
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 35.202.82.174:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:91,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-5082617-1651556460-14-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 3: 10206
		2: dead (exit status 7)
		4: 10247
		1: 11706
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

Same failure on other branches

roachtest: backup/mixed-version failed #78399 roachtest: backup/mixed-version failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-05T05:48:12Z

roachtest.backup/mixed-version failed with artifacts on release-21.1 @ bf2bef135b5be9c73ccc445f2d996781ee59e94f:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/mixed-version/run_1
	test.go:288,assertions.go:262,assertions.go:1332,require.go:1035,backup.go:186,versionupgrade.go:200,backup.go:249,test_runner.go:733: 
			Error Trace:	backup.go:186
			            				versionupgrade.go:200
			            				backup.go:249
			            				test_runner.go:733
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 35.223.55.175:26257: connect: connection refused
			Test:       	backup/mixed-version

	cluster.go:1667,context.go:91,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-5109205-1651729368-03-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: 9943
		1: 11053
		3: 9935
		2: dead (exit status 7)
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/mixed-version

Same failure on other branches

roachtest: backup/mixed-version failed #78399 roachtest: backup/mixed-version failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

stevendanna · 2022-10-14T14:33:21Z

I'm closing this for now. The underlying issue is a test-specific failure on an old branch.

cockroach-teamcity added branch-release-21.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Dec 1, 2021

cockroach-teamcity added this to the 21.1 milestone Dec 1, 2021

blathers-crl bot added the T-disaster-recovery label Dec 1, 2021

livlobo assigned adityamaru Dec 1, 2021

stevendanna removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Dec 2, 2021

adityamaru assigned stevendanna and unassigned adityamaru Dec 6, 2021

cockroach-teamcity mentioned this issue Dec 11, 2021

roachtest: backup/mixed-version failed #73708

Closed

cockroach-teamcity mentioned this issue Dec 16, 2021

roachtest: backup/mixed-version failed #73897

Closed

cockroach-teamcity mentioned this issue Feb 5, 2022

roachtest: backup/mixed-version failed #76101

Closed

cockroach-teamcity mentioned this issue Mar 24, 2022

roachtest: backup/mixed-version failed #78399

Closed

cockroach-teamcity mentioned this issue Mar 29, 2022

roachtest: backup/mixed-version failed #78721

Closed

jlinder added sync-me-3 and removed sync-me-3 labels May 24, 2022

stevendanna closed this as completed Oct 14, 2022

github-project-automation bot added this to Disaster Recovery Backlog Aug 28, 2024

github-project-automation bot moved this to Done in Disaster Recovery Backlog Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: backup/mixed-version failed #73327

roachtest: backup/mixed-version failed #73327

cockroach-teamcity commented Dec 1, 2021 •

edited by cockroach-jira-scripts

Loading

stevendanna commented Dec 2, 2021

stevendanna commented Dec 2, 2021

cockroach-teamcity commented Dec 6, 2021

adityamaru commented Dec 6, 2021

cockroach-teamcity commented Dec 14, 2021

stevendanna commented Dec 14, 2021

cockroach-teamcity commented Dec 18, 2021

cockroach-teamcity commented Jan 16, 2022

cockroach-teamcity commented Jan 20, 2022

cockroach-teamcity commented Jan 28, 2022

cockroach-teamcity commented Feb 5, 2022

cockroach-teamcity commented Feb 18, 2022

cockroach-teamcity commented Mar 10, 2022

cockroach-teamcity commented Apr 8, 2022

cockroach-teamcity commented Apr 12, 2022

cockroach-teamcity commented Apr 16, 2022

cockroach-teamcity commented Apr 19, 2022

cockroach-teamcity commented Apr 20, 2022

cockroach-teamcity commented May 3, 2022

cockroach-teamcity commented May 5, 2022

stevendanna commented Oct 14, 2022

roachtest: backup/mixed-version failed #73327

roachtest: backup/mixed-version failed #73327

Comments

cockroach-teamcity commented Dec 1, 2021 • edited by cockroach-jira-scripts Loading

stevendanna commented Dec 2, 2021

stevendanna commented Dec 2, 2021

cockroach-teamcity commented Dec 6, 2021

adityamaru commented Dec 6, 2021

cockroach-teamcity commented Dec 14, 2021

stevendanna commented Dec 14, 2021

cockroach-teamcity commented Dec 18, 2021

cockroach-teamcity commented Jan 16, 2022

cockroach-teamcity commented Jan 20, 2022

cockroach-teamcity commented Jan 28, 2022

cockroach-teamcity commented Feb 5, 2022

cockroach-teamcity commented Feb 18, 2022

cockroach-teamcity commented Mar 10, 2022

cockroach-teamcity commented Apr 8, 2022

cockroach-teamcity commented Apr 12, 2022

cockroach-teamcity commented Apr 16, 2022

cockroach-teamcity commented Apr 19, 2022

cockroach-teamcity commented Apr 20, 2022

cockroach-teamcity commented May 3, 2022

cockroach-teamcity commented May 5, 2022

stevendanna commented Oct 14, 2022

cockroach-teamcity commented Dec 1, 2021 •

edited by cockroach-jira-scripts

Loading