Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: c2c/kv0 failed #97903

Closed
cockroach-teamcity opened this issue Mar 2, 2023 · 10 comments
Closed

roachtest: c2c/kv0 failed #97903

cockroach-teamcity opened this issue Mar 2, 2023 · 10 comments
Assignees
@benbardin
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 2, 2023 •
edited by exalate-issue-sync bot
Loading

roachtest.c2c/kv0 failed with artifacts on master @ 20e2adda3c76c7172dd986c871df0ae9a346918f:

	            				/go/src/github.com/cockroachdb/cockroach/cluster_to_cluster.go:529
	            				/go/src/github.com/cockroachdb/cockroach/test_runner.go:979
	            				/go/src/github.com/cockroachdb/cockroach/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	monitor failure: monitor task failed: t.Fatal() was called
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:585
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:529
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:979
	            	Wraps: (2) monitor failure
	            	Wraps: (3) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).wait.func2
	            	  | 	main/pkg/cmd/roachtest/monitor.go:171
	            	Wraps: (4) monitor task failed
	            	Wraps: (5) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).Go.func1.1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:101
	            	  | runtime.gopanic
	            	  | 	GOROOT/src/runtime/panic.go:884
	            	  | main.(*testImpl).Fatalf
	            	  | 	main/pkg/cmd/roachtest/test_impl.go:298
	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*Row).Scan
	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:218
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:570
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:105
	            	  | [...repeated from below...]
	            	Wraps: (6) attached stack trace
	            	  -- stack trace:
	            	  | main.init
	            	  | 	main/pkg/cmd/roachtest/monitor.go:80
	            	  | runtime.doInit
	            	  | 	GOROOT/src/runtime/proc.go:6348
	            	  | runtime.main
	            	  | 	GOROOT/src/runtime/proc.go:233
	            	  | runtime.goexit
	            	  | 	GOROOT/src/runtime/asm_amd64.s:1594
	            	Wraps: (7) t.Fatal() was called
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *withstack.withStack (7) *errutil.leafError
	Test:       	c2c/kv0
(require.go:1264).NoError: FailNow called

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-24952

Epic CRDB-21942
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Mar 2, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Mar 2, 2023
@cockroach-teamcity
Copy link
Member Author

roachtest.c2c/kv0 failed with artifacts on master @ 20e2adda3c76c7172dd986c871df0ae9a346918f:

	            				/go/src/github.com/cockroachdb/cockroach/cluster_to_cluster.go:529
	            				/go/src/github.com/cockroachdb/cockroach/test_runner.go:979
	            				/go/src/github.com/cockroachdb/cockroach/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	monitor failure: monitor task failed: t.Fatal() was called
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:585
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:529
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:979
	            	Wraps: (2) monitor failure
	            	Wraps: (3) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).wait.func2
	            	  | 	main/pkg/cmd/roachtest/monitor.go:171
	            	Wraps: (4) monitor task failed
	            	Wraps: (5) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).Go.func1.1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:101
	            	  | runtime.gopanic
	            	  | 	GOROOT/src/runtime/panic.go:884
	            	  | main.(*testImpl).Fatalf
	            	  | 	main/pkg/cmd/roachtest/test_impl.go:298
	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*Row).Scan
	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:218
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:570
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:105
	            	  | [...repeated from below...]
	            	Wraps: (6) attached stack trace
	            	  -- stack trace:
	            	  | main.init
	            	  | 	main/pkg/cmd/roachtest/monitor.go:80
	            	  | runtime.doInit
	            	  | 	GOROOT/src/runtime/proc.go:6348
	            	  | runtime.main
	            	  | 	GOROOT/src/runtime/proc.go:233
	            	  | runtime.goexit
	            	  | 	GOROOT/src/runtime/asm_amd64.s:1594
	            	Wraps: (7) t.Fatal() was called
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *withstack.withStack (7) *errutil.leafError
	Test:       	c2c/kv0
(require.go:1264).NoError: FailNow called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.c2c/kv0 failed with artifacts on master @ fcea283ebca17a6d923c5d4b0401697438b77dbd:

	            				/go/src/github.com/cockroachdb/cockroach/cluster_to_cluster.go:529
	            				/go/src/github.com/cockroachdb/cockroach/test_runner.go:975
	            				/go/src/github.com/cockroachdb/cockroach/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	monitor failure: monitor task failed: t.Fatal() was called
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:585
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:529
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:975
	            	Wraps: (2) monitor failure
	            	Wraps: (3) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).wait.func2
	            	  | 	main/pkg/cmd/roachtest/monitor.go:171
	            	Wraps: (4) monitor task failed
	            	Wraps: (5) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).Go.func1.1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:101
	            	  | runtime.gopanic
	            	  | 	GOROOT/src/runtime/panic.go:884
	            	  | main.(*testImpl).Fatalf
	            	  | 	main/pkg/cmd/roachtest/test_impl.go:298
	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*Row).Scan
	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:218
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:570
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:105
	            	  | [...repeated from below...]
	            	Wraps: (6) attached stack trace
	            	  -- stack trace:
	            	  | main.init
	            	  | 	main/pkg/cmd/roachtest/monitor.go:80
	            	  | runtime.doInit
	            	  | 	GOROOT/src/runtime/proc.go:6348
	            	  | runtime.main
	            	  | 	GOROOT/src/runtime/proc.go:233
	            	  | runtime.goexit
	            	  | 	GOROOT/src/runtime/asm_amd64.s:1594
	            	Wraps: (7) t.Fatal() was called
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *withstack.withStack (7) *errutil.leafError
	Test:       	c2c/kv0
(require.go:1264).NoError: FailNow called

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.c2c/kv0 failed with artifacts on master @ fcea283ebca17a6d923c5d4b0401697438b77dbd:

	            				/go/src/github.com/cockroachdb/cockroach/cluster_to_cluster.go:529
	            				/go/src/github.com/cockroachdb/cockroach/test_runner.go:975
	            				/go/src/github.com/cockroachdb/cockroach/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	monitor failure: monitor task failed: t.Fatal() was called
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:585
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:529
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:975
	            	Wraps: (2) monitor failure
	            	Wraps: (3) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).wait.func2
	            	  | 	main/pkg/cmd/roachtest/monitor.go:171
	            	Wraps: (4) monitor task failed
	            	Wraps: (5) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).Go.func1.1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:101
	            	  | runtime.gopanic
	            	  | 	GOROOT/src/runtime/panic.go:884
	            	  | main.(*testImpl).Fatalf
	            	  | 	main/pkg/cmd/roachtest/test_impl.go:298
	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*Row).Scan
	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:218
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:570
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:105
	            	  | [...repeated from below...]
	            	Wraps: (6) attached stack trace
	            	  -- stack trace:
	            	  | main.init
	            	  | 	main/pkg/cmd/roachtest/monitor.go:80
	            	  | runtime.doInit
	            	  | 	GOROOT/src/runtime/proc.go:6348
	            	  | runtime.main
	            	  | 	GOROOT/src/runtime/proc.go:233
	            	  | runtime.goexit
	            	  | 	GOROOT/src/runtime/asm_amd64.s:1594
	            	Wraps: (7) t.Fatal() was called
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *withstack.withStack (7) *errutil.leafError
	Test:       	c2c/kv0
(require.go:1264).NoError: FailNow called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.c2c/kv0 failed with artifacts on master @ cf14ad694ee562676f53e36fa8495206c3aed61f:

	            				/go/src/github.com/cockroachdb/cockroach/cluster_to_cluster.go:529
	            				/go/src/github.com/cockroachdb/cockroach/test_runner.go:975
	            				/go/src/github.com/cockroachdb/cockroach/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	monitor failure: monitor task failed: t.Fatal() was called
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:585
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:529
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:975
	            	Wraps: (2) monitor failure
	            	Wraps: (3) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).wait.func2
	            	  | 	main/pkg/cmd/roachtest/monitor.go:171
	            	Wraps: (4) monitor task failed
	            	Wraps: (5) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).Go.func1.1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:101
	            	  | runtime.gopanic
	            	  | 	GOROOT/src/runtime/panic.go:884
	            	  | main.(*testImpl).Fatalf
	            	  | 	main/pkg/cmd/roachtest/test_impl.go:298
	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*Row).Scan
	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:218
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:570
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:105
	            	  | [...repeated from below...]
	            	Wraps: (6) attached stack trace
	            	  -- stack trace:
	            	  | main.init
	            	  | 	main/pkg/cmd/roachtest/monitor.go:80
	            	  | runtime.doInit
	            	  | 	GOROOT/src/runtime/proc.go:6348
	            	  | runtime.main
	            	  | 	GOROOT/src/runtime/proc.go:233
	            	  | runtime.goexit
	            	  | 	GOROOT/src/runtime/asm_amd64.s:1594
	            	Wraps: (7) t.Fatal() was called
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *withstack.withStack (7) *errutil.leafError
	Test:       	c2c/kv0
(require.go:1264).NoError: FailNow called

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.c2c/kv0 failed with artifacts on master @ cf14ad694ee562676f53e36fa8495206c3aed61f:

	            				/go/src/github.com/cockroachdb/cockroach/cluster_to_cluster.go:529
	            				/go/src/github.com/cockroachdb/cockroach/test_runner.go:975
	            				/go/src/github.com/cockroachdb/cockroach/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	monitor failure: monitor task failed: t.Fatal() was called
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:585
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:529
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:975
	            	Wraps: (2) monitor failure
	            	Wraps: (3) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).wait.func2
	            	  | 	main/pkg/cmd/roachtest/monitor.go:171
	            	Wraps: (4) monitor task failed
	            	Wraps: (5) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).Go.func1.1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:101
	            	  | runtime.gopanic
	            	  | 	GOROOT/src/runtime/panic.go:884
	            	  | main.(*testImpl).Fatalf
	            	  | 	main/pkg/cmd/roachtest/test_impl.go:298
	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*Row).Scan
	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:218
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:570
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:105
	            	  | [...repeated from below...]
	            	Wraps: (6) attached stack trace
	            	  -- stack trace:
	            	  | main.init
	            	  | 	main/pkg/cmd/roachtest/monitor.go:80
	            	  | runtime.doInit
	            	  | 	GOROOT/src/runtime/proc.go:6348
	            	  | runtime.main
	            	  | 	GOROOT/src/runtime/proc.go:233
	            	  | runtime.goexit
	            	  | 	GOROOT/src/runtime/asm_amd64.s:1594
	            	Wraps: (7) t.Fatal() was called
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *withstack.withStack (7) *errutil.leafError
	Test:       	c2c/kv0
(require.go:1264).NoError: FailNow called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.c2c/kv0 failed with artifacts on master @ 14b43be03c1c246765be17aa15245493a42c401d:

	            				/go/src/github.com/cockroachdb/cockroach/cluster_to_cluster.go:529
	            				/go/src/github.com/cockroachdb/cockroach/test_runner.go:975
	            				/go/src/github.com/cockroachdb/cockroach/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	monitor failure: monitor task failed: t.Fatal() was called
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:585
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:529
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:975
	            	Wraps: (2) monitor failure
	            	Wraps: (3) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).wait.func2
	            	  | 	main/pkg/cmd/roachtest/monitor.go:171
	            	Wraps: (4) monitor task failed
	            	Wraps: (5) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).Go.func1.1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:101
	            	  | runtime.gopanic
	            	  | 	GOROOT/src/runtime/panic.go:884
	            	  | main.(*testImpl).Fatalf
	            	  | 	main/pkg/cmd/roachtest/test_impl.go:298
	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*Row).Scan
	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:218
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:570
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:105
	            	  | [...repeated from below...]
	            	Wraps: (6) attached stack trace
	            	  -- stack trace:
	            	  | main.init
	            	  | 	main/pkg/cmd/roachtest/monitor.go:80
	            	  | runtime.doInit
	            	  | 	GOROOT/src/runtime/proc.go:6348
	            	  | runtime.main
	            	  | 	GOROOT/src/runtime/proc.go:233
	            	  | runtime.goexit
	            	  | 	GOROOT/src/runtime/asm_amd64.s:1594
	            	Wraps: (7) t.Fatal() was called
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *withstack.withStack (7) *errutil.leafError
	Test:       	c2c/kv0
(require.go:1264).NoError: FailNow called

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.c2c/kv0 failed with artifacts on master @ 14b43be03c1c246765be17aa15245493a42c401d:

	            				/go/src/github.com/cockroachdb/cockroach/cluster_to_cluster.go:529
	            				/go/src/github.com/cockroachdb/cockroach/test_runner.go:975
	            				/go/src/github.com/cockroachdb/cockroach/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	monitor failure: monitor task failed: t.Fatal() was called
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:585
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:529
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:975
	            	Wraps: (2) monitor failure
	            	Wraps: (3) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).wait.func2
	            	  | 	main/pkg/cmd/roachtest/monitor.go:171
	            	Wraps: (4) monitor task failed
	            	Wraps: (5) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).Go.func1.1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:101
	            	  | runtime.gopanic
	            	  | 	GOROOT/src/runtime/panic.go:884
	            	  | main.(*testImpl).Fatalf
	            	  | 	main/pkg/cmd/roachtest/test_impl.go:298
	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*Row).Scan
	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:218
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:570
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:105
	            	  | [...repeated from below...]
	            	Wraps: (6) attached stack trace
	            	  -- stack trace:
	            	  | main.init
	            	  | 	main/pkg/cmd/roachtest/monitor.go:80
	            	  | runtime.doInit
	            	  | 	GOROOT/src/runtime/proc.go:6348
	            	  | runtime.main
	            	  | 	GOROOT/src/runtime/proc.go:233
	            	  | runtime.goexit
	            	  | 	GOROOT/src/runtime/asm_amd64.s:1594
	            	Wraps: (7) t.Fatal() was called
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *withstack.withStack (7) *errutil.leafError
	Test:       	c2c/kv0
(require.go:1264).NoError: FailNow called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.c2c/kv0 failed with artifacts on master @ 43a25a09f92408accdeb3cfed1eb35e02d375877:

	            				/go/src/github.com/cockroachdb/cockroach/cluster_to_cluster.go:529
	            				/go/src/github.com/cockroachdb/cockroach/test_runner.go:975
	            				/go/src/github.com/cockroachdb/cockroach/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	monitor failure: monitor task failed: t.Fatal() was called
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:585
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:529
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:975
	            	Wraps: (2) monitor failure
	            	Wraps: (3) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).wait.func2
	            	  | 	main/pkg/cmd/roachtest/monitor.go:171
	            	Wraps: (4) monitor task failed
	            	Wraps: (5) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).Go.func1.1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:101
	            	  | runtime.gopanic
	            	  | 	GOROOT/src/runtime/panic.go:884
	            	  | main.(*testImpl).Fatalf
	            	  | 	main/pkg/cmd/roachtest/test_impl.go:298
	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*Row).Scan
	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:218
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:570
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:105
	            	  | [...repeated from below...]
	            	Wraps: (6) attached stack trace
	            	  -- stack trace:
	            	  | main.init
	            	  | 	main/pkg/cmd/roachtest/monitor.go:80
	            	  | runtime.doInit
	            	  | 	GOROOT/src/runtime/proc.go:6348
	            	  | runtime.main
	            	  | 	GOROOT/src/runtime/proc.go:233
	            	  | runtime.goexit
	            	  | 	GOROOT/src/runtime/asm_amd64.s:1594
	            	Wraps: (7) t.Fatal() was called
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *withstack.withStack (7) *errutil.leafError
	Test:       	c2c/kv0
(require.go:1264).NoError: FailNow called

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.c2c/kv0 failed with artifacts on master @ 43a25a09f92408accdeb3cfed1eb35e02d375877:

	            				/go/src/github.com/cockroachdb/cockroach/cluster_to_cluster.go:529
	            				/go/src/github.com/cockroachdb/cockroach/test_runner.go:975
	            				/go/src/github.com/cockroachdb/cockroach/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	monitor failure: monitor task failed: t.Fatal() was called
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:585
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:529
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:975
	            	Wraps: (2) monitor failure
	            	Wraps: (3) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).wait.func2
	            	  | 	main/pkg/cmd/roachtest/monitor.go:171
	            	Wraps: (4) monitor task failed
	            	Wraps: (5) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).Go.func1.1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:101
	            	  | runtime.gopanic
	            	  | 	GOROOT/src/runtime/panic.go:884
	            	  | main.(*testImpl).Fatalf
	            	  | 	main/pkg/cmd/roachtest/test_impl.go:298
	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*Row).Scan
	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:218
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.compareTenantFingerprintsAtTimestamp.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:570
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:105
	            	  | [...repeated from below...]
	            	Wraps: (6) attached stack trace
	            	  -- stack trace:
	            	  | main.init
	            	  | 	main/pkg/cmd/roachtest/monitor.go:80
	            	  | runtime.doInit
	            	  | 	GOROOT/src/runtime/proc.go:6348
	            	  | runtime.main
	            	  | 	GOROOT/src/runtime/proc.go:233
	            	  | runtime.goexit
	            	  | 	GOROOT/src/runtime/asm_amd64.s:1594
	            	Wraps: (7) t.Fatal() was called
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *withstack.withStack (7) *errutil.leafError
	Test:       	c2c/kv0
(require.go:1264).NoError: FailNow called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@benbardin benbardin mentioned this issue Mar 7, 2023
Merged
@benbardin
Copy link
Collaborator

Will be fixed by #97916

craig bot pushed a commit that referenced this issue Mar 7, 2023
@stevendanna
Merge #97916
0dd1e91 
97916: kv: opt into Elastic CPU limiting resume spans r=adityamaru,arulajmani a=stevendanna

In #96691, we changed ExportRequest such that it returns resume spans that result from the elastic CPU limiter all the way to the caller.

This has at least two problems:

1) In a mixed-version state, the caller might not yet know how to
   handle resume spans. This could result in incomplete responses
   erroneous being used as if they were full responses.

2) The DistSender inspects a request to determine whether it may stop
   early. If it shouldn't be able to stop early, then the request is
   split up, possibly sent in parallel, and all responses are
   combined. The code which combines responses asserts that neither side
   has a resume span.  As a result, we've seen failures such as

        crdb_internal.fingerprint(): combining
        /Tenant/2/Table/106/1/-{8403574544142222370/0-37656332809536692}
        with /Tenant/{2/Table/106/1/436440321206557763/0-3}

   since the change was made.

Here, we add a new request header field to allow callers to indicate whether they are prepared to accept resume spans. Further, we add that new field into the logic in DistSender which decides how to process requests.

The downside here is that crdb_internal.fingerprint won't have its requests sent in parallel.

Release note: None

Fixes #97886,  #97903

Epic: none

Co-authored-by: Steven Danna <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benbardin benbardin

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
23.1
Development

No branches or pull requests
3 participants
@benbardin @stevendanna @cockroach-teamcity