Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: c2c/tpcc/warehouses=500/duration=10/cutover=5 failed #103939

Closed
cockroach-teamcity opened this issue May 26, 2023 · 5 comments · Fixed by #104131
Closed

roachtest: c2c/tpcc/warehouses=500/duration=10/cutover=5 failed #103939

cockroach-teamcity opened this issue May 26, 2023 · 5 comments · Fixed by #104131
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 26, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.c2c/tpcc/warehouses=500/duration=10/cutover=5 failed with artifacts on master @ 54058c8455ca84f6734890dae883f535b7d75104:

test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=500/duration=10/cutover=5/run_1
(latency_verifier.go:192).assertValid: latency never dropped to acceptable steady level: 2m0s
(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:823
	            				main/pkg/cmd/roachtest/monitor.go:105
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
	            				GOROOT/src/runtime/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	context canceled
	Test:       	c2c/tpcc/warehouses=500/duration=10/cutover=5
(require.go:1360).NoError: FailNow called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-28267
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery labels May 26, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone May 26, 2023
@cockroach-teamcity
Copy link
Member Author

roachtest.c2c/tpcc/warehouses=500/duration=10/cutover=5 failed with artifacts on master @ 9d6e7baebdc0d8b1a57ce8dc158fec35b48e9e2a:

test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=500/duration=10/cutover=5/run_1
(latency_verifier.go:192).assertValid: latency never dropped to acceptable steady level: 2m0s
(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:833
	            				main/pkg/cmd/roachtest/monitor.go:105
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
	            				GOROOT/src/runtime/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	context canceled
	Test:       	c2c/tpcc/warehouses=500/duration=10/cutover=5
(require.go:1360).NoError: FailNow called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue May 27, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.c2c/tpcc/warehouses=500/duration=10/cutover=5 failed with artifacts on master @ dd8dbfc4fa61763dc4fa8fbc19d3a7f2a30bb1e2:

test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=500/duration=10/cutover=5/run_1
(latency_verifier.go:192).assertValid: latency never dropped to acceptable steady level: 2m0s
(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:833
	            				main/pkg/cmd/roachtest/monitor.go:105
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
	            				GOROOT/src/runtime/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	context canceled
	Test:       	c2c/tpcc/warehouses=500/duration=10/cutover=5
(require.go:1360).NoError: FailNow called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.c2c/tpcc/warehouses=500/duration=10/cutover=5 failed with artifacts on master @ 87c677569f7af6e72834d68e67493efbd72971ac:

test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=500/duration=10/cutover=5/run_1
(latency_verifier.go:192).assertValid: latency never dropped to acceptable steady level: 2m0s
(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:833
	            				main/pkg/cmd/roachtest/monitor.go:105
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
	            				GOROOT/src/runtime/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	context canceled
	Test:       	c2c/tpcc/warehouses=500/duration=10/cutover=5
(require.go:1360).NoError: FailNow called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.c2c/tpcc/warehouses=500/duration=10/cutover=5 failed with artifacts on master @ e48d8a376760498c20aa922ddd3b52b5b31dcb19:

test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=500/duration=10/cutover=5/run_1
(latency_verifier.go:192).assertValid: latency never dropped to acceptable steady level: 2m0s
(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:833
	            				main/pkg/cmd/roachtest/monitor.go:105
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
	            				GOROOT/src/runtime/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	context canceled
	Test:       	c2c/tpcc/warehouses=500/duration=10/cutover=5
(require.go:1360).NoError: FailNow called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@stevendanna stevendanna mentioned this issue May 31, 2023
Merged
@cockroach-teamcity
Copy link
Member Author

roachtest.c2c/tpcc/warehouses=500/duration=10/cutover=5 failed with artifacts on master @ 906e7ade9d3891a974ab02e89b4cb69c35ffda13:

test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=500/duration=10/cutover=5/run_1
(latency_verifier.go:192).assertValid: latency never dropped to acceptable steady level: 2m0s
(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:833
	            				main/pkg/cmd/roachtest/monitor.go:105
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
	            				GOROOT/src/runtime/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	context canceled
	Test:       	c2c/tpcc/warehouses=500/duration=10/cutover=5
(require.go:1360).NoError: FailNow called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue May 31, 2023
@stevendanna
Merge #104131
a879627 
104131: roachtest: handle zero replicated timestamp correctly r=msbutler a=stevendanna

A zero-valued hlc.Timestamp converts to a non-zero-values Go timestamp since hlc.Timestamps are based on the Unix epoch.

As a result, the waitForReplicatedTime function would immediately return, as a result we were not actually waiting for the initial scan.

Possibly Fixes #103939

Epic: none

Release note: None

Co-authored-by: Steven Danna <[email protected]>
@craig craig bot closed this as completed in d3e05b7 May 31, 2023
@blathers-crl blathers-crl bot mentioned this issue May 31, 2023
Merged
blathers-crl bot pushed a commit that referenced this issue May 31, 2023
@stevendanna
roachtest: handle zero replicated timestamp correctly
190e12d 
A zero-valued hlc.Timestamp converts to a non-zero-values Go timestamp
since hlc.Timestamps are based on the Unix epoch.

As a result, the waitForReplicatedTime function would immediately
return, as a result we were not actually waiting for the initial scan.

Possibly Fixes #103939

Epic: none

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
23.1
Development

Successfully merging a pull request may close this issue.

roachtest: handle zero replicated timestamp correctly stevendanna/cockroach
1 participant
@cockroach-teamcity