Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: c2c/BulkOps/full failed #115409

Closed
cockroach-teamcity opened this issue Dec 1, 2023 · 2 comments
Closed

roachtest: c2c/BulkOps/full failed #115409

cockroach-teamcity opened this issue Dec 1, 2023 · 2 comments
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Dec 1, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.c2c/BulkOps/full failed with artifacts on master @ 02e46d54d0b0bf63f43592709d551534edb54be6:

(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/backup.go:753
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/backup.go:869
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:367
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:674
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:898
	            				main/pkg/cmd/roachtest/monitor.go:119
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
	            				src/runtime/asm_amd64.s:1650
	Error:      	Received unexpected error:
	            	expected job status canceled, but got reverting
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runBackupMVCCRangeTombstones.func1.1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/backup.go:765
	            	  | github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do
	            	  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:172
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runBackupMVCCRangeTombstones.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/backup.go:753
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runBackupMVCCRangeTombstones
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/backup.go:869
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.replicateBulkOps.runDriver
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:367
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).runWorkload
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:674
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main.func2
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:898
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:119
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1650
	            	Wraps: (2) expected job status canceled, but got reverting
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: read tcp 172.17.0.3:40878 -> 34.148.69.87:26257: read: connection reset by peer
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-34015
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery labels Dec 1, 2023
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Dec 1, 2023
@cockroach-teamcity cockroach-teamcity mentioned this issue Dec 1, 2023
Closed
@msbutler
Copy link
Collaborator

msbutler commented Dec 1, 2023

Another instance of the spanFrontier NPE bug, from node 6's logs:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1ee1b7b]

goroutine 39104 [running]:
panic({0x5775ae0?, 0xb1ca010?})
	GOROOT/src/runtime/panic.go:1017 +0x3ac fp=0xc01ab05b40 sp=0xc01ab05a90 pc=0x49c4cc
runtime.panicmem(...)
	GOROOT/src/runtime/panic.go:261
runtime.sigpanic()
	GOROOT/src/runtime/signal_unix.go:861 +0x378 fp=0xc01ab05ba0 sp=0xc01ab05b40 pc=0x4b3bd8
github.com/cockroachdb/cockroach/pkg/util/span.(*iterator).findNextOverlap(0xc01ab05d90, 0xc00197bb00?)
	github.com/cockroachdb/cockroach/pkg/util/span/frontierentry_interval_btree.go:1109 +0xdb fp=0xc01ab05c10 sp=0xc01ab05ba0 pc=0x1ee1b7b
github.com/cockroachdb/cockroach/pkg/util/span.(*iterator).FirstOverlap(0xc01ab05d90, 0xc00197a080?)
	github.com/cockroachdb/cockroach/pkg/util/span/frontierentry_interval_btree.go:1071 +0xf4 fp=0xc01ab05c40 sp=0xc01ab05c10 pc=0x1ee1774
github.com/cockroachdb/cockroach/pkg/util/span.(*btreeFrontier).forward(0xc0165aed70, {{0xc00b35faf0, 0x9, 0x10}, {0xc00b35f978, 0x2, 0x8}}, {0x179cae518499cddf, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/util/span/frontier.go:467 +0x1e5 fp=0xc01ab05e30 sp=0xc01ab05c40 pc=0x1eda2e5
github.com/cockroachdb/cockroach/pkg/util/span.(*btreeFrontier).Forward(0xc0165aed70, {{0xc00b35faf0, 0x9, 0x10}, {0xc00b35f978, 0x2, 0x8}}, {0x179cae518499cddf, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/util/span/frontier.go:300 +0x135 fp=0xc01ab05eb0 sp=0xc01ab05e30 pc=0x1ed8fb5
github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest.newStreamIngestionFrontierProcessor({0x78a1a98, 0xc0165aec80}, 0xc02921a780, 0x0?, {{0x179cae5180744849, 0x0, 0x0}, {0xc02f794400, 0x3d, 0x40}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_frontier_processor.go:107 +0x1c5 fp=0xc01ab06070 sp=0xc01ab05eb0 pc=0x4f4c125
github.com/cockroachdb/cockroach/pkg/sql/rowexec.NewProcessor({0x78a1a98, 0xc0165aec80}, 0x0?, 0x0?, 0xc010a901d8, 0x0?, {0xc00169f2e0?, 0x1, 0x0?}, {0xc00145b300, ...})
	github.com/cockroachdb/cockroach/pkg/sql/rowexec/processors.go:369 +0x783 fp=0xc01ab08390 sp=0xc01ab06070 pc=0x25bce83
github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*rowBasedFlow).makeProcessorAndOutput(0xc00a96eca8, {0x78a1a98, 0xc0165aec80}, 0xc010a901c0, {0xc00169f2e0, 0x1, 0x1})
	github.com/cockroachdb/cockroach/pkg/sql/rowflow/row_based_flow.go:254 +0x31a fp=0xc01ab08480 sp=0xc01ab08390 pc=0x31bd59a
github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*rowBasedFlow).setupProcessors(0xc00a96eca8, {0x78a1a98, 0xc0165aec80}, 0xc005d016e8, {0xc01cd5eff0, 0x2, 0x2})
	github.com/cockroachdb/cockroach/pkg/sql/rowflow/row_based_flow.go:93 +0x1f6 fp=0xc01ab085a8 sp=0xc01ab08480 pc=0x31bc956
github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*rowBasedFlow).Setup(0xc00a96eca8, {0x78a1ad0?, 0xc01cd5ef90?}, 0x5531300?, 0x60?)
	github.com/cockroachdb/cockroach/pkg/sql/rowflow/row_based_flow.go:75 +0xa9 fp=0xc01ab08610 sp=0xc01ab085a8 pc=0x31bc6e9
github.com/cockroachdb/cockroach/pkg/sql/distsql.(*ServerImpl).setupFlow(0xc00baedc00, {0x78a1ad0, 0xc01cd5ec90}, 0xc0043efc00, 0xc0015ac120?, 0xc0075d8180, 0xc005d016c0, {0x7870ba8, 0xc005a0e380}, {0x0, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/distsql/server.go:429 +0x1634 fp=0xc01ab08d48 sp=0xc01ab08610 pc=0x31c49d4
github.com/cockroachdb/cockroach/pkg/sql/distsql.(*ServerImpl).SetupLocalSyncFlow(0x61f511b?, {0x78a1ad0, 0xc01cd5ec90}, 0x800000000?, 0x617ceb8?, {0x7870ba8, 0xc005a0e380}, {0x0, 0x0}, {0xc019d7b800, ...})
	github.com/cockroachdb/cockroach/pkg/sql/distsql/server.go:597 +0x11a fp=0xc01ab08e10 sp=0xc01ab08d48 pc=0x31c5dba
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).setupFlows(0xc001a71e00, {0x78a1ad0, 0xc01cd5ec90}, 0xc019d7b800, 0xc006b02370, 0x0, 0xc01cd5ecc0, 0xc005a0e380, {0xc019d7b800, 0x0, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:469 +0x745 fp=0xc01ab094e8 sp=0xc01ab08e10 pc=0x3638885
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc001a71e00, {0x78a1ad0, 0xc01cd5ec90}, 0xc006b02370, 0x0, 0xc02f570180, 0xc005a0e380, 0xc019d7b800, 0x0)
	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:863 +0x814 fp=0xc01ab09e38 sp=0xc01ab094e8 pc=0x363a0f4
github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest.startDistIngestion.func6({0x78a1a98?, 0xc00b7c20f0?})
	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_dist.go:200 +0x319 fp=0xc01ab09f58 sp=0xc01ab09e38 pc=0x4f47a19
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x22 fp=0xc01ab09f78 sp=0xc01ab09f58 pc=0x1e8a082
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x56 fp=0xc01ab09fe0 sp=0xc01ab09f78 pc=0x1b05416
runtime.goexit()
	src/runtime/asm_amd64.s:1650 +0x1 fp=0xc01ab09fe8 sp=0xc01ab09fe0 pc=0x4d3121
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 3597
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0x96

@msbutler
Copy link
Collaborator

msbutler commented Dec 1, 2023

dup of #115411

@msbutler msbutler closed this as completed Dec 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
24.1
Development

No branches or pull requests
2 participants
@cockroach-teamcity @msbutler