Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pkg/ccl/backupccl/backuprand/backuprand_test: TestBackupRestoreRandomDataRoundtrips failed #115483

Closed
cockroach-teamcity opened this issue Dec 2, 2023 · 2 comments · Fixed by #115509
Closed

pkg/ccl/backupccl/backuprand/backuprand_test: TestBackupRestoreRandomDataRoundtrips failed #115483

cockroach-teamcity opened this issue Dec 2, 2023 · 2 comments · Fixed by #115509
Assignees
@miretskiy
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Dec 2, 2023 •
edited by cockroach-jira-scripts
Loading

pkg/ccl/backupccl/backuprand/backuprand_test.TestBackupRestoreRandomDataRoundtrips failed with artifacts on master @ 74ae9a18d82fdc80e5e9b71b3f06b258ae5cb91b:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1edc7bb]

Stack:

goroutine 895514 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc1?, {0x79a8420, 0xc00e2db320})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x58544c0?, 0xb11fbf0?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/util/span.(*iterator).findNextOverlap(0xc005c5cf58, 0xc0061b9e00?)
	github.com/cockroachdb/cockroach/pkg/util/span/frontierentry_interval_btree.go:1109 +0xdb
github.com/cockroachdb/cockroach/pkg/util/span.(*iterator).FirstOverlap(0xc005c5cf58, 0x2?)
	github.com/cockroachdb/cockroach/pkg/util/span/frontierentry_interval_btree.go:1071 +0xf4
github.com/cockroachdb/cockroach/pkg/util/span.(*btreeFrontier).forward(0xc0055f5f90, {{0xc012c90378, 0x2, 0x8}, {0xc012c90380, 0x2, 0x8}}, {0x179cf0a386531626, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/util/span/frontier.go:467 +0x27b
github.com/cockroachdb/cockroach/pkg/util/span.(*btreeFrontier).Forward(0xc0055f5f90, {{0xc012c90378, 0x2, 0x8}, {0xc012c90380, 0x2, 0x8}}, {0x179cf0a386531626, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/util/span/frontier.go:300 +0x18b
github.com/cockroachdb/cockroach/pkg/ccl/backupccl.checkCoverage({0xc01092e558?, 0x1?}, {0xc0143ed400?, 0xc0025f7009?, 0x1?}, {0xc00f16ae00, 0x1, 0x1?})
	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_span_coverage.go:63 +0x571
github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restore({0x79a8420, 0xc00460f0b0}, {0x7a011e0, 0xc0040020a0}, {0xc00f16ae00, 0x1, 0x1}, {0xc01092e558, 0x1, 0x1}, ...)
	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:285 +0x27d
github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restoreWithRetry({0x79a8420, 0xc00460f0b0}, {0x7a011e0, 0xc0040020a0}, {0xc00f16ae00, 0x1, 0x1}, {0xc01092e558, 0x1, 0x1}, ...)
	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:172 +0x478
github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).doResume(0xc012a92bd0, {0x79a8420, 0xc00460f0b0}, {0x5eace80?, 0xc0040020a0})
	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1801 +0x195c
github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).Resume(0xc012a92bd0, {0x79a8420, 0xc00460f0b0}, {0x5eace80?, 0xc0040020a0?})
	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1586 +0x4a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2(0xc01481a340?, 0x0?, 0xc005c5f838, {0x7986540?, 0xc012a92bd0?}, {0x79a8420?, 0xc00460f0b0?}, {0x5eace80?, 0xc0040020a0?})
	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1565 +0xc2
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc000e41400, {0x79a8420, 0xc00e2db710}, {0x5eace80, 0xc0040020a0}, {0x7986540, 0xc012a92bd0}, 0xc01481a340, {0x62807e2, 0x7}, ...)
	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1566 +0xa2d
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc000e41400, {0x79a8420, 0xc00e2db710}, {0x7986540, 0xc012a92bd0}, 0xc01481a340, {0x62807e2, 0x7}, {0xc012164090, 0x16})
	github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:447 +0x552
github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2({0xc0056a3800?, 0x0?})
	github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:905 +0xe7
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 887666
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x415
Log preceding fatal error 

=== RUN   TestBackupRestoreRandomDataRoundtrips
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/4de3d9455ce580fd52f59602b3ef6e62/logTestBackupRestoreRandomDataRoundtrips1860539140
    test_log_scope.go:81: use -show-logs to present logs inline
*
* ERROR: a panic has occurred!
* runtime error: invalid memory address or nil pointer dereference
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:914
*   | runtime.panicmem
*   | 	GOROOT/src/runtime/panic.go:261
*   | runtime.sigpanic
*   | 	GOROOT/src/runtime/signal_unix.go:861
*   | github.com/cockroachdb/cockroach/pkg/util/span.(*iterator).findNextOverlap
*   | 	github.com/cockroachdb/cockroach/pkg/util/span/frontierentry_interval_btree.go:1109
*   | github.com/cockroachdb/cockroach/pkg/util/span.(*iterator).FirstOverlap
*   | 	github.com/cockroachdb/cockroach/pkg/util/span/frontierentry_interval_btree.go:1071
*   | github.com/cockroachdb/cockroach/pkg/util/span.(*btreeFrontier).forward
*   | 	github.com/cockroachdb/cockroach/pkg/util/span/frontier.go:467
*   | github.com/cockroachdb/cockroach/pkg/util/span.(*btreeFrontier).Forward
*   | 	github.com/cockroachdb/cockroach/pkg/util/span/frontier.go:300
*   | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.checkCoverage
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_span_coverage.go:63
*   | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restore
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:285
*   | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restoreWithRetry
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:172
*   | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).doResume
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1801
*   | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).Resume
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1586
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
*   | 	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1565
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
*   | 	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1566
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
*   | 	github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:447
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2
*   | 	github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:905
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1650
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-34052
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery labels Dec 2, 2023
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Dec 2, 2023
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/backupccl/backuprand/backuprand_test.TestBackupRestoreRandomDataRoundtrips failed with artifacts on master @ 8b246d0009d40d7ecad14df14bc2fb61b31f4be5:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1edc7bb]

Stack:

goroutine 1005065 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xcd?, {0x79a85c0, 0xc013ad0480})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x5854600?, 0xb120bf0?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/util/span.(*iterator).findNextOverlap(0xc006400f58, 0xc0018fc880?)
	github.com/cockroachdb/cockroach/pkg/util/span/frontierentry_interval_btree.go:1109 +0xdb
github.com/cockroachdb/cockroach/pkg/util/span.(*iterator).FirstOverlap(0xc006400f58, 0x2?)
	github.com/cockroachdb/cockroach/pkg/util/span/frontierentry_interval_btree.go:1071 +0xf4
github.com/cockroachdb/cockroach/pkg/util/span.(*btreeFrontier).forward(0xc00dc65cc0, {{0xc010a26be0, 0x2, 0x8}, {0xc010a26be8, 0x2, 0x8}}, {0x179d8cf79282b6f8, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/util/span/frontier.go:467 +0x27b
github.com/cockroachdb/cockroach/pkg/util/span.(*btreeFrontier).Forward(0xc00dc65cc0, {{0xc010a26be0, 0x2, 0x8}, {0xc010a26be8, 0x2, 0x8}}, {0x179d8cf79282b6f8, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/util/span/frontier.go:300 +0x18b
github.com/cockroachdb/cockroach/pkg/ccl/backupccl.checkCoverage({0xc010cc5388?, 0x1?}, {0xc010d37800?, 0xc005053058?, 0x1?}, {0xc0042f2a80, 0x1, 0x1?})
	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_span_coverage.go:63 +0x571
github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restore({0x79a85c0, 0xc00cc73350}, {0x7a01380, 0xc00600c188}, {0xc0042f2a80, 0x1, 0x1}, {0xc010cc5388, 0x1, 0x1}, ...)
	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:285 +0x27d
github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restoreWithRetry({0x79a85c0, 0xc00cc73350}, {0x7a01380, 0xc00600c188}, {0xc0042f2a80, 0x1, 0x1}, {0xc010cc5388, 0x1, 0x1}, ...)
	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:172 +0x478
github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).doResume(0xc0055ef570, {0x79a85c0, 0xc00cc73350}, {0x5eacfc0?, 0xc00600c188})
	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1801 +0x195c
github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).Resume(0xc0055ef570, {0x79a85c0, 0xc00cc73350}, {0x5eacfc0?, 0xc00600c188?})
	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1586 +0x4a
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2(0xc004e3bba0?, 0x0?, 0xc006403838, {0x79866e0?, 0xc0055ef570?}, {0x79a85c0?, 0xc00cc73350?}, {0x5eacfc0?, 0xc00600c188?})
	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1565 +0xc2
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc000d12a00, {0x79a85c0, 0xc013ad07b0}, {0x5eacfc0, 0xc00600c188}, {0x79866e0, 0xc0055ef570}, 0xc004e3bba0, {0x6280922, 0x7}, ...)
	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1566 +0xa2d
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc000d12a00, {0x79a85c0, 0xc013ad07b0}, {0x79866e0, 0xc0055ef570}, 0xc004e3bba0, {0x6280922, 0x7}, {0xc014e12198, 0x16})
	github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:447 +0x552
github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2({0xc00f332800?, 0xc00f5507b0?})
	github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:905 +0xe7
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 996019
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x415
Log preceding fatal error 

=== RUN   TestBackupRestoreRandomDataRoundtrips
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/4de3d9455ce580fd52f59602b3ef6e62/logTestBackupRestoreRandomDataRoundtrips2828438660
    test_log_scope.go:81: use -show-logs to present logs inline
*
* ERROR: a panic has occurred!
* runtime error: invalid memory address or nil pointer dereference
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:914
*   | runtime.panicmem
*   | 	GOROOT/src/runtime/panic.go:261
*   | runtime.sigpanic
*   | 	GOROOT/src/runtime/signal_unix.go:861
*   | github.com/cockroachdb/cockroach/pkg/util/span.(*iterator).findNextOverlap
*   | 	github.com/cockroachdb/cockroach/pkg/util/span/frontierentry_interval_btree.go:1109
*   | github.com/cockroachdb/cockroach/pkg/util/span.(*iterator).FirstOverlap
*   | 	github.com/cockroachdb/cockroach/pkg/util/span/frontierentry_interval_btree.go:1071
*   | github.com/cockroachdb/cockroach/pkg/util/span.(*btreeFrontier).forward
*   | 	github.com/cockroachdb/cockroach/pkg/util/span/frontier.go:467
*   | github.com/cockroachdb/cockroach/pkg/util/span.(*btreeFrontier).Forward
*   | 	github.com/cockroachdb/cockroach/pkg/util/span/frontier.go:300
*   | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.checkCoverage
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_span_coverage.go:63
*   | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restore
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:285
*   | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.restoreWithRetry
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:172
*   | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).doResume
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1801
*   | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).Resume
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1586
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
*   | 	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1565
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
*   | 	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1566
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
*   | 	github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:447
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2
*   | 	github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:905
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1650
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Dec 4, 2023

cc @cockroachdb/cdc

@blathers-crl blathers-crl bot added the A-cdc Change Data Capture label Dec 4, 2023
@exalate-issue-sync exalate-issue-sync bot removed the A-cdc Change Data Capture label Dec 4, 2023
@miretskiy miretskiy mentioned this issue Dec 4, 2023
Merged
@craig craig bot closed this as completed in #115509 Dec 4, 2023
@craig craig bot closed this as completed in cac8b23 Dec 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@miretskiy miretskiy

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc
Projects
None yet
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

span: Re-initialize iterator when forwarding miretskiy/cockroach
3 participants
@dt @cockroach-teamcity @miretskiy