Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/streamingccl/streamingest: TestStreamingAutoReplan failed #115512

Closed
cockroach-teamcity opened this issue Dec 3, 2023 · 4 comments · Fixed by #115509
Closed

ccl/streamingccl/streamingest: TestStreamingAutoReplan failed #115512

cockroach-teamcity opened this issue Dec 3, 2023 · 4 comments · Fixed by #115509
Assignees
@miretskiy
Labels
A-cdc Change Data Capture branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Dec 3, 2023 •
edited by cockroach-jira-scripts
Loading

ccl/streamingccl/streamingest.TestStreamingAutoReplan failed with artifacts on master @ 74ae9a18d82fdc80e5e9b71b3f06b258ae5cb91b:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x219a85b]

Stack:

goroutine 169810065 [running]:
github.com/cockroachdb/cockroach/pkg/util/span.(*iterator).findNextOverlap(0xc00eac5938, 0xc0176fe980?)
	github.com/cockroachdb/cockroach/pkg/util/span/frontierentry_interval_btree.go:1109 +0xdb
github.com/cockroachdb/cockroach/pkg/util/span.(*iterator).FirstOverlap(0xc00eac5938, 0x7?)
	github.com/cockroachdb/cockroach/pkg/util/span/frontierentry_interval_btree.go:1071 +0xf4
github.com/cockroachdb/cockroach/pkg/util/span.(*btreeFrontier).forward(0xc01487c690, {{0xc00e6de390, 0x7, 0x8}, {0xc00e6de398, 0x7, 0x8}}, {0x179d472582907f5c, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/util/span/frontier.go:467 +0x27b
github.com/cockroachdb/cockroach/pkg/util/span.(*btreeFrontier).Forward(0xc01487c690, {{0xc00e6de390, 0x7, 0x8}, {0xc00e6de398, 0x7, 0x8}}, {0x179d472582907f5c, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/util/span/frontier.go:300 +0x18b
github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest.(*streamIngestionProcessor).bufferCheckpoint(0xc012cb3400, {{0x7a32490?, 0xc017f3e210?}, {0x6387ba4?, 0xc00eac5e92?}})
	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor.go:871 +0x303
github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest.(*streamIngestionProcessor).handleEvent(0xc012cb3400, {{0x7a32490?, 0xc017f3e210?}, {0x6387ba4?, 0x96a285?}})
	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor.go:710 +0x2bc
github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest.(*streamIngestionProcessor).consumeEvents(0xc012cb3400, {0x179d4711e36b69c9?, 0xc019195860?})
	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor.go:655 +0xef
github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest.(*streamIngestionProcessor).Start.func3({0x7a00da0?, 0xc01ee94d70?})
	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor.go:464 +0x72
github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest.(*streamIngestionProcessor).Start.Group.GoCtx.func9()
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x22
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 169804549
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0x96
Log preceding fatal error 

=== RUN   TestStreamingAutoReplan
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestStreamingAutoReplan1059297502
    test_log_scope.go:81: use -show-logs to present logs inline

pkg/ccl/streamingccl/replicationtestutils/testutils.go:150: ((*TenantStreamingClusters).init)
	NOTICE: .WaitForTenantCapabilities() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().WaitForTenantCapabilities() instead.

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-34073
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery labels Dec 3, 2023
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Dec 3, 2023
@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestStreamingAutoReplan failed with artifacts on master @ 79a9849c37695b49653920b1c3aa76d5a8cb71d5:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x219a85b]

Stack:

goroutine 172824226 [running]:
github.com/cockroachdb/cockroach/pkg/util/span.(*iterator).findNextOverlap(0xc0206e5938, 0xc013a84f80?)
	github.com/cockroachdb/cockroach/pkg/util/span/frontierentry_interval_btree.go:1109 +0xdb
github.com/cockroachdb/cockroach/pkg/util/span.(*iterator).FirstOverlap(0xc0206e5938, 0x7?)
	github.com/cockroachdb/cockroach/pkg/util/span/frontierentry_interval_btree.go:1071 +0xf4
github.com/cockroachdb/cockroach/pkg/util/span.(*btreeFrontier).forward(0xc017a44410, {{0xc01cb1fff0, 0x7, 0x8}, {0xc01cb1fff8, 0x7, 0x8}}, {0x179d95ecbe479e76, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/util/span/frontier.go:467 +0x27b
github.com/cockroachdb/cockroach/pkg/util/span.(*btreeFrontier).Forward(0xc017a44410, {{0xc01cb1fff0, 0x7, 0x8}, {0xc01cb1fff8, 0x7, 0x8}}, {0x179d95ecbe479e76, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/util/span/frontier.go:300 +0x18b
github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest.(*streamIngestionProcessor).bufferCheckpoint(0xc0128f2000, {{0x7a32630?, 0xc01af1a648?}, {0x6387d19?, 0xc01706be92?}})
	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor.go:871 +0x303
github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest.(*streamIngestionProcessor).handleEvent(0xc0128f2000, {{0x7a32630?, 0xc01af1a648?}, {0x6387d19?, 0x96a285?}})
	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor.go:710 +0x2bc
github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest.(*streamIngestionProcessor).consumeEvents(0xc0128f2000, {0x179d95d409277639?, 0xc0156b1360?})
	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor.go:655 +0xef
github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest.(*streamIngestionProcessor).Start.func3({0x7a00f40?, 0xc0170d1680?})
	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest/stream_ingestion_processor.go:464 +0x72
github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamingest.(*streamIngestionProcessor).Start.Group.GoCtx.func9()
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x22
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 172820981
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0x96
Log preceding fatal error 

=== RUN   TestStreamingAutoReplan
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestStreamingAutoReplan2069512678
    test_log_scope.go:81: use -show-logs to present logs inline

pkg/ccl/streamingccl/replicationtestutils/testutils.go:150: ((*TenantStreamingClusters).init)
	NOTICE: .WaitForTenantCapabilities() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().WaitForTenantCapabilities() instead.

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@dt
Copy link
Member

dt commented Dec 4, 2023

#115411

@dt
Copy link
Member

dt commented Dec 4, 2023

#115509

@dt dt linked a pull request Dec 4, 2023 that will close this issue
span: Re-initialize iterator when forwarding #115509
Merged
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Dec 4, 2023

cc @cockroachdb/cdc

@blathers-crl blathers-crl bot added the A-cdc Change Data Capture label Dec 4, 2023
@miretskiy miretskiy mentioned this issue Dec 4, 2023
Merged
@craig craig bot closed this as completed in cac8b23 Dec 4, 2023
@craig craig bot closed this as completed in #115509 Dec 4, 2023
@rharding6373 rharding6373 added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@miretskiy miretskiy

Labels
A-cdc Change Data Capture branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc
Projects
None yet
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

span: Re-initialize iterator when forwarding miretskiy/cockroach
4 participants
@dt @rharding6373 @cockroach-teamcity @miretskiy