Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/streamingccl/streamproducer: TestStreamPartition failed #100414

Closed
cockroach-teamcity opened this issue Apr 2, 2023 · 1 comment · Fixed by #100731
Closed

ccl/streamingccl/streamproducer: TestStreamPartition failed #100414

cockroach-teamcity opened this issue Apr 2, 2023 · 1 comment · Fixed by #100731
Assignees
@msbutler
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-disaster-recovery
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Apr 2, 2023 •
edited by cockroach-jira-scripts
Loading

ccl/streamingccl/streamproducer.TestStreamPartition failed with artifacts on release-23.1 @ 63b683eab3e0ebfff812bb54fdac936c101d9383:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4a0b443]

Stack:

goroutine 8602051 [running]:
testing.tRunner.func1.2({0x524a9c0, 0x9f00990})
	GOROOT/src/testing/testing.go:1396 +0x24e
testing.tRunner.func1()
	GOROOT/src/testing/testing.go:1399 +0x39f
panic({0x524a9c0, 0x9f00990})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/replicationtestutils.(*ReplicationFeed).ObserveKey(0xc00ccfe810, {0x7035178, 0xc000086058}, {0xc007ed21e0?, 0x70fe168?, 0xc007096a00?})
	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/replicationtestutils/replication_helpers.go:110 +0xc3
github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamproducer_test.TestStreamPartition.func2(0x7035178?)
	github.com/cockroachdb/cockroach/pkg/ccl/streamingccl/streamproducer_test/pkg/ccl/streamingccl/streamproducer/replication_stream_test.go:406 +0x6bb
testing.tRunner(0xc00b22c1a0, 0xc005252600)
	GOROOT/src/testing/testing.go:1446 +0x10b
created by testing.(*T).Run
	GOROOT/src/testing/testing.go:1493 +0x35f
Log preceding fatal error 

=== RUN   TestStreamPartition
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/9ce24496530642208197dda818d65c9e/logTestStreamPartition3544692374
    test_log_scope.go:79: use -show-logs to present logs inline
--- FAIL: TestStreamPartition (250.33s)
=== RUN   TestStreamPartition/stream-table
    --- FAIL: TestStreamPartition/stream-table (120.08s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-26421
@cockroach-teamcity cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 2, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Apr 2, 2023
@msbutler
Copy link
Collaborator

msbutler commented Apr 5, 2023

This panic occurs when replicationFeed.ObserveKey is about to return a key after calling the internal consumeUntil function. This function is supposed to error or return the key, but I've found a bug in which consumeUntil swallows any error surfaced by the feed and returns without an expected key. This bug is likely why the panic here occurred.

I wasn't able to reproduce this under stress and race on 23.1 and master, though I'll patch this test so we can get a more descriptive error next time.

@msbutler msbutler mentioned this issue Apr 5, 2023
Merged
@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 16, 2023
Closed
msbutler added a commit to msbutler/cockroach that referenced this issue Apr 20, 2023
@msbutler
c2c: clean up ReplicationFeed error handling
5051370 
Previously, the replicationFeed test helper had methods that would swallow
errors, making it impossible to debug certain test failures. This patch cleans
up the internals of this test helper and prevents error swallowing.

Fixes cockroachdb#100414

Release note: None
craig bot pushed a commit that referenced this issue Apr 20, 2023
@msbutler @jayshrivastava @nvanbenschoten
Merge #100731 #101860 #101936
ccc9d02 
100731: c2c: clean up ReplicationFeed error handling r=lidorcarmel a=msbutler

Previously, the replicationFeed test helper had methods that would swallow errors, making it impossible to debug certain test failures. This patch cleans up the internals of this test helper and prevents error swallowing.

Fixes #100414

Release note: None

101860: util/parquet: add support for arrays r=miretskiy a=jayshrivastava

This change extends and refactors the util/parquet library to be able to read and write arrays.

Release note: None

Informs: #99028
Epic: https://cockroachlabs.atlassian.net/browse/CRDB-15071

101936: kv: deflake and unskip TestStoreResolveMetrics r=arulajmani a=nvanbenschoten

Fixes #98404.

The test had begun flaking after #98044 because we now perform more async intent resolution operations when starting a cluster. Specifically, we perform additional async intent resolution operations in service of jobs updates. These updates perform SELECT FOR UPDATE queries over the new `system.job_info` table, but then perform a 1-phase commit.

To deflake the test, we clear the intent resolution metrics after server startup.

Release note: None

Co-authored-by: Michael Butler <[email protected]>
Co-authored-by: Jayant Shrivastava <[email protected]>
Co-authored-by: Nathan VanBenschoten <[email protected]>
@craig craig bot closed this as completed in 8571abc Apr 20, 2023
@blathers-crl blathers-crl bot mentioned this issue Apr 20, 2023
Merged
msbutler added a commit that referenced this issue Apr 25, 2023
@msbutler
c2c: clean up ReplicationFeed error handling
cdd467b 
Previously, the replicationFeed test helper had methods that would swallow
errors, making it impossible to debug certain test failures. This patch cleans
up the internals of this test helper and prevents error swallowing.

Fixes #100414

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@msbutler msbutler

Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
23.1
Development

Successfully merging a pull request may close this issue.

c2c: clean up ReplicationFeed error handling msbutler/cockroach
2 participants
@cockroach-teamcity @msbutler