Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/streamingccl/streamproducer: TestStreamAddSSTable failed #102523

Closed
cockroach-teamcity opened this issue Apr 28, 2023 · 4 comments · Fixed by #102593
Closed

ccl/streamingccl/streamproducer: TestStreamAddSSTable failed #102523

cockroach-teamcity opened this issue Apr 28, 2023 · 4 comments · Fixed by #102593
Assignees
@msbutler
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test T-disaster-recovery
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Apr 28, 2023 •
edited by cockroach-jira-scripts
Loading

ccl/streamingccl/streamproducer.TestStreamAddSSTable failed with artifacts on master @ 3b1bc872bb4bf8e5eb2ad8790ba8ec781167dd1f:

=== RUN   TestStreamAddSSTable
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/9ce24496530642208197dda818d65c9e/logTestStreamAddSSTable910410699
    test_log_scope.go:79: use -show-logs to present logs inline
    replication_stream_test.go:563: -- test log scope end --
    replication_stream_test.go:563: Leaked goroutine: goroutine 5947038 [select]:
        github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
        	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c
        created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
        	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
--- FAIL: TestStreamAddSSTable (26.09s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-27490
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-disaster-recovery labels Apr 28, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Apr 28, 2023
@dt
Copy link
Member

dt commented Apr 28, 2023

did we expect this to be fixed by #102258 ?

@msbutler
Copy link
Collaborator

Yeah, but perhaps this github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1() is apart of a different leaky goroutine. I can take this an manage the backport.

@msbutler msbutler assigned msbutler and unassigned lidorcarmel Apr 28, 2023
@msbutler
Copy link
Collaborator

yeah, the og issue detected 2 leaky goroutines. #97346
So the original patch only addressed one leaky goroutine.

msbutler added a commit to msbutler/cockroach that referenced this issue Apr 28, 2023
@msbutler
c2c: deflake and skip a few unit tests
5cb1684 
Fixes cockroachdb#102523
Informs cockroachdb#102286
Informs cockroachdb#86206

Release note: none
@msbutler msbutler mentioned this issue Apr 28, 2023
Merged
msbutler added a commit to msbutler/cockroach that referenced this issue Apr 28, 2023
@msbutler
c2c: skip a few unit tests
bfeaf2b 
Informs cockroachdb#102523
Informs cockroachdb#102286
Informs cockroachdb#86206

Release note: none
@msbutler
Copy link
Collaborator

jk, there may be a real leak here. investigating further and will skip the test for now.

craig bot pushed a commit that referenced this issue Apr 28, 2023
@msbutler
Merge #102593
d6946ef 
102593: c2c: skip and deflake a few unit tests r=benbardin a=msbutler

c2c: skip a few unit tests
Informs #102523
Informs #102286
Informs #86206

Release note: none

c2c: close pgconn with correct context in test infra
When a user opens a pgx connection, they pass it a context that must be used to
close the connection, else pgx's contextWatcher will run indefinetly on a
seperate goroutine.  Previously, the `startReplication()` test helper was
initializing its own context to open pgx connection, and did not pass this
context back to the caller, and consequently, the user would close the
connection with the incorrect context, leading to a leaky goroutine.

This patch allows the user to pass a context in the startReplication(),
preventing the user from closing the connection with the wrong goroutine.

Fixes #102523

Release note: None

Co-authored-by: Michael Butler <[email protected]>
@craig craig bot closed this as completed in e7d0336 Apr 28, 2023
blathers-crl bot pushed a commit that referenced this issue Apr 28, 2023
@msbutler
c2c: skip a few unit tests
c9455f5 
Informs #102523
Informs #102286
Informs #86206

Release note: none
blathers-crl bot pushed a commit that referenced this issue Apr 28, 2023
@msbutler
c2c: close pgconn with correct context in test infra
1c24966 
When a user opens a pgx connection, they pass it a context that must be used to
close the connection, else pgx's contextWatcher will run indefinetly on a
seperate goroutine.  Previously, the `startReplication()` test helper was
initializing its own context to open pgx connection, and did not pass this
context back to the caller, and consequently, the user would close the
connection with the incorrect context, leading to a leaky goroutine.

This patch allows the user to pass a context in the startReplication(),
preventing the user from closing the connection with the wrong goroutine.

Fixes #102523

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Apr 28, 2023
Merged
fqazi pushed a commit to fqazi/cockroach that referenced this issue May 11, 2023
@msbutler @fqazi
Merge cockroachdb#102593
cd5a7e4 
102593: c2c: skip and deflake a few unit tests r=benbardin a=msbutler

c2c: skip a few unit tests
Informs cockroachdb#102523
Informs cockroachdb#102286
Informs cockroachdb#86206

Release note: none

c2c: close pgconn with correct context in test infra
When a user opens a pgx connection, they pass it a context that must be used to
close the connection, else pgx's contextWatcher will run indefinetly on a
seperate goroutine.  Previously, the `startReplication()` test helper was
initializing its own context to open pgx connection, and did not pass this
context back to the caller, and consequently, the user would close the
connection with the incorrect context, leading to a leaky goroutine.

This patch allows the user to pass a context in the startReplication(),
preventing the user from closing the connection with the wrong goroutine.

Fixes cockroachdb#102523

Release note: None

Co-authored-by: Michael Butler <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@msbutler msbutler

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
23.1
Development

Successfully merging a pull request may close this issue.

c2c: skip and deflake a few unit tests msbutler/cockroach
4 participants
@dt @cockroach-teamcity @msbutler @lidorcarmel