Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql/catalog/lease: TestRangefeedUpdatesHandledProperlyInTheFaceOfRaces failed [test deadlocks in LeaseAcquiredEvent testing knob] #129097

Closed
cockroach-teamcity opened this issue Aug 15, 2024 · 3 comments · Fixed by #129302
Closed

sql/catalog/lease: TestRangefeedUpdatesHandledProperlyInTheFaceOfRaces failed [test deadlocks in LeaseAcquiredEvent testing knob] #129097

cockroach-teamcity opened this issue Aug 15, 2024 · 3 comments · Fixed by #129302
Assignees
@rafiss
Labels
branch-master Failures and bugs on the master branch. branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.1.4-rc branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 15, 2024 •
edited by cockroach-jira-scripts
Loading

sql/catalog/lease.TestRangefeedUpdatesHandledProperlyInTheFaceOfRaces failed with artifacts on master @ 5907123373f975b4c4eb3695f0222c1912ed8b2a:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestRangefeedUpdatesHandledProperlyInTheFaceOfRaces (13m30s)

Stack:

goroutine 220949 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x30c
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x38
Log preceding fatal error 

=== RUN   TestRangefeedUpdatesHandledProperlyInTheFaceOfRaces
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1e31bde6494190748168afe7a747530a/logTestRangefeedUpdatesHandledProperlyInTheFaceOfRaces2618326280
    test_log_scope.go:81: use -show-logs to present logs inline

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-41371
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Aug 15, 2024
@rafiss
Copy link
Collaborator

rafiss commented Aug 19, 2024 

goroutine 185669 [chan receive, 13 minutes]:
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease_test.TestRangefeedUpdatesHandledProperlyInTheFaceOfRaces.func2({0xffff6ce5da90?, 0x4005a76b08?}, {0x5261800?, 0x1?})
	github.com/cockroachdb/cockroach/pkg/sql/catalog/lease_test/pkg/sql/catalog/lease/lease_test.go:2426 +0x108
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.storage.acquire({0x4007aa2200, {0x63f5c90, 0x4007261320}, 0x4014298e40, 0x4002c1a000, {{0x9885d20, 0x9885d40, {0x1}}, {0x9885d20}, 0x0}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/storage.go:284 +0x628
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.acquireNodeLease.func1({0x6418ff8, 0x402a317950})
	github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/lease.go:665 +0x1d0
github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall.func1({0x6418ff8?, 0x402a317950?})
	github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:389 +0x38
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask(0x4007260090, {0x6418ff8, 0x402a317950}, {0x63b62e0?, 0x400a86c000?}, 0x4004748f40)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:319 +0xac
github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall(0x4007de95c0, {0x6419030?, 0x402620c930?}, 0x400db9c000, {0x401ae125a7, 0x3}, {0x4007260090?, 0xa0?}, 0x401a6e9bc0)
	github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:388 +0x244
created by github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).DoChan in goroutine 185634
	github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:358 +0x484

goroutine 185680 [IO wait, 13 minutes]:
internal/poll.runtime_pollWait(0xffff65c725f8, 0x72)
	GOROOT/src/runtime/netpoll.go:345 +0xa0
internal/poll.(*pollDesc).wait(0x401ab6d900?, 0x4004c91c00?, 0x0)
	GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	GOROOT/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x401ab6d900, {0x4004c91c00, 0x700, 0x700})
	GOROOT/src/internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x401ab6d900, {0x4004c91c00?, 0xffff6b698e20?, 0x40084ed4b8?})
	GOROOT/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x4029588c08, {0x4004c91c00?, 0x4007467318?, 0x421dfc?})
	GOROOT/src/net/net.go:185 +0x34
crypto/tls.(*atLeastReader).Read(0x40084ed4b8, {0x4004c91c00?, 0x0?, 0x40084ed4b8?})
	GOROOT/src/crypto/tls/conn.go:806 +0x40
bytes.(*Buffer).ReadFrom(0x40186509b0, {0x63ddac0, 0x40084ed4b8})
	GOROOT/src/bytes/buffer.go:211 +0x90
crypto/tls.(*Conn).readFromUntil(0x4018650708, {0x63dd800, 0x4029588c08}, 0x40074673f0?)
	GOROOT/src/crypto/tls/conn.go:828 +0xd0
crypto/tls.(*Conn).readRecordOrCCS(0x4018650708, 0x0)
	GOROOT/src/crypto/tls/conn.go:626 +0x35c
crypto/tls.(*Conn).readRecord(...)
	GOROOT/src/crypto/tls/conn.go:588
crypto/tls.(*Conn).Read(0x4018650708, {0x4003e86000, 0x1000, 0x935064?})
	GOROOT/src/crypto/tls/conn.go:1370 +0x168
bufio.(*Reader).Read(0x4006c6b1a0, {0x400f7f3228, 0x5, 0x2a?})
	GOROOT/src/bufio/bufio.go:241 +0x1b4
io.ReadAtLeast({0x63d4000, 0x4006c6b1a0}, {0x400f7f3228, 0x5, 0x200}, 0x5)
	GOROOT/src/io/io.go:335 +0xa0
io.ReadFull(...)
	GOROOT/src/io/io.go:354
github.com/lib/pq.(*conn).recvMessage(0x400f7f3208, 0x4007467850)
	github.com/lib/pq/external/com_github_lib_pq/conn.go:1004 +0xc4
github.com/lib/pq.(*conn).recv1Buf(0x400f7f3208, 0x4007467850)
	github.com/lib/pq/external/com_github_lib_pq/conn.go:1059 +0x2c
github.com/lib/pq.(*conn).recv1(...)
	github.com/lib/pq/external/com_github_lib_pq/conn.go:1086
github.com/lib/pq.(*conn).simpleExec(0x400f7f3208, {0x4f4caee?, 0x34e?})
	github.com/lib/pq/external/com_github_lib_pq/conn.go:664 +0x17c
github.com/lib/pq.(*conn).Exec(0x400f7f3208, {0x4f4caee, 0x24}, {0x99ba7a0, 0x0, 0x0})
	github.com/lib/pq/external/com_github_lib_pq/conn.go:921 +0x194
github.com/lib/pq.(*conn).ExecContext(0x400f7f3208, {0x6418e28, 0x99ba7a0}, {0x4f4caee, 0x24}, {0x99ba7a0, 0x0, 0x4009fd7900?})
	github.com/lib/pq/external/com_github_lib_pq/conn_go18.go:46 +0x110
database/sql.ctxDriverExec({0x6418e28?, 0x99ba7a0?}, {0xffff6b6ae818?, 0x400f7f3208?}, {0x0?, 0x0?}, {0x4f4caee?, 0x2faf080?}, {0x99ba7a0?, 0x4000000000000000?, ...})
	GOROOT/src/database/sql/ctxutil.go:31 +0xac
database/sql.(*DB).execDC.func2()
	GOROOT/src/database/sql/sql.go:1703 +0x114
database/sql.withLock({0x63e9150, 0x40031eb170}, 0x4004748cf8)
	GOROOT/src/database/sql/sql.go:3530 +0x7c
database/sql.(*DB).execDC(0x4005ba55f0?, {0x6418e28, 0x99ba7a0}, 0x40031eb170, 0x4015958de0?, {0x4f4caee, 0x24}, {0x0, 0x0, 0x0})
	GOROOT/src/database/sql/sql.go:1698 +0x178
database/sql.(*DB).exec(0x4005ba55f0, {0x6418e28, 0x99ba7a0}, {0x4f4caee, 0x24}, {0x0, 0x0, 0x0}, 0x30?)
	GOROOT/src/database/sql/sql.go:1683 +0xa0
database/sql.(*DB).ExecContext.func1(0x1a?)
	GOROOT/src/database/sql/sql.go:1662 +0x54
database/sql.(*DB).retry(0x400981d5dc?, 0x4007467f00)
	GOROOT/src/database/sql/sql.go:1566 +0x4c
database/sql.(*DB).ExecContext(0x0?, {0x6418e28?, 0x99ba7a0?}, {0x4f4caee?, 0x4014793d40?}, {0x0?, 0xb60a40?, 0x4001d7a440?})
	GOROOT/src/database/sql/sql.go:1661 +0x94
database/sql.(*DB).Exec(...)
	GOROOT/src/database/sql/sql.go:1675
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease_test.TestRangefeedUpdatesHandledProperlyInTheFaceOfRaces.func4()
	github.com/cockroachdb/cockroach/pkg/sql/catalog/lease_test/pkg/sql/catalog/lease/lease_test.go:2463 +0x58
created by github.com/cockroachdb/cockroach/pkg/sql/catalog/lease_test.TestRangefeedUpdatesHandledProperlyInTheFaceOfRaces in goroutine 182042
	github.com/cockroachdb/cockroach/pkg/sql/catalog/lease_test/pkg/sql/catalog/lease/lease_test.go:2462 +0x66c

this part of the test seems to have a deadlock, cc @fqazi

cockroach/pkg/sql/catalog/lease/lease_test.go

Lines 2415 to 2426 in 51a7038

LeaseStoreTestingKnobs: lease.StorageTestingKnobs{
LeaseAcquiredEvent: func(desc catalog.Descriptor, _ error) {
// Block the lease acquisition for the desc after the leasing
// transaction has been issued. We'll wait to unblock it until after
// the new version has been published and that even has been received.
if desc.GetID() != interestingTable.Load().(descpb.ID) {
return
}
blocked := make(chan struct{})
select {
case blockLeaseAcquisitionOfInterestingTable <- blocked:
<-blocked

@rafiss rafiss added P-2 Issues/test failures with a fix SLA of 3 months and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Aug 19, 2024
@rafiss rafiss changed the title sql/catalog/lease: TestRangefeedUpdatesHandledProperlyInTheFaceOfRaces failed sql/catalog/lease: TestRangefeedUpdatesHandledProperlyInTheFaceOfRaces failed [test deadlocks in LeaseAcquiredEvent testing knob] Aug 19, 2024
@rafiss rafiss self-assigned this Aug 20, 2024
@rafiss rafiss mentioned this issue Aug 20, 2024
Merged
@craig craig bot closed this as completed in d5329dc Aug 20, 2024
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Aug 20, 2024

Based on the specified backports for linked PR #129302, I applied the following new label(s) to this issue: branch-release-23.2, branch-release-24.1, branch-release-24.2. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 labels Aug 20, 2024
This was referenced Aug 20, 2024
Merged
Merged
@rafiss rafiss removed the branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 label Aug 20, 2024
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Aug 27, 2024

Based on the specified backports for linked PR #129302, I applied the following new label(s) to this issue: branch-release-24.1.4-rc. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot mentioned this issue Aug 27, 2024
Merged
@spilchen spilchen mentioned this issue Nov 26, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rafiss rafiss

Labels
branch-master Failures and bugs on the master branch. branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.1.4-rc branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
No open projects
SQL Foundations
Status: Triage
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

lease: fix goroutine starvation caused by test cluster setting rafiss/cockroach
2 participants
@rafiss @cockroach-teamcity