Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

time,runtime: too many concurrent timer firings for short time.Ticker [1.23 backport] #69882

Closed
gopherbot opened this issue Oct 14, 2024 · 3 comments
Labels
CherryPickApproved Used during the release process for point releases compiler/runtime Issues related to the Go compiler and/or runtime.
Milestone

Comments

@gopherbot
Copy link
Contributor

@mknyszek requested issue #69880 to be considered for backport to the next 1.23 minor release.

@gopherbot Please open a backport issue for Go 1.23.

This is a serious issue that affects users with no reasonable workaround. This issues does not affect Go 1.22.

@gopherbot gopherbot added the CherryPickCandidate Used during the release process for point releases label Oct 14, 2024
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Oct 14, 2024
@gopherbot gopherbot added this to the Go1.23.3 milestone Oct 14, 2024
@gopherbot
Copy link
Contributor Author

Change https://go.dev/cl/620137 mentions this issue: [release-branch.go1.23] runtime: don't frob isSending for tickers

@prattmic prattmic added the CherryPickApproved Used during the release process for point releases label Oct 23, 2024
@gopherbot gopherbot removed the CherryPickCandidate Used during the release process for point releases label Oct 23, 2024
gopherbot pushed a commit that referenced this issue Oct 23, 2024
The Ticker Stop and Reset methods don't report a value,
so we don't need to track whether they are interrupting a send.

This includes a test that used to fail about 2% of the time on
my laptop when run under x/tools/cmd/stress.

For #69880
Fixes #69882

Change-Id: Ic6d14b344594149dd3c24b37bbe4e42e83f9a9ad
Reviewed-on: https://go-review.googlesource.com/c/go/+/620136
LUCI-TryBot-Result: Go LUCI <[email protected]>
Reviewed-by: Ian Lance Taylor <[email protected]>
Auto-Submit: Michael Knyszek <[email protected]>
Auto-Submit: Ian Lance Taylor <[email protected]>
Reviewed-by: Michael Knyszek <[email protected]>
(cherry picked from commit 48849e0)
Reviewed-on: https://go-review.googlesource.com/c/go/+/620137
Reviewed-by: Dmitri Shuralyov <[email protected]>
@gopherbot
Copy link
Contributor Author

Closed by merging CL 620137 (commit 8d79bf7) to release-branch.go1.23.

@gopherbot
Copy link
Contributor Author

Change https://go.dev/cl/621856 mentions this issue: [release-branch.go1.23] runtime,time: use atomic.Int32 for isSending

gopherbot pushed a commit that referenced this issue Oct 25, 2024
This change switches isSending to be an atomic.Int32 instead of an
atomic.Uint8. The Int32 version is managed as a counter, which is
something that we couldn't do with Uint8 without adding a new intrinsic
which may not be available on all architectures.

That is, instead of only being able to support 8 concurrent timer
firings on the same timer because we only have 8 independent bits to set
for each concurrent timer firing, we can now have 2^31-1 concurrent
timer firings before running into any issues. Like the fact that each
bit-set was matched with a clear, here we match increments with
decrements to indicate that we're in the "sending on a channel" critical
section in the timer code, so we can report the correct result back on
Stop or Reset.

We choose an Int32 instead of a Uint32 because it's easier to check for
obviously bad values (negative values are always bad) and 2^31-1
concurrent timer firings should be enough for anyone.

Previously, we avoided anything bigger than a Uint8 because we could
pack it into some padding in the runtime.timer struct. But it turns out
that the type that actually matters, runtime.timeTimer, is exactly 96
bytes in size. This means its in the next size class up in the 112 byte
size class because of an allocation header. We thus have some free space
to work with. This change increases the size of this struct from 96
bytes to 104 bytes.

(I'm not sure if runtime.timer is often allocated directly, but if it
is, we get lucky in the same way too. It's exactly 80 bytes in size,
which means its in the 96-byte size class, leaving us with some space to
work with.)

Fixes #69978
For #69969.
Related to #69880 and #69312 and #69882.

Change-Id: I9fd59cb6a69365c62971d1f225490a65c58f3e77
Cq-Include-Trybots: luci.golang.try:go1.23-linux-amd64-longtest
Reviewed-on: https://go-review.googlesource.com/c/go/+/621616
Reviewed-by: Ian Lance Taylor <[email protected]>
Auto-Submit: Michael Knyszek <[email protected]>
LUCI-TryBot-Result: Go LUCI <[email protected]>
(cherry picked from commit 6a49f81)
Reviewed-on: https://go-review.googlesource.com/c/go/+/621856
Auto-Submit: Ian Lance Taylor <[email protected]>
Reviewed-by: Michael Pratt <[email protected]>
Sidddddarth pushed a commit to rudderlabs/rudder-server that referenced this issue Nov 11, 2024
Upgrade to Go1.23.3, to address fatal crash observed in production.

golang/go#69882
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CherryPickApproved Used during the release process for point releases compiler/runtime Issues related to the Go compiler and/or runtime.
Projects
None yet
Development

No branches or pull requests

2 participants