Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: cpu contention on jwk reads + suppress duplicate jwk generation #3870

Merged
merged 1 commit into from
Nov 4, 2024

Conversation

terev
Copy link
Contributor

@terev terev commented Nov 1, 2024

Using the reproduction steps included in #3863 I'm able to confirm this fixes the read contention:

bombardier -d 30s -t 30s -a -c 270 -l http://localhost:4444/.well-known/openid-configuration
Bombarding http://localhost:4444/.well-known/openid-configuration for 30s using 270 connection(s)
[==============================================================================================================================================================] 30s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec      3768.36    2589.36   12964.59
  Latency       71.76ms    29.92ms   579.56ms
  Latency Distribution
     50%    67.30ms
     75%    82.45ms
     90%    98.98ms
     95%   113.15ms
     99%   184.51ms
  HTTP codes:
    1xx - 0, 2xx - 112946, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:     7.70MB/s

I also noticed the .well-known/jwks.json route generates extra keys (if hammered before initialized) so I changed that handler to utilize the same duplicate suppression code.

Related issue(s)

Closes #3863

Checklist

  • I have read the contributing guidelines.
  • I have referenced an issue containing the design document if my change
    introduces a new feature.
  • I am following the
    contributing code guidelines.
  • I have read the security policy.
  • I confirm that this pull request does not address a security
    vulnerability. If this pull request addresses a security vulnerability, I
    confirm that I got the approval (please contact
    [email protected]) from the maintainers to push
    the changes.
  • I have added tests that prove my fix is effective or that my feature
    works.
  • I have added or changed the documentation.

Further Comments

@terev terev changed the title fix: suppress duplicate jwk generation with singleflight group fix: cpu contention on jwk reads + suppress duplicate jwk generation Nov 1, 2024
@terev terev marked this pull request as ready for review November 1, 2024 08:13
jwk/helper_test.go Outdated Show resolved Hide resolved
…te JWKs

Previously each concurrent caller would need to lock a shared mutex when reading or writing a given JWK set.
The read path now doesn't require locking a mutex at all and instead
returns valid query results directly.
The write path is now protected by a concurrency control mechanism
(using x/sync/singleflight) to ensure only one JWK set is generated and
persisted.
Note: Duplicate JWK sets may still be improperly generated if running
more than one Hydra instance in a high traffic environment.
@aeneasr aeneasr merged commit d5f65c5 into ory:master Nov 4, 2024
28 checks passed
@aeneasr
Copy link
Member

aeneasr commented Nov 4, 2024

Great job everyone, thanks!

@terev
Copy link
Contributor Author

terev commented Nov 4, 2024

🎉 Thanks for the well written initial report @awill1988 !

@terev terev deleted the suppress-duplicate-jwk-gen branch November 4, 2024 15:30
@awill1988
Copy link
Contributor

Awesome fix @terev!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

GetOrGenerateKeys locking leads to significant service degredation with high request saturation
3 participants