Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[server, installer] Make PrebuildRateLimiter period configurable #14975

Merged
merged 1 commit into from
Nov 30, 2022

Conversation

geropl
Copy link
Member

@geropl geropl commented Nov 25, 2022

Description

Effectively changes the rate limiter from being

Limit:  50,
Period: 60,

to be

Limit:  100,
Period: 600,

Related Issue(s)

Fixes #14791

How to test

  • setup a project
  • change the server ConfigMap to be set to limit: 2, and kubectl rollout restart deployment/server
  • trigger 3 prebuilds on that project withing 5 minutes
  • notice how the 3rd gets rejected (you can use the ...project/events endpoint for that)

Release Notes

NONE

Documentation

Werft options:

  • /werft with-local-preview
    If enabled this will build install/preview
  • /werft with-preview
  • /werft with-large-vm
  • /werft with-integration-tests=all
    Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh

@werft-gitpod-dev-com
Copy link

started the job as gitpod-build-gpl-14791-leaky.2 because the annotations in the pull request description changed
(with .werft/ from main)

@geropl geropl force-pushed the gpl/14791-leaky branch 3 times, most recently from 89479f0 to 89f6ab9 Compare November 29, 2022 08:24
@geropl geropl marked this pull request as ready for review November 29, 2022 09:57
@geropl geropl requested a review from a team November 29, 2022 09:57
@github-actions github-actions bot added the team: webapp Issue belongs to the WebApp team label Nov 29, 2022
cloneURL: string,
): PrebuildRateLimiterConfig {
// First we use any explicit overrides for a given cloneURL
let config = rateLimiters[cloneURL];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be the scenario where we would want to change the rate limiting for one cloneURL but not for all?

Copy link
Member Author

@geropl geropl Nov 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this is the same code as before, and I'd argue out of scope for this PR.
The only change here is to pull up the time period to be configurable with changing the installer code, which is justified by the argument made above.

Ultimately, I think the whole approach is ill-suited for what we want to achieve. And I feel we have alignment on that we want to pursue a controller-pased/async approach for handling prebuild triggers instead.

But as long as we are not there yet (and it's still unclear how to prioritize that work) it's important to me that we have the ability to quickly and confidently react to incidents like the one we had the other week.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think having a default, which can be overriden for specific clone URLs makes perfect sense. It exists largely for the case where you get hit hard, system pages you and you want to adjust the config only to mitigate the issue. But adjusting for all is outside of the scope of that incident and would require deeper investigation.

Comment on lines 256 to 268
PrebuildLimiter: PrebuildRateLimiters{
// default limit for all cloneURLs
"*": 50,
"*": PrebuildRateLimiterConfig{
Limit: 50,
Period: 300,
},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wanna extend the configmap_test.go to check for this? Given there was no change to the test file with this change suggests there's no test coverage for this.

@easyCZ
Copy link
Member

easyCZ commented Nov 29, 2022

/hold for discussion

@geropl
Copy link
Member Author

geropl commented Nov 30, 2022

/unhold

@werft-gitpod-dev-com
Copy link

started the job as gitpod-build-gpl-14791-leaky.8 because the annotations in the pull request description changed
(with .werft/ from main)

@geropl
Copy link
Member Author

geropl commented Nov 30, 2022

/werft run

👍 started the job as gitpod-build-gpl-14791-leaky.9
(with .werft/ from main)

@roboquat roboquat merged commit bafbaab into main Nov 30, 2022
@roboquat roboquat deleted the gpl/14791-leaky branch November 30, 2022 10:44
@roboquat roboquat added deployed: webapp Meta team change is running in production deployed Change is completely running in production labels Dec 1, 2022
@atduarte
Copy link
Contributor

FYI @geropl removed the release notes as this does not impact the end user of gitpod.io or dedicated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
deployed: webapp Meta team change is running in production deployed Change is completely running in production release-note-none size/L team: webapp Issue belongs to the WebApp team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[prebuilds] More effective rate-limit (Rate-limit prebuilds per user?)
5 participants