Request billing project quota increase and/or mitigate

[spiffxp]

Guess what I just hit while working #851

$ ./ensure-e2e-projects.sh
Ensuring boskos-janitor is empowered
  Ensuring external ip address exists for boskos-metrics service in prow build cluster
Ensuring e2e projects exist and are appropriately configured
  Ensuring e2e project exists and is appropriately configured: k8s-infra-e2e-scale-001
    Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/k8s-infra-e2e-scale-001].
    Waiting for [operations/cp.8556783790996840528] to finish...
    ..done.
    ERROR: (gcloud.beta.billing.projects.link) FAILED_PRECONDITION: Precondition check failed.
    - '@type': type.googleapis.com/google.rpc.QuotaFailure
      violations:
      - description: 'Cloud billing quota exceeded: https://support.google.com/code/contact/billing_quota_increase'
        subject: billingAccounts/018801-93540E-22A20E

Going to https://support.google.com/code/contact/billing_quota_increase leads me to a form to fill out:

• Name
  • (I'm happy to file this request if we approve of this)
• Company name or website
  • @idvoretskyi @thockin CNCF? Google?
• Email addresses that will be used to create projects
  • I will enter in the e-mail addresses of the members of [email protected]
• How many projects are being requested? I can request 5, 10, 20, 50, 100, or manually enter more (which "might require additional review")
  • 100
• What kind of services will these project use?
  • Paid Services like Compute Engine, BigQuery, Cloud SQL, Cloud Storage, MAPs API etc.
• Billing account IDs for which additional quota is needed
  • I'll put in the value from infra/gcp/lib.sh
• Any other things we need to be aware of to help us understand the request? (Include any information that will help us with verification, like why you need additional quota, the Project IDs of any other projects where you made payments, or a Google sales contact):

These GCP projects are being used to host project infrastructure for the open source Kubernetes project (https://kubernetes.io), e.g.
- build and host project artifacts (e.g. https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io)
- host project services (e.g. DNS for Kubernetes domains https://github.com/kubernetes/k8s.io/tree/master/dns)
- host/isolate e2e test resources

The goal is for these resources to live in the CNCF GCP org instead of the google.com GCP org

ref: https://www.cncf.io/announcement/2018/08/29/cncf-receives-9-million-cloud-credit-grant-from-google/
ref: https://cloud.google.com/blog/products/gcp/google-cloud-grants-9m-in-credits-for-the-operation-of-the-kubernetes-project
ref: https://github.com/kubernetes/k8s.io/

Existing project IDs:
(list of our project ids)

Google contact:
(thockin?)

WDYT, should I fill the form out as described above? Should someone else? Should the answers be different?

/assign @idvoretskyi @thockin @spiffxp

[spiffxp]

We likely require more than 100, but I feel like it'd be better to take 100 now and revisit when we get to needing more.

Rough guess at what we'd need in total:

• If we stick with project per subproject-artifact, we expect that could grow to meet the number of kubernetes subprojects; call it an upper limit of 160 if I go by the number of repos we have
• Current boskos pools for prow.k8s.io ballpark max out at:
  • 100 for GCE project usage
  • 5 for GPU project usage
  • 10 for Ingress project usage
  • 10 for Node e2e project usage
  • 10 for Scalability project usage
  • 30 for Scalability presubmit project usage
• Totals up to 325
• We currently have 100 projects listed in the latest audit PR
  • 43 are "staging" projects
  • 42 are "e2e" projects

So we may find we need to request 225 more in total.

[thockin]

Go ahead and fill it out. Many quota requests are automatically approved based on various signals. But even if this needs manual review, I don't see why it would be denied.

[spiffxp]

filed

[spiffxp]

Haven't heard anything back as of this morning

[spiffxp]

FWIW I just checked and am still having problems

$ ./ensure-staging-storage.sh e2e-test-images
Configuring staging: e2e-test-images
  Ensuring project exists: k8s-staging-e2e-test-images
  ERROR: (gcloud.beta.billing.projects.link) FAILED_PRECONDITION: Precondition check failed.
  - '@type': type.googleapis.com/google.rpc.QuotaFailure
    violations:
    - description: 'Cloud billing quota exceeded: https://support.google.com/code/contact/billing_quota_increase'
      subject: billingAccounts/REDACTED

[spiffxp]

@idvoretskyi I think at this point both @thockin and I have filled out this form; just to increase the signal, would you mind filling out as a representative for the CNCF?

[idvoretskyi]

@spiffxp sure, of course!

[idvoretskyi]

Requested, will keep us updated.

cc @spiffxp @thockin

[spiffxp]

We're attempting to escalate internally as well

[idvoretskyi]

Quota granted. However:

Based on all the factors, we have approved an additional quota of 100 projects.

Is this enough for now or we have to work on adding more?

[bartsmykla]

It's enough for now, but if it will take 19 days everytime we hit the limit to bump it I would argue we should work on adding more now

[spiffxp]

/priority important-soon

The immediate need has been met but I agree we're going to want more with some breathing room. I estimated above that we would want ballpark 325 in total. Maybe round that up to 400, meaning we want to ask for 200 more?

[idvoretskyi]

Re-requested 350 projects, will track this question and report back once I'll have some news.

[bartsmykla]

@idvoretskyi did you get any response?

[idvoretskyi]

@bartsmykla unfortunately, no. @thockin can you escalate or should I?

[thockin]

Did you get any metadata or ticket ID or anything?

[idvoretskyi]

@thockin @bartsmykla

UPD. I've re-checked my inbox and it seems that the quota request was granted. Apologies for the confusion, for some reason this email went out of my radar.

[idvoretskyi]

@spiffxp can you please check if the current quota is enough?

[spiffxp]

I don't think I have a way to see our quota, the only way to tell if it's enough would be to start provisioning a lot of projects ahead of actually having the time to allocate/use them. Projects for presubmits would probably do that, but I very much do not have the bandwidth to prepare for what moving presubmits over here would entail.

I am satisfied suggesting we are going to assume we have 350 until we find out otherwise.

[spiffxp]

I'm going to start creating a bunch more projects in anticipation of handling kubernetes/kubernetes presubmits here, so I expect if there's a limit that's less than 350 I'll hit it pretty quickly

[spiffxp]

/close
We haven't had project quota issues for a while, and while working through #1132 I believe put us on a better escalation path for future quota increases.

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
We haven't had project quota issues for a while, and while working through #1132 I believe put us on a better escalation path for future quota increases.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.