Switch to C3 machine types for prow build cluster

[upodroid]

/cc @ameukam @BenTheElder @xmudrii

We should get better performance out of these instances.

C3 instance types are a bit tricky. Our options are:

• c3d-highmem-8-lssd which provides 8 cores, 64GB of ram and a single local SSDs
• c3-standard-8-lssd which provides 8 cores, 32GB of ram and a two local SSDs

I'm in favour offering 7 cores and 28GB to all our prowjobs across the board. This setting guarantees a dedicated node for testing. The remaining core belongs to us to run our stuff on it

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• infra/gcp/terraform/OWNERS [upodroid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[BenTheElder]

I also used a custom type to insert 2 extra cores for all the extra daemonset we need to run.

We really shouldn't need 2 cores of daemonsets ...?
/hold

We should be able to tighten this up, discarding 20% of the node as overhead is excessive.

[BenTheElder]

Maybe we can import from the existing metrics collection?

[upodroid]

According to the calculations here:
We could adopt c3-standard-8-lssd right now and disable calico. I'm not sure why we need a network policy enabled on the build cluster. The capacity being used by calico would be available for node-exporter

[BenTheElder]

[...] and disable calico. I'm not sure why we need a network policy enabled on the build cluster. The capacity being used by calico would be available for node-exporter

That sounds good to me (have not reviewed machine types yet). I don't recall an discussion about enabling network policy and I don't think we need it, 400m CPU is quite a bit back to fit other daemonsets cc @ameukam

[BenTheElder]

Just checked: k8s-prow-builds in google.com (cluster: default) has calico network policy disabled.

[upodroid]

One more question, are we ok with nodes having 32gb memory instead of 64gb?

[BenTheElder]

EDIT: jinx
c3-standard-8-lssd has lower memory / core (32gb for 8 cores) versus n1-highmem-8 (52 gb for 8 cores).

I'm not sure if that's a problem for any of our workloads or not. I think we might have a few that are a bit lopsided in memory>cpu requests currently

[upodroid]

Apply plan:

1. apply the 1st commit using terraform
2. Cordon and Drain the old nodepool in the console
3. Apply the 2nd commit

After lgtm is applied on the PR.

[BenTheElder]

I think we should probably query the jobs to see if any have 30+ GB or a larger memory : core ratio than c3, if so, we can do n2?

[ameukam]

[...] and disable calico. I'm not sure why we need a network policy enabled on the build cluster. The capacity being used by calico would be available for node-exporter

That sounds good to me (have not reviewed machine types yet). I don't recall an discussion about enabling network policy and I don't think we need it, 400m CPU is quite a bit back to fit other daemonsets cc @ameukam

Network Policy was enabled by default at the time the cluster was created. We never took the opportunity to disable it. +1 for the disablement.

[upodroid]

Instead of tweaking the prowjobs, lets go with N2 instances(Ice Lake platform) and keep the same machine-type and merge that change this weekend.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle rotten
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten