infra/gcp/aaa: add k8s-infra-prow #2235

ameukam · 2021-06-16T21:58:04Z

Deploy k8s-infra prow instance for configuration focused on k8s.io repo.

Signed-off-by: Arnaud Meukam [email protected]

ameukam · 2021-07-22T05:52:53Z

/test pull-k8sio-verify

spiffxp

I feel like maybe you grabbed a copy from or were at least inspired by my attempts at setting up github.com/bashfire/prow-config. The problem is that's fallen way stale at this point. I would recommend a refresh from https://github.com/kubernetes/test-infra/tree/master/config/prow just to be sure

I dropped some notes to self that I'll try and answer as followup

apps/prow/cluster/deck_deployment.yaml

apps/prow/cluster/prow-externalsecrets.yaml

apps/prow/cluster/prow_ingress.yaml

apps/prow/cluster/sinker_deployment.yaml

apps/prow/cluster/statusreconciler_deployment.yaml

apps/prow/config.yaml

apps/prow/plugins.yaml

spiffxp · 2021-07-27T01:31:21Z

apps/prow/prowjobs/kubernetes/k8s.io/periodics.yaml

@@ -0,0 +1,23 @@
+periodics:


nit: I am more used to seeing apps/prow/jobs/{org}/{repo}

we're gonna want to move this to be top-level or easy to find anyway, so maybe the particular name right now doesn't matter as much

We'll update location when I'll add prow self-maintenance jobs.

apps/prow/plugins.yaml

spiffxp

one nit, just the one, otherwise it LGTM

spiffxp · 2021-08-05T22:27:56Z

apps/prow/config.yaml

+        default_org: "kubernetes"
+        default_repo: "k8s.io"


I edited the suggestion to remove these two lines, I'd like to see this change accepted before lgtm

Add kubernetes manifests for k8s-infra-prow Move ghproxy to cluster folder Add configuration for k8s-infra-prow Add plugins configuration for k8s-infra-prow Ensure deploy.sh targets cluster folder Add ingress and GKE managed certificate Add a Makefile for prow config and plugins deployments. Add a Makefile with commands allowing to update prow config and plugins once changes are merged. Create and Add Github HMAC Token to GCP Secrets Manager of project kubernetes-public. Create Github OAuth config and add it to GCP Secrets Manager. Fix yamllint

ameukam · 2021-08-06T15:04:44Z

one nit, just the one, otherwise it LGTM

@spiffxp removed the lines.

spiffxp

/approve
/lgtm

k8s-ci-robot · 2021-08-06T17:38:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ameukam, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~apps/prow/OWNERS~~ [ameukam,spiffxp]
~~groups/OWNERS~~ [spiffxp]
~~infra/gcp/OWNERS~~ [ameukam,spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

spiffxp · 2021-08-06T18:07:43Z

This failed to auto-deploy because RBAC:

Failed job: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-k8sio-deploy-app-prow/1423700739766620160
Open issue is: Prow: allow deployment of kubernetes-external-secrets without roles/container.admin #2218
PR to workaround is: infra/gcp: prow-deployer as container.admin #2483

spiffxp · 2021-08-06T18:09:30Z

Retriggered job via re-run button https://prow.k8s.io/?job=post-k8sio-deploy-app-prow

spiffxp · 2021-08-06T18:10:44Z

OK I think the remaining errors are token / typo related https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-k8sio-deploy-app-prow/1423707644174012416

will leave to @ameukam to fix for now

Fix typos found during deployment of : kubernetes#2235

Fix typos and mistakes found during deployment of : kubernetes#2235

ameukam · 2021-08-06T23:26:54Z

Some issues we have post-deployment :

sinker & prow-controller-manager:

E0807 22:07:54.495104       1 reflector.go:138] external/io_k8s_client_go/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:prow:sinker" cannot list resource "pods" in API group "" in the namespace "k8s-infra-test-pods"

deck need to allowed to push objects on gs://k8s-testgrid:

kubectl -n prow logs -l app=deck
...
{"component":"deck","error":"Get \"https://storage.googleapis.com/k8s-testgrid/config\": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: The caller does not have permission\nThis error could be caused by a missing IAM policy binding on the target IAM service account.\nFor more information, refer to the Workload Identity documentation:\n\thttps://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to\n\n`","file":"prow/spyglass/testgrid.go:50","func":"k8s.io/test-infra/prow/spyglass.(*TestGrid).Start.func1","level":"error","msg":"Couldn't update TestGrid config.","path":"gs://k8s-testgrid/config","severity":"error","time":"2021-08-06T21:50:08Z"}

@spiffxp I believe this is under a Google GCP Org. IAM binding may need to be done by Googlers.

tide need to push objects to gs://k8s-infra-prow-results.

{"component":"tide","error":"error initializing history client from \"gs://k8s-infra-prow-results/tide-history.json\": open: Get \"https://storage.googleapis.com/k8s-infra-prow-results/tide-history.json\": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: The caller does not have permission\nThis error could be caused by a missing IAM policy binding on the target IAM service account.\nFor more information, refer to the Workload Identity documentation:\n\thttps://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to\n\n`","file":"prow/cmd/tide/main.go:176","func":"main.main","level":"fatal","msg":"Error creating Tide controller.","severity":"fatal","time":"2021-08-06T22:55:38Z"}

Fix typos and mistakes found during deployment of : kubernetes#2235

ameukam · 2021-08-08T12:00:30Z

Ref: #1394

Followup of kubernetes#2235. Ref: kubernetes#1394. sinker & prow-controller-manager require a namespace is present and they can list and update the pods presents in this namespace: ```shell E0807 22:07:54.495104 1 reflector.go:138] external/io_k8s_client_go/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:prow:sinker" cannot list resource "pods" in API group "" in the namespace "k8s-infra-test-pods" ``` Signed-off-by: Arnaud Meukam <[email protected]>

k8s-ci-robot requested review from dims and spiffxp June 16, 2021 21:58

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jun 16, 2021

ameukam force-pushed the aaa-k8s-infra-prow branch 3 times, most recently from 30b4b55 to 1376465 Compare June 20, 2021 13:00

ameukam mentioned this pull request Jul 16, 2021

prow: Add k8s-infra-build-clusters-kubeconfig ExternalSecret #2357

Merged

ameukam force-pushed the aaa-k8s-infra-prow branch from 1376465 to eb9c129 Compare July 21, 2021 22:54

k8s-ci-robot added the sig/testing Categorizes an issue or PR as relevant to SIG Testing. label Jul 21, 2021

ameukam changed the title ~~[WIP] infra/gcp/aaa: add k8s-infra-prow~~ infra/gcp/aaa: add k8s-infra-prow Jul 21, 2021

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 21, 2021

ameukam force-pushed the aaa-k8s-infra-prow branch from eb9c129 to 63aa983 Compare July 21, 2021 23:07

ameukam force-pushed the aaa-k8s-infra-prow branch 2 times, most recently from 6dd07f9 to c0cefd5 Compare July 22, 2021 20:08

k8s-ci-robot added area/access Define who has access to what via IAM bindings, role bindings, policy, etc. sig/security Categorizes an issue or PR as relevant to SIG Security. and removed approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jul 22, 2021

ameukam mentioned this pull request Jul 26, 2021

audit: update as of 2021-07-28 #2385

Merged

ameukam force-pushed the aaa-k8s-infra-prow branch from b7e7710 to b691f90 Compare July 26, 2021 20:52

spiffxp force-pushed the aaa-k8s-infra-prow branch from e3f8856 to f2d5f65 Compare July 27, 2021 00:43

spiffxp reviewed Jul 27, 2021

View reviewed changes

ameukam mentioned this pull request Jul 28, 2021

DNS entries for k8s-infra-prow.k8s.io #2404

Merged

ameukam force-pushed the aaa-k8s-infra-prow branch from f2d5f65 to e943787 Compare August 4, 2021 22:48

k8s-ci-robot added area/apps Application management, code in apps/ area/apps/prow k8s-infra-prow.k8s.io, code in apps/prow/ labels Aug 4, 2021

k8s-ci-robot added the area/infra Infrastructure management, infrastructure design, code in infra/ label Aug 4, 2021

spiffxp mentioned this pull request Aug 5, 2021

audit: update as of 2021-08-05 #2456

Merged

spiffxp suggested changes Aug 5, 2021

View reviewed changes

k8s-ci-robot assigned spiffxp Aug 5, 2021

ameukam force-pushed the aaa-k8s-infra-prow branch from e943787 to d68d628 Compare August 6, 2021 07:02

spiffxp approved these changes Aug 6, 2021

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 6, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 6, 2021

spiffxp mentioned this pull request Aug 6, 2021

infra/gcp: mv clusters to terraform #2482

Merged

k8s-ci-robot merged commit 931fb05 into kubernetes:main Aug 6, 2021

k8s-ci-robot added this to the v1.23 milestone Aug 6, 2021

spiffxp mentioned this pull request Aug 6, 2021

infra/gcp: prow-deployer as container.admin #2483

Merged

ameukam added a commit to ameukam/k8s.io that referenced this pull request Aug 6, 2021

apps/prow: Fix errors and typos of k8s-infra-prow

9db74b3

Fix typos found during deployment of : kubernetes#2235

ameukam added a commit to ameukam/k8s.io that referenced this pull request Aug 6, 2021

apps/prow: Fix errors and typos of k8s-infra-prow

de41806

Fix typos and mistakes found during deployment of : kubernetes#2235

ameukam mentioned this pull request Aug 6, 2021

apps/prow: Fix errors and typos of k8s-infra-prow #2487

Merged

ameukam added a commit to ameukam/k8s.io that referenced this pull request Aug 6, 2021

apps/prow: Fix errors and typos of k8s-infra-prow

609346d

Fix typos and mistakes found during deployment of : kubernetes#2235

ameukam added a commit to ameukam/k8s.io that referenced this pull request Aug 7, 2021

apps/prow: Fix errors and typos of k8s-infra-prow

4f13103

Fix typos and mistakes found during deployment of : kubernetes#2235

ameukam mentioned this pull request Aug 9, 2021

k8s-infra-prow: allow deck to access to gs://k8s-testgrid #2497

Closed

ameukam mentioned this pull request Aug 9, 2021

apps/prow: Add additional kubernetes namespace #2500

Merged

ameukam mentioned this pull request Aug 11, 2021

Setup a k8s-infra prow instance #1394

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

infra/gcp/aaa: add k8s-infra-prow #2235

infra/gcp/aaa: add k8s-infra-prow #2235

ameukam commented Jun 16, 2021 •

edited

Loading

ameukam commented Jul 22, 2021

spiffxp left a comment

spiffxp Jul 27, 2021

ameukam Aug 4, 2021

spiffxp left a comment

spiffxp Aug 5, 2021

ameukam commented Aug 6, 2021

spiffxp left a comment

k8s-ci-robot commented Aug 6, 2021

spiffxp commented Aug 6, 2021 •

edited

Loading

spiffxp commented Aug 6, 2021

spiffxp commented Aug 6, 2021

ameukam commented Aug 6, 2021 •

edited

Loading

ameukam commented Aug 8, 2021

infra/gcp/aaa: add k8s-infra-prow #2235

infra/gcp/aaa: add k8s-infra-prow #2235

Conversation

ameukam commented Jun 16, 2021 • edited Loading

ameukam commented Jul 22, 2021

spiffxp left a comment

Choose a reason for hiding this comment

spiffxp Jul 27, 2021

Choose a reason for hiding this comment

ameukam Aug 4, 2021

Choose a reason for hiding this comment

spiffxp left a comment

Choose a reason for hiding this comment

spiffxp Aug 5, 2021

Choose a reason for hiding this comment

ameukam commented Aug 6, 2021

spiffxp left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Aug 6, 2021

spiffxp commented Aug 6, 2021 • edited Loading

spiffxp commented Aug 6, 2021

spiffxp commented Aug 6, 2021

ameukam commented Aug 6, 2021 • edited Loading

ameukam commented Aug 8, 2021

ameukam commented Jun 16, 2021 •

edited

Loading

spiffxp commented Aug 6, 2021 •

edited

Loading

ameukam commented Aug 6, 2021 •

edited

Loading