Chores: upload credential-provider-config.yaml to artifact store in ci build #4280

MartinForReal · 2023-11-16T07:34:54Z

What type of PR is this?
/kind failing-test
/kind cleanup

What this PR does / why we need it:

Uploading credentials-config yaml to azure blob storage and fix flaky e2e scenarios which uses oot azure credential provider.

In recent e2e build of cloud provider project, Cluster node may fail to initialize the control plane because credential provider config is invalid and kubelet crashes.

Nov 15 12:40:06.473151 capz-oh7kxt-control-plane-klvq7 kubelet[2278]: E1115 12:40:06.473135    2278 kuberuntime_manager.go:274] "Failed to register CRI auth plugins" err="error decoding config /var/lib/kubelet/credential-provider-config.yaml: Object 'Kind' is missing in '404: Not Found'"

The config is downloaded from git repo when the node is provisioned.

cluster-api-provider-azure/templates/test/ci/cluster-template-prow-ci-version.yaml

Line 116 in c6c977e

    
                   curl --retry 10 --retry-delay 5 -Lo /var/lib/kubelet/credential-provider-config.yaml https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/examples/out-of-tree/credential-provider-config.yaml

But the commit may reside in forked repo. And provision script will get 404 when it tries to download config file in this scenario.
The config file may be a part of changes. We should upload it to blob store to make sure config file is validated too.

Also, status code of http response is sent to stdout for debug purpose.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

cherry-pick candidate

TODOs:

squashed commits
includes documentation
adds unit tests

Release note:

NONE

lzhecheng · 2023-11-16T07:40:38Z

/lgtm

k8s-ci-robot · 2023-11-16T07:40:45Z

LGTM label has been added.

Git tree hash: 23fe5c0f4a99e5fc9cfcfd6b80725b0b18ad1e33

codecov · 2023-11-16T07:42:45Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (c6c977e) 59.55% compared to head (7731499) 59.55%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4280   +/-   ##
=======================================
  Coverage   59.55%   59.55%           
=======================================
  Files         185      185           
  Lines       18782    18782           
=======================================
  Hits        11186    11186           
  Misses       6981     6981           
  Partials      615      615

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

MartinForReal · 2023-11-16T07:43:11Z

/assign @nojnhuh

lzhecheng · 2023-11-16T07:53:56Z

/cherrypick release-1.11

k8s-infra-cherrypick-robot · 2023-11-16T07:53:58Z

@lzhecheng: once the present PR merges, I will cherry-pick it on top of release-1.11 in a new PR and assign it to you.

In response to this:

/cherrypick release-1.11

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sonasingh46 · 2023-11-16T07:54:02Z

templates/test/ci/prow-ci-version/patches/oot-credential-provider-win.yaml

@@ -8,7 +8,7 @@
      mkdir C:\var\lib\kubelet\credential-provider
      curl.exe --retry 10 --retry-delay 5 -L "https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/${AZURE_BLOB_CONTAINER_NAME}/${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/azure-acr-credential-provider.exe" --output C:\var\lib\kubelet\credential-provider\acr-credential-provider.exe
      cp C:\var\lib\kubelet\credential-provider\acr-credential-provider.exe C:\var\lib\kubelet\credential-provider\acr-credential-provider
-      curl.exe --retry 10 --retry-delay 5 -L https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/examples/out-of-tree/credential-provider-config-win.yaml --output C:\var\lib\kubelet\credential-provider-config.yaml
+      curl.exe --retry 10 --retry-delay 5 -L "https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/${AZURE_BLOB_CONTAINER_NAME}/${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/credential-provider-config-win.yaml" --output C:\var\lib\kubelet\credential-provider-config.yaml


nit: Is there any way we can also print the http code like we do for linux? No need to block the PR on this -- but was just curious.

no for now..

sonasingh46

/lgtm

sonasingh46 · 2023-11-16T08:02:53Z

/retest

sonasingh46 · 2023-11-16T08:09:18Z

@MartinForReal -- Can you please write a PR description for record, something on the lines of why this change is needed so that folks can get some context who may be seeing for the first time?

Something on lines -- uploading credentials-config yaml to azure blob storage and using that in CI from azure blob storage and not from the github repository so that the downloads are fast and not flaky and hence stable CI ??

sonasingh46 · 2023-11-16T08:11:41Z

Also Netlify deploy is failing? I did not get an option to re trigger this. Does anyone know the root issue or how to re trigger this? cc @mboersma

MartinForReal · 2023-11-16T08:34:47Z

/hold

MartinForReal · 2023-11-16T09:05:36Z

/retest

…i build

MartinForReal · 2023-11-16T10:05:47Z

/unhold

CecileRobertMichon

/lgtm
/approve
/cherry-pick release-1.11

k8s-ci-robot · 2023-11-16T16:55:13Z

LGTM label has been added.

Git tree hash: 3aee7b5fe2cee70b4bc00adf471fc15761ec5d94

k8s-ci-robot · 2023-11-16T16:55:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CecileRobertMichon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [CecileRobertMichon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-infra-cherrypick-robot · 2023-11-16T16:59:46Z

@lzhecheng: #4280 failed to apply on top of branch "release-1.11":

Applying: chores: upload credential-provider-config.yaml to artifact store in ci build
Using index info to reconstruct a base tree...
A	templates/test/ci/cluster-template-prow-ci-version-dual-stack.yaml
A	templates/test/ci/cluster-template-prow-ci-version-ipv6.yaml
M	templates/test/ci/cluster-template-prow-ci-version.yaml
M	templates/test/ci/cluster-template-prow-machine-pool-ci-version.yaml
M	templates/test/ci/prow-machine-pool-ci-version/patches/kubeadm-bootstrap-windows-k8s-ci-binaries.yaml
M	templates/test/ci/prow-machine-pool-ci-version/patches/machine-pool-ci-version.yaml
M	templates/test/dev/cluster-template-custom-builds-machine-pool.yaml
M	templates/test/dev/cluster-template-custom-builds.yaml
M	templates/test/dev/custom-builds-machine-pool/patches/custom-builds.yaml
M	templates/test/dev/custom-builds-machine-pool/patches/kubeadm-bootstrap-machine-pool-windows-k8s-pr-binaries.yaml
M	templates/test/dev/patches/control-plane-custom-builds.yaml
Falling back to patching base and 3-way merge...
Auto-merging templates/test/dev/patches/control-plane-custom-builds.yaml
Auto-merging templates/test/dev/custom-builds-machine-pool/patches/kubeadm-bootstrap-machine-pool-windows-k8s-pr-binaries.yaml
Auto-merging templates/test/dev/custom-builds-machine-pool/patches/custom-builds.yaml
Auto-merging templates/test/dev/cluster-template-custom-builds.yaml
Auto-merging templates/test/dev/cluster-template-custom-builds-machine-pool.yaml
Auto-merging templates/test/ci/prow-machine-pool-ci-version/patches/machine-pool-ci-version.yaml
Auto-merging templates/test/ci/prow-machine-pool-ci-version/patches/kubeadm-bootstrap-windows-k8s-ci-binaries.yaml
Auto-merging templates/test/ci/cluster-template-prow-machine-pool-ci-version.yaml
Auto-merging templates/test/ci/cluster-template-prow-ci-version.yaml
CONFLICT (modify/delete): templates/test/ci/cluster-template-prow-ci-version-ipv6.yaml deleted in HEAD and modified in chores: upload credential-provider-config.yaml to artifact store in ci build. Version chores: upload credential-provider-config.yaml to artifact store in ci build of templates/test/ci/cluster-template-prow-ci-version-ipv6.yaml left in tree.
CONFLICT (modify/delete): templates/test/ci/cluster-template-prow-ci-version-dual-stack.yaml deleted in HEAD and modified in chores: upload credential-provider-config.yaml to artifact store in ci build. Version chores: upload credential-provider-config.yaml to artifact store in ci build of templates/test/ci/cluster-template-prow-ci-version-dual-stack.yaml left in tree.
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 chores: upload credential-provider-config.yaml to artifact store in ci build
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherrypick release-1.11

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot requested review from CecileRobertMichon and Jont828 November 16, 2023 07:35

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Nov 16, 2023

k8s-ci-robot assigned lzhecheng Nov 16, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 16, 2023

k8s-ci-robot assigned nojnhuh Nov 16, 2023

sonasingh46 reviewed Nov 16, 2023

View reviewed changes

k8s-ci-robot assigned sonasingh46 Nov 16, 2023

MartinForReal changed the title ~~Chores: upload credential-provider-config.yaml to artifact store in ci build~~ [draft]Chores: upload credential-provider-config.yaml to artifact store in ci build Nov 16, 2023

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 16, 2023

chores: upload credential-provider-config.yaml to artifact store in c…

7731499

…i build

MartinForReal force-pushed the main branch from 7e553fb to 7731499 Compare November 16, 2023 09:22

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 16, 2023

k8s-ci-robot requested a review from lzhecheng November 16, 2023 09:23

k8s-ci-robot requested review from nojnhuh and sonasingh46 November 16, 2023 09:23

MartinForReal changed the title ~~[draft]Chores: upload credential-provider-config.yaml to artifact store in ci build~~ Chores: upload credential-provider-config.yaml to artifact store in ci build Nov 16, 2023

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 16, 2023

CecileRobertMichon reviewed Nov 16, 2023

View reviewed changes

k8s-ci-robot assigned CecileRobertMichon Nov 16, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 16, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 16, 2023

k8s-ci-robot merged commit 3973bb6 into kubernetes-sigs:main Nov 16, 2023
28 checks passed

k8s-ci-robot added this to the v1.12 milestone Nov 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Chores: upload credential-provider-config.yaml to artifact store in ci build #4280

Chores: upload credential-provider-config.yaml to artifact store in ci build #4280

MartinForReal commented Nov 16, 2023 •

edited

Loading

lzhecheng commented Nov 16, 2023

k8s-ci-robot commented Nov 16, 2023

codecov bot commented Nov 16, 2023 •

edited

Loading

MartinForReal commented Nov 16, 2023

lzhecheng commented Nov 16, 2023

k8s-infra-cherrypick-robot commented Nov 16, 2023

sonasingh46 Nov 16, 2023

MartinForReal Nov 16, 2023

sonasingh46 Nov 16, 2023

sonasingh46 left a comment

sonasingh46 commented Nov 16, 2023

sonasingh46 commented Nov 16, 2023

sonasingh46 commented Nov 16, 2023

MartinForReal commented Nov 16, 2023

MartinForReal commented Nov 16, 2023

MartinForReal commented Nov 16, 2023

CecileRobertMichon left a comment

k8s-ci-robot commented Nov 16, 2023

k8s-ci-robot commented Nov 16, 2023

k8s-infra-cherrypick-robot commented Nov 16, 2023

Chores: upload credential-provider-config.yaml to artifact store in ci build #4280

Chores: upload credential-provider-config.yaml to artifact store in ci build #4280

Conversation

MartinForReal commented Nov 16, 2023 • edited Loading

lzhecheng commented Nov 16, 2023

k8s-ci-robot commented Nov 16, 2023

codecov bot commented Nov 16, 2023 • edited Loading

Codecov Report

MartinForReal commented Nov 16, 2023

lzhecheng commented Nov 16, 2023

k8s-infra-cherrypick-robot commented Nov 16, 2023

sonasingh46 Nov 16, 2023

Choose a reason for hiding this comment

MartinForReal Nov 16, 2023

Choose a reason for hiding this comment

sonasingh46 Nov 16, 2023

Choose a reason for hiding this comment

sonasingh46 left a comment

Choose a reason for hiding this comment

sonasingh46 commented Nov 16, 2023

sonasingh46 commented Nov 16, 2023

sonasingh46 commented Nov 16, 2023

MartinForReal commented Nov 16, 2023

MartinForReal commented Nov 16, 2023

MartinForReal commented Nov 16, 2023

CecileRobertMichon left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 16, 2023

k8s-ci-robot commented Nov 16, 2023

k8s-infra-cherrypick-robot commented Nov 16, 2023

MartinForReal commented Nov 16, 2023 •

edited

Loading

codecov bot commented Nov 16, 2023 •

edited

Loading