Default v3 scaffold manager pod got permission denied error #3028

astraw99 · 2022-10-18T03:08:02Z

What broke? What's expected?

Based on the quick start doc, did the make manifests, make install, then
make docker-build docker-push IMG=xxx/xxx:tag, make deploy IMG=xxx/xxx:tag, then
in the cluster the controller-pod got error:

1.6656687258729894e+09  ERROR   controller-runtime.client.config        unable to get kubeconfig        {"error": "open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied"}
sigs.k8s.io/controller-runtime/pkg/client/config.GetConfigOrDie
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/client/config/config.go:153
main.main
        /workspace/main.go:68
runtime.main
        /usr/local/go/src/runtime/proc.go:250

Reproducing this issue

Based on the quick start doc, did the make manifests, make install, then
make docker-build docker-push IMG=xxx/xxx:tag, make deploy IMG=xxx/xxx:tag, then
in the cluster will reproduce this issue.

My cluster K8s version is:

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:25:06Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}

Kubebuilder version is:

Version: main.version{KubeBuilderVersion:"3.7.0", KubernetesVendor:"1.24.1", GitCommit:"3bfc84ec8767fa760d1771ce7a0cb05a9a8f6286", BuildDate:"2022-09-20T17:21:57Z", GoOs:"darwin", GoArch:"amd64"}

With these context, will replicate the error.

KubeBuilder (CLI) Version

Version: main.version{KubeBuilderVersion:"3.7.0", KubernetesVendor:"1.24.1", GitCommit:"3bfc84ec8767fa760d1771ce7a0cb05a9a8f6286", BuildDate:"2022-09-20T17:21:57Z", GoOs:"darwin", GoArch:"amd64"}

PROJECT version

No response

Plugin versions

No response

Other versions

No response

Extra Labels

No response

The text was updated successfully, but these errors were encountered:

camilamacedo86 · 2022-10-18T15:19:05Z

Hi @astraw99,

This error is NOT reproducible with the default scaffold and with all mocks generated in the e2e tests. Also, the default scaffold does NOT use a service account. So, the following some questions:

a) Are you able to reproduce this issue by following the quick start and creating a new project generated with the latest Kubebuilder version?
b) Did you create your project with Kubebuilder CLI 3.7.0 version or not? Could you please add the PROJECT file content so that we can check what plugin version are you using?

astraw99 · 2022-10-19T03:31:39Z

@camilamacedo86
Seems it is a common issue, also faced in cert-manager/cert-manager#5513

Tested if removed this base image part:

kubebuilder/pkg/plugins/golang/v3/scaffolds/internal/templates/dockerfile.go

Lines 68 to 71 in 61611b2

    
           FROM gcr.io/distroless/static:nonroot 
        
           WORKDIR / 
        
           COPY --from=builder /workspace/manager . 
        
           USER 65532:65532

Just use golang:1.19 base image, then manager will not encounter this issue.
PTAL thanks.

camilamacedo86 · 2022-10-21T14:19:08Z

HI @astraw99,

The description provided here does enable certmanager and it seems not accurate.
The error described by you can NOT be reproduced with the steps provided.

See that we indeed have the cert-manager enabled in the e2e tests and the issue is NOT faced.

Can you please provide all steps required for we create a new project from the scratch and face this issue?

Regards your comment in the other issue:

Only faced the sa permission error, opened a new issue #3028 to clarify it.

The default scaffold does have not SA. So, how can that can be faced by default?

astraw99 · 2022-10-22T16:35:10Z

@camilamacedo86
Yes, scaffold does not have SA, but the pod startup need to open SA token file:

open /var/run/secrets/kubernetes.io/serviceaccount/token

to build InClusterConfig, then encountered this SA permission denied error.

Tested the manager pod works well in K8s v1.25, but it got this issue on K8s v1.21.
Seems it is a permission bug which have been fixed in higher K8s verison. Or maybe there is some other reason?

camilamacedo86 · 2022-10-23T02:54:36Z

Hi @astraw99,

to build InClusterConfig, then encountered this SA permission denied error.

Could you please provide an example?
From the scratch, we need to create a project and do what customization on top to face this error.
Are you using cert-manager? if so, did you change the default version scaffolded by the CLI? What version are you using?

Tested the manager pod works well in K8s v1.25, but it got this issue on K8s v1.21.

The e2e tests run against now only from 1.23. However, a few months ago we tested the default scaffolds from 1.16 and we never got this error.

On top of that, if that is an issue that is not faced anymore why would we change the default scaffold to fix that? What about we add this one to a FAQ section as well? See: https://github.com/kubernetes-sigs/kubebuilder/issues?q=is%3Aissue+is%3Aopen+FAq

astraw99 · 2022-10-23T13:33:28Z

@camilamacedo86 Found the SA token permission related issues and PRs:
kubernetes/kubernetes#74565
kubernetes/kubernetes#82573
kubernetes/enhancements#1598
kubernetes/kubernetes#89193

From the latest code:
https://github.com/kubernetes/kubernetes/blob/641197dc9562d8affd428d05046a8921b3a367e0/pkg/volume/projected/projected.go#L319-L327
We can see the DefaultMode = 0644.

The default securityContext set by manager.yaml is:

securityContext:
  runAsNonRoot: true

Checked in my cluster, the permission of SA token file with the above set is 0600:

# ls -la /var/run/secrets/kubernetes.io/serviceaccount/..data/token 
-rw-------    1 root     root           994 Oct 23 12:57 /var/run/secrets/kubernetes.io/serviceaccount/..data/token

So if not set fsGroup, the controller pod would get this issue error.

camilamacedo86 · 2022-10-26T16:01:44Z

Hi @astraw99,

Still not clear for me when the error is faced. If we create a project from the scratch and run it against k8s version x will it happen and if we test against version > X then it is solved? Could you please clarify?

astraw99 · 2022-10-27T02:44:02Z

Yes, we can reproduce by create a project from scatch, then do the make manifests, make install, make docker-build docker-push IMG=xxx/xxx:tag, make deploy IMG=xxx/xxx:tag, will reproduce the error.
The K8s version is:

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:25:06Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}

The KB version is:

Version: main.version{KubeBuilderVersion:"3.7.0", KubernetesVendor:"1.24.1", GitCommit:"3bfc84ec8767fa760d1771ce7a0cb05a9a8f6286", BuildDate:"2022-09-20T17:21:57Z", GoOs:"darwin", GoArch:"amd64"}

camilamacedo86 · 2022-10-27T08:58:43Z

Hi @astraw99,

So the error ONLY occurs on versions < 1.22? When the issue was solved in k8s?

astraw99 · 2022-10-27T09:33:01Z

Hi @astraw99,

So the error ONLY occurs on versions < 1.22? When the issue was solved in k8s?

Yes, I think so. My version is 1.21.
Tested it works weill in 1.24 and 1.25, but I don't have 1.22, 1.23, or < 1.21 versions to test.

camilamacedo86 · 2022-10-27T11:34:40Z

Hi @astraw99,

Here into the PR for the doc #3044 you added that it also sorted out with the PR set proper file permission for projected service account volume #89193. If so, that was merged on 5 May 2020 and we could check what releases have it or not.

However, note that these changes are in tag 1.21.0 released on 8 Apr 2021.
https://github.com/kubernetes/kubernetes/blob/v1.21.0/pkg/securitycontext/util.go#L127-L144

So, it might. not be the right fix.

Therefore, for the doc explanation into the PR #3044 I'd suggest we add only face with old k8s versions instead of the version. Also, I would recommend not adding the PR fix since it might be not the right one.

astraw99 · 2022-10-27T13:04:13Z

In the PR #3017, there is a "may" doc info, so it might be proper to add this PR.
PTAL thanks.

camilamacedo86 · 2022-10-31T21:27:47Z

Hi @astraw99,

The issue does not occur as you checked in the latest version of k8s.
So, I do not think that we should change the scaffolds with a workaround to solve issues for old k8s versions.
Also, we could address this scenario with the FAQ doc: #3044

So, wdyt? Could we close this one? Is that make sense?

astraw99 · 2022-11-01T02:26:12Z

OK, we can close it now.

astraw99 added the kind/bug Categorizes issue or PR as related to a bug. label Oct 18, 2022

astraw99 mentioned this issue Oct 18, 2022

🐛: Fix controller-pod SA permission denied error #3017

Closed

camilamacedo86 added the triage/needs-information Indicates an issue needs more information in order to work on it. label Oct 18, 2022

astraw99 mentioned this issue Oct 19, 2022

Got errors on kubebuilder v3 quick start #3016

Closed

astraw99 mentioned this issue Oct 20, 2022

Deploy of cert-manager-webhook/cainjector:v1.9.1 got permission error cert-manager/cert-manager#5513

Closed

astraw99 mentioned this issue Oct 25, 2022

📖 Add kubebuilder FAQ section #3044

Merged

astraw99 closed this as completed Nov 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Default v3 scaffold manager pod got permission denied error #3028

Default v3 scaffold manager pod got permission denied error #3028

astraw99 commented Oct 18, 2022

camilamacedo86 commented Oct 18, 2022

astraw99 commented Oct 19, 2022 •

edited

Loading

camilamacedo86 commented Oct 21, 2022 •

edited

Loading

astraw99 commented Oct 22, 2022

camilamacedo86 commented Oct 23, 2022 •

edited

Loading

astraw99 commented Oct 23, 2022

camilamacedo86 commented Oct 26, 2022

astraw99 commented Oct 27, 2022

camilamacedo86 commented Oct 27, 2022

astraw99 commented Oct 27, 2022

camilamacedo86 commented Oct 27, 2022

astraw99 commented Oct 27, 2022

camilamacedo86 commented Oct 31, 2022

astraw99 commented Nov 1, 2022

Default v3 scaffold manager pod got permission denied error #3028

Default v3 scaffold manager pod got permission denied error #3028

Comments

astraw99 commented Oct 18, 2022

What broke? What's expected?

Reproducing this issue

KubeBuilder (CLI) Version

PROJECT version

Plugin versions

Other versions

Extra Labels

camilamacedo86 commented Oct 18, 2022

astraw99 commented Oct 19, 2022 • edited Loading

camilamacedo86 commented Oct 21, 2022 • edited Loading

astraw99 commented Oct 22, 2022

camilamacedo86 commented Oct 23, 2022 • edited Loading

astraw99 commented Oct 23, 2022

camilamacedo86 commented Oct 26, 2022

astraw99 commented Oct 27, 2022

camilamacedo86 commented Oct 27, 2022

astraw99 commented Oct 27, 2022

camilamacedo86 commented Oct 27, 2022

astraw99 commented Oct 27, 2022

camilamacedo86 commented Oct 31, 2022

astraw99 commented Nov 1, 2022

astraw99 commented Oct 19, 2022 •

edited

Loading

camilamacedo86 commented Oct 21, 2022 •

edited

Loading

camilamacedo86 commented Oct 23, 2022 •

edited

Loading