Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unauthorized when listing k8s resources in tenant #1100

Closed
StanMichel opened this issue Jun 6, 2024 · 3 comments
Closed

Unauthorized when listing k8s resources in tenant #1100

StanMichel opened this issue Jun 6, 2024 · 3 comments
Assignees
@oliverbaehler
Labels
question Further information is requested

Comments

@StanMichel
Copy link

Bug description

I've created a tenant in a clean Kubernetes cluster, by following the docs. Afterwards, I ran the hack/create-user.sh to create a dummy kubeconfig file. With this file exported, I could always list the pods in the newly created tenant. As of today, I get the following error:

Note: This always worked for me until this morning

KUBECONFIG=alice-oil.kubeconfig kubectl get pods
Error from server (Forbidden): pods is forbidden: User "alice" cannot list resource "pods" in API group "" in the namespace "default"

I tested this on a clean Kind & Debian environment.

How to reproduce

Steps to reproduce the behavior:

In a clean Kind environment, I executed the following commands:

  1. kubectl apply -f https://raw.githubusercontent.com/clastix/capsule/master/config/install.yaml
kubectl create -f - << EOF
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: oil
spec:
  owners:
  - name: alice
    kind: User
EOF

Note: I also tried apiVersion: capsule.clastix.io/v1beta1

  1. kubectl get tenants (to verify if the tenant oil is active)
  2. ./create-user.sh alice oil
  3. KUBECONFIG=alice-oil.kubeconfig kubectl get pods (This results in a 403)

We've tested this on several macos machines.
I'm running Sonoma 14.5

Expected behavior

Not being forbidden when I list the pods - or another k8s resource - when logged in as the tenant owner.

Logs

kubectl -n capsule-system logs deploy/capsule-controller-manager
2024/06/06 10:43:49 maxprocs: Updating GOMAXPROCS=1: using minimum allowed GOMAXPROCS
2024-06-06T10:43:49.186Z        INFO    setup   Capsule Version v0.6.2 9c1b948
2024-06-06T10:43:49.186Z        INFO    setup   Build from: https://github.com/projectcapsule/capsule
2024-06-06T10:43:49.186Z        INFO    setup   Build date: 2024-03-28T20:14:03
2024-06-06T10:43:49.186Z        INFO    setup   Go Version: go1.21.8
2024-06-06T10:43:49.186Z        INFO    setup   Go OS/Arch: linux/arm64
2024-06-06T10:43:49.225Z        INFO    controllers.TLS Skipping TLS certificate generation as it is still valid
2024-06-06T10:43:49.225Z        INFO    controllers.TLS Updating caBundle in webhooks and crd
2024-06-06T10:43:49.229Z        INFO    controllers.TLS Updating capsule operator pods
2024-06-06T10:43:49.294Z        INFO    controller-runtime.builder      skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called       {"GVK": "capsule.clastix.io/v1beta1, Kind=Tenant"}
2024-06-06T10:43:49.294Z        INFO    controller-runtime.builder      skip registering a validating webhook, object does not implement admission.Validator or WithValidator wasn't called     {"GVK": "capsule.clastix.io/v1beta1, Kind=Tenant"}
2024-06-06T10:43:49.294Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/convert"}
2024-06-06T10:43:49.294Z        INFO    controller-runtime.builder      Conversion webhook enabled      {"GVK": "capsule.clastix.io/v1beta1, Kind=Tenant"}
2024-06-06T10:43:49.317Z        INFO    setup   skipping setup of Indexer ingress.HostnamePath for object *v1beta1.Ingress      {"error": "failed to get API group resources: unable to retrieve the complete list of server APIs: extensions/v1beta1: the server could not find the requested resource"}
2024-06-06T10:43:49.318Z        INFO    setup   skipping setup of Indexer ingress.HostnamePath for object *v1beta1.Ingress      {"error": "failed to get API group resources: unable to retrieve the complete list of server APIs: networking.k8s.io/v1beta1: the server could not find the requested resource"}
2024-06-06T10:43:49.318Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/pods"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/namespaces"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/ingresses"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/persistentvolumeclaims"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/services"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/tenantresource-objects"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/networkpolicies"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/tenants"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/namespace-owner-reference"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/cordoning"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/nodes"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/defaults"}
2024-06-06T10:43:49.319Z        INFO    setup   starting manager
2024-06-06T10:43:49.319Z        INFO    controller-runtime.metrics      Starting metrics server
2024-06-06T10:43:49.319Z        INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": ":8080", "secure": false}
2024-06-06T10:43:49.319Z        INFO    starting server {"kind": "health probe", "addr": "[::]:10080"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Starting webhook server
2024-06-06T10:43:49.320Z        INFO    controller-runtime.certwatcher  Updated current TLS certificate
2024-06-06T10:43:49.320Z        INFO    controller-runtime.webhook      Serving webhook server  {"host": "", "port": 9443}
2024-06-06T10:43:49.320Z        INFO    controller-runtime.certwatcher  Starting certificate watcher
I0606 10:43:49.425509       1 leaderelection.go:250] attempting to acquire leader lease capsule-system/42c733ea.clastix.capsule.io...
I0606 10:44:07.379255       1 leaderelection.go:260] successfully acquired lease capsule-system/42c733ea.clastix.capsule.io
2024-06-06T10:44:07.381Z        INFO    Starting EventSource    {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "source": "kind source: *v1.Secret"}
2024-06-06T10:44:07.381Z        INFO    Starting EventSource    {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "source": "kind source: *v1.ValidatingWebhookConfiguration"}
2024-06-06T10:44:07.381Z        INFO    Starting EventSource    {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "source": "kind source: *v1.MutatingWebhookConfiguration"}
2024-06-06T10:44:07.381Z        INFO    Starting EventSource    {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "source": "kind source: *v1.CustomResourceDefinition"}
2024-06-06T10:44:07.381Z        INFO    Starting Controller     {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret"}
2024-06-06T10:44:07.382Z        DEBUG   events  capsule-controller-manager-58884475b6-fj56t_b45e0ee1-e567-4ea1-b217-baec1dd034a0 became leader  {"type": "Normal", "object": {"kind":"Lease","namespace":"capsule-system","name":"42c733ea.clastix.capsule.io","uid":"a7b5a715-f021-4f3f-8f05-9e5738900151","apiVersion":"coordination.k8s.io/v1","resourceVersion":"703"}, "reason": "LeaderElection"}
2024-06-06T10:44:07.382Z        INFO    controllers.Rbac        setting up ClusterRoles {"ClusterRole": "capsule-namespace-provisioner"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant", "source": "kind source: *v1beta2.Tenant"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant", "source": "kind source: *v1.Namespace"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant", "source": "kind source: *v1.NetworkPolicy"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant", "source": "kind source: *v1.LimitRange"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant", "source": "kind source: *v1.ResourceQuota"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant", "source": "kind source: *v1.RoleBinding"}
2024-06-06T10:44:07.384Z        INFO    Starting Controller     {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1.ClusterRoleBinding"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1beta2.CapsuleConfiguration"}
2024-06-06T10:44:07.384Z        INFO    Starting Controller     {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "clusterrole", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRole", "source": "kind source: *v1.ClusterRole"}
2024-06-06T10:44:07.384Z        INFO    Starting Controller     {"controller": "clusterrole", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRole"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "endpoints", "controllerGroup": "", "controllerKind": "Endpoints", "source": "kind source: *v1.Endpoints"}
2024-06-06T10:44:07.385Z        INFO    Starting Controller     {"controller": "endpoints", "controllerGroup": "", "controllerKind": "Endpoints"}
2024-06-06T10:44:07.385Z        INFO    Starting EventSource    {"controller": "service", "controllerGroup": "", "controllerKind": "Service", "source": "kind source: *v1.Service"}
2024-06-06T10:44:07.385Z        INFO    Starting Controller     {"controller": "service", "controllerGroup": "", "controllerKind": "Service"}
2024-06-06T10:44:07.385Z        INFO    Starting EventSource    {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2024-06-06T10:44:07.385Z        INFO    Starting Controller     {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod"}
2024-06-06T10:44:07.385Z        INFO    Starting EventSource    {"controller": "endpointslice", "controllerGroup": "discovery.k8s.io", "controllerKind": "EndpointSlice", "source": "kind source: *v1.EndpointSlice"}
2024-06-06T10:44:07.385Z        INFO    Starting Controller     {"controller": "endpointslice", "controllerGroup": "discovery.k8s.io", "controllerKind": "EndpointSlice"}
2024-06-06T10:44:07.385Z        INFO    Starting EventSource    {"controller": "capsuleconfiguration", "controllerGroup": "capsule.clastix.io", "controllerKind": "CapsuleConfiguration", "source": "kind source: *v1beta2.CapsuleConfiguration"}
2024-06-06T10:44:07.385Z        INFO    Starting Controller     {"controller": "capsuleconfiguration", "controllerGroup": "capsule.clastix.io", "controllerKind": "CapsuleConfiguration"}
2024-06-06T10:44:07.385Z        INFO    Starting EventSource    {"controller": "persistentvolume", "controllerGroup": "", "controllerKind": "PersistentVolume", "source": "kind source: *v1.PersistentVolume"}
2024-06-06T10:44:07.385Z        INFO    Starting Controller     {"controller": "persistentvolume", "controllerGroup": "", "controllerKind": "PersistentVolume"}
2024-06-06T10:44:07.386Z        INFO    Starting EventSource    {"controller": "tenantresource", "controllerGroup": "capsule.clastix.io", "controllerKind": "TenantResource", "source": "kind source: *v1beta2.TenantResource"}
2024-06-06T10:44:07.386Z        INFO    Starting Controller     {"controller": "tenantresource", "controllerGroup": "capsule.clastix.io", "controllerKind": "TenantResource"}
2024-06-06T10:44:07.386Z        INFO    Starting EventSource    {"controller": "globaltenantresource", "controllerGroup": "capsule.clastix.io", "controllerKind": "GlobalTenantResource", "source": "kind source: *v1beta2.GlobalTenantResource"}
2024-06-06T10:44:07.386Z        INFO    Starting EventSource    {"controller": "globaltenantresource", "controllerGroup": "capsule.clastix.io", "controllerKind": "GlobalTenantResource", "source": "kind source: *v1beta2.Tenant"}
2024-06-06T10:44:07.386Z        INFO    Starting Controller     {"controller": "globaltenantresource", "controllerGroup": "capsule.clastix.io", "controllerKind": "GlobalTenantResource"}
2024-06-06T10:44:07.619Z        INFO    Starting workers        {"controller": "tenantresource", "controllerGroup": "capsule.clastix.io", "controllerKind": "TenantResource", "worker count": 1}
2024-06-06T10:44:07.619Z        INFO    Starting workers        {"controller": "globaltenantresource", "controllerGroup": "capsule.clastix.io", "controllerKind": "GlobalTenantResource", "worker count": 1}
2024-06-06T10:44:07.620Z        INFO    controllers.Rbac        setting up ClusterRoles {"ClusterRole": "capsule-namespace-deleter"}
2024-06-06T10:44:07.622Z        INFO    controllers.Rbac        setting up ClusterRoleBindings
2024-06-06T10:44:07.623Z        INFO    Starting workers        {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "worker count": 1}
2024-06-06T10:44:07.678Z        INFO    Starting workers        {"controller": "capsuleconfiguration", "controllerGroup": "capsule.clastix.io", "controllerKind": "CapsuleConfiguration", "worker count": 1}
2024-06-06T10:44:07.678Z        INFO    controllers.CapsuleConfiguration        CapsuleConfiguration reconciliation started     {"request.name": "capsule-default"}
2024-06-06T10:44:07.678Z        INFO    controllers.CapsuleConfiguration        CapsuleConfiguration reconciliation finished    {"request.name": "capsule-default"}
2024-06-06T10:44:07.678Z        INFO    Starting workers        {"controller": "endpointslice", "controllerGroup": "discovery.k8s.io", "controllerKind": "EndpointSlice", "worker count": 1}
2024-06-06T10:44:07.680Z        INFO    Starting workers        {"controller": "endpoints", "controllerGroup": "", "controllerKind": "Endpoints", "worker count": 1}
2024-06-06T10:44:07.680Z        INFO    Starting workers        {"controller": "service", "controllerGroup": "", "controllerKind": "Service", "worker count": 1}
2024-06-06T10:44:07.680Z        INFO    Starting workers        {"controller": "persistentvolume", "controllerGroup": "", "controllerKind": "PersistentVolume", "worker count": 1}
2024-06-06T10:44:07.692Z        INFO    Starting workers        {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant", "worker count": 1}
2024-06-06T10:44:07.707Z        INFO    controllers.Tenant      Ensuring limit resources count is updated       {"Request.Name": "oil"}
2024-06-06T10:44:07.707Z        INFO    controllers.Tenant      Ensuring all Namespaces are collected   {"Request.Name": "oil"}
2024-06-06T10:44:07.711Z        INFO    controllers.Tenant      Starting processing of Namespaces       {"Request.Name": "oil", "items": 0}
2024-06-06T10:44:07.711Z        INFO    controllers.Tenant      Starting processing of Network Policies {"Request.Name": "oil"}
2024-06-06T10:44:07.711Z        INFO    controllers.Tenant      Starting processing of Limit Ranges     {"Request.Name": "oil", "items": 0}
2024-06-06T10:44:07.711Z        INFO    controllers.Tenant      Starting processing of Resource Quotas  {"Request.Name": "oil", "items": 0}
2024-06-06T10:44:07.711Z        INFO    controllers.Tenant      Ensuring RoleBindings for Owners and Tenant     {"Request.Name": "oil"}
2024-06-06T10:44:07.711Z        INFO    controllers.Tenant      Ensuring Namespace count        {"Request.Name": "oil"}
2024-06-06T10:44:07.714Z        INFO    controllers.Tenant      Tenant reconciling completed    {"Request.Name": "oil"}
2024-06-06T10:44:07.720Z        INFO    Starting workers        {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "worker count": 1}
2024-06-06T10:44:07.720Z        INFO    Starting workers        {"controller": "clusterrole", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRole", "worker count": 1}
2024-06-06T10:44:07.720Z        INFO    Starting workers        {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "worker count": 1}
2024-06-06T10:44:07.721Z        INFO    controllers.Tenant      Ensuring limit resources count is updated       {"Request.Name": "oil"}
2024-06-06T10:44:07.721Z        INFO    controllers.Tenant      Ensuring all Namespaces are collected   {"Request.Name": "oil"}
2024-06-06T10:44:07.724Z        INFO    controllers.TLS Skipping TLS certificate generation as it is still valid        {"Request.Namespace": "capsule-system", "Request.Name": "capsule-tls"}
2024-06-06T10:44:07.724Z        INFO    controllers.TLS Updating caBundle in webhooks and crd   {"Request.Namespace": "capsule-system", "Request.Name": "capsule-tls"}
2024-06-06T10:44:07.724Z        INFO    controllers.Tenant      Starting processing of Namespaces       {"Request.Name": "oil", "items": 0}
2024-06-06T10:44:07.724Z        INFO    controllers.Tenant      Starting processing of Network Policies {"Request.Name": "oil"}
2024-06-06T10:44:07.724Z        INFO    controllers.Tenant      Starting processing of Limit Ranges     {"Request.Name": "oil", "items": 0}
2024-06-06T10:44:07.724Z        INFO    controllers.Tenant      Starting processing of Resource Quotas  {"Request.Name": "oil", "items": 0}
2024-06-06T10:44:07.724Z        INFO    controllers.Tenant      Ensuring RoleBindings for Owners and Tenant     {"Request.Name": "oil"}
2024-06-06T10:44:07.724Z        INFO    controllers.Tenant      Ensuring Namespace count        {"Request.Name": "oil"}
2024-06-06T10:44:07.727Z        INFO    controllers.TLS Updating capsule operator pods  {"Request.Namespace": "capsule-system", "Request.Name": "capsule-tls"}
2024-06-06T10:44:07.727Z        INFO    controllers.Tenant      Tenant reconciling completed    {"Request.Name": "oil"}
2024-06-06T10:44:07.821Z        INFO    controllers.TLS Reconciliation completed, processing back in 4247h58m50.178222333s      {"Request.Namespace": "capsule-system", "Request.Name": "capsule-tls"}

Additional context

  • Capsule version: (capsule --version) - the newest
  • Helm Chart version: (helm list -n capsule-system) - not installed
  • Kubernetes version: (kubectl version):
kubectl version
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2
@StanMichel StanMichel added blocked-needs-validation Issue need triage and validation bug Something isn't working labels Jun 6, 2024
@oliverbaehler oliverbaehler self-assigned this Jun 6, 2024
@oliverbaehler oliverbaehler added question Further information is requested and removed bug Something isn't working blocked-needs-validation Issue need triage and validation labels Jun 6, 2024
@oliverbaehler
Copy link
Collaborator

oliverbaehler commented Jun 6, 2024 •
edited
Loading

Hi, there's multiple issues with your approach:

  1. We mainly support installation via Helm Chart, for everything else we can't offer support: https://artifacthub.io/packages/helm/projectcapsule/capsule

The command you are expecting to work kubectl get pods rightfully does not work anymore, because the namespace default is not in a tenant, where alice is an owner of.

I recommend you first going through the documentation:
https://projectcapsule.dev/docs/tenants/

Alice needs a dedicated namespace:

KUBECONFIG=alice-oil.kubeconfig kubectl create ns oil-dev
KUBECONFIG=alice-oil.kubeconfig kubectl get pod -n oil-dev

@StanMichel
Copy link
Author

StanMichel commented Jun 6, 2024 •
edited
Loading

Creating the dedicated namespaces and listing the pods in that namespace as you suggested didn't work with my configuration, but after a clean installation and installing Capsule throughout helm worked for me!

Thanks for your time to response as quickly as you did :) But might I suggest that the supported approach will be reflected in the documentation? The first option in the Getting started section is to install throughout the YAML file.

@oliverbaehler
Copy link
Collaborator

Bad timing from your side :P, we are on the final steps of migrating to the helm chart #1065

The documentation will also be deprecated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@oliverbaehler oliverbaehler

Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@oliverbaehler @StanMichel