master/v4.0.0/c40ab9c operator does not appear to create a functional console #479

jhoblitt · 2021-02-19T18:39:54Z

Expected Behavior

A working console.

Current Behavior

The new non-tenant specific console is in accessible via an ingress resource:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minio-console-ingress
  namespace: minio-operator
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt"
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/proxy-body-size: 1024m
spec:
  tls:
  - hosts:
    - s3console.cp.lsst.org
    secretName: minio-console-ingress-tls
  rules:
  - host: s3console.cp.lsst.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: minio-tenant-1-console
            port:
              number: 9090

The console may be accessed via the kubectl minio proxy command (this is somewhat inconvenient for me as I normal run administrative commands from a remote host but I resorted to installing the kubectl plugin on a local desktop). However, after logging in with the supplied jwt token this error message is displayed:

When trying to create a new tenant via the console, the creation is stuck at specifying the namespace. If the namespace already exists or not does not seem to change the behavior. The "next" button is always grayed out.

Steps to Reproduce (for bugs)

Build kubectl plugin and operator docker image from source (c40ab9c)
Install dev operator build:

kubectl create ns minio-operator
kubectl minio init -n minio-operator --image=jhoblitt/minio-operator:v3.0.29-70-gc40ab9c-2

Your Environment

Version used (minio-operator): c40ab9c
Environment name and version (e.g. kubernetes v1.17.2): rke 1.19

The text was updated successfully, but these errors were encountered:

jhoblitt · 2021-02-19T19:04:56Z

logs make it look like an rbac setup issue:

$ kubectl -n minio-operator logs console-7494c75898-6b7ft
2021-02-19 18:02:35.669526 I | 2021/02/19 18:02:35 server.go:129: Serving console at http://[::]:9090
2021-02-19 18:26:02.884081 I | 2021/02/19 18:26:02 error.go:44: original error: tenants.minio.min.io is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot list resource "tenants" in API group "minio.min.io" at the cluster scope
2021-02-19 18:26:09.925516 I | 2021/02/19 18:26:09 error.go:44: original error: license not found
2021-02-19 18:26:09.925538 I | 2021/02/19 18:26:09 error.go:124: debugging error: secrets "subnet-license" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "secrets" in API group "" in the namespace "minio-operator"
2021-02-19 18:26:12.734256 I | 2021/02/19 18:26:12 error.go:44: original error: tenants.minio.min.io is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot list resource "tenants" in API group "minio.min.io" at the cluster scope
2021-02-19 18:26:27.706397 I | 2021/02/19 18:26:27 error.go:44: original error: license not found
2021-02-19 18:26:27.706417 I | 2021/02/19 18:26:27 error.go:124: debugging error: secrets "subnet-license" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "secrets" in API group "" in the namespace "minio-operator"
2021-02-19 18:26:28.846171 I | 2021/02/19 18:26:28 error.go:44: original error: tenants.minio.min.io is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot list resource "tenants" in API group "minio.min.io" at the cluster scope
2021-02-19 18:26:55.014750 I | 2021/02/19 18:26:55 error.go:44: original error: nodes is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot list resource "nodes" in API group "" at the cluster scope
2021-02-19 18:27:08.471312 I | 2021/02/19 18:27:08 error.go:44: original error: resourcequotas "minio-tenant-storagequota" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "resourcequotas" in API group "" in the namespace "minio-tenant"
2021-02-19 18:27:09.008350 I | 2021/02/19 18:27:09 error.go:44: original error: resourcequotas "minio-tenant-1-storagequota" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "resourcequotas" in API group "" in the namespace "minio-tenant-1"
2021-02-19 18:27:56.124584 I | 2021/02/19 18:27:56 error.go:44: original error: resourcequotas "minio-tenant--storagequota" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "resourcequotas" in API group "" in the namespace "minio-tenant-"
2021-02-19 18:27:57.030858 I | 2021/02/19 18:27:57 error.go:44: original error: resourcequotas "minio-tenant-2-storagequota" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "resourcequotas" in API group "" in the namespace "minio-tenant-2"
2021-02-19 18:28:02.117582 I | 2021/02/19 18:28:02 error.go:44: original error: resourcequotas "minio--storagequota" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "resourcequotas" in API group "" in the namespace "minio-"
2021-02-19 18:28:03.191003 I | 2021/02/19 18:28:03 error.go:44: original error: resourcequotas "minio-te-storagequota" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "resourcequotas" in API group "" in the namespace "minio-te"
2021-02-19 18:28:04.387881 I | 2021/02/19 18:28:04 error.go:44: original error: resourcequotas "minio-ten-storagequota" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "resourcequotas" in API group "" in the namespace "minio-ten"
2021-02-19 18:28:06.403957 I | 2021/02/19 18:28:06 error.go:44: original error: resourcequotas "minio-tenant-1-storagequota" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "resourcequotas" in API group "" in the namespace "minio-tenant-1"

cesnietor · 2021-02-19T19:54:37Z

Hi @jhoblitt, for number 3. right now you need to have a Resource quota so that an available storage class can be selected.
Can you please try creating a resourcequota.yaml file with the following content:
Supposing the namespace you are trying to use already exists (else you have to create it in advance):

apiVersion: v1
kind: ResourceQuota
metadata:
  name: default-storagequota
  namespace: minio-tenant-1
spec:
  hard:
    standard.storageclass.storage.k8s.io/requests.storage: "9223372036854775807"

then applying it like

kubectl apply -f resourcequota.yaml

The number 9223372036854775807 is basically setting the limit for the storageclass standard, you can modify the number depending your requirements/available storage.

EDIT: With current operator, there is no need to create a resourcequota by yourself.

dvaldivia · 2021-02-19T20:01:56Z

@cesnietor I think we removed that requirement recently, does the minio-tenant-1 namespace exists?

jhoblitt · 2021-02-19T20:01:57Z

@cesnietor Sure. I killed the operator pod to reset the logging:

$ kubectl delete ns minio-tenant-1
namespace "minio-tenant-1" deleted
$ kubectl create ns minio-tenant-1
namespace/minio-tenant-1 created
$ cat rquota.yaml 
apiVersion: v1
kind: ResourceQuota
metadata:
  name: default-storagequota
  namespace: minio-tenant-1
spec:
  hard:
    standard.storageclass.storage.k8s.io/requests.storage: "9223372036854775807"
$ kubectl apply -f rquota.yaml 
resourcequota/default-storagequota created
$ kubectl -n minio-operator delete pod minio-operator-79fb7887b4-tt4xm 
pod "minio-operator-79fb7887b4-tt4xm" deleted
$ kubectl minio tenant create \
>   minio-tenant-1 \
>   --servers          3 \
>   --volumes          12 \
>   --capacity         1Ti \
>   --namespace        minio-tenant-1 \
>   --storage-class    local-storage
W0219 12:59:32.673903 1098279 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition

Tenant 'minio-tenant-1' created in 'minio-tenant-1' Namespace

  Username: admin 
  Password: 3673b98d-0477-4d04-b8a5-a6c44c0b8df9 
  Note: Copy the credentials to a secure location. MinIO will not display these again.

+-------------+------------------------+----------------+--------------+--------------+
| APPLICATION | SERVICE NAME           | NAMESPACE      | SERVICE TYPE | SERVICE PORT |
+-------------+------------------------+----------------+--------------+--------------+
| MinIO       | minio                  | minio-tenant-1 | ClusterIP    | 443          |
| Console     | minio-tenant-1-console | minio-tenant-1 | ClusterIP    | 9443         |
+-------------+------------------------+----------------+--------------+--------------+

# wait a bit

$ kubectl -n minio-tenant-1 get pod
No resources found in minio-tenant-1 namespace.
 $ kubectl -n minio-operator logs minio-operator-79fb7887b4-7ddbx  | grep -v deprecated
I0219 19:58:01.120593       1 main.go:72] Starting MinIO Operator
I0219 19:58:01.354854       1 main.go:139] caBundle on CRD updated
I0219 19:58:01.355839       1 main-controller.go:250] Setting up event handlers
I0219 19:58:01.355897       1 main-controller.go:634] Starting Tenant controller
I0219 19:58:01.355903       1 main-controller.go:637] Waiting for informer caches to sync
I0219 19:58:01.365780       1 main-controller.go:598] operator TLS secret not found%!(EXTRA string=secrets "operator-tls" not found)
I0219 19:58:01.379457       1 csr.go:217] Start polling for certificate of csr/operator-minio-operator-csr, every 5s, timeout after 20m0s
I0219 19:58:01.456000       1 main-controller.go:642] Starting workers
E0219 19:59:38.269921       1 main-controller.go:720] error syncing 'minio-tenant-1/minio-tenant-1': secrets "operator-tls" not found
E0219 19:59:48.250049       1 main-controller.go:720] error syncing 'minio-tenant-1/minio-tenant-1': secrets "operator-tls" not found
E0219 20:00:48.254844       1 main-controller.go:720] error syncing 'minio-tenant-1/minio-tenant-1': secrets "operator-tls" not found

jhoblitt · 2021-02-19T20:03:30Z

@cesnietor @dvaldivia sorry, I got confused as to which issue this was. I'll try the console and report back.

jhoblitt · 2021-02-19T20:08:05Z

Looks like no change:

$ kubectl -n minio-tenant-1 get resourcequota
NAME                   AGE     REQUEST                                                                        LIMIT
default-storagequota   4m40s   standard.storageclass.storage.k8s.io/requests.storage: 0/9223372036854775807   
$ kubectl -n minio-operator get pods
NAME                              READY   STATUS    RESTARTS   AGE
console-7494c75898-6b7ft          1/1     Running   0          122m
minio-operator-79fb7887b4-7ddbx   1/1     Running   0          7m15s
$ kubectl -n minio-operator delete pod console-7494c75898-6b7ft
pod "console-7494c75898-6b7ft" deleted
$ kubectl -n minio-operator get pod
NAME                              READY   STATUS    RESTARTS   AGE
console-7494c75898-krpx4          1/1     Running   0          11s
minio-operator-79fb7887b4-7ddbx   1/1     Running   0          7m39s
[chonchon] ~/github/k8s-cookbook/chonchon/minio $ kubectl minio proxy
Starting port forward of the Console UI.

To connect open a browser and go to http://localhost:9090

Current JWT to login: eyJhbGciOiJSUzI1NiIsImtpZCI6IndXcVZrTnJoTGhDQjN0ZWQ1a1A0d3RrZjM5aGYxTDRHSXBpZ3F5SDZ4VDgifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaW5pby1vcGVyYXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjb25zb2xlLXNhLXRva2VuLWMycDlzIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNvbnNvbGUtc2EiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI2ZDExOWIyYS1kMDA0LTQzMGUtYjA3My05ZWI4NjQxOWZmODYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6bWluaW8tb3BlcmF0b3I6Y29uc29sZS1zYSJ9.LD_TQWTTSwRClsxJNXlUk7pMnP1lO_lTJ7Po8ZEMtTnPTdzV_O4bGj8J7WyTYQoIJlw1_fAoo8FV_Bc8cFMK-uah4BazF3cRcLu73q9JEUMKnKK7OnvMMc1qiLLK0g9d0Kn5-vxV7obEftiAQ2Hn1OUIJN0RH1yE05oFjULiLvkAEfHK7t2M2bNbmHVGnn-4hkRjRTumJIu66OdCEuFJ1Hoi1iw83uNJ7yigZ0q8tD9n-HlF9WZsQdItuNU_2u-r_CfVKO-f3h8o5NclnkC9YbPHvDHTKXwjurDMex7VObrhdFz9KBqRACSjqZ-lWiMkK_wPtx07bAk6SqiWGN94VQ

Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
Handling connection for 9090
Handling connection for 9090
Handling connection for 9090
^C
$ kubectl -n minio-operator logs console-7494c75898-krpx4
2021-02-19 20:05:31.263648 I | 2021/02/19 20:05:31 server.go:129: Serving console at http://[::]:9090
2021-02-19 20:05:58.903145 I | 2021/02/19 20:05:58 token.go:145: cipher: message authentication failed
2021-02-19 20:05:58.903164 I | 2021/02/19 20:05:58 token.go:93: encrypted session token claims not in the right format
2021-02-19 20:05:58.903171 I | 2021/02/19 20:05:58 configure_console.go:78: session token internal data is malformed
2021-02-19 20:06:04.542710 I | 2021/02/19 20:06:04 error.go:44: original error: tenants.minio.min.io is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot list resource "tenants" in API group "minio.min.io" at the cluster scope
2021-02-19 20:06:07.200584 I | 2021/02/19 20:06:07 error.go:44: original error: nodes is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot list resource "nodes" in API group "" at the cluster scope
2021-02-19 20:06:11.125867 I | 2021/02/19 20:06:11 error.go:44: original error: resourcequotas "t-storagequota" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "resourcequotas" in API group "" in the namespace "t"
2021-02-19 20:06:14.783173 I | 2021/02/19 20:06:14 error.go:44: original error: resourcequotas "minio-tenan-storagequota" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "resourcequotas" in API group "" in the namespace "minio-tenan"
2021-02-19 20:06:15.910201 I | 2021/02/19 20:06:15 error.go:44: original error: resourcequotas "minio-tenant-storagequota" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "resourcequotas" in API group "" in the namespace "minio-tenant"
2021-02-19 20:06:18.434332 I | 2021/02/19 20:06:18 error.go:44: original error: resourcequotas "minio-tenant1-storagequota" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "resourcequotas" in API group "" in the namespace "minio-tenant1"
2021-02-19 20:06:26.678440 I | 2021/02/19 20:06:26 error.go:44: original error: resourcequotas "minio-tenant-1-storagequota" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "resourcequotas" in API group "" in the namespace "minio-tenant-1"

dvaldivia · 2021-02-19T20:16:55Z

@jhoblitt the main error is the following

2021-02-19 20:06:26.678440 I | 2021/02/19 20:06:26 error.go:44: original error: resourcequotas "minio-tenant-1-storagequota" is forbidden: User "system:serviceaccount:minio-operator:console-sa" cannot get resource "resourcequotas" in API group "" in the namespace "minio-tenant-1"

How did you install the operator? It sounds like the service account console-sa doesn't have permission to access resourcequotas

jhoblitt · 2021-02-19T20:35:45Z

@dvaldivia I replaced the krew synlimk for kubectl-minio with my build of master and than ran:

kubectl minio delete -n minio-operator
kubectl delete ns minio-operator
kubectl create ns minio-operator
kubectl minio init -n minio-operator --image=jhoblitt/minio-operator:v3.0.29-70-gc40ab9c-2

Do you think this might be caused by #473 leaving resources behind?

dvaldivia · 2021-02-19T20:39:28Z

it's possible @jhoblitt, do you wanna try installing via kubectl ?

kubectl delete -k github.com/minio/operator/\?ref\=v3.0.29"
kubectl apply -k github.com/minio/operator/\?ref\=v3.0.29"

cesnietor · 2021-02-19T20:43:34Z

@jhoblitt So I've tried the commands with kubectl plugin and indeed the some resources like service accounts are not created correctly.
In fact the console that you are running might be old I guess you apply it separately? cause, yes, you shouldn't have had to add the resourcequota.
I ran it using kustomize and work fine. Creating the service accounts correctly.

k get serviceaccounts 
NAME             SECRETS   AGE
console-sa       1         4m37s
default          1         13m
minio-operator   1         4m37s

In the meantime you can run it as @dvaldivia said or try with kustomize by modifying the kustomization.yaml for your image.
like:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonAnnotations:
  operator.min.io/authors: "MinIO, Inc."
  operator.min.io/license: "AGPLv3"
  operator.min.io/support: "https://subnet.min.io"

# Configure repo and tag of MinIO Operator Image
images:
  - name: minio/operator
    newName: <your-image>
    newTag: <your-tag>

namespace: <your-ns>

resources:
  - resources/namespace.yaml
  - resources/service-account.yaml
  - resources/cluster-role.yaml
  - resources/cluster-role-binding.yaml
  - resources/base/crds/minio.min.io_tenants.yaml
  - resources/service.yaml
  - resources/deployment.yaml
  - resources/console-ui.yaml

And then run

 kustomize build . | k apply -f -

We'll track the issue with the kubectl plugin but you can try to use that in the meantime, hope it helps.

dvaldivia · 2021-02-19T20:46:01Z

what @cesnietor proposes is a good solution

cesnietor · 2021-02-19T20:46:30Z

the main reason might be cause plugin is attached to version v3.0.29

kubectl krew list
PLUGIN  VERSION
krew    v0.4.1
minio   v3.0.29

And building with master which is ahead of 3.0.29 @dvaldivia we need to update plugin only and test. (once new release is ready).
Else, you can try with an older operator image (3.0.29 instead of master) but you would be missing out of the new stuff.

jhoblitt · 2021-02-19T20:58:56Z

@dvaldivia Sure, I can try to to install with apply or customize. I'm trying to get the console ldap support working which was merged after 3.0.29 was tagged but I can try the yaml from master.

@cesnietor note that I am not installing the plugin with krew. I am just copying in my build of the plugin from master to into the krew bin dir.

 ~/.krew/bin $ ll
total 48008
lrwxrwxrwx 1 jhoblitt jhoblitt       43 Feb 19 11:22 kubectl-krew -> /home/jhoblitt/.krew/store/krew/v0.4.1/krew
-rwxrwxr-x 1 jhoblitt jhoblitt 49158220 Feb 19 11:25 kubectl-minio
lrwxrwxrwx 1 jhoblitt jhoblitt       54 Feb 19 11:24 kubectl-minio-v3.0.29 -> /home/jhoblitt/.krew/store/minio/v3.0.29/kubectl-minio
 ~/.krew/bin $

Maybe it would be a good idea to add a version sub-command to the plugin?

 ~/.krew/bin $ ./kubectl-minio version
Error: unknown command "version" for "minio"
Run 'minio --help' for usage.
``

dvaldivia · 2021-02-19T22:18:15Z

if you want to deploy the latest, you could build your own container

TAG=jhoblitt/operator:v4.0.0 make build && docker push jhoblitt/operator:v4.0.0

and then clone this repo, modify resources/deployment.yaml to match your image and then

kustomize build . | k apply -f -

jhoblitt · 2021-02-19T22:58:16Z

@dvaldivia I am aware and that's what I did this morning. From the initial ticket I'm using kubectl minio init -n minio-operator --image=jhoblitt/minio-operator:v3.0.29-70-gc40ab9c-2. I just didn't bother to override the dev tag logic in the Makefile this morning: https://hub.docker.com/repository/docker/jhoblitt/minio-operator . c40ab9c was the tip of master as of this morning.

cesnietor · 2021-02-19T23:26:32Z

@jhoblitt the issue for the serviceaccount is a clusterrolebinding which was not being set to the defined namespace. this is being addressed in the pr above ^.

Alevsk · 2021-02-20T00:07:26Z

@jhoblitt hi, for issue #1 if UI is loading on screen that means requests are correctly routed to the service via the ingress controller (at least the client was able to download the js), can you show here the output of:

kubectl get svc --all-namespaces

also that initial screen that shows the 503, does it keeps loading forever? Were you able to authenticate using the k8s service account token? how did you get to the second screen?

jhoblitt · 2021-02-20T00:15:38Z

@cesnietor I've build your PR and pushed it to docker hub as jhoblitt/minio-operator:v3.0.29-73-g11f0964. Probably not going to be able to test it today though.

@Alevsk Re #1. I just checked and it is still showing the same page with the "503" message on it and a spinning circle symbol. And I see where you going... the service name is wrong.

$ kubectl get svc --all-namespaces
NAMESPACE        NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
cert-manager     cert-manager                         ClusterIP      10.43.169.25    <none>           9402/TCP                     10d
cert-manager     cert-manager-webhook                 ClusterIP      10.43.249.198   <none>           443/TCP                      10d
default          kubernetes                           ClusterIP      10.43.0.1       <none>           443/TCP                      10d
ingress-nginx    ingress-nginx-controller             LoadBalancer   10.43.211.179   139.229.160.56   80:32662/TCP,443:30699/TCP   10d
ingress-nginx    ingress-nginx-controller-admission   ClusterIP      10.43.29.220    <none>           443/TCP                      10d
kube-system      kube-dns                             ClusterIP      10.43.0.10      <none>           53/UDP,53/TCP,9153/TCP       10d
kube-system      metrics-server                       ClusterIP      10.43.138.226   <none>           443/TCP                      10d
minio-operator   console                              ClusterIP      10.43.215.65    <none>           9090/TCP,9443/TCP            6h10m
minio-operator   operator                             ClusterIP      10.43.28.102    <none>           4222/TCP,4233/TCP            6h10m

This is extremely strange... what is the ingress has to be pointed at nothing now. If I do an f5 reload in FF I get a gateway error... so this must be some sort of browser caching.

jhoblitt · 2021-02-20T00:18:29Z

@Alevsk Good call. #1 was an id10t error (and some crazy browser caching of a tenant console). Fixing the ingress resource got it working.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minio-console-ingress
  namespace: minio-operator
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt"
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/proxy-body-size: 1024m
spec:
  tls:
  - hosts:
    - s3console.cp.lsst.org
    secretName: minio-console-ingress-tls
  rules:
  - host: s3console.cp.lsst.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: console
            port:
              number: 9090

cesnietor · 2021-02-22T21:46:33Z

Closing this issue since all issues are being resolved in PR #485, please reopen or file new issue if the bug persists.

jhoblitt · 2021-02-23T16:04:44Z

@dvaldivia / @cesnietor / @Alevsk I have tried installing the operator using kustomize from 9ab6c08 with the operator image tag as the only modification.

Doing it this way means the JWT token isn't fished out for you but it looks like is data.token in the secret console-sa-token-xxxx. It might be worth documenting this if if use of kustomize will be encouraged.

However, using kustomize directly means that the fix from #485 isn't picked up at all:

2021-02-22 17:16:58.418566 I | 2021/02/22 17:16:58 server.go:129: Serving console at http://[::]:9090
2021-02-23 15:53:01.382282 I | 2021/02/23 15:53:01 error.go:44: original error: tenants.minio.min.io is forbidden: User "system:serviceaccount:minio-operator:default" cannot list resource "tenants" in API group "minio.min.io" at the cluster scope
2021-02-23 15:54:06.416187 I | 2021/02/23 15:54:06 error.go:44: original error: nodes is forbidden: User "system:serviceaccount:minio-operator:default" cannot list resource "nodes" in API group "" at the cluster scope
2021-02-23 15:54:19.905651 I | 2021/02/23 15:54:19 error.go:44: original error: resourcequotas "minio--storagequota" is forbidden: User "system:serviceaccount:minio-operator:default" cannot get resource "resourcequotas" in API group "" in the namespace "minio-"
2021-02-23 15:54:20.701342 I | 2021/02/23 15:54:20 error.go:44: original error: resourcequotas "minio-te-storagequota" is forbidden: User "system:serviceaccount:minio-operator:default" cannot get resource "resourcequotas" in API group "" in the namespace "minio-te"
2021-02-23 15:54:22.340674 I | 2021/02/23 15:54:22 error.go:44: original error: resourcequotas "minio-tenant-2-storagequota" is forbidden: User "system:serviceaccount:minio-operator:default" cannot get resource "resourcequotas" in API group "" in the namespace "minio-tenant-2"

I will try again with kubectl-minio since #486 has been merged.

jhoblitt · 2021-02-23T16:16:00Z

@cesnietor It appears #485 caused a new problem. I am unable to re-open this ticket. Would you like to open a new issue?

I rebuilt kubectl-minio from bfa9605.

$ cp kubectl-minio/kubectl-minio ~/.krew/bin/kubectl-minio
$ kubectl minio delete -n minio-operator

Are you sure you want to delete ALL the MinIO Tenants and MinIO Operator?: Y
$ kubectl get clusterrolebinding minio-operator-binding
Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "minio-operator-binding" not found
$ kubectl minio init -n minio-operator --image=jhoblitt/minio-operator:v3.0.29-76-gbfa9605
MinIO Operator Namespace minio-operator: created
CustomResourceDefinition tenants.minio.min.io: created
ClusterRole minio-operator-role: created
ServiceAccount minio-operator: created
ClusterRoleBinding minio-operator-binding: created
MinIO Operator Service operator: created
MinIO Operator Deployment minio-operator: created
MinIO Console Deployment: created
-----------------

To open Operator UI, start a port forward using this command:

kubectl minio proxy

-----------------
$ kubectl minio proxy
Starting port forward of the Console UI.

To connect open a browser and go to http://localhost:9090

Current JWT to login: eyJhbGciOiJSUzI1NiIsImtpZCI6IndXcVZrTnJoTGhDQjN0ZWQ1a1A0d3RrZjM5aGYxTDRHSXBpZ3F5SDZ4VDgifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaW5pby1vcGVyYXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjb25zb2xlLXNhLXRva2VuLXp3emJtIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNvbnNvbGUtc2EiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI2YWJkYTI5Mi01Yjk3LTRjZGYtOTA5OC04ZTUwNThkOGQzYjgiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6bWluaW8tb3BlcmF0b3I6Y29uc29sZS1zYSJ9.xNXShL_pEuYXztoP2dE8HnehkZgWObBJ8q2h-QdMtMJU5ilE_dV6k6U8uZFbIU172CFt6WDNB3j3I9PHgnbc4MaI7Vw3ZvoxGbhRF6duq7BQGVNY-dIdxaj50BkijFTSsRZY2_K2DRQu99v453YxJo-LahF_n0gSX7mjgXdVXlPe86cy6kojjPMtNDdrMCWcuEMDVDUdfkw-TQmqtKOUyQQvpKBxn8Eh3ctXD2bstmSnaVaaLY8TwOCpbZfoHOgAwnvjsdXFrPu9LlFtXyx98I0LdT47aybrlOol1aQi_nJw_tnSFkpfUi5BnoRS3Z_ZM_siaiswwGMqrZch7x_Tmw

The good news is that issue 2) is resolved:

However, issue 3 is still present but with a different log messages:

$ kubectl -n minio-operator logs console-7494c75898-w5m6f 
2021-02-23 16:09:24.817512 I | 2021/02/23 16:09:24 server.go:129: Serving console at http://[::]:9090
2021-02-23 16:09:58.111314 I | 2021/02/23 16:09:58 token.go:145: cipher: message authentication failed
2021-02-23 16:09:58.111331 I | 2021/02/23 16:09:58 token.go:93: encrypted session token claims not in the right format
2021-02-23 16:09:58.111337 I | 2021/02/23 16:09:58 configure_console.go:78: session token internal data is malformed
2021-02-23 16:10:02.528772 I | 2021/02/23 16:10:02 token.go:145: cipher: message authentication failed
2021-02-23 16:10:02.528789 I | 2021/02/23 16:10:02 token.go:93: encrypted session token claims not in the right format
2021-02-23 16:10:02.528796 I | 2021/02/23 16:10:02 configure_console.go:78: session token internal data is malformed
2021-02-23 16:10:34.890896 I | 2021/02/23 16:10:34 error.go:44: original error: namespaces "minio-te" not found
2021-02-23 16:10:36.454651 I | 2021/02/23 16:10:36 error.go:44: original error: namespaces "minio-tenant-1" not found

cesnietor · 2021-02-23T16:18:40Z

@jhoblitt does the namespace minio-tenant-1 exist? it should be created in advance manually.
In the previous error reported it was having a forbidden message but now it is just not finding it.

jhoblitt · 2021-02-23T16:30:37Z

@cesnietor Aww, creating a namespace with kubectl does get the console past that error. That's something is something I would normally expect a gui wizard to take care of.

I'm not sure why kustomize failed to generate working resources. I'm guessing that the clusterrolebinding having default as the default namespace might be the cause.

jhoblitt · 2021-02-23T16:43:16Z

This is about as far as I can go with the console for the moment as I've got 3 physical nodes to test with. minio/console#610

I may have to spin up VMs.

cesnietor · 2021-02-23T17:16:06Z

@dvaldivia / @cesnietor / @Alevsk I have tried installing the operator using kustomize from 9ab6c08 with the operator image tag as the only modification.

Doing it this way means the JWT token isn't fished out for you but it looks like is data.token in the secret console-sa-token-xxxx. It might be worth documenting this if if use of kustomize will be encouraged.

However, using kustomize directly means that the fix from #485 isn't picked up at all:

2021-02-22 17:16:58.418566 I | 2021/02/22 17:16:58 server.go:129: Serving console at http://[::]:9090
2021-02-23 15:53:01.382282 I | 2021/02/23 15:53:01 error.go:44: original error: tenants.minio.min.io is forbidden: User "system:serviceaccount:minio-operator:default" cannot list resource "tenants" in API group "minio.min.io" at the cluster scope
2021-02-23 15:54:06.416187 I | 2021/02/23 15:54:06 error.go:44: original error: nodes is forbidden: User "system:serviceaccount:minio-operator:default" cannot list resource "nodes" in API group "" at the cluster scope
2021-02-23 15:54:19.905651 I | 2021/02/23 15:54:19 error.go:44: original error: resourcequotas "minio--storagequota" is forbidden: User "system:serviceaccount:minio-operator:default" cannot get resource "resourcequotas" in API group "" in the namespace "minio-"
2021-02-23 15:54:20.701342 I | 2021/02/23 15:54:20 error.go:44: original error: resourcequotas "minio-te-storagequota" is forbidden: User "system:serviceaccount:minio-operator:default" cannot get resource "resourcequotas" in API group "" in the namespace "minio-te"
2021-02-23 15:54:22.340674 I | 2021/02/23 15:54:22 error.go:44: original error: resourcequotas "minio-tenant-2-storagequota" is forbidden: User "system:serviceaccount:minio-operator:default" cannot get resource "resourcequotas" in API group "" in the namespace "minio-tenant-2"

I will try again with kubectl-minio since #486 has been merged.

@jhoblitt did you also specify the namespace on the kustomization yaml? not only the image?
I've just tested it with master from operator doing the proper changes like I mentioned in #479 (comment) and the clusterrolebinding is being created correctly assigning the namespace to the role:

 k get clusterrolebinding console-sa-binding        
NAME                 ROLE                          AGE
console-sa-binding   ClusterRole/console-sa-role   29s

...
        ],
        "name": "console-sa-binding",
        "resourceVersion": "6438",
        "selfLink": "/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/console-sa-binding",
        "uid": "054028b3-f511-468c-aabf-2aaf013d1d22"
    },
    "roleRef": {
        "apiGroup": "rbac.authorization.k8s.io",
        "kind": "ClusterRole",
        "name": "console-sa-role"
    },
    "subjects": [
        {
            "kind": "ServiceAccount",
            "name": "console-sa",
            "namespace": "minio-operator"
        }
    ]
}

Please let me know if setting the namespace on kustomization yaml works, else could you please paste your kustomization.yaml (hiding any specific things) in case there is something different I'm missing.

harshavardhana assigned dvaldivia and Alevsk Feb 19, 2021

harshavardhana assigned cesnietor Feb 19, 2021

cesnietor mentioned this issue Feb 19, 2021

Fix ClusterRoleBinding namespace setting on kubectl-minio #485

Merged

cesnietor linked a pull request Feb 20, 2021 that will close this issue

Fix ClusterRoleBinding namespace setting on kubectl-minio #485

Merged

harshavardhana added the release-4.0 label Feb 22, 2021

cesnietor closed this as completed Feb 22, 2021

harshavardhana added the fixed label Feb 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

master/v4.0.0/c40ab9c operator does not appear to create a functional console #479

master/v4.0.0/c40ab9c operator does not appear to create a functional console #479

jhoblitt commented Feb 19, 2021

jhoblitt commented Feb 19, 2021

cesnietor commented Feb 19, 2021 •

edited

Loading

dvaldivia commented Feb 19, 2021

jhoblitt commented Feb 19, 2021

jhoblitt commented Feb 19, 2021

jhoblitt commented Feb 19, 2021 •

edited

Loading

dvaldivia commented Feb 19, 2021

jhoblitt commented Feb 19, 2021

dvaldivia commented Feb 19, 2021

cesnietor commented Feb 19, 2021

dvaldivia commented Feb 19, 2021

cesnietor commented Feb 19, 2021 •

edited

Loading

jhoblitt commented Feb 19, 2021

dvaldivia commented Feb 19, 2021 •

edited

Loading

jhoblitt commented Feb 19, 2021

cesnietor commented Feb 19, 2021

Alevsk commented Feb 20, 2021

jhoblitt commented Feb 20, 2021

jhoblitt commented Feb 20, 2021

cesnietor commented Feb 22, 2021

jhoblitt commented Feb 23, 2021

jhoblitt commented Feb 23, 2021

cesnietor commented Feb 23, 2021 •

edited

Loading

jhoblitt commented Feb 23, 2021

jhoblitt commented Feb 23, 2021

cesnietor commented Feb 23, 2021

master/v4.0.0/c40ab9c operator does not appear to create a functional console #479

master/v4.0.0/c40ab9c operator does not appear to create a functional console #479

Comments

jhoblitt commented Feb 19, 2021

Expected Behavior

Current Behavior

Steps to Reproduce (for bugs)

Your Environment

jhoblitt commented Feb 19, 2021

cesnietor commented Feb 19, 2021 • edited Loading

dvaldivia commented Feb 19, 2021

jhoblitt commented Feb 19, 2021

jhoblitt commented Feb 19, 2021

jhoblitt commented Feb 19, 2021 • edited Loading

dvaldivia commented Feb 19, 2021

jhoblitt commented Feb 19, 2021

dvaldivia commented Feb 19, 2021

cesnietor commented Feb 19, 2021

dvaldivia commented Feb 19, 2021

cesnietor commented Feb 19, 2021 • edited Loading

jhoblitt commented Feb 19, 2021

dvaldivia commented Feb 19, 2021 • edited Loading

jhoblitt commented Feb 19, 2021

cesnietor commented Feb 19, 2021

Alevsk commented Feb 20, 2021

jhoblitt commented Feb 20, 2021

jhoblitt commented Feb 20, 2021

cesnietor commented Feb 22, 2021

jhoblitt commented Feb 23, 2021

jhoblitt commented Feb 23, 2021

cesnietor commented Feb 23, 2021 • edited Loading

jhoblitt commented Feb 23, 2021

jhoblitt commented Feb 23, 2021

cesnietor commented Feb 23, 2021

cesnietor commented Feb 19, 2021 •

edited

Loading

jhoblitt commented Feb 19, 2021 •

edited

Loading

cesnietor commented Feb 19, 2021 •

edited

Loading

dvaldivia commented Feb 19, 2021 •

edited

Loading

cesnietor commented Feb 23, 2021 •

edited

Loading