requestAutoCert makes tenant stuck in "Provisioning default buckets", operator can't find Kubernetes CA (CN=kubernetes)

[krishgu]

Trying to use a default setup of the minio-operator and a tenant, using Helm charts version 5.0.10. The operator is installed in minio-operator namespace, and tenant is appzero in tenant-ns namespace, with 3 buckets defined. This is the tenant yaml file. the requestAutoCert is true by default in the Tenant helm values.yaml

tenant:
  name: appzero
  image:
  ...
  mountPath: /apps/minio-data
  subPath: /appzero
  configuration:
    name: appzero-env-configuration
  pools:
...
  metrics:
    enabled: true
  features:
    bucketDNS: false
    domains: { }
    enableSFTP: true
  buckets: 
    - name: "bkt1"
    - name: "bkt2"
    - name: "bkt3"
  prometheusOperator: false
  ingress:
    api:
      enabled: true
      ingressClassName: "nginx"
      host: minio.local
    console:
      enabled: true
      ingressClassName: "nginx"
      host: minio-console.local

Expected Behavior

I expect this to work, with this tenant and the three buckets created

Current Behavior

1. The tenant Status is in "Provisioning Default buckets"

status:
  availableReplicas: 0
  certificates:
    autoCertEnabled: true
    customCertificates: {}
  currentState: Provisioning default buckets
  drivesOffline: 13
  drivesOnline: 3
  healthMessage: Service Unavailable
  healthStatus: red

2. additionally, there are three services created for the tenant in the tenant-ns namespace. appzero-console, appzero-hl, and just minio . Why is the minio service created without the prefix of the tenant name (should be appzero-minio)? Is it because of the ingress configuration for the tenant?

3. an unrelated -- if I set prometheusOperator: true, it looks at the default namespace, is there a way to specify the namespace for the Prometheus?

4. Additional information

• The operator gets stuck in

I1031 23:10:23.883357       1 event.go:298] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tenant-ns", Name:"appzero", UID:"xxxx", APIVersion:"minio.min.io/v2", ResourceVersion:"22030816", FieldPath:""}): type: 'Warning' reason: 'BucketsCreatedFailed' Buckets creation failed: Put "https://minio.tenant-ns.svc.cluster.local/bkt1/": dial tcp 10.107.23.215:443: connect: connection refused

• Doing curl from the operator-leader pod, shows that it can't find the kubernetes ca.crt, with CN=kubernetes, get stuck in SSL certificate problem: unable to get local issuer certificate
  ** the minio.tenant-ns service is showing subject: O=system:nodes; CN=system:node:*.appzero-hl.tenant-ns.svc.cluster.local start date: Oct 31 23:05:16 2023 GMT expire date: Oct 30 23:05:16 2024 GMT issuer: CN=kubernetes
• curl to the endpoint without cert-validation (curl -k https://minio.tenant-ns.svc.cluster.local/bkt1) works. curl uses only /etc/pki/tls/certs/ca-bundle.crt to validate certs

Possible Solution

• we can add the SSL_CERTS_DIR to include /var/run/kubernetes.io/secrets/serviceaccounts as one option, but shouldn't that be the default?
• I also haven't tried setting requestAutoCert: false so as to get a closure to this issue
• Will try removing the ingress settings to observe the behavior

Steps to Reproduce (for bugs)

Context

Thank you!

Regression

Your Environment

• Version used (minio-operator): 5.0.17
• Environment name and version (e.g. kubernetes v1.17.2): v1.21.4
• Server type and version:
• Operating System and version (uname -a):
• Link to your deployment file:

[krishgu]

https://github.com/minio/operator#4-connect-to-the-tenant says that the ca.crt must be copied over. Let me try that

May use the Go config (instead use SSL_CERTS_DIR https://pkg.go.dev/crypto/x509)

[krishgu]

Setting SSL_CERTS_DIR and CURL_CA_BUNDLE environment variables to the operator pod solves this.

But any input on the "minio" service-naming convention and about Prometheus integration will be appreciated

[krishgu]

The prometheus integration seems to be resolved by setting PROMETHEUS_NAMESPACE env var to the operator. But the Tenant is in a continuous/infinite loop w/ the [provisionedBucketStatus]. (https://github.com/minio/operator/blob/fc3d3f4b9039d749b58bc48ec79dd2e311dac205/pkg/controller/status.go#L183C72-L183C72).

Operator checks the tenant every 5 seconds, and there is some issue there, should it stop at the error in helper.

01:23:56.898891       1 event.go:298] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tenant-ns", Name:"appzero", UID:"c01528e2-5535-4824-a387-280e8bbc7e12", APIVersion:"minio.min.io/v2", ResourceVersion:"22515138", FieldPath:""}): type: 'Normal' reason: 'BucketsCreated' Buckets created


01:24:01.914981       1 helper.go:779] Your previous request to create the named bucket succeeded and you already own it.
01:24:01.916895       1 helper.go:779] Your previous request to create the named bucket succeeded and you already own it.
01:24:01.918757       1 helper.go:779] Your previous request to create the named bucket succeeded and you already own it.
01:24:01.924142       1 event.go:298] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tenant-ns", Name:"appzero", UID:"c01528e2-5535-4824-a387-280e8bbc7e12", APIVersion:"minio.min.io/v2", ResourceVersion:"22515152", FieldPath:""}): type: 'Normal' reason: 'BucketsCreated' Buckets created


01:24:06.969098       1 helper.go:779] Your previous request to create the named bucket succeeded and you already own it.
01:24:06.971086       1 helper.go:779] Your previous request to create the named bucket succeeded and you already own it.
01:24:06.973388       1 helper.go:779] Your previous request to create the named bucket succeeded and you already own it.
01:24:06.977325       1 status.go:197] Hit conflict issue, getting latest version of tenant
01:24:06.984899       1 event.go:298] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tenant-ns", Name:"appzero", UID:"c01528e2-5535-4824-a387-280e8bbc7e12", APIVersion:"minio.min.io/v2", ResourceVersion:"22515166", FieldPath:""}): type: 'Normal' reason: 'BucketsCreated' Buckets created

[krishgu]

this is version 5.0.10 -- looks like there are some code changes on this feature, as part #1840 ? should we not do provisionedBuckets until then?

[krishgu]

the "Hit Conflict" is a dupe of #1829 and PR #1837 .. all others were resolved. closing this.

can we get an EBF release please? Thanks again!