Updating a minio tenant image creates downtime

[fpetkovski]

Expected Behavior

I would expect that updating the minio image for a tenant to be seamless and not lead to any downtime.

Current Behavior

The current behavior when updating the .spec.image field of a tenant is that the operator updates the statefulset image backing the tenant pool. This causes pods of the statefulset to roll one after another. While pods are rolling, pretty much all requests to MinIO start to fail.

Even the console becomes very slow and sometimes times out.

Possible Solution

Steps to Reproduce (for bugs)

Create a tenant with a single pool, 16 nodes and 1 drive per node.
Update the image for the tenant and wait for the statefulset pods to start rolling.

Requests against MinIO will fail as long as pods are rolling.

Context

We see the following logs coming from all pods

API: SYSTEM.grid
Time: 13:01:29 UTC 12/06/2024
DeploymentID: 4fd70aef-1659-4ec6-b6ac-546812282941
Error: grid: http://tenant-pool-0-15.tenant-hl.minio.svc.cluster.local:9000 re-connecting to http://tenant-pool-0-0.tenant-hl.minio.svc.cluster.local:9000: dial tcp 10.136.14.157:9000: i/o timeout (*net.OpError) Sleeping 0s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()

I suspect this happens because the healtcheck takes up to 30s to mark a peer as unhealthy: https://github.com/minio/minio/blob/master/internal/rest/client.go#L471-L503. During that time, I don't think the resiliency mechanism kicks in, and requests are still routed to the peer.

This is made worse by the fact that as pods roll, new peers constantly become unavailable, but are not marked as unhealthy on time, which just extends the period for which MinIO as a whole is unavailable.

Your Environment

• Version used (minio-operator): minio/operator:v6.0.4
• Environment name and version (e.g. kubernetes v1.17.2): v1.29.9-gke

[allanrogerr]

Add a readinessProbe to your tenant spec e.g.

  readiness:
    httpGet: 
      path: /minio/health/cluster
      port: 9000
      scheme: HTTP
    initialDelaySeconds: 5
    periodSeconds: 1

See both
https://min.io/docs/minio/linux/operations/monitoring/healthcheck-probe.html
https://min.io/docs/minio/kubernetes/upstream/reference/operator-crd.html#tenantspec

sample-tenant.txt

[fpetkovski]

This makes sense, thanks.

I do wonder though why the operator is not configuring the probe automatically. I would expect an operator to manage a well-defined deployment by default, and it would be a much better user experience if the readiness probe was not something a user has to worry about since it is an internal detail of MinIO.

[allanrogerr]

The operator does not configure your tenant readiness. This is an opt-in configuration in the tenant yaml; depending on the when you consider your tenant to be actually ready e.g. based on expected network timeouts, or some other complex logic.

[cesnietor]

please feel free to reopen if the issue persists after trying the suggested solution.

[fpetkovski]

Thanks for your help @allanrogerr