unknown field "spec.pools[0].volumeClaimTemplate.metadata.creationTimestamp" when trying to upgrade a tenant

[fgleixner]

I try to upgrade a tenant using the current minio Operator:

# kubectl minio version
v5.0.9
# kubectl minio tenant list

Tenant 'bibinfra', Namespace 'minio-bibinfra', Total capacity 6.0 TiB

  Current status: Initialized 
  MinIO version: minio/minio:RELEASE.2023-06-23T20-26-00Z 

# kubectl minio tenant upgrade bibinfra --namespace minio-bibinfra --image minio/minio:RELEASE.2023-09-07T02-05-02Z 
? Upgrade is a one way process. Are you sure to upgrade Tenant 'bibinfra/minio-bibinfra' from version minio/minio:RELEASE.2023-06-23T20-26-00Z to minio/minio:RELEASE.2023-09-07T02-05-02Z? [y/N] y█

Upgrading Tenant 'bibinfra/minio-bibinfra'

W0915 17:42:02.255890   67532 warnings.go:70] unknown field "spec.pools[0].volumeClaimTemplate.metadata.creationTimestamp"
Error: Operation cannot be fulfilled on tenants.minio.min.io "bibinfra": the object has been modified; please apply your changes to the latest version and try again

Some issues say, this is already solved, but why do i still see this problem?

Expected Behavior

Upgrade should work

Current Behavior

Unable to upgrade the tenant

Possible Solution

Unknown

Steps to Reproduce (for bugs)

Don't know

Context

Trying to upgrade

Regression

Probably

Your Environment

• Version used (minio-operator): 5.0.9
• Environment name and version (e.g. kubernetes v1.17.2): kubernetes v1.26.5
• Server type and version: Different
• Operating System and version (uname -a): Linux kube-dar1-15 5.14.21-150500.55.7-default #1 SMP PREEMPT_DYNAMIC Mon Jul 10 18:53:13 UTC 2023 (4204a3a) x86_64 x86_64 x86_64 GNU/Linux
• Link to your deployment file:

[jiuker]

Share the minio-operator's deployments yaml plz. @fgleixner

[fgleixner]

deployment.txt

Here is the output, i get when adding the -o switch. I changed some sensitive infos: domain names and certificate serials.

[jiuker]

deployment.txt

Here is the output, i get when adding the -o switch. I changed some sensitive infos: domain names and certificate serials.

I mean minio-operator. Not tenant. @fgleixner

[fgleixner]

deployment.txt

Ah, here it is. Sorry for the delay!

[jiuker]

How to install minio-operator ? @fgleixner I can't reproduce it. I need more information.

[fgleixner]

I installed it with krew

kubectl minio init -n minio-operator --cluster-domain searchkube.cluster

[anastazya]

Following, i have the same message.

[jiuker]

warnings.go Doesn't exist in our code, it should be an intermediate log from the kube client. If it doesn't have any effect, I think it can be ignored. @anastazya @fgleixner
And I can't reproduce it.

[anastazya]

warnings.go Doesn't exist in our code, it should be an intermediate log from the kube client. If it doesn't have any effect, I think it can be ignored. @anastazya @fgleixner
And I can't reproduce it.

I will dig deeper and if i find the cause i will get back here, can't have that on my prod. cluster, it pollutes the log.

[jiuker]

@anastazya any updates?

[anastazya]

@anastazya any updates?

i don't yet have a separate cluster to test, this is in production and we decided not to touch that since it works.

[anastazya]

@anastazya any updates?

i don't yet have a separate cluster to test, this is in production and we decided not to touch that since it works.

In the end we moved all the data and traffic to a backup location, deleted all updated the operator and created a new tennant. The initial one had become very unstable and throwing a lot of errors like this with no explanation so far. We are investigating the stability of VERSION: v5.0.10 now.

[jiuker]

ial one had become very unstable and throwing a lot of errors like this with no explanation

How to upgrade the operator? @anastazya And could you share the logs?

[fgleixner]

Hi, i upgraded the operator using this documentation:
https://min.io/docs/minio/kubernetes/upstream/operations/install-deploy-manage/upgrade-minio-operator.html
Finally the upgrade of the operator is done with:
kubectl minio init

What i realized is, that the Tenant CRD has not been upgraded, its last update was from 2022-12-02 and it does not contain the spec.pools[].volumeClaimTemplate.metadata.creationTimestamp definition.

But also the crds here https://github.com/minio/operator/blob/master/resources/base/crds/minio.min.io_tenants.yaml do not contain the creationTimestamp property at this path.

Why does kubectl minio tenant upgrade create this creationTimestamp? Is this a bug?

[fgleixner]

I managed to upgrade the tenant by changing the Image name in the minio-operator console. Editing the Tenant using kubectl did not work.

[jiuker]

@fgleixner could you share the kubectl get crd tenants.minio.min.io -oyaml?

[fgleixner]

Here it is
crd_tenants.yaml.gz

[jiuker]

crd_tenants.yaml.gz

Thanks. We have a pr to fix this.