argocd-k8s-auth calling removed v1beta1 api

[rwong2888]

Using argocd 2.4.2

I am on gke 1.21 and trying to prepare for 1.22 where apis have been removed.

I recently have implemented argocd-k8s-auth as mentioned here, here, and here.

In my gke audit logs, I am seeing our argo service account with workload identity call v1beta1 every 10 minutes starting on the day argocd-k8s-auth was implemented

authorizationInfo: [
0: {
granted: true
permission: "io.k8s.extensions.v1beta1.ingresses.watch"
resource: "extensions/v1beta1/ingresses"
}]
methodName: "io.k8s.extensions.v1beta1.ingresses.watch"
requestMetadata: {
callerIp: "35.194.76.176"
callerSuppliedUserAgent: "argocd-application-controller/v0.0.0 (linux/amd64) kubernetes/$Format"
}

I believe this to be a blocker to upgrading to gke 1.22 and would like argocd to use the ga v1 apis.

[dspeck1]

We are seeing the same issue with these beta APIs, GKE and argo.

/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings
/apis/extensions/v1beta1/ingresses

[jannfis]

This is the normal behavior of the application controller. It establishes a watch on all available resources in the clusters it manages. It is not an API used by the application controller, or any other Argo CD component.

As long as none of your applications manages a deprecated resource, you can safely upgrade your cluster version and the messages will disappear (because the API is not available anymore).

[rwong2888]

@jannfis can you explain why I didn't see logs for it before setting up argocd-k8s-auth?

[toVersus]

@rwong2888
The application controller is calling the Ingress v1beta1 APIs regardless of setting up argocd-k8s-auth or not. You can see the same logs described in #8121 in either case. The difference is a caller of the deprecated APIs. Before using argocd-k8s-auth mechanism, I guess your application controller was calling the deprecated APIs using the Bearer token of the argocd-manager service account in kube-system namespace automatically created by argocd CLI (i.e. argocd cluster add command). After setting up argocd-k8s-auth, your application controller has started to call it using GCP service account binding to the Kubernetes service account.

Then, there are three different problems here:

1. Recently, GKE introduced the new feature called Deprecation Insights, which prevents your cluster from automatically upgrading to GKE 1.22 since application controller is calling the deprecated APIs. But as @jannfis said, unless you manage deprecated resources using Argo CD or other controllers on your clusters call the deprecated APIs explicitly, you can safely upgrade your clusters to GKE 1.22 or above without any problems. But, you should manually upgrade the cluster becuase GKE pauses the automatic upgrades. (We manually upgraded our clusters to GKE 1.22 last week without any troubles.)

2. Deprecation Insights feature is in preview and has some issues. One thing is that we cannot ensure that the caller of the Deprecated APIs detected by Deprecation Insights due to the following constraints GCP support told us:

   Admin Activity / Audit log does not turn on all k8s request logging by default. However, our deprecation notification scans for all usages of deprecated API usage.

   I'm not sure but the warning messages from the client-go (extensions/v1beta1 Ingress deprecation #8121) are the most useful way to verify the suspicious clients calling the deprecated APIs on your cluster at this writing.

3. We see the same GKE Audit logs after migrating to argocd-k8s-auth and the message shows Connection closed early. These logs are not limited to the Deprecated APIs and you can see the same logs for other stable APIs. I'm not sure but something wrong is happened on the application controller. The argocd-k8s-auth periodically refreshes the token to communicate with the Kubernetes API server so we might need to add a caching logic to argocd-k8s-auth and reuse the same token unless it expires as gke-gcloud-auth-plugin does, or do we need to implement a graceful reconnection logic for watch API?

I think 1 and 2 are not related to the Argo CD, so you should ask the GCP support if needed. But regarding 3, it can be improved by Argo CD side. (I think it is better to create a new Issue for 3.)

[rwong2888]

Thank you @toVersus that was very thorough. Funny you mention #3 as I have been seeing that as well. Opened 9794

I will manually upgrade to 1.22 and report back here.

[rwong2888]

I am still seeing some hits on a remote cluster post upgrade for the connection closed early. I don't think it is a big deal. Do I have to upgrade my argocd cluster to 1.22 to remove completely @toVersus ?

[toVersus]

I am still seeing some hits on a remote cluster post upgrade for the connection closed early.

Ah sorry, I forgot to share the most important thing. That's because the GKE reverts the removal of Ingress v1beta1 API since 1.22.7-gke.300. (I think this life-prolonging treatment taken by GKE ended up causing a lot of confusion.) So, you will continue to see the connection closed early audit logs until GKE 1.23 or later. Also, the deprecation insights may still show the alerts after upgrade. We confirm that Migrate to supported APIs messages are displayed on some of our GKE 1.22 clusters view. As I said earlier, you can safely ignore those messages.

https://cloud.google.com/kubernetes-engine/docs/deprecations/apis-1-22#ingress-v122
GKE clusters that upgrade from GKE version 1.21 and earlier to versions 1.22.7-gke.300 and later can still use the Beta API versions until version 1.23.

The available APIs on v1.22.8-gke.202 cluster:

❯ kubectl version
(...)
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8-gke.202", GitCommit:"88deae00580af268497b9656f216cb092b630563", GitTreeState:"clean", BuildDate:"2022-06-03T03:27:52Z", GoVersion:"go1.16.14b7", Compiler:"gc", Platform:"linux/amd64"}

# The deprecated extensions/v1beta1 Ingress API is still listed
❯ kubectl api-resources | grep ingresses
ingresses                         ing                extensions/v1beta1                     true         Ingress
ingresses                         ing                networking.k8s.io/v1                   true         Ingress

Do I have to upgrade my argocd cluster to 1.22 to remove completely

No, the calls of Ingress v1beta1 APIs on a remote cluster has nothing to do with the version of Argo CD cluster. If you upgrade your Argo CD cluster to 1.22, you will still see the same audit logs or deprecation insights message on a remote cluster.

[rwong2888]

Thank you @toVersus , that was very thorough. Closing the issue