Error with permissions

[rajeshkodali]

kubectl get hpa
sqs-consumer-scaler Deployment/<> /30 (avg) 1 10 1 4h

Warning   FailedComputeMetricsReplicas   horizontal-pod-autoscaler                               failed to get <<name>> external metric: unable to get external metric default/<<name>>/nil: unable to fetch metrics from external metrics API: <<name>>.external.metrics.k8s.io is forbidden: User "system:anonymous" cannot list resource "<<resource>>" in API group "external.metrics.k8s.io" in the namespace "default"

[chankh]

looks like it is missing some permissions, did you deploy the full adapter.yaml? There are service account, role bindings defined.

[rajeshkodali]

Yes, I did.

Here are role bindings and roles:

k8s-cloudwatch-adapter-resource-reader 6d
k8s-cloudwatch-adapter:crd-metrics-reader 6d
k8s-cloudwatch-adapter:external-metrics-reader 6d
k8s-cloudwatch-adapter:system:auth-delegator 6d

[joesbigidea]

We experienced this issue as well. EKS with K8s 1.14. We reverted to d385146 as that had worked for us in the past with the same setup and it resolved the issue.

[chankh]

Hi @joesbigidea do you mean it worked fine with v0.8.0?

[joesbigidea]

That's correct, the head version with 0.9.0 gave us this issue:
Warning FailedComputeMetricsReplicas 1s horizontal-pod-autoscaler failed to get external metric <<name>>: unable to get external metric custom-metrics/<<name>>/nil: unable to fetch metrics from external metrics API: <<name>>.external.metrics.k8s.io is forbidden: User "system:anonymous" cannot list resource "<<name>>" in API group "external.metrics.k8s.io" in the namespace "custom-metrics"

Reverting to the version with v0.8.0 resolved the issue. It's working fine with that version.

[rajeshkodali]

Digging deeper looks like kiam is blocking the requests. I have created image with curl which gave me hints on what's happening. Now taking it again from the top.

After reverting to previous version, v0.8.0 everything works.