Add a counter for API server watch failures

[nckturner]

• New metrics package
• Use new registry in test setup to avoid duplicate metric registration

Replaces: #387

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nckturner

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [nckturner]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[wongma7]

expose the metrics over http?

https://pkg.go.dev/github.com/prometheus/client_golang/prometheus/promauto

func main() {
go Random()
http.Handle("/metrics", promhttp.Handler())
http.ListenAndServe(":1971", nil)
}

[wongma7]

or i guess they are already exposed but show a curl example : D

[nckturner]

With MountedFile:

$ curl -k https://localhost:21362/metrics | grep "configmap_watch_failures"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                                                                            
                                 Dload  Upload   Total   Spent    Left  Speed                                                                                                                              
100  7915    0  7915    0     0   # HELP aws_iam_authenticator_configmap_watch_failures EKS Configmap watch failures                                                                                       
85# TYPE aws_iam_authenticator_configmap_watch_failures counter                                                                                                                                            
8aws_iam_authenticator_configmap_watch_failures 0                                                                                                                                                          
k      0 --:--:-- --:--:-- --:--:--  858k        

And with EKSConfigMap, but API server not running, after a few seconds:

$ curl -k https://localhost:21362/metrics |grep "configmap_watch_failures"                                                                            [22:53:08]
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                                                                            
                                 Dload  Upload   Total   Spent    Left  Speed                                                                                                                              
100  6416    0  6416# HELP aws_iam_authenticator_configmap_watch_failures EKS Configmap watch failures                                                                                                     
  # TYPE aws_iam_authenticator_configmap_watch_failures counter                                                                                                                                            
  aws_iam_authenticator_configmap_watch_failures 2                                                                                                                                                         

[wongma7]

/lgtm

[jyotimahapatra]

/lgtm