Enhance metrics endpoint

[prashanth26]

Issue

The metric endpoints currently only expose the machine objects managed by the machine controller manager. This could be more useful to track MCM freezing and # of API calls

Solution

The metrics endpoint should expose more detailed metrics for MCM.
Like

• # of machines, machineSet, machineDeployment
• # of API calls to the cloud provider
• Freeze status of MCM

[fsniper]

I was also working on this issue and may be the metrics from the below list can also be considered? I am ready to start on this and try to add some metrics if you have not already started working on this.

* crd:machinedeployment: count, age|created, availableReplicas, readyReplicas, replicas, updatedReplicas
* crd:machineset: count, availableReplicas, failedMachines, fullyLabeledReplicas, readyReplicas, replicas, age|created
* crd:machine: age|created, count, conditions, status

[prashanth26]

Hi @fsniper ,

We are happy to have you pitch in. You are more than welcome to take this issue up. The metrics you have mentioned above does make sense to me. In addition to that, if you are able to track the # of API calls to cloud provider (keeping a global atomic integer counter?) and also expose the freeze status of the controller it would be great.

Thanks & Regards,
Prashanth

[fsniper]

• # of API calls to the cloud provider
• # of machines, machineSet, machineDeployment
• machine info, created (age), conditions, phase
• machineset info, created (age), status, failed_machines
• machinedeployment: info, created (age), status, conditions, failed_machines
• Freeze status of MCM

[prashanth26]

Reopening issue to have a second look at this, and see how they can be helpful while alerting.

[prashanth26]

Closing in favor of - https://app.zenhub.com/workspace/o/gardener/machine-controller-manager/issues/211