Add webhook to trigger status reconciliations faster

[a-thaler]

Description
As part of #425 the detection of a bad situation by prometheus should result in an immediate status change in the related k8s resource. As the manager is pulling prometheus, that can take up to one minute. Goal is to introduce a push notification by leveraging the alertmanager protocol to have an immediate reaction in the manager.

Details

• Have a dedicated a path exposing a webhook
• Configure prometheus to use the webhook as alertmanager endpoint
• The webhook will trigger reconcilation
• Have a metric counting the invocations, being labeled with the alert type
• Have a test assuring that the webhook gets called, verified by metrics

Criterias

• An alert on prometheus gets propagated immediately (before you had to wait for next reconcilation)
• Operator can see usage of webhook via metrics
• the self-monitor is allowed to access the manager port of the webhook only, no other port

Links
https://github.com/kyma-project/telemetry-manager/pull/753/files

[skhalash]

All acceptance criteria were implemented except the last one:

• Since it's not feasible to whitelist Kubernetes API server IP in the self-monitor network policy, the existing "loose" policy is left unchanged.