Create Prometheus alerts for applications health

[ghost]

Our current Prometheus installation has alerts for default Kubernetes cluster health like CPU and Memory usage and Pods status but we don't have alerts for the internals of applications like:

• Nginx response time
• Zookeeper quorum Size (suggestions)
• Kafka active brokers
• HDFS (suggestions)

We should create those alerts to get notified in case of any issue as soon as possible.

(Please add any other metrics and alerts that you think is necessary)

[yatharthranjan]

Some things that could be useful -

• Kafka Connector Healths.
• HDFS health (Number of live data nodes) and Capacity remaining.
• Disk usage for block devices.
• HDFS restructure time taken and run failure.
• Kafka brokers throughput and multiple topic under replication.Although these are provided in kafka-manager too but no alerting there.

Edit:

• Kafka producer and consumer metrics could also be useful. For example, consumer lag.