Allow changing the livenessPrope settings for prometheus exporter

[benediktarnold]

Hi!

My prometheus exporter pod was constantly restarting because the liveness probe timed out. I've noticed a high cpu and memory usage on the exporter pod as well even though I'm only scraping a 3 node solr cluster. Since #83 does not allow to change the liveness probe settings on SolrPrometheusExporter resources, I've deployed the pod manually with a much less aggressive liveness probe configuration like:

    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /metrics
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 20
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 5

With that setting the CPU and memory usage decreased a lot. Please see the attached grafana screenshot of the cpu and memory usage of the prometheus exporter pod. Green is the pod deployed with SolrPrometheusExporter and orange is the pod I've deployed manually with the above settings.

Please allow to overwrite livenessProbe settings on SolrPrometheusExporter resources.

Keep up the good work and thanks very much!
Ben

[HoustonPutman]

Thanks for suggesting this @benediktarnold , and especially backing it up with such clear results.

I think that adding support for changing all the probes for the PrometheusExporter is a no-brainer, and should be very simple to do. But if the performance using the default probes is this bad, we should probably look at options for changing the default behavior as well.

Maybe one solution is to add an actual health-check handler to the PrometheusExporter itself (in Solr), that just checks that the metrics cache is healthy, and the jetty server responds.

A second question, what version of the prometheus exporter are you using? It seems weird to me that additional calls to /metrics would be that expensive, given that all the metrics should be cached in the first place. Maybe it's very expensive to generate the long response?

[benediktarnold]

Maybe one solution is to add an actual health-check handler to the PrometheusExporter itself (in Solr), that just checks that the metrics cache is healthy, and the jetty server responds.

That seems to be a good idea. I'll look into the code and will come up with a PR or at least an issue in the solr repo.

A second question, what version of the prometheus exporter are you using? It seems weird to me that additional calls to /metrics would be that expensive, given that all the metrics should be cached in the first place. Maybe it's very expensive to generate the long response?

I thought so as well. I'm using the exporter that comes with the solr 8.7.0 Images. I'm about to upgrade to 8.9.0 and I'll let you know if that brings any advantages. I made a snapshot of a metric response once and it was about 24MB (~100 small collections on 3 solr nodes). The peak CPU usage of the exporter is far bigger than the CPU usage of my whole solr cluster.