Scala Stream Collector: expose Prometheus metrics

[christoph-buente]

In order to monitor the parts of the real-time stack, it would be very beneficial to have a way to monitor each step of the pipeline. For example, how many requests got into the collector, how much got out to kinesis, how many instances are running in each stage? I know we can send some metrics into the collector itself, but it feels wrong to send data to an endpoint, that i actually want to monitor itself.

I would really like to see support for prometheus for collector, enricher and storage sinks. All it requires is a http server listening to a given port and expose some metrics in pre-defined text format.

What do you think?

https://github.com/prometheus/prometheus

[alexanderdean]

Yes good idea, we have had customers asking for this... Any suggestions on the metrics?

[christoph-buente]

For the collector, i think the usual suspects for any http endpoint:

• http_requests_total{path, code, method}
• http_request_duration_seconds_count
• http_request_duration_seconds_sum
• http_request_duration_seconds_bucket(histogram percentiles)
• http_request_size_bytes
• http_request_size_bytes_count
• http_request_size_bytes_sum

Also for all parts reading from and writing to kinesis/kafka

• kinesis_events_write_total{type,format,retry,failure/success}
• kinesis_events_read_total{type,format,retry,failure/success}

For all parts in common:

• build_info{scalaversion,javaversion,version,gitsha}

Add anything that the parts send as data metrics to the collector right now.