Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scala Stream Collector: expose Prometheus metrics #3421

Closed
christoph-buente opened this issue Sep 7, 2017 · 2 comments
Closed

Scala Stream Collector: expose Prometheus metrics #3421

christoph-buente opened this issue Sep 7, 2017 · 2 comments

Comments

@christoph-buente
Copy link
Contributor

In order to monitor the parts of the real-time stack, it would be very beneficial to have a way to monitor each step of the pipeline. For example, how many requests got into the collector, how much got out to kinesis, how many instances are running in each stage? I know we can send some metrics into the collector itself, but it feels wrong to send data to an endpoint, that i actually want to monitor itself.

I would really like to see support for prometheus for collector, enricher and storage sinks. All it requires is a http server listening to a given port and expose some metrics in pre-defined text format.

What do you think?

https://github.com/prometheus/prometheus

@alexanderdean
Copy link
Member

Yes good idea, we have had customers asking for this... Any suggestions on the metrics?

@alexanderdean alexanderdean changed the title Add support for prometheus metrics Placeholder for adding support for Prometheus metrics Sep 7, 2017
@christoph-buente
Copy link
Contributor Author

christoph-buente commented Sep 8, 2017

For the collector, i think the usual suspects for any http endpoint:

  • http_requests_total{path, code, method}
  • http_request_duration_seconds_count
  • http_request_duration_seconds_sum
  • http_request_duration_seconds_bucket(histogram percentiles)
  • http_request_size_bytes
  • http_request_size_bytes_count
  • http_request_size_bytes_sum

Also for all parts reading from and writing to kinesis/kafka

  • kinesis_events_write_total{type,format,retry,failure/success}
  • kinesis_events_read_total{type,format,retry,failure/success}

For all parts in common:

  • build_info{scalaversion,javaversion,version,gitsha}

Add anything that the parts send as data metrics to the collector right now.

@BenFradet BenFradet changed the title Placeholder for adding support for Prometheus metrics Scala Stream Collector: expose Prometheus metrics Feb 11, 2019
lukeindykiewicz pushed a commit to snowplow/stream-collector that referenced this issue Jun 4, 2020
lukeindykiewicz pushed a commit to snowplow/stream-collector that referenced this issue Jun 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants