Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix timer/histogram metrics #41

Closed
pjenvey opened this issue Jul 26, 2018 · 2 comments
Closed

Fix timer/histogram metrics #41

pjenvey opened this issue Jul 26, 2018 · 2 comments
Assignees
Labels
bug Something isn't working p1

Comments

@pjenvey
Copy link
Member

pjenvey commented Jul 26, 2018

autopush-rs's metrics go through dogstatsd vs the direct datadog HTTP API like autopush-py.

For whatever reason:

Note: DogStatsD does NOT implement the following from StatsD:

    Gauge deltas (see this issue)
    Timers as a native metric type (though it does support them via histograms)

Even though Timers, which exist in StatsD, are a sub-set of histograms in DogStatsD.

I'm not sure why dogstatsd doesn't handle timers for us (I imagine that's what the python datadog is doing for us under the covers/via the HTTP API), so we'll need to switch to the histogram type ourselves

@pjenvey pjenvey added bug Something isn't working p1 labels Jul 26, 2018
@pjenvey pjenvey self-assigned this Jul 26, 2018
@pjenvey pjenvey changed the title Switch timer metrics to histogram Fix timer/histogram metrics Jul 27, 2018
@pjenvey
Copy link
Member Author

pjenvey commented Jul 27, 2018

#42 switched to histogram yet we continue to not see either type show up on datadog

@pjenvey
Copy link
Member Author

pjenvey commented Aug 1, 2018

The difference w/ dogstatsd is it allows overriding the datadog API host field via a tag: https://help.datadoghq.com/hc/en-us/articles/218349043-How-to-remove-the-host-tag-when-submitting-metrics-via-dogstatsD

We include a host tag w/ the lifespan metric. In rust the value's the node's internal IP address. It looks like this causes other tags (most importantly 'language:rust' and 'env:prod') to become disassociated with this overridden host (the metrics are there under these overriden raw IPs host)

Python autopush specifies the Host header from the request as this value instead, always something like "push.services.mozilla.com". The python datadog lib does something different here, according to the datadog ui it seems to duplicate the metrics for both the host tag we specify and its regular host value (ec2 instance id).

Long story short: we shouldn't include a host tag in either impl., "push.services.mozilla.com" isn't a useful data point. Nor is it useful in our Session INFO log. The MozLogs format includes a Hostname field anyway (exact value of datadog's)

pjenvey added a commit that referenced this issue Aug 2, 2018
the tag overwrites datadog's API host field. it isn't a useful value
for metrics or logs anyway

Closes #41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working p1
Projects
None yet
Development

No branches or pull requests

2 participants