Prometheus Never Prints the Buffer

[grippy]

Been scratching my head at why I'm not seeing metrics sent to Prometheus.

Looks like this strbuf is created for the metrics but never passed to `self.flush_inner
https://github.com/fralalonde/dipstick/blob/master/src/output/prometheus.rs#L104
https://github.com/fralalonde/dipstick/blob/master/src/output/prometheus.rs#L137

I'm not familiar with the inner workings on this library to submit a patch just yet. If I can figure it out, I'll update this Issue.

[fralalonde]

I'll look right into it.

[fralalonde]

Looks like a refactoring's gone wrong somewhere along the way. I've fixed the buffer issue and will be making sure that proper http requests are made. I'll publish a crate update right after that.

[grippy]

@fralalonde +1 I just fixed this locally... I don't know what you had before but this is what I ended up with...

Please note: the prometheus pushgateway is returning 400 w/ a body of "text format parsing error in line 1: unexpected end of input stream\n" because of the missing '\n' at the end up of the buffer.

    fn print(&self, metric: &PrometheusMetric, value: MetricValue, labels: Labels) {

        let scaled_value = value / metric.scale;
        let value_str = scaled_value.to_string();

        let mut buffer = self.buffer.borrow_mut();

        // prometheus format be like `http_requests_total{method="post",code="200"} 1027 1395066363000`
        buffer.push_str(&metric.prefix);

        let labels_map = labels.into_map();
        if !labels_map.is_empty() {
            buffer.push('{');
            let mut i = labels_map.into_iter();
            let mut next = i.next();
            while let Some((k, v)) = next {
                buffer.push_str(&k);
                buffer.push_str("=\"");
                buffer.push_str(&v);
                next = i.next();
                if next.is_some() {
                    buffer.push_str("\",");
                } else {
                    buffer.push('"');
                }
            }
            buffer.push_str("} ");
        } else {
            buffer.push(' ');
        }
        buffer.push_str(&value_str);
        buffer.push('\n');

        if buffer.len() > BUFFER_FLUSH_THRESHOLD {
            metrics::PROMETHEUS_OVERFLOW.mark();
            warn!(
                "Prometheus Buffer Size Exceeded: {}",
                BUFFER_FLUSH_THRESHOLD
            );
            let _ = self.flush_inner(buffer);
        } else if !self.is_buffered() {
            if let Err(e) = self.flush_inner(buffer) {
                debug!("Could not send to Prometheus {}", e)
            }
        }
    }

[fralalonde]

I just released v0.7.7, let me know how it runs... I'm happy to have someone helping me test Prometheus!

[fralalonde]

Make that 0.7.8...

[grippy]

@fralalonde One more issue if you didn't fix it:

We should change the http call to post (instead of get which isn't supported from what I can tell).
https://github.com/prometheus/pushgateway/blob/master/README.md#put-method
https://github.com/fralalonde/dipstick/blob/master/src/output/prometheus.rs#L163

I'll try out your new release! Thanks for the fix.

I might have some docs coming your way. I've been playing around with using the Proxy => AtomicBucket => MultiOutput pipelines. The kicker here is you need to you use flush otherwise the Prometheus print code panics trying to use self.buffer.borrow_mut() if you hammer the pushgateway with too many metrics.

It took me a little while to figure out. Might as well pass it on.

[grippy]

Btw, still seeing this error in the pushgateway response:

TRACE dipstick::output::prometheus > 400 "text format parsing error in line 1: unexpected end of input stream\n"

See the PR here:
#71

[fralalonde]

I merged your PR, made some edits to the handbook regarding Proxy and published a new version. The new example you included is pretty nice too. I should document the other examples the same way.

[grippy]

This looks good! Thanks for the quick turnaround.