Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Sets initial count of task finished state to zero. This enables acquiring the rate from zero to one (particularly useful if you want to alert on any failures). We're using the Prometheus statsd-exporter. Since counters are usually used with a PromQL function like `rate`, it's important that counters are initialized at zero, otherwise when a task finishes the rate function will not have a previous value to compare the state count to. For example, what we'd like to do: ``` sum by (dag_id, task_id) (rate(airflow_ti_finish{state='failed'}[1h])) > 0 ``` This tells us the failure rate of tasks over time. What I've tried to do instead to ensure the metric captures the change from zero to one: ``` (sum by (dag_id, task_id) (rate(airflow_ti_finish{state='failed'}[1h])) > 0) or sum by (dag_id, task_id) (airflow_ti_finish{state='failed'} != 0 unless (airflow_ti_finish{state='failed'} offset 1m)) ``` Two useful posts on this subject: https://www.robustperception.io/why-predeclare-metrics https://www.section.io/blog/beware-prometheus-counters-that-do-not-begin-at-zero/
- Loading branch information