From 93abf6b6f8245f5bcf43a9a1d5deb7be9dc44272 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 10:54:20 +0200 Subject: [PATCH 01/26] Add blurb to readme about identifying commits --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 2d70b23..23ad6e3 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,7 @@ See [Why Autometrics?](https://github.com/autometrics-dev#why-autometrics) for m - 💡 Writes Prometheus queries so you can understand the data generated without knowing PromQL - 🔗 Create links to live Prometheus charts directly into each functions docstrings (with tooltips coming soon!) +- [🔍 Identify commits](#identifying-commits-that-introduced-problems) that introduced errors or increased latency - [🚨 Define alerts](#alerts--slos) using SLO best practices directly in your source code - [📊 Grafana dashboards](#dashboards) work out of the box to visualize the performance of instrumented functions & SLOs - [⚙️ Configurable](#metrics-libraries) metric collection library (`opentelemetry`, `prometheus`, or `metrics`) From 49f6d90dc9fa6a5ab949c1219659227afac8a595 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 10:55:16 +0200 Subject: [PATCH 02/26] Remove "coming soon" from readme item on adding links to live Prom charts --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 23ad6e3..108d59d 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ See [Why Autometrics?](https://github.com/autometrics-dev#why-autometrics) for m most useful metrics - 💡 Writes Prometheus queries so you can understand the data generated without knowing PromQL -- 🔗 Create links to live Prometheus charts directly into each functions docstrings (with tooltips coming soon!) +- 🔗 Create links to live Prometheus charts directly into each functions docstrings - [🔍 Identify commits](#identifying-commits-that-introduced-problems) that introduced errors or increased latency - [🚨 Define alerts](#alerts--slos) using SLO best practices directly in your source code - [📊 Grafana dashboards](#dashboards) work out of the box to visualize the performance of instrumented functions & SLOs From 41d764922ca54a8582342f0bfc29827268f3a01e Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 11:00:40 +0200 Subject: [PATCH 03/26] Add WIP section to the readme on using build info (version/commit) --- README.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 108d59d..0ab1e58 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ See [Why Autometrics?](https://github.com/autometrics-dev#why-autometrics) for m most useful metrics - 💡 Writes Prometheus queries so you can understand the data generated without knowing PromQL -- 🔗 Create links to live Prometheus charts directly into each functions docstrings +- 🔗 Create links to live Prometheus charts directly into each function's docstring - [🔍 Identify commits](#identifying-commits-that-introduced-problems) that introduced errors or increased latency - [🚨 Define alerts](#alerts--slos) using SLO best practices directly in your source code - [📊 Grafana dashboards](#dashboards) work out of the box to visualize the performance of instrumented functions & SLOs @@ -115,6 +115,22 @@ Configure the crate that autometrics will use to produce metrics by using one of - `opentelemetry` - (enabled by default, can also be explicitly set using the AUTOMETRICS_TRACKER="OPEN_TELEMETERY" env var) uses - `prometheus` -(using the AUTOMETRICS_TRACKER env var set to "PROMETHEUS") +## Identifying commits that introduced problems + +Autometrics makes it easy to identify if a specific version or commit introduced errors or increased latencies. + +It uses a separate metric (`build_info`) to track the version and, optionally, git commit of your service. It then writes queries that group metrics by the `version` and `commit` labels so you can spot correlations between those and potential issues. + +The `version` is collected from the `...TODO...` environment variable. You can override this by setting the runtime environment variable `AUTOMETRICS_VERSION`. + +This follows the method outlined in [Exposing the software version to Prometheus](https://www.robustperception.io/exposing-the-software-version-to-prometheus/). + +To set the `commit`, you can either set the run-time environment variable `AUTOMETRICS_COMMIT`, or have it set automatically using `...TODO...` + +```py +# TODO +``` + ## Development of the package This package uses [poetry](https://python-poetry.org) as a package manager, with all dependencies separated into three groups: From cd381709fad910bba01f1b462f94e67ac16edbf0 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 11:03:42 +0200 Subject: [PATCH 04/26] Add constants --- src/autometrics/constants.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/autometrics/constants.py b/src/autometrics/constants.py index 025228a..f3d5bc9 100644 --- a/src/autometrics/constants.py +++ b/src/autometrics/constants.py @@ -2,17 +2,22 @@ COUNTER_NAME = "function.calls.count" HISTOGRAM_NAME = "function.calls.duration" +BUILD_INFO_NAME = "build.info" COUNTER_NAME_PROMETHEUS = COUNTER_NAME.replace(".", "_") HISTOGRAM_NAME_PROMETHEUS = HISTOGRAM_NAME.replace(".", "_") +BUILD_INFO_NAME_PROMETHEUS = BUILD_INFO_NAME.replace(".", "_") COUNTER_DESCRIPTION = "Autometrics counter for tracking function calls" HISTOGRAM_DESCRIPTION = "Autometrics histogram for tracking function call duration" +BUILD_INFO_DESCRIPTION = "Autometrics info metric for tracking software version and build details" # The following constants are used to create the labels OBJECTIVE_NAME = "objective.name" OBJECTIVE_PERCENTILE = "objective.percentile" OBJECTIVE_LATENCY_THRESHOLD = "objective.latency_threshold" +VERSION_KEY = "version" +COMMIT_KEY = "commit" # The values are updated to use underscores instead of periods to avoid issues with prometheus. # A similar thing is done in the rust library, which supports multiple exporters From 3d5cc9ce7d3b6a33ee33e73008e053b699a606be Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 11:10:42 +0200 Subject: [PATCH 05/26] Initialize Prometheus Gauge for build_info --- src/autometrics/tracker/prometheus.py | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/src/autometrics/tracker/prometheus.py b/src/autometrics/tracker/prometheus.py index 855f1ff..a40386c 100644 --- a/src/autometrics/tracker/prometheus.py +++ b/src/autometrics/tracker/prometheus.py @@ -1,16 +1,20 @@ import time from typing import Optional -from prometheus_client import Counter, Histogram +from prometheus_client import Counter, Histogram, Gauge from .tracker import Result from ..constants import ( COUNTER_NAME_PROMETHEUS, HISTOGRAM_NAME_PROMETHEUS, + BUILD_INFO_NAME_PROMETHEUS, COUNTER_DESCRIPTION, HISTOGRAM_DESCRIPTION, + BUILD_INFO_DESCRIPTION, OBJECTIVE_NAME_PROMETHEUS, OBJECTIVE_PERCENTILE_PROMETHEUS, OBJECTIVE_LATENCY_THRESHOLD_PROMETHEUS, + COMMIT_KEY, + VERSION_KEY ) from ..objectives import Objective @@ -41,6 +45,14 @@ class PrometheusTracker: OBJECTIVE_LATENCY_THRESHOLD_PROMETHEUS, ], ) + prom_gague = Gauge( + BUILD_INFO_NAME_PROMETHEUS, + BUILD_INFO_DESCRIPTION, + [ + COMMIT_KEY, + VERSION_KEY + ] + ) def _count( self, @@ -93,6 +105,10 @@ def _histogram( threshold, ).observe(duration) + def _build_info(self): + """Observe the build info.""" + pass + # def start(self, function: str = None, module: str = None): # """Start tracking metrics for a function call.""" # pass From 3cee1694e56bb913910504b4282f45dc51ea58b5 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 11:18:33 +0200 Subject: [PATCH 06/26] Add updown counter for build info to otel tracker --- src/autometrics/constants.py | 5 ++++- src/autometrics/tracker/opentelemetry.py | 12 ++++++++++++ src/autometrics/tracker/prometheus.py | 9 ++------- 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/src/autometrics/constants.py b/src/autometrics/constants.py index f3d5bc9..b45e6c9 100644 --- a/src/autometrics/constants.py +++ b/src/autometrics/constants.py @@ -2,6 +2,7 @@ COUNTER_NAME = "function.calls.count" HISTOGRAM_NAME = "function.calls.duration" +# NOTE - The Rust implementation does not use `build.info`, instead opts for just `build_info` BUILD_INFO_NAME = "build.info" COUNTER_NAME_PROMETHEUS = COUNTER_NAME.replace(".", "_") @@ -10,7 +11,9 @@ COUNTER_DESCRIPTION = "Autometrics counter for tracking function calls" HISTOGRAM_DESCRIPTION = "Autometrics histogram for tracking function call duration" -BUILD_INFO_DESCRIPTION = "Autometrics info metric for tracking software version and build details" +BUILD_INFO_DESCRIPTION = ( + "Autometrics info metric for tracking software version and build details" +) # The following constants are used to create the labels OBJECTIVE_NAME = "objective.name" diff --git a/src/autometrics/tracker/opentelemetry.py b/src/autometrics/tracker/opentelemetry.py index 6c13096..f41a89d 100644 --- a/src/autometrics/tracker/opentelemetry.py +++ b/src/autometrics/tracker/opentelemetry.py @@ -4,6 +4,7 @@ Meter, Counter, Histogram, + UpDownCounter, set_meter_provider, ) @@ -21,6 +22,8 @@ COUNTER_NAME, HISTOGRAM_DESCRIPTION, HISTOGRAM_NAME, + BUILD_INFO_NAME, + BUILD_INFO_DESCRIPTION, OBJECTIVE_NAME, OBJECTIVE_PERCENTILE, OBJECTIVE_LATENCY_THRESHOLD, @@ -39,6 +42,7 @@ class OpenTelemetryTracker: __counter_instance: Counter __histogram_instance: Histogram + __up_down_counter_instance: UpDownCounter def __init__(self): exporter = PrometheusMetricReader("") @@ -60,6 +64,10 @@ def __init__(self): name=HISTOGRAM_NAME, description=HISTOGRAM_DESCRIPTION, ) + self.__up_down_counter_instance = meter.create_up_down_counter( + name=BUILD_INFO_NAME, + description=BUILD_INFO_DESCRIPTION, + ) def __count( self, @@ -116,6 +124,10 @@ def __histogram( }, ) + def _build_info(self): + """Observe the build info.""" + pass + def finish( self, start_time: float, diff --git a/src/autometrics/tracker/prometheus.py b/src/autometrics/tracker/prometheus.py index a40386c..ff5a876 100644 --- a/src/autometrics/tracker/prometheus.py +++ b/src/autometrics/tracker/prometheus.py @@ -14,7 +14,7 @@ OBJECTIVE_PERCENTILE_PROMETHEUS, OBJECTIVE_LATENCY_THRESHOLD_PROMETHEUS, COMMIT_KEY, - VERSION_KEY + VERSION_KEY, ) from ..objectives import Objective @@ -46,12 +46,7 @@ class PrometheusTracker: ], ) prom_gague = Gauge( - BUILD_INFO_NAME_PROMETHEUS, - BUILD_INFO_DESCRIPTION, - [ - COMMIT_KEY, - VERSION_KEY - ] + BUILD_INFO_NAME_PROMETHEUS, BUILD_INFO_DESCRIPTION, [COMMIT_KEY, VERSION_KEY] ) def _count( From 936ed06ef8bfbf5a0592405191c4eda07c6f8f73 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 12:00:03 +0200 Subject: [PATCH 07/26] Implement set_build_info for OTEL and Prom, and call when we set the default tracker --- src/autometrics/tracker/opentelemetry.py | 10 ++++++++-- src/autometrics/tracker/prometheus.py | 4 ++-- src/autometrics/tracker/tracker.py | 6 ++++++ 3 files changed, 16 insertions(+), 4 deletions(-) diff --git a/src/autometrics/tracker/opentelemetry.py b/src/autometrics/tracker/opentelemetry.py index f41a89d..b1d9fb2 100644 --- a/src/autometrics/tracker/opentelemetry.py +++ b/src/autometrics/tracker/opentelemetry.py @@ -124,9 +124,15 @@ def __histogram( }, ) - def _build_info(self): + def set_build_info(self, commit: str, version: str): """Observe the build info.""" - pass + self.__up_down_counter_instance.add( + 1.0, + attributes={ + "commit": commit, + "version": version, + }, + ) def finish( self, diff --git a/src/autometrics/tracker/prometheus.py b/src/autometrics/tracker/prometheus.py index ff5a876..b7c902e 100644 --- a/src/autometrics/tracker/prometheus.py +++ b/src/autometrics/tracker/prometheus.py @@ -100,9 +100,9 @@ def _histogram( threshold, ).observe(duration) - def _build_info(self): + def set_build_info(self, commit: str, version: str): """Observe the build info.""" - pass + self.prom_gague.labels(commit, version).set(1.0) # def start(self, function: str = None, module: str = None): # """Start tracking metrics for a function call.""" diff --git a/src/autometrics/tracker/tracker.py b/src/autometrics/tracker/tracker.py index 307dc17..ceb7aa8 100644 --- a/src/autometrics/tracker/tracker.py +++ b/src/autometrics/tracker/tracker.py @@ -65,6 +65,12 @@ def default_tracker(): tracker: TrackMetrics = default_tracker() +# FIXME - move somewhere else +tracker.set_build_info( + commit=os.getenv("AUTOMETRICS_COMMIT") or "", + version=os.getenv("AUTOMETRICS_VERSION") or "", +) + def get_tracker() -> TrackMetrics: """Get the tracker type.""" From 44ac3c513ad699df4bfe4ecb268e2cfa5bf564cc Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 12:03:04 +0200 Subject: [PATCH 08/26] Update README --- README.md | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 0ab1e58..4826385 100644 --- a/README.md +++ b/README.md @@ -117,19 +117,15 @@ Configure the crate that autometrics will use to produce metrics by using one of ## Identifying commits that introduced problems +> This follows the method outlined in [Exposing the software version to Prometheus](https://www.robustperception.io/exposing-the-software-version-to-prometheus/). + Autometrics makes it easy to identify if a specific version or commit introduced errors or increased latencies. It uses a separate metric (`build_info`) to track the version and, optionally, git commit of your service. It then writes queries that group metrics by the `version` and `commit` labels so you can spot correlations between those and potential issues. The `version` is collected from the `...TODO...` environment variable. You can override this by setting the runtime environment variable `AUTOMETRICS_VERSION`. -This follows the method outlined in [Exposing the software version to Prometheus](https://www.robustperception.io/exposing-the-software-version-to-prometheus/). - -To set the `commit`, you can either set the run-time environment variable `AUTOMETRICS_COMMIT`, or have it set automatically using `...TODO...` - -```py -# TODO -``` +To set the `commit`, expose the run-time environment variable `AUTOMETRICS_COMMIT`. ## Development of the package From 2d21708b5136c17787636d388eced2ab0109c1e2 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 12:41:13 +0200 Subject: [PATCH 09/26] Move set_build_info call into create_tracker --- src/autometrics/tracker/tracker.py | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/src/autometrics/tracker/tracker.py b/src/autometrics/tracker/tracker.py index ceb7aa8..de28961 100644 --- a/src/autometrics/tracker/tracker.py +++ b/src/autometrics/tracker/tracker.py @@ -37,16 +37,26 @@ class TrackerType(Enum): def create_tracker(tracker_type: TrackerType) -> TrackMetrics: """Create a tracker""" + tracker_instance = None if tracker_type == TrackerType.OPENTELEMETRY: # pylint: disable=import-outside-toplevel from .opentelemetry import OpenTelemetryTracker - return OpenTelemetryTracker() + tracker_instance = OpenTelemetryTracker() elif tracker_type == TrackerType.PROMETHEUS: # pylint: disable=import-outside-toplevel from .prometheus import PrometheusTracker - return PrometheusTracker() + tracker_instance = PrometheusTracker() + + # NOTE - Only set the build info when the tracker is initialized + if tracker_instance: + tracker_instance.set_build_info( + commit=os.getenv("AUTOMETRICS_COMMIT") or "", + version=os.getenv("AUTOMETRICS_VERSION") or "", + ) + + return tracker_instance def get_tracker_type() -> TrackerType: @@ -65,12 +75,6 @@ def default_tracker(): tracker: TrackMetrics = default_tracker() -# FIXME - move somewhere else -tracker.set_build_info( - commit=os.getenv("AUTOMETRICS_COMMIT") or "", - version=os.getenv("AUTOMETRICS_VERSION") or "", -) - def get_tracker() -> TrackMetrics: """Get the tracker type.""" From 0d7db6e65e69359de092e0aeef372c0748d8244b Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 13:58:59 +0200 Subject: [PATCH 10/26] Update prometheus queries --- src/autometrics/prometheus_url.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/src/autometrics/prometheus_url.py b/src/autometrics/prometheus_url.py index f05d74f..ec68923 100644 --- a/src/autometrics/prometheus_url.py +++ b/src/autometrics/prometheus_url.py @@ -3,6 +3,8 @@ from typing import Optional from dotenv import load_dotenv +ADD_BUILD_INFO_LABELS = "* on (instance, job) group_left(version, commit) build_info" + def cleanup_url(url: str) -> str: """Remove the trailing slash if there is one.""" @@ -26,9 +28,9 @@ def __init__( def create_urls(self): """Create the prometheus query urls for the function and module.""" - request_rate_query = f'sum by (function, module) (rate (function_calls_count_total{{function="{self.function_name}",module="{self.module_name}"}}[5m]))' - latency_query = f'sum by (le, function, module) (rate(function_calls_duration_bucket{{function="{self.function_name}",module="{self.module_name}"}}[5m]))' - error_ratio_query = f'sum by (function, module) (rate (function_calls_count_total{{function="{self.function_name}",module="{self.module_name}", result="error"}}[5m])) / {request_rate_query}' + request_rate_query = f'sum by (function, module, commit, version) (rate (function_calls_count_total{{function="{self.function_name}",module="{self.module_name}"}}[5m]) {ADD_BUILD_INFO_LABELS})' + latency_query = f'sum by (le, function, module, commit, version) (rate(function_calls_duration_bucket{{function="{self.function_name}",module="{self.module_name}"}}[5m]) {ADD_BUILD_INFO_LABELS})' + error_ratio_query = f'sum by (function, module, commit, version) (rate (function_calls_count_total{{function="{self.function_name}",module="{self.module_name}", result="error"}}[5m]) {ADD_BUILD_INFO_LABELS}) / {request_rate_query}' queries = { "Request rate URL": request_rate_query, From 02ad205f43264df6c25b33ca9fd1c751421aaf86 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 14:24:43 +0200 Subject: [PATCH 11/26] Update prometheus URL tests --- src/autometrics/test_prometheus_url.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/src/autometrics/test_prometheus_url.py b/src/autometrics/test_prometheus_url.py index 256599c..86a7d8c 100644 --- a/src/autometrics/test_prometheus_url.py +++ b/src/autometrics/test_prometheus_url.py @@ -24,11 +24,13 @@ def test_create_prometheus_url_with_default_url(default_url_generator: Generator def test_create_urls_with_default_url(default_url_generator: Generator): urls = default_url_generator.create_urls() - # print(urls.keys()) + print(urls.keys()) + print(urls.values()) + result = { - "Request rate URL": "http://localhost:9090/graph?g0.expr=sum%20by%20%28function%2C%20module%29%20%28rate%20%28function_calls_count_total%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%7D%5B5m%5D%29%29&g0.tab=0", - "Latency URL": "http://localhost:9090/graph?g0.expr=sum%20by%20%28le%2C%20function%2C%20module%29%20%28rate%28function_calls_duration_bucket%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%7D%5B5m%5D%29%29&g0.tab=0", - "Error Ratio URL": "http://localhost:9090/graph?g0.expr=sum%20by%20%28function%2C%20module%29%20%28rate%20%28function_calls_count_total%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%2C%20result%3D%22error%22%7D%5B5m%5D%29%29%20/%20sum%20by%20%28function%2C%20module%29%20%28rate%20%28function_calls_count_total%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%7D%5B5m%5D%29%29&g0.tab=0", + "Request rate URL": "http://localhost:9090/graph?g0.expr=sum%20by%20%28function%2C%20module%2C%20commit%2C%20version%29%20%28rate%20%28function_calls_count_total%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%7D%5B5m%5D%29%20%2A%20on%20%28instance%2C%20job%29%20group_left%28version%2C%20commit%29%20build_info%29&g0.tab=0", + "Latency URL": "http://localhost:9090/graph?g0.expr=sum%20by%20%28le%2C%20function%2C%20module%2C%20commit%2C%20version%29%20%28rate%28function_calls_duration_bucket%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%7D%5B5m%5D%29%20%2A%20on%20%28instance%2C%20job%29%20group_left%28version%2C%20commit%29%20build_info%29&g0.tab=0", + "Error Ratio URL": "http://localhost:9090/graph?g0.expr=sum%20by%20%28function%2C%20module%2C%20commit%2C%20version%29%20%28rate%20%28function_calls_count_total%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%2C%20result%3D%22error%22%7D%5B5m%5D%29%20%2A%20on%20%28instance%2C%20job%29%20group_left%28version%2C%20commit%29%20build_info%29%20/%20sum%20by%20%28function%2C%20module%2C%20commit%2C%20version%29%20%28rate%20%28function_calls_count_total%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%7D%5B5m%5D%29%20%2A%20on%20%28instance%2C%20job%29%20group_left%28version%2C%20commit%29%20build_info%29&g0.tab=0", } assert result == urls From 9a3b55aa7e634b8a0e0e30196be23dff73be53b2 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 15:14:29 +0200 Subject: [PATCH 12/26] Add test for build_info gauge for prometheus tracker (skipped test for otel tracker) --- src/autometrics/test_prometheus_url.py | 4 +- src/autometrics/tracker/prometheus.py | 4 +- src/autometrics/tracker/test_tracker.py | 69 ++++++++++++++++++++++++- 3 files changed, 72 insertions(+), 5 deletions(-) diff --git a/src/autometrics/test_prometheus_url.py b/src/autometrics/test_prometheus_url.py index 86a7d8c..e7d20ac 100644 --- a/src/autometrics/test_prometheus_url.py +++ b/src/autometrics/test_prometheus_url.py @@ -24,8 +24,8 @@ def test_create_prometheus_url_with_default_url(default_url_generator: Generator def test_create_urls_with_default_url(default_url_generator: Generator): urls = default_url_generator.create_urls() - print(urls.keys()) - print(urls.values()) + # print(urls.keys()) + # print(urls.values()) result = { "Request rate URL": "http://localhost:9090/graph?g0.expr=sum%20by%20%28function%2C%20module%2C%20commit%2C%20version%29%20%28rate%20%28function_calls_count_total%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%7D%5B5m%5D%29%20%2A%20on%20%28instance%2C%20job%29%20group_left%28version%2C%20commit%29%20build_info%29&g0.tab=0", diff --git a/src/autometrics/tracker/prometheus.py b/src/autometrics/tracker/prometheus.py index b7c902e..2f20423 100644 --- a/src/autometrics/tracker/prometheus.py +++ b/src/autometrics/tracker/prometheus.py @@ -45,7 +45,7 @@ class PrometheusTracker: OBJECTIVE_LATENCY_THRESHOLD_PROMETHEUS, ], ) - prom_gague = Gauge( + prom_gauge = Gauge( BUILD_INFO_NAME_PROMETHEUS, BUILD_INFO_DESCRIPTION, [COMMIT_KEY, VERSION_KEY] ) @@ -102,7 +102,7 @@ def _histogram( def set_build_info(self, commit: str, version: str): """Observe the build info.""" - self.prom_gague.labels(commit, version).set(1.0) + self.prom_gauge.labels(commit, version).set(1) # def start(self, function: str = None, module: str = None): # """Start tracking metrics for a function call.""" diff --git a/src/autometrics/tracker/test_tracker.py b/src/autometrics/tracker/test_tracker.py index 2f8d252..71fe266 100644 --- a/src/autometrics/tracker/test_tracker.py +++ b/src/autometrics/tracker/test_tracker.py @@ -1,7 +1,12 @@ +from prometheus_client.exposition import generate_latest +import pytest + from .opentelemetry import OpenTelemetryTracker from .prometheus import PrometheusTracker -from .tracker import default_tracker +from .tracker import default_tracker, create_tracker, TrackerType + +tracker_types = [TrackerType.PROMETHEUS, TrackerType.OPENTELEMETRY] def test_default_tracker(monkeypatch): @@ -22,3 +27,65 @@ def test_default_tracker(monkeypatch): monkeypatch.setenv("AUTOMETRICS_TRACKER", "something_else") tracker = default_tracker() assert isinstance(tracker, OpenTelemetryTracker) + + +def test_create_prometheus_tracker_set_build_info(monkeypatch): + """Test that create_tracker calls set_build_info using env vars.""" + + commit = "d6abce3" + version = "1.0.1" + + monkeypatch.setenv("AUTOMETRICS_COMMIT", commit) + monkeypatch.setenv("AUTOMETRICS_VERSION", version) + + prom_tracker = create_tracker(TrackerType.PROMETHEUS) + assert isinstance(prom_tracker, PrometheusTracker) + + blob = generate_latest() + assert blob is not None + data = blob.decode("utf-8") + + prom_build_info = f"""build_info{{commit="{commit}",version="{version}"}} 1.0""" + assert prom_build_info in data + + monkeypatch.delenv("AUTOMETRICS_VERSION", raising=False) + monkeypatch.delenv("AUTOMETRICS_COMMIT", raising=False) + + +def test_create_otel_tracker_set_build_info(monkeypatch): + """Test that create_tracker calls set_build_info using env vars.""" + + pytest.skip("Skipping test because I am unser how to inspect otel metrics") + + commit = "a29a178" + version = "0.0.1" + + monkeypatch.setenv("AUTOMETRICS_COMMIT", commit) + monkeypatch.setenv("AUTOMETRICS_VERSION", version) + + # FIXME - Creating this again (after having created it in test_default_tracker) logs a warning, since you're not suppose to ever re-set the MeterProvider in the Otel tracker + otel_tracker = create_tracker(TrackerType.OPENTELEMETRY) + assert isinstance(otel_tracker, OpenTelemetryTracker) + + blob = generate_latest() + assert blob is not None + data = blob.decode("utf-8") + + prom_build_info = f"""build_info{{commit="{commit}",version="{version}"}} 1.0""" + assert prom_build_info in data + + monkeypatch.delenv("AUTOMETRICS_VERSION", raising=False) + monkeypatch.delenv("AUTOMETRICS_COMMIT", raising=False) + + +# def test_create_tracker_set_build_info_empty(monkeypatch): +# """Test that create_tracker calls set_build_info with empty strings when none is present.""" + +# monkeypatch.delenv("AUTOMETRICS_VERSION", raising=False) +# monkeypatch.delenv("AUTOMETRICS_COMMIT", raising=False) + +# otel_tracker = create_tracker(TrackerType.OPENTELEMETRY) +# assert isinstance(otel_tracker, OpenTelemetryTracker) + +# prom_tracker = create_tracker() +# assert isinstance(prom_tracker, PrometheusTracker) From cd7448665fd8115945c84bd70ff3b9a0378a3ab0 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 16:03:32 +0200 Subject: [PATCH 13/26] Checkpoint (saving work for later reference) --- src/autometrics/constants.py | 3 +-- src/autometrics/tracker/opentelemetry.py | 9 +++++++++ src/autometrics/tracker/prometheus.py | 4 ++-- src/autometrics/tracker/test_tracker.py | 20 +++++++++++++++----- 4 files changed, 27 insertions(+), 9 deletions(-) diff --git a/src/autometrics/constants.py b/src/autometrics/constants.py index b45e6c9..7399ee0 100644 --- a/src/autometrics/constants.py +++ b/src/autometrics/constants.py @@ -3,11 +3,10 @@ COUNTER_NAME = "function.calls.count" HISTOGRAM_NAME = "function.calls.duration" # NOTE - The Rust implementation does not use `build.info`, instead opts for just `build_info` -BUILD_INFO_NAME = "build.info" +BUILD_INFO_NAME = "build_info" COUNTER_NAME_PROMETHEUS = COUNTER_NAME.replace(".", "_") HISTOGRAM_NAME_PROMETHEUS = HISTOGRAM_NAME.replace(".", "_") -BUILD_INFO_NAME_PROMETHEUS = BUILD_INFO_NAME.replace(".", "_") COUNTER_DESCRIPTION = "Autometrics counter for tracking function calls" HISTOGRAM_DESCRIPTION = "Autometrics histogram for tracking function call duration" diff --git a/src/autometrics/tracker/opentelemetry.py b/src/autometrics/tracker/opentelemetry.py index b1d9fb2..e8c96d6 100644 --- a/src/autometrics/tracker/opentelemetry.py +++ b/src/autometrics/tracker/opentelemetry.py @@ -6,6 +6,7 @@ Histogram, UpDownCounter, set_meter_provider, + get_meter_provider, ) from opentelemetry.sdk.metrics import MeterProvider @@ -68,6 +69,7 @@ def __init__(self): name=BUILD_INFO_NAME, description=BUILD_INFO_DESCRIPTION, ) + self.__exporter = exporter def __count( self, @@ -124,6 +126,13 @@ def __histogram( }, ) + def _collect(self): + self.__exporter.collect() + + def _get_metrics_data(self): + result = self.__exporter._collector._metrics_datas + return result + def set_build_info(self, commit: str, version: str): """Observe the build info.""" self.__up_down_counter_instance.add( diff --git a/src/autometrics/tracker/prometheus.py b/src/autometrics/tracker/prometheus.py index 2f20423..f52f0dc 100644 --- a/src/autometrics/tracker/prometheus.py +++ b/src/autometrics/tracker/prometheus.py @@ -6,7 +6,7 @@ from ..constants import ( COUNTER_NAME_PROMETHEUS, HISTOGRAM_NAME_PROMETHEUS, - BUILD_INFO_NAME_PROMETHEUS, + BUILD_INFO_NAME, COUNTER_DESCRIPTION, HISTOGRAM_DESCRIPTION, BUILD_INFO_DESCRIPTION, @@ -46,7 +46,7 @@ class PrometheusTracker: ], ) prom_gauge = Gauge( - BUILD_INFO_NAME_PROMETHEUS, BUILD_INFO_DESCRIPTION, [COMMIT_KEY, VERSION_KEY] + BUILD_INFO_NAME, BUILD_INFO_DESCRIPTION, [COMMIT_KEY, VERSION_KEY] ) def _count( diff --git a/src/autometrics/tracker/test_tracker.py b/src/autometrics/tracker/test_tracker.py index 71fe266..4a9ac13 100644 --- a/src/autometrics/tracker/test_tracker.py +++ b/src/autometrics/tracker/test_tracker.py @@ -11,7 +11,7 @@ def test_default_tracker(monkeypatch): """Test the default tracker type.""" - + pytest.skip() monkeypatch.delenv("AUTOMETRICS_TRACKER", raising=False) tracker = default_tracker() assert isinstance(tracker, OpenTelemetryTracker) @@ -32,6 +32,8 @@ def test_default_tracker(monkeypatch): def test_create_prometheus_tracker_set_build_info(monkeypatch): """Test that create_tracker calls set_build_info using env vars.""" + pytest.skip() + commit = "d6abce3" version = "1.0.1" @@ -53,9 +55,13 @@ def test_create_prometheus_tracker_set_build_info(monkeypatch): def test_create_otel_tracker_set_build_info(monkeypatch): - """Test that create_tracker calls set_build_info using env vars.""" - - pytest.skip("Skipping test because I am unser how to inspect otel metrics") + """ + Test that create_tracker calls set_build_info using env vars. + Note that the OTEL collector translates metrics to Prometheus. + """ + # pytest.skip( + # "Skipping test because OTEL collector does not create a gauge when it translates to Prometheus" + # ) commit = "a29a178" version = "0.0.1" @@ -67,10 +73,14 @@ def test_create_otel_tracker_set_build_info(monkeypatch): otel_tracker = create_tracker(TrackerType.OPENTELEMETRY) assert isinstance(otel_tracker, OpenTelemetryTracker) + # otel_tracker._collect() + blob = generate_latest() assert blob is not None data = blob.decode("utf-8") - + # print(data) + for item in otel_tracker._get_metrics_data(): + print(item) prom_build_info = f"""build_info{{commit="{commit}",version="{version}"}} 1.0""" assert prom_build_info in data From f20e973f0a1a9adb664d25e60bb300f61f2f967c Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 16:44:50 +0200 Subject: [PATCH 14/26] Update otel tracker and tracker tests after finding otel prometheus bug --- src/autometrics/tracker/opentelemetry.py | 8 -------- src/autometrics/tracker/test_tracker.py | 17 +++++------------ 2 files changed, 5 insertions(+), 20 deletions(-) diff --git a/src/autometrics/tracker/opentelemetry.py b/src/autometrics/tracker/opentelemetry.py index e8c96d6..388f010 100644 --- a/src/autometrics/tracker/opentelemetry.py +++ b/src/autometrics/tracker/opentelemetry.py @@ -6,7 +6,6 @@ Histogram, UpDownCounter, set_meter_provider, - get_meter_provider, ) from opentelemetry.sdk.metrics import MeterProvider @@ -126,13 +125,6 @@ def __histogram( }, ) - def _collect(self): - self.__exporter.collect() - - def _get_metrics_data(self): - result = self.__exporter._collector._metrics_datas - return result - def set_build_info(self, commit: str, version: str): """Observe the build info.""" self.__up_down_counter_instance.add( diff --git a/src/autometrics/tracker/test_tracker.py b/src/autometrics/tracker/test_tracker.py index 4a9ac13..c616189 100644 --- a/src/autometrics/tracker/test_tracker.py +++ b/src/autometrics/tracker/test_tracker.py @@ -11,7 +11,7 @@ def test_default_tracker(monkeypatch): """Test the default tracker type.""" - pytest.skip() + monkeypatch.delenv("AUTOMETRICS_TRACKER", raising=False) tracker = default_tracker() assert isinstance(tracker, OpenTelemetryTracker) @@ -30,9 +30,7 @@ def test_default_tracker(monkeypatch): def test_create_prometheus_tracker_set_build_info(monkeypatch): - """Test that create_tracker calls set_build_info using env vars.""" - - pytest.skip() + """Test that create_tracker (for a Prometheus tracker) calls set_build_info using env vars.""" commit = "d6abce3" version = "1.0.1" @@ -56,11 +54,11 @@ def test_create_prometheus_tracker_set_build_info(monkeypatch): def test_create_otel_tracker_set_build_info(monkeypatch): """ - Test that create_tracker calls set_build_info using env vars. + Test that create_tracker (for an OTEL tracker) calls set_build_info using env vars. Note that the OTEL collector translates metrics to Prometheus. """ # pytest.skip( - # "Skipping test because OTEL collector does not create a gauge when it translates to Prometheus" + # "Skipping test because OTEL collector does not create a gauge when it translates UpDownCounter to Prometheus" # ) commit = "a29a178" @@ -69,18 +67,13 @@ def test_create_otel_tracker_set_build_info(monkeypatch): monkeypatch.setenv("AUTOMETRICS_COMMIT", commit) monkeypatch.setenv("AUTOMETRICS_VERSION", version) - # FIXME - Creating this again (after having created it in test_default_tracker) logs a warning, since you're not suppose to ever re-set the MeterProvider in the Otel tracker otel_tracker = create_tracker(TrackerType.OPENTELEMETRY) assert isinstance(otel_tracker, OpenTelemetryTracker) - # otel_tracker._collect() - blob = generate_latest() assert blob is not None data = blob.decode("utf-8") - # print(data) - for item in otel_tracker._get_metrics_data(): - print(item) + prom_build_info = f"""build_info{{commit="{commit}",version="{version}"}} 1.0""" assert prom_build_info in data From c36262085acc1622d12c477e58b9f7991c6ca327 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 17:01:46 +0200 Subject: [PATCH 15/26] Update env vars we mention in README (still needs more thought) --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 4826385..9b70707 100644 --- a/README.md +++ b/README.md @@ -123,9 +123,7 @@ Autometrics makes it easy to identify if a specific version or commit introduced It uses a separate metric (`build_info`) to track the version and, optionally, git commit of your service. It then writes queries that group metrics by the `version` and `commit` labels so you can spot correlations between those and potential issues. -The `version` is collected from the `...TODO...` environment variable. You can override this by setting the runtime environment variable `AUTOMETRICS_VERSION`. - -To set the `commit`, expose the run-time environment variable `AUTOMETRICS_COMMIT`. +The `version` is read from the `AUTOMETRICS_VERSION` environment variable, and the `commit` value uses the environment variable `AUTOMETRICS_COMMIT`. ## Development of the package @@ -162,4 +160,6 @@ poetry run black . poetry run pyright # Run the tests using pytest poetry run pytest +# Run a single test, and clear the cache +poetry run pytest --cache-clear -k test_tracker ``` From 034406aa8da736e42ca5401f85e6313117027f33 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 17:31:27 +0200 Subject: [PATCH 16/26] Ensure set_build_info is only called once --- src/autometrics/tracker/opentelemetry.py | 17 +++++++++-------- src/autometrics/tracker/prometheus.py | 8 ++++++-- src/autometrics/tracker/test_tracker.py | 10 ++++------ 3 files changed, 19 insertions(+), 16 deletions(-) diff --git a/src/autometrics/tracker/opentelemetry.py b/src/autometrics/tracker/opentelemetry.py index 388f010..9ce7d89 100644 --- a/src/autometrics/tracker/opentelemetry.py +++ b/src/autometrics/tracker/opentelemetry.py @@ -68,7 +68,7 @@ def __init__(self): name=BUILD_INFO_NAME, description=BUILD_INFO_DESCRIPTION, ) - self.__exporter = exporter + self._has_set_build_info = False def __count( self, @@ -127,13 +127,14 @@ def __histogram( def set_build_info(self, commit: str, version: str): """Observe the build info.""" - self.__up_down_counter_instance.add( - 1.0, - attributes={ - "commit": commit, - "version": version, - }, - ) + if not self._has_set_build_info: + self.__up_down_counter_instance.add( + 1.0, + attributes={ + "commit": commit, + "version": version, + }, + ) def finish( self, diff --git a/src/autometrics/tracker/prometheus.py b/src/autometrics/tracker/prometheus.py index f52f0dc..e1688db 100644 --- a/src/autometrics/tracker/prometheus.py +++ b/src/autometrics/tracker/prometheus.py @@ -49,6 +49,9 @@ class PrometheusTracker: BUILD_INFO_NAME, BUILD_INFO_DESCRIPTION, [COMMIT_KEY, VERSION_KEY] ) + def __init__(self) -> None: + self._has_set_build_info = False + def _count( self, func_name: str, @@ -101,8 +104,9 @@ def _histogram( ).observe(duration) def set_build_info(self, commit: str, version: str): - """Observe the build info.""" - self.prom_gauge.labels(commit, version).set(1) + """Observe the build info. Should only be called once per tracker instance""" + if not self._has_set_build_info: + self.prom_gauge.labels(commit, version).set(1) # def start(self, function: str = None, module: str = None): # """Start tracking metrics for a function call.""" diff --git a/src/autometrics/tracker/test_tracker.py b/src/autometrics/tracker/test_tracker.py index c616189..24305fc 100644 --- a/src/autometrics/tracker/test_tracker.py +++ b/src/autometrics/tracker/test_tracker.py @@ -4,9 +4,7 @@ from .opentelemetry import OpenTelemetryTracker from .prometheus import PrometheusTracker -from .tracker import default_tracker, create_tracker, TrackerType - -tracker_types = [TrackerType.PROMETHEUS, TrackerType.OPENTELEMETRY] +from .tracker import default_tracker, create_tracker, set_tracker, TrackerType def test_default_tracker(monkeypatch): @@ -57,9 +55,9 @@ def test_create_otel_tracker_set_build_info(monkeypatch): Test that create_tracker (for an OTEL tracker) calls set_build_info using env vars. Note that the OTEL collector translates metrics to Prometheus. """ - # pytest.skip( - # "Skipping test because OTEL collector does not create a gauge when it translates UpDownCounter to Prometheus" - # ) + pytest.skip( + "Skipping test because OTEL collector does not create a gauge when it translates UpDownCounter to Prometheus" + ) commit = "a29a178" version = "0.0.1" From b1eef93538fbe638977291738c50a195722b8d80 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 19:36:46 +0200 Subject: [PATCH 17/26] Fix "call once" logic of set_build_info --- src/autometrics/tracker/opentelemetry.py | 1 + src/autometrics/tracker/prometheus.py | 1 + 2 files changed, 2 insertions(+) diff --git a/src/autometrics/tracker/opentelemetry.py b/src/autometrics/tracker/opentelemetry.py index 9ce7d89..82d27f6 100644 --- a/src/autometrics/tracker/opentelemetry.py +++ b/src/autometrics/tracker/opentelemetry.py @@ -128,6 +128,7 @@ def __histogram( def set_build_info(self, commit: str, version: str): """Observe the build info.""" if not self._has_set_build_info: + self._has_set_build_info = True self.__up_down_counter_instance.add( 1.0, attributes={ diff --git a/src/autometrics/tracker/prometheus.py b/src/autometrics/tracker/prometheus.py index e1688db..bf2f09f 100644 --- a/src/autometrics/tracker/prometheus.py +++ b/src/autometrics/tracker/prometheus.py @@ -106,6 +106,7 @@ def _histogram( def set_build_info(self, commit: str, version: str): """Observe the build info. Should only be called once per tracker instance""" if not self._has_set_build_info: + self._has_set_build_info = True self.prom_gauge.labels(commit, version).set(1) # def start(self, function: str = None, module: str = None): From a071249e0228027078161b127283a058236adf2b Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 19:38:41 +0200 Subject: [PATCH 18/26] Remove commented out test --- src/autometrics/tracker/test_tracker.py | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/src/autometrics/tracker/test_tracker.py b/src/autometrics/tracker/test_tracker.py index 24305fc..9097f05 100644 --- a/src/autometrics/tracker/test_tracker.py +++ b/src/autometrics/tracker/test_tracker.py @@ -77,16 +77,3 @@ def test_create_otel_tracker_set_build_info(monkeypatch): monkeypatch.delenv("AUTOMETRICS_VERSION", raising=False) monkeypatch.delenv("AUTOMETRICS_COMMIT", raising=False) - - -# def test_create_tracker_set_build_info_empty(monkeypatch): -# """Test that create_tracker calls set_build_info with empty strings when none is present.""" - -# monkeypatch.delenv("AUTOMETRICS_VERSION", raising=False) -# monkeypatch.delenv("AUTOMETRICS_COMMIT", raising=False) - -# otel_tracker = create_tracker(TrackerType.OPENTELEMETRY) -# assert isinstance(otel_tracker, OpenTelemetryTracker) - -# prom_tracker = create_tracker() -# assert isinstance(prom_tracker, PrometheusTracker) From 5bcc0ba8c50b78518077d2c3bc3fe4ede75f0d80 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 22:34:22 +0200 Subject: [PATCH 19/26] Update changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2b8a347..33adf8a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ### Added +- Support for build_info metrics in Prometheus (#35) - OpenTelemetry Support (#28) - Fly.io example (#26) - Django example (#22) From 894c224da59ce3db4b11c1baaee5ca701fdf2fc7 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 23:21:34 +0200 Subject: [PATCH 20/26] Remove unnecessary commented-out print statements --- src/autometrics/test_prometheus_url.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/autometrics/test_prometheus_url.py b/src/autometrics/test_prometheus_url.py index e7d20ac..a16af1e 100644 --- a/src/autometrics/test_prometheus_url.py +++ b/src/autometrics/test_prometheus_url.py @@ -24,9 +24,6 @@ def test_create_prometheus_url_with_default_url(default_url_generator: Generator def test_create_urls_with_default_url(default_url_generator: Generator): urls = default_url_generator.create_urls() - # print(urls.keys()) - # print(urls.values()) - result = { "Request rate URL": "http://localhost:9090/graph?g0.expr=sum%20by%20%28function%2C%20module%2C%20commit%2C%20version%29%20%28rate%20%28function_calls_count_total%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%7D%5B5m%5D%29%20%2A%20on%20%28instance%2C%20job%29%20group_left%28version%2C%20commit%29%20build_info%29&g0.tab=0", "Latency URL": "http://localhost:9090/graph?g0.expr=sum%20by%20%28le%2C%20function%2C%20module%2C%20commit%2C%20version%29%20%28rate%28function_calls_duration_bucket%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%7D%5B5m%5D%29%20%2A%20on%20%28instance%2C%20job%29%20group_left%28version%2C%20commit%29%20build_info%29&g0.tab=0", From b58510feeae73e77e71baa6c98d270d856c149d6 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 23:25:40 +0200 Subject: [PATCH 21/26] Add set_build_info to the TrackMetrics Protocol --- src/autometrics/tracker/opentelemetry.py | 1 - src/autometrics/tracker/prometheus.py | 1 - src/autometrics/tracker/tracker.py | 3 +++ 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/src/autometrics/tracker/opentelemetry.py b/src/autometrics/tracker/opentelemetry.py index 82d27f6..985bdd1 100644 --- a/src/autometrics/tracker/opentelemetry.py +++ b/src/autometrics/tracker/opentelemetry.py @@ -126,7 +126,6 @@ def __histogram( ) def set_build_info(self, commit: str, version: str): - """Observe the build info.""" if not self._has_set_build_info: self._has_set_build_info = True self.__up_down_counter_instance.add( diff --git a/src/autometrics/tracker/prometheus.py b/src/autometrics/tracker/prometheus.py index bf2f09f..688f760 100644 --- a/src/autometrics/tracker/prometheus.py +++ b/src/autometrics/tracker/prometheus.py @@ -104,7 +104,6 @@ def _histogram( ).observe(duration) def set_build_info(self, commit: str, version: str): - """Observe the build info. Should only be called once per tracker instance""" if not self._has_set_build_info: self._has_set_build_info = True self.prom_gauge.labels(commit, version).set(1) diff --git a/src/autometrics/tracker/tracker.py b/src/autometrics/tracker/tracker.py index de28961..61b54e3 100644 --- a/src/autometrics/tracker/tracker.py +++ b/src/autometrics/tracker/tracker.py @@ -16,6 +16,9 @@ class Result(Enum): class TrackMetrics(Protocol): """Protocol for tracking metrics.""" + def set_build_info(self, commit: str, version: str): + """Observe the build info. Should only be called once per tracker instance""" + def finish( self, start_time: float, From 7163175ad8c70d8dd829cfad88a3a9ee1f9e3517 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Wed, 10 May 2023 23:41:46 +0200 Subject: [PATCH 22/26] Fix build_info query based off of autometrics-dev/autometrics-shared#8 --- src/autometrics/prometheus_url.py | 2 +- src/autometrics/test_prometheus_url.py | 8 +++++--- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/src/autometrics/prometheus_url.py b/src/autometrics/prometheus_url.py index ec68923..1d92c74 100644 --- a/src/autometrics/prometheus_url.py +++ b/src/autometrics/prometheus_url.py @@ -3,7 +3,7 @@ from typing import Optional from dotenv import load_dotenv -ADD_BUILD_INFO_LABELS = "* on (instance, job) group_left(version, commit) build_info" +ADD_BUILD_INFO_LABELS = "* on (instance, job) group_left(version, commit) (last_over_time(build_info[1s]) or on (instance, job) up)" def cleanup_url(url: str) -> str: diff --git a/src/autometrics/test_prometheus_url.py b/src/autometrics/test_prometheus_url.py index a16af1e..74ab469 100644 --- a/src/autometrics/test_prometheus_url.py +++ b/src/autometrics/test_prometheus_url.py @@ -24,10 +24,12 @@ def test_create_prometheus_url_with_default_url(default_url_generator: Generator def test_create_urls_with_default_url(default_url_generator: Generator): urls = default_url_generator.create_urls() + print(urls) + result = { - "Request rate URL": "http://localhost:9090/graph?g0.expr=sum%20by%20%28function%2C%20module%2C%20commit%2C%20version%29%20%28rate%20%28function_calls_count_total%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%7D%5B5m%5D%29%20%2A%20on%20%28instance%2C%20job%29%20group_left%28version%2C%20commit%29%20build_info%29&g0.tab=0", - "Latency URL": "http://localhost:9090/graph?g0.expr=sum%20by%20%28le%2C%20function%2C%20module%2C%20commit%2C%20version%29%20%28rate%28function_calls_duration_bucket%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%7D%5B5m%5D%29%20%2A%20on%20%28instance%2C%20job%29%20group_left%28version%2C%20commit%29%20build_info%29&g0.tab=0", - "Error Ratio URL": "http://localhost:9090/graph?g0.expr=sum%20by%20%28function%2C%20module%2C%20commit%2C%20version%29%20%28rate%20%28function_calls_count_total%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%2C%20result%3D%22error%22%7D%5B5m%5D%29%20%2A%20on%20%28instance%2C%20job%29%20group_left%28version%2C%20commit%29%20build_info%29%20/%20sum%20by%20%28function%2C%20module%2C%20commit%2C%20version%29%20%28rate%20%28function_calls_count_total%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%7D%5B5m%5D%29%20%2A%20on%20%28instance%2C%20job%29%20group_left%28version%2C%20commit%29%20build_info%29&g0.tab=0", + "Request rate URL": "http://localhost:9090/graph?g0.expr=sum%20by%20%28function%2C%20module%2C%20commit%2C%20version%29%20%28rate%20%28function_calls_count_total%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%7D%5B5m%5D%29%20%2A%20on%20%28instance%2C%20job%29%20group_left%28version%2C%20commit%29%20%28last_over_time%28build_info%5B1s%5D%29%20or%20on%20%28instance%2C%20job%29%20up%29%29&g0.tab=0", + "Latency URL": "http://localhost:9090/graph?g0.expr=sum%20by%20%28le%2C%20function%2C%20module%2C%20commit%2C%20version%29%20%28rate%28function_calls_duration_bucket%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%7D%5B5m%5D%29%20%2A%20on%20%28instance%2C%20job%29%20group_left%28version%2C%20commit%29%20%28last_over_time%28build_info%5B1s%5D%29%20or%20on%20%28instance%2C%20job%29%20up%29%29&g0.tab=0", + "Error Ratio URL": "http://localhost:9090/graph?g0.expr=sum%20by%20%28function%2C%20module%2C%20commit%2C%20version%29%20%28rate%20%28function_calls_count_total%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%2C%20result%3D%22error%22%7D%5B5m%5D%29%20%2A%20on%20%28instance%2C%20job%29%20group_left%28version%2C%20commit%29%20%28last_over_time%28build_info%5B1s%5D%29%20or%20on%20%28instance%2C%20job%29%20up%29%29%20/%20sum%20by%20%28function%2C%20module%2C%20commit%2C%20version%29%20%28rate%20%28function_calls_count_total%7Bfunction%3D%22myFunction%22%2Cmodule%3D%22myModule%22%7D%5B5m%5D%29%20%2A%20on%20%28instance%2C%20job%29%20group_left%28version%2C%20commit%29%20%28last_over_time%28build_info%5B1s%5D%29%20or%20on%20%28instance%2C%20job%29%20up%29%29&g0.tab=0", } assert result == urls From c44a6213628e4b73cfdc6b014cf0d39788438bcb Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Thu, 11 May 2023 13:50:46 +0200 Subject: [PATCH 23/26] Rename create_tracker to init_tracker --- src/autometrics/tracker/test_tracker.py | 14 +++++++------- src/autometrics/tracker/tracker.py | 6 +++--- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/src/autometrics/tracker/test_tracker.py b/src/autometrics/tracker/test_tracker.py index 9097f05..a51728e 100644 --- a/src/autometrics/tracker/test_tracker.py +++ b/src/autometrics/tracker/test_tracker.py @@ -4,7 +4,7 @@ from .opentelemetry import OpenTelemetryTracker from .prometheus import PrometheusTracker -from .tracker import default_tracker, create_tracker, set_tracker, TrackerType +from .tracker import default_tracker, init_tracker, TrackerType def test_default_tracker(monkeypatch): @@ -27,8 +27,8 @@ def test_default_tracker(monkeypatch): assert isinstance(tracker, OpenTelemetryTracker) -def test_create_prometheus_tracker_set_build_info(monkeypatch): - """Test that create_tracker (for a Prometheus tracker) calls set_build_info using env vars.""" +def test_init_prometheus_tracker_set_build_info(monkeypatch): + """Test that init_tracker (for a Prometheus tracker) calls set_build_info using env vars.""" commit = "d6abce3" version = "1.0.1" @@ -36,7 +36,7 @@ def test_create_prometheus_tracker_set_build_info(monkeypatch): monkeypatch.setenv("AUTOMETRICS_COMMIT", commit) monkeypatch.setenv("AUTOMETRICS_VERSION", version) - prom_tracker = create_tracker(TrackerType.PROMETHEUS) + prom_tracker = init_tracker(TrackerType.PROMETHEUS) assert isinstance(prom_tracker, PrometheusTracker) blob = generate_latest() @@ -50,9 +50,9 @@ def test_create_prometheus_tracker_set_build_info(monkeypatch): monkeypatch.delenv("AUTOMETRICS_COMMIT", raising=False) -def test_create_otel_tracker_set_build_info(monkeypatch): +def test_init_otel_tracker_set_build_info(monkeypatch): """ - Test that create_tracker (for an OTEL tracker) calls set_build_info using env vars. + Test that init_tracker (for an OTEL tracker) calls set_build_info using env vars. Note that the OTEL collector translates metrics to Prometheus. """ pytest.skip( @@ -65,7 +65,7 @@ def test_create_otel_tracker_set_build_info(monkeypatch): monkeypatch.setenv("AUTOMETRICS_COMMIT", commit) monkeypatch.setenv("AUTOMETRICS_VERSION", version) - otel_tracker = create_tracker(TrackerType.OPENTELEMETRY) + otel_tracker = init_tracker(TrackerType.OPENTELEMETRY) assert isinstance(otel_tracker, OpenTelemetryTracker) blob = generate_latest() diff --git a/src/autometrics/tracker/tracker.py b/src/autometrics/tracker/tracker.py index 61b54e3..b22179d 100644 --- a/src/autometrics/tracker/tracker.py +++ b/src/autometrics/tracker/tracker.py @@ -38,7 +38,7 @@ class TrackerType(Enum): PROMETHEUS = "prometheus" -def create_tracker(tracker_type: TrackerType) -> TrackMetrics: +def init_tracker(tracker_type: TrackerType) -> TrackMetrics: """Create a tracker""" tracker_instance = None if tracker_type == TrackerType.OPENTELEMETRY: @@ -73,7 +73,7 @@ def get_tracker_type() -> TrackerType: def default_tracker(): """Setup the default tracker.""" preferred_tracker = get_tracker_type() - return create_tracker(preferred_tracker) + return init_tracker(preferred_tracker) tracker: TrackMetrics = default_tracker() @@ -87,4 +87,4 @@ def get_tracker() -> TrackMetrics: def set_tracker(tracker_type: TrackerType): """Set the tracker type.""" global tracker - tracker = create_tracker(tracker_type) + tracker = init_tracker(tracker_type) From c91606170eb3ff9ac84613267eeb221fa0bcbe78 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Thu, 11 May 2023 13:53:36 +0200 Subject: [PATCH 24/26] Use types instead of ifs --- src/autometrics/tracker/tracker.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/autometrics/tracker/tracker.py b/src/autometrics/tracker/tracker.py index b22179d..f4bd05b 100644 --- a/src/autometrics/tracker/tracker.py +++ b/src/autometrics/tracker/tracker.py @@ -40,7 +40,8 @@ class TrackerType(Enum): def init_tracker(tracker_type: TrackerType) -> TrackMetrics: """Create a tracker""" - tracker_instance = None + + tracker_instance: TrackMetrics if tracker_type == TrackerType.OPENTELEMETRY: # pylint: disable=import-outside-toplevel from .opentelemetry import OpenTelemetryTracker @@ -53,11 +54,10 @@ def init_tracker(tracker_type: TrackerType) -> TrackMetrics: tracker_instance = PrometheusTracker() # NOTE - Only set the build info when the tracker is initialized - if tracker_instance: - tracker_instance.set_build_info( - commit=os.getenv("AUTOMETRICS_COMMIT") or "", - version=os.getenv("AUTOMETRICS_VERSION") or "", - ) + tracker_instance.set_build_info( + commit=os.getenv("AUTOMETRICS_COMMIT") or "", + version=os.getenv("AUTOMETRICS_VERSION") or "", + ) return tracker_instance From b754103d996d27216bb620c6b7273d30e09cf734 Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Thu, 11 May 2023 13:54:51 +0200 Subject: [PATCH 25/26] Update pyright --- poetry.lock | 8 ++++---- pyproject.toml | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/poetry.lock b/poetry.lock index 9f4ba17..01691d0 100644 --- a/poetry.lock +++ b/poetry.lock @@ -948,14 +948,14 @@ tomli = {version = ">=1.1.0", markers = "python_version < \"3.11\""} [[package]] name = "pyright" -version = "1.1.306" +version = "1.1.307" description = "Command line wrapper for pyright" category = "dev" optional = false python-versions = ">=3.7" files = [ - {file = "pyright-1.1.306-py3-none-any.whl", hash = "sha256:008eb2a29584ae274a154d749cf81476a3073fb562a462eac8d43a753378b9db"}, - {file = "pyright-1.1.306.tar.gz", hash = "sha256:16d5d198be64de497d5f9002000a271176c381e21b977ca5566cf779b643c9ed"}, + {file = "pyright-1.1.307-py3-none-any.whl", hash = "sha256:6b360d2e018311bdf8acea73ef1f21bf0b5b502345aa94bc6763cf197b2e75b3"}, + {file = "pyright-1.1.307.tar.gz", hash = "sha256:b7a8734fad4a2438b8bb0dfbe462f529c9d4eb31947bdae85b9b4e7a97ff6a49"}, ] [package.dependencies] @@ -1430,4 +1430,4 @@ testing = ["big-O", "flake8 (<5)", "jaraco.functools", "jaraco.itertools", "more [metadata] lock-version = "2.0" python-versions = "^3.8" -content-hash = "ffab8d31818912728e687168957e18888fe71d72ea6493a38c87ecbc6f5a6258" +content-hash = "1cb7ea3cfe833febf8bd996651c5b5a7feb9b1d49b457f12bd52ffdae0239598" diff --git a/pyproject.toml b/pyproject.toml index 2c607d0..649d182 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -27,7 +27,7 @@ typing-extensions = "^4.5.0" optional = true [tool.poetry.group.dev.dependencies] -pyright = "^1.1.302" +pyright = "^1.1.307" pytest = "^7.3.0" pytest-asyncio = "^0.21.0" black = "^23.3.0" From 4cb6cd4f77bb3d30a8ea05d2fd105a4064e75e7e Mon Sep 17 00:00:00 2001 From: Brett Beutell Date: Thu, 11 May 2023 14:00:31 +0200 Subject: [PATCH 26/26] Update README to mention OpenTelemetry tracker does not work with build_info --- README.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9b70707..b42ef07 100644 --- a/README.md +++ b/README.md @@ -113,11 +113,14 @@ def api_handler(): Configure the crate that autometrics will use to produce metrics by using one of the following feature flags: - `opentelemetry` - (enabled by default, can also be explicitly set using the AUTOMETRICS_TRACKER="OPEN_TELEMETERY" env var) uses -- `prometheus` -(using the AUTOMETRICS_TRACKER env var set to "PROMETHEUS") +- `prometheus` - (using the AUTOMETRICS_TRACKER env var set to "PROMETHEUS") ## Identifying commits that introduced problems -> This follows the method outlined in [Exposing the software version to Prometheus](https://www.robustperception.io/exposing-the-software-version-to-prometheus/). +> **NOTE** - As of writing, `build_info` will not work correctly when using the default tracker (`AUTOMETRICS_TRACKER=OPEN_TELEMETRY`). +> This will be fixed once the following PR is merged on the opentelemetry-python project: https://github.com/open-telemetry/opentelemetry-python/pull/3306 +> +> autometrics-py will track support for build_info using the OpenTelemetry tracker via #38 Autometrics makes it easy to identify if a specific version or commit introduced errors or increased latencies. @@ -125,6 +128,8 @@ It uses a separate metric (`build_info`) to track the version and, optionally, g The `version` is read from the `AUTOMETRICS_VERSION` environment variable, and the `commit` value uses the environment variable `AUTOMETRICS_COMMIT`. +This follows the method outlined in [Exposing the software version to Prometheus](https://www.robustperception.io/exposing-the-software-version-to-prometheus/). + ## Development of the package This package uses [poetry](https://python-poetry.org) as a package manager, with all dependencies separated into three groups: