Add logfire.instrument_system_metrics()

[alexmojaki]

Breaking change!

With this, you must call logfire.instrument_system_metrics() to enable basic system metrics, previously system metrics were always sent if you had the appropriate package installed.

---

Background:

In preparation for starting to charge we looked our usage from our larger beta users and realised a lot of companies were sending us a lot of system metrics which we are guessing they didn't care about, but would be charged a lot for.

This was because the standard opentelemetry-instrumentation-system-metrics logic was collecting around 400 metrics per minute per host — stuff like this:

metric_name	count
system.disk.operations	74
system.disk.time	74
system.disk.io	74
system.cpu.time	64
system.cpu.utilization	64
system.memory.utilization	10
system.memory.usage	10
system.network.io	6
system.network.dropped_packets	6
system.network.errors	6

(This is from looking at a machine running on render)

Most people don't need 219 datapoints on disk usage every minute!

Solution

The solution (described in detail in the docs changes in this PR) are:

1. Make system metrics opt-in, logfire will now only send them if you call logfire.instrument_system_metrics()
2. Send a curated set of the most useful system metrics, reducing the metric volume by around 98% in common cases

Feedback on whether we should adjust the set of metrics sent in "basic" mode very welcome.

[cloudflare-workers-and-pages]

Deploying logfire-docs with    Cloudflare Pages

Latest commit:	0ba76a7
Status:	✅  Deploy successful!
Preview URL:	https://5e05a01f.logfire-docs.pages.dev
Branch Preview URL:	https://alex-instrument-system-metri.logfire-docs.pages.dev

View logs

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (5d9a16f) to head (0ba76a7).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #373   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          124       126    +2     
  Lines         9193      9256   +63     
  Branches      1198      1206    +8     
=========================================
+ Hits          9193      9256   +63     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[alexmojaki]

The current 'Basic System Metrics' dashboard template won't work with this new set of reduced metrics. I will create a PR with a new template. That PR will need to be ready to merge before these changes can be merged, and it should be deployed approximately when this is released.

When users upgrade the SDK, system metrics will stop being collected, if they were being collected at all. Since their collection was the implicit default, we can't warn them about this in code, and must rely on the changelog and announcements in slack. If the only thing that users do is add logfire.instrument_system_metrics() then existing basic system metrics dashboards will still not work, so we need to make it clear that they also need to switch to the new template.

The old template is still useful though, because it works with OTEL conventions, so system metrics collected by means other than the logfire SDK can be used with the old template. So we should keep and document both. We need to figure out how to name them to clarify the difference, e.g. 'Basic System Metrics (Logfire)' and 'Basic System Metrics (OpenTelemetry)'.

[alexmojaki]

PR here: https://github.com/pydantic/platform/pull/3838

[samuelcolvin]

Sorry to be so slow, otherwise looks great.