system-metrics.md

Semantic Conventions for System Metrics

Status: Experimental

This document describes instruments and attributes for common system level metrics in OpenTelemetry. Consider the general metric semantic conventions when creating instruments not explicitly defined in the specification.

The system.* namespace SHOULD be exclusively used to report hosts' metrics. The system.* namespace SHOULD only be used when the metrics are collected from within the target system. (physical servers, virtual machines etc). Metrics collected from technology-specific, well-defined APIs (e.g. Kubelet's API or container runtimes) should be reported under their respective namespace (e.g. k8s., container.). Resource attributes related to a host, SHOULD be reported under the host.* namespace.

• Processor Metrics
  • Metric: system.cpu.time
  • Metric: system.cpu.utilization
  • Metric: system.cpu.physical.count
  • Metric: system.cpu.logical.count
  • Metric: system.cpu.frequency
• Memory Metrics
  • Metric: system.memory.usage
  • Metric: system.memory.limit
  • Metric: system.memory.utilization
• Paging/Swap Metrics
  • Metric: system.paging.usage
  • Metric: system.paging.utilization
  • Metric: system.paging.faults
  • Metric: system.paging.operations
• Disk Controller Metrics
  • Metric: system.disk.io
  • Metric: system.disk.operations
  • Metric: system.disk.io_time
  • Metric: system.disk.operation_time
  • Metric: system.disk.merged
• Filesystem Metrics
  • Metric: system.filesystem.usage
  • Metric: system.filesystem.utilization
• Network Metrics
  • Metric: system.network.dropped
  • Metric: system.network.packets
  • Metric: system.network.errors
  • Metric: system.network.io
  • Metric: system.network.connections
• Aggregate System Process Metrics
  • Metric: system.process.count
  • Metric: system.process.created
• system.{os}. - OS Specific System Metrics
  • Metric: system.linux.memory.available

Warning Existing instrumentations and collector that are using v1.21.0 of this document (or prior):

• SHOULD NOT adopt any breaking changes from document until the system semantic conventions are marked stable. Conventions include, but are not limited to, attributes, metric names, and unit of measure.
• SHOULD introduce a control mechanism to allow users to opt-in to the new conventions once the migration plan is finalized.

Processor Metrics

Description: System level processor metrics captured under the namespace system.cpu.

Metric: system.cpu.time

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.cpu.time	Counter	s	Seconds each logical CPU spent on each mode

Attribute	Type	Description	Examples	Requirement Level	Stability
system.cpu.logical_number	int	The logical CPU number [0..n-1]	1	Recommended
system.cpu.state	string	The CPU state for this data point. A system's CPU SHOULD be characterized either by data points with no state labels, or only data points with state labels.	idle; interrupt	Recommended

system.cpu.state has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value	Description	Stability
user	user
system	system
nice	nice
idle	idle
iowait	iowait
interrupt	interrupt
steal	steal

Metric: system.cpu.utilization

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.cpu.utilization	Gauge	1	Difference in system.cpu.time since the last measurement, divided by the elapsed time and number of logical CPUs

Attribute	Type	Description	Examples	Requirement Level	Stability
system.cpu.logical_number	int	The logical CPU number [0..n-1]	1	Recommended
system.cpu.state	string	The CPU state for this data point. A system's CPU SHOULD be characterized either by data points with no state labels, or only data points with state labels.	idle; interrupt	Recommended

system.cpu.state has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value	Description	Stability
user	user
system	system
nice	nice
idle	idle
iowait	iowait
interrupt	interrupt
steal	steal

Metric: system.cpu.physical.count

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.cpu.physical.count	UpDownCounter	{cpu}	Reports the number of actual physical processor cores on the hardware

Metric: system.cpu.logical.count

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.cpu.logical.count	UpDownCounter	{cpu}	Reports the number of logical (virtual) processor cores created by the operating system to manage multitasking

Metric: system.cpu.frequency

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.cpu.frequency	Gauge	{Hz}	Reports the current frequency of the CPU in Hz

Attribute	Type	Description	Examples	Requirement Level	Stability
system.cpu.logical_number	int	The logical CPU number [0..n-1]	1	Recommended

Memory Metrics

Description: System level memory metrics capture under the namespace system.memory. This does not include paging/swap memory.

Metric: system.memory.usage

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.memory.usage	UpDownCounter	By	Reports memory in use by state. [1]

[1]: The sum over all system.memory.state values SHOULD equal the total memory available on the system, that is system.memory.limit.

Attribute	Type	Description	Examples	Requirement Level	Stability
system.memory.state	string	The memory state	free; cached	Recommended

system.memory.state has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value	Description	Stability
used	used
free	free
shared	shared
buffers	buffers
cached	cached

Metric: system.memory.limit

This metric is opt-in.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.memory.limit	UpDownCounter	By	Total memory available in the system. [1]

[1]: Its value SHOULD equal the sum of system.memory.state over all states.

Metric: system.memory.utilization

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.memory.utilization	Gauge	1

Attribute	Type	Description	Examples	Requirement Level	Stability
system.memory.state	string	The memory state	free; cached	Recommended

system.memory.state has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value	Description	Stability
used	used
free	free
shared	shared
buffers	buffers
cached	cached

Paging/Swap Metrics

Description: System level paging/swap memory metrics captured under the namespace system.paging.

Metric: system.paging.usage

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.paging.usage	UpDownCounter	By	Unix swap or windows pagefile usage

Attribute	Type	Description	Examples	Requirement Level	Stability
system.paging.state	string	The memory paging state	free	Recommended

system.paging.state MUST be one of the following:

Value	Description	Stability
used	used
free	free

Metric: system.paging.utilization

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.paging.utilization	Gauge	1

Attribute	Type	Description	Examples	Requirement Level	Stability
system.paging.state	string	The memory paging state	free	Recommended

system.paging.state MUST be one of the following:

Value	Description	Stability
used	used
free	free

Metric: system.paging.faults

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.paging.faults	Counter	{fault}

Attribute	Type	Description	Examples	Requirement Level	Stability
system.paging.type	string	The memory paging type	minor	Recommended

system.paging.type MUST be one of the following:

Value	Description	Stability
major	major
minor	minor

Metric: system.paging.operations

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.paging.operations	Counter	{operation}

Attribute	Type	Description	Examples	Requirement Level	Stability
system.paging.direction	string	The paging access direction	in	Recommended
system.paging.type	string	The memory paging type	minor	Recommended

system.paging.direction MUST be one of the following:

Value	Description	Stability
in	in
out	out

system.paging.type MUST be one of the following:

Value	Description	Stability
major	major
minor	minor

Disk Controller Metrics

Description: System level disk performance metrics captured under the namespace system.disk.

Metric: system.disk.io

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.disk.io	Counter	By

Attribute	Type	Description	Examples	Requirement Level	Stability
disk.io.direction	string	The disk IO operation direction.	read	Recommended
system.device	string	The device identifier	(identifier)	Recommended

disk.io.direction MUST be one of the following:

Value	Description	Stability
read	read
write	write

Metric: system.disk.operations

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.disk.operations	Counter	{operation}

Attribute	Type	Description	Examples	Requirement Level	Stability
disk.io.direction	string	The disk IO operation direction.	read	Recommended
system.device	string	The device identifier	(identifier)	Recommended

disk.io.direction MUST be one of the following:

Value	Description	Stability
read	read
write	write

Metric: system.disk.io_time

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.disk.io_time	Counter	s	Time disk spent activated [1]

[1]: The real elapsed time ("wall clock") used in the I/O path (time from operations running in parallel are not counted). Measured as:

• Linux: Field 13 from procfs-diskstats
• Windows: The complement of "Disk% Idle Time" performance counter: uptime * (100 - "Disk\% Idle Time") / 100

Attribute	Type	Description	Examples	Requirement Level	Stability
system.device	string	The device identifier	(identifier)	Recommended

Metric: system.disk.operation_time

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.disk.operation_time	Counter	s	Sum of the time each operation took to complete [1]

[1]: Because it is the sum of time each request took, parallel-issued requests each contribute to make the count grow. Measured as:

• Linux: Fields 7 & 11 from procfs-diskstats
• Windows: "Avg. Disk sec/Read" perf counter multiplied by "Disk Reads/sec" perf counter (similar for Writes)

Attribute	Type	Description	Examples	Requirement Level	Stability
disk.io.direction	string	The disk IO operation direction.	read	Recommended
system.device	string	The device identifier	(identifier)	Recommended

disk.io.direction MUST be one of the following:

Value	Description	Stability
read	read
write	write

Metric: system.disk.merged

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.disk.merged	Counter	{operation}

Attribute	Type	Description	Examples	Requirement Level	Stability
disk.io.direction	string	The disk IO operation direction.	read	Recommended
system.device	string	The device identifier	(identifier)	Recommended

disk.io.direction MUST be one of the following:

Value	Description	Stability
read	read
write	write

Filesystem Metrics

Description: System level filesystem metrics captured under the namespace system.filesystem.

Metric: system.filesystem.usage

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.filesystem.usage	UpDownCounter	By

Attribute	Type	Description	Examples	Requirement Level	Stability
system.device	string	The device identifier	(identifier)	Recommended
system.filesystem.mode	string	The filesystem mode	rw, ro	Recommended
system.filesystem.mountpoint	string	The filesystem mount path	/mnt/data	Recommended
system.filesystem.state	string	The filesystem state	used	Recommended
system.filesystem.type	string	The filesystem type	ext4	Recommended

system.filesystem.state MUST be one of the following:

Value	Description	Stability
used	used
free	free
reserved	reserved

system.filesystem.type has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value	Description	Stability
fat32	fat32
exfat	exfat
ntfs	ntfs
refs	refs
hfsplus	hfsplus
ext4	ext4

Metric: system.filesystem.utilization

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.filesystem.utilization	Gauge	1

Attribute	Type	Description	Examples	Requirement Level	Stability
system.device	string	The device identifier	(identifier)	Recommended
system.filesystem.mode	string	The filesystem mode	rw, ro	Recommended
system.filesystem.mountpoint	string	The filesystem mount path	/mnt/data	Recommended
system.filesystem.state	string	The filesystem state	used	Recommended
system.filesystem.type	string	The filesystem type	ext4	Recommended

system.filesystem.state MUST be one of the following:

Value	Description	Stability
used	used
free	free
reserved	reserved

system.filesystem.type has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value	Description	Stability
fat32	fat32
exfat	exfat
ntfs	ntfs
refs	refs
hfsplus	hfsplus
ext4	ext4

Network Metrics

Description: System level network metrics captured under the namespace system.network.

Metric: system.network.dropped

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.network.dropped	Counter	{packet}	Count of packets that are dropped or discarded even though there was no error [1]

[1]: Measured as:

• Linux: the drop column in /proc/dev/net (source)
• Windows: InDiscards/OutDiscards from GetIfEntry2

Attribute	Type	Description	Examples	Requirement Level	Stability
network.io.direction	string	The network IO operation direction.	transmit	Recommended
system.device	string	The device identifier	(identifier)	Recommended

network.io.direction MUST be one of the following:

Value	Description	Stability
transmit	transmit
receive	receive

Metric: system.network.packets

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.network.packets	Counter	{packet}

Attribute	Type	Description	Examples	Requirement Level	Stability
network.io.direction	string	The network IO operation direction.	transmit	Recommended
system.device	string	The device identifier	(identifier)	Recommended

network.io.direction MUST be one of the following:

Value	Description	Stability
transmit	transmit
receive	receive

Metric: system.network.errors

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.network.errors	Counter	{error}	Count of network errors detected [1]

[1]: Measured as:

• Linux: the errs column in /proc/dev/net (source).
• Windows: InErrors/OutErrors from GetIfEntry2.

Attribute	Type	Description	Examples	Requirement Level	Stability
network.io.direction	string	The network IO operation direction.	transmit	Recommended
system.device	string	The device identifier	(identifier)	Recommended

network.io.direction MUST be one of the following:

Value	Description	Stability
transmit	transmit
receive	receive

Metric: system.network.io

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.network.io	Counter	By

Attribute	Type	Description	Examples	Requirement Level	Stability
network.io.direction	string	The network IO operation direction.	transmit	Recommended
system.device	string	The device identifier	(identifier)	Recommended

network.io.direction MUST be one of the following:

Value	Description	Stability
transmit	transmit
receive	receive

Metric: system.network.connections

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.network.connections	UpDownCounter	{connection}

Attribute	Type	Description	Examples	Requirement Level	Stability
network.transport	string	OSI transport layer or inter-process communication method. [1]	tcp; udp	Recommended
system.device	string	The device identifier	(identifier)	Recommended
system.network.state	string	A stateless protocol MUST NOT set this attribute	close_wait	Recommended

[1]: The value SHOULD be normalized to lowercase.

Consider always setting the transport when setting a port number, since a port number is ambiguous without knowing the transport. For example different processes could be listening on TCP port 12345 and UDP port 12345.

network.transport has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value	Description	Stability
tcp	TCP
udp	UDP
pipe	Named or anonymous pipe.
unix	Unix domain socket

system.network.state MUST be one of the following:

Value	Description	Stability
close	close
close_wait	close_wait
closing	closing
delete	delete
established	established
fin_wait_1	fin_wait_1
fin_wait_2	fin_wait_2
last_ack	last_ack
listen	listen
syn_recv	syn_recv
syn_sent	syn_sent
time_wait	time_wait

Aggregate System Process Metrics

Description: System level aggregate process metrics captured under the namespace system.process. For metrics at the individual process level, see process metrics.

Metric: system.process.count

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.process.count	UpDownCounter	{process}	Total number of processes in each state

Attribute	Type	Description	Examples	Requirement Level	Stability
system.process.status	string	The process state, e.g., Linux Process State Codes	running	Recommended

system.process.status has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value	Description	Stability
running	running
sleeping	sleeping
stopped	stopped
defunct	defunct

Metric: system.process.created

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.process.created	Counter	{process}	Total number of processes created over uptime of the host

system.{os}. - OS Specific System Metrics

Instrument names for system level metrics that have different and conflicting meaning across multiple OSes should be prefixed with system.{os}. and follow the hierarchies listed above for different entities like CPU, memory, and network.

For example, UNIX load average over a given interval is not well standardized and its value across different UNIX like OSes may vary despite being under similar load:

Without getting into the vagaries of every Unix-like operating system in existence, the load average more or less represents the average number of processes that are in the running (using the CPU) or runnable (waiting for the CPU) states. One notable exception exists: Linux includes processes in uninterruptible sleep states, typically waiting for some I/O activity to complete. This can markedly increase the load average on Linux systems.

(source of quote, linux source code)

An instrument for load average over 1 minute on Linux could be named system.linux.cpu.load_1m, reusing the cpu name proposed above and having an {os} prefix to split this metric across OSes.

Metric: system.linux.memory.available

Name	Instrument Type	Unit (UCUM)	Description	Stability
system.linux.memory.available	UpDownCounter	By	An estimate of how much memory is available for starting new applications, without causing swapping [1]

[1]: This is an alternative to system.memory.usage metric with state=free. Linux starting from 3.14 exports "available" memory. It takes "free" memory as a baseline, and then factors in kernel-specific values. This is supposed to be more accurate than just "free" memory. For reference, see the calculations here. See also MemAvailable in /proc/meminfo.