The node_memory_Cached_bytes metric collected by Node Exporter differs by an order of magnitude compared to the value observed using the free command.

[woodoow]

Host operating system: output of uname -a

Linux xxx 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter command line flags

node_exporter log output

Jun 01 23:21:15 systemd[1]: Stopping node_export...
Jun 01 23:21:15 systemd[1]: Stopped node_export.
Jun 01 23:21:38 systemd[1]: Started node_export.
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.098Z caller=node_exporter.go:193 level=info msg="Starting node_exporter" version="(version=1.8.1, branch=HEAD, revision=400c397993
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.098Z caller=node_exporter.go:194 level=info msg="Build context" build_context="(go=go1.22.3, platform=linux/amd64, user=root@7afbf
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.098Z caller=node_exporter.go:196 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unpriv
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.099Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(z
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude"
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" fla
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:111 level=info msg="Enabled collectors"
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=arp
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=bcache
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=bonding
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=btrfs
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=conntrack
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=cpu
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=cpufreq
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=diskstats
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=dmi
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=edac
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=entropy
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=fibrechannel
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=filefd
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=filesystem
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=hwmon
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=infiniband
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=ipvs
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=loadavg
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=mdadm
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=meminfo
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=netclass
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=netdev
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=netstat
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=nfs
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=nfsd
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=nvme
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=os
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=powersupplyclass
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=pressure
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=rapl
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=schedstat
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=selinux
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=sockstat
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=softnet
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=stat
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=tapestats
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=textfile
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=thermal_zone
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=time
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=timex
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=udp_queues
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=uname
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=vmstat
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=watchdog
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=xfs
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=zfs
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9100
Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9100

Are you running node_exporter in Docker?

What did you do that produced an error?

[root@www node_exporter]# free -b
total used free shared buff/cache available
Mem: 8201244672 2074763264 382656512 383348736 5743824896 5429821440

What did you expect to see?

The actual value of buffer and the collected value differ by an order of magnitude.

What did you see instead?

HELP node_memory_Cached_bytes Memory information field Cached_bytes.

TYPE node_memory_Cached_bytes gauge

node_memory_Cached_bytes 5.14207744e+08

[woodoow]

[root@ node_exporter]# ./node_exporter --version
node_exporter, version 1.8.1 (branch: HEAD, revision: 400c397)
build user: root@7afbff271a3f
build date: 20240521-18:36:22
go version: go1.22.3
platform: linux/amd64
tags: unknown

[SuperQ]

What do you get for cat /proc/meminfo at the same time as when you capture the metrics?

[woodoow]

What do you get for cat /proc/meminfo at the same time as when you capture the metrics?

/proc/meminfo
[root@www ~]# cat /proc/meminfo
MemTotal: 8009028 kB
MemFree: 326216 kB
MemAvailable: 5335908 kB
Buffers: 0 kB
Cached: 566940 kB
SwapCached: 8292 kB
Active: 1297436 kB
Inactive: 1048700 kB
Active(anon): 1178748 kB
Inactive(anon): 967744 kB
Active(file): 118688 kB

The value for Cached 566940 kB, is incorrect.What can I do about it?

[SuperQ]

That is what the kernel reports, so if you can prove it's wrong, it's a kernel bug. Either way, the node_exporter only reports what the kernel tells it to. The issue is not in the node_exporter.