Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

node_exporter_metrics' netdev timestamp doesn't update when inserting label in processor #9565

Closed
GlassBil opened this issue Nov 8, 2024 · 0 comments · Fixed by #9632
Closed

node_exporter_metrics' netdev timestamp doesn't update when inserting label in processor #9565

GlassBil opened this issue Nov 8, 2024 · 0 comments · Fixed by #9632
Assignees
@cosmo0920
Labels
status: waiting-for-triage
Milestone
Fluent Bit v3.2.2

Comments

@GlassBil
Copy link

GlassBil commented Nov 8, 2024

Bug Report

Describe the bug
The metric timestamp always reuses the first timestamp when inserting a label using metrics processor and then selecting the netdev metric after. If label is inserted after metrics_selector it updates the timestamp as expected.

To Reproduce
Minimal config:

service:
  flush: 5
  daemon: off
  log_level: info

pipeline:
  inputs:
    - name: node_exporter_metrics
      tag: metrics.node_metrics
      scrape_interval: 10s
      metrics: netdev
      processors:
        metrics:
          - name: labels
            insert: foo bar
          - name: metrics_selector
            metric_name: /node_network_(transmit|receive)_bytes_total/
            action: include

  outputs:
    - name: stdout
      match: '*'

Output:

2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="lo",foo="bar"} = 72974224473
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="dummy0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="eth0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="eth1",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="eth2",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="eth3",foo="bar"} = 6250404559365
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="can0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="can1",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="docker0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="br-4521a4325322",foo="bar"} = 40322446624
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="vethafb2f8e",foo="bar"} = 1421814568
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="vethf3f28ee",foo="bar"} = 151845108
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="veth4c03345",foo="bar"} = 488909550
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="lo",foo="bar"} = 72974224473
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="dummy0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="eth0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="eth1",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="eth2",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="eth3",foo="bar"} = 271969416143
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="can0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="can1",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="docker0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="br-4521a4325322",foo="bar"} = 47413716083
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="vethafb2f8e",foo="bar"} = 1342829263
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="vethf3f28ee",foo="bar"} = 127696069
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="veth4c03345",foo="bar"} = 876682996
----- NEXT UPDATE -----
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="lo",foo="bar"} = 72974224473
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="dummy0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="eth0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="eth1",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="eth2",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="eth3",foo="bar"} = 6250404559365
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="can0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="can1",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="docker0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="br-4521a4325322",foo="bar"} = 40322446624
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="vethafb2f8e",foo="bar"} = 1421814568
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="vethf3f28ee",foo="bar"} = 151845108
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="veth4c03345",foo="bar"} = 488909550
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="lo",foo="bar"} = 72974224473
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="dummy0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="eth0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="eth1",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="eth2",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="eth3",foo="bar"} = 271969416143
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="can0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="can1",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="docker0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="br-4521a4325322",foo="bar"} = 47413716083
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="vethafb2f8e",foo="bar"} = 1342829263
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="vethf3f28ee",foo="bar"} = 127696069
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="veth4c03345",foo="bar"} = 876682996
----- NEXT UPDATE -----
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="lo",foo="bar"} = 72974224473
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="dummy0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="eth0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="eth1",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="eth2",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="eth3",foo="bar"} = 6250404559365
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="can0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="can1",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="docker0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="br-4521a4325322",foo="bar"} = 40322446624
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="vethafb2f8e",foo="bar"} = 1421814568
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="vethf3f28ee",foo="bar"} = 151845108
2024-11-07T23:54:45.285863659Z node_network_receive_bytes_total{device="veth4c03345",foo="bar"} = 488909550
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="lo",foo="bar"} = 72974224473
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="dummy0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="eth0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="eth1",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="eth2",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="eth3",foo="bar"} = 271969416143
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="can0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="can1",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="docker0",foo="bar"} = 0
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="br-4521a4325322",foo="bar"} = 47413716083
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="vethafb2f8e",foo="bar"} = 1342829263
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="vethf3f28ee",foo="bar"} = 127696069
2024-11-07T23:54:45.285863659Z node_network_transmit_bytes_total{device="veth4c03345",foo="bar"} = 876682996

If we swap the order of "labels" and "metrics_selector" it works as expected:

service:
  flush: 5
  daemon: off
  log_level: info

pipeline:
  inputs:
    - name: node_exporter_metrics
      tag: metrics.node_metrics
      scrape_interval: 10s
      metrics: netdev
      processors:
        metrics:
          - name: metrics_selector
            metric_name: /node_network_(transmit|receive)_bytes_total/
            action: include
          - name: labels
            insert: foo bar

  outputs:
    - name: stdout
      match: '*'

Output:

2024-11-08T00:01:12.285883501Z node_network_receive_bytes_total{device="lo",foo="bar"} = 72987186709
2024-11-08T00:01:12.285883501Z node_network_receive_bytes_total{device="dummy0",foo="bar"} = 0
2024-11-08T00:01:12.285883501Z node_network_receive_bytes_total{device="eth0",foo="bar"} = 0
2024-11-08T00:01:12.285883501Z node_network_receive_bytes_total{device="eth1",foo="bar"} = 0
2024-11-08T00:01:12.285883501Z node_network_receive_bytes_total{device="eth2",foo="bar"} = 0
2024-11-08T00:01:12.285883501Z node_network_receive_bytes_total{device="eth3",foo="bar"} = 6251705170896
2024-11-08T00:01:12.285883501Z node_network_receive_bytes_total{device="can0",foo="bar"} = 0
2024-11-08T00:01:12.285883501Z node_network_receive_bytes_total{device="can1",foo="bar"} = 0
2024-11-08T00:01:12.285883501Z node_network_receive_bytes_total{device="docker0",foo="bar"} = 0
2024-11-08T00:01:12.285883501Z node_network_receive_bytes_total{device="br-4521a4325322",foo="bar"} = 40328809038
2024-11-08T00:01:12.285883501Z node_network_receive_bytes_total{device="vethafb2f8e",foo="bar"} = 1426506969
2024-11-08T00:01:12.285883501Z node_network_receive_bytes_total{device="vethf3f28ee",foo="bar"} = 152809611
2024-11-08T00:01:12.285883501Z node_network_receive_bytes_total{device="veth4c03345",foo="bar"} = 492106268
2024-11-08T00:01:12.285883501Z node_network_transmit_bytes_total{device="lo",foo="bar"} = 72987186709
2024-11-08T00:01:12.285883501Z node_network_transmit_bytes_total{device="dummy0",foo="bar"} = 0
2024-11-08T00:01:12.285883501Z node_network_transmit_bytes_total{device="eth0",foo="bar"} = 0
2024-11-08T00:01:12.285883501Z node_network_transmit_bytes_total{device="eth1",foo="bar"} = 0
2024-11-08T00:01:12.285883501Z node_network_transmit_bytes_total{device="eth2",foo="bar"} = 0
2024-11-08T00:01:12.285883501Z node_network_transmit_bytes_total{device="eth3",foo="bar"} = 272026374396
2024-11-08T00:01:12.285883501Z node_network_transmit_bytes_total{device="can0",foo="bar"} = 0
2024-11-08T00:01:12.285883501Z node_network_transmit_bytes_total{device="can1",foo="bar"} = 0
2024-11-08T00:01:12.285883501Z node_network_transmit_bytes_total{device="docker0",foo="bar"} = 0
2024-11-08T00:01:12.285883501Z node_network_transmit_bytes_total{device="br-4521a4325322",foo="bar"} = 47426946334
2024-11-08T00:01:12.285883501Z node_network_transmit_bytes_total{device="vethafb2f8e",foo="bar"} = 1351454563
2024-11-08T00:01:12.285883501Z node_network_transmit_bytes_total{device="vethf3f28ee",foo="bar"} = 128556612
2024-11-08T00:01:12.285883501Z node_network_transmit_bytes_total{device="veth4c03345",foo="bar"} = 882252450
----- NEXT UPDATE -----
2024-11-08T00:01:22.285820951Z node_network_receive_bytes_total{device="lo",foo="bar"} = 72987457899
2024-11-08T00:01:22.285820951Z node_network_receive_bytes_total{device="dummy0",foo="bar"} = 0
2024-11-08T00:01:22.285820951Z node_network_receive_bytes_total{device="eth0",foo="bar"} = 0
2024-11-08T00:01:22.285820951Z node_network_receive_bytes_total{device="eth1",foo="bar"} = 0
2024-11-08T00:01:22.285820951Z node_network_receive_bytes_total{device="eth2",foo="bar"} = 0
2024-11-08T00:01:22.285820951Z node_network_receive_bytes_total{device="eth3",foo="bar"} = 6251738706825
2024-11-08T00:01:22.285820951Z node_network_receive_bytes_total{device="can0",foo="bar"} = 0
2024-11-08T00:01:22.285820951Z node_network_receive_bytes_total{device="can1",foo="bar"} = 0
2024-11-08T00:01:22.285820951Z node_network_receive_bytes_total{device="docker0",foo="bar"} = 0
2024-11-08T00:01:22.285820951Z node_network_receive_bytes_total{device="br-4521a4325322",foo="bar"} = 40328943969
2024-11-08T00:01:22.285820951Z node_network_receive_bytes_total{device="vethafb2f8e",foo="bar"} = 1426598828
2024-11-08T00:01:22.285820951Z node_network_receive_bytes_total{device="vethf3f28ee",foo="bar"} = 152833019
2024-11-08T00:01:22.285820951Z node_network_receive_bytes_total{device="veth4c03345",foo="bar"} = 492182679
2024-11-08T00:01:22.285820951Z node_network_transmit_bytes_total{device="lo",foo="bar"} = 72987457899
2024-11-08T00:01:22.285820951Z node_network_transmit_bytes_total{device="dummy0",foo="bar"} = 0
2024-11-08T00:01:22.285820951Z node_network_transmit_bytes_total{device="eth0",foo="bar"} = 0
2024-11-08T00:01:22.285820951Z node_network_transmit_bytes_total{device="eth1",foo="bar"} = 0
2024-11-08T00:01:22.285820951Z node_network_transmit_bytes_total{device="eth2",foo="bar"} = 0
2024-11-08T00:01:22.285820951Z node_network_transmit_bytes_total{device="eth3",foo="bar"} = 272027803413
2024-11-08T00:01:22.285820951Z node_network_transmit_bytes_total{device="can0",foo="bar"} = 0
2024-11-08T00:01:22.285820951Z node_network_transmit_bytes_total{device="can1",foo="bar"} = 0
2024-11-08T00:01:22.285820951Z node_network_transmit_bytes_total{device="docker0",foo="bar"} = 0
2024-11-08T00:01:22.285820951Z node_network_transmit_bytes_total{device="br-4521a4325322",foo="bar"} = 47427239862
2024-11-08T00:01:22.285820951Z node_network_transmit_bytes_total{device="vethafb2f8e",foo="bar"} = 1351625782
2024-11-08T00:01:22.285820951Z node_network_transmit_bytes_total{device="vethf3f28ee",foo="bar"} = 128575685
2024-11-08T00:01:22.285820951Z node_network_transmit_bytes_total{device="veth4c03345",foo="bar"} = 882398167
----- NEXT UPDATE -----
2024-11-08T00:01:32.285887883Z node_network_receive_bytes_total{device="lo",foo="bar"} = 72987800501
2024-11-08T00:01:32.285887883Z node_network_receive_bytes_total{device="dummy0",foo="bar"} = 0
2024-11-08T00:01:32.285887883Z node_network_receive_bytes_total{device="eth0",foo="bar"} = 0
2024-11-08T00:01:32.285887883Z node_network_receive_bytes_total{device="eth1",foo="bar"} = 0
2024-11-08T00:01:32.285887883Z node_network_receive_bytes_total{device="eth2",foo="bar"} = 0
2024-11-08T00:01:32.285887883Z node_network_receive_bytes_total{device="eth3",foo="bar"} = 6251771357740
2024-11-08T00:01:32.285887883Z node_network_receive_bytes_total{device="can0",foo="bar"} = 0
2024-11-08T00:01:32.285887883Z node_network_receive_bytes_total{device="can1",foo="bar"} = 0
2024-11-08T00:01:32.285887883Z node_network_receive_bytes_total{device="docker0",foo="bar"} = 0
2024-11-08T00:01:32.285887883Z node_network_receive_bytes_total{device="br-4521a4325322",foo="bar"} = 40329097464
2024-11-08T00:01:32.285887883Z node_network_receive_bytes_total{device="vethafb2f8e",foo="bar"} = 1426716723
2024-11-08T00:01:32.285887883Z node_network_receive_bytes_total{device="vethf3f28ee",foo="bar"} = 152856675
2024-11-08T00:01:32.285887883Z node_network_receive_bytes_total{device="veth4c03345",foo="bar"} = 492254545
2024-11-08T00:01:32.285887883Z node_network_transmit_bytes_total{device="lo",foo="bar"} = 72987800501
2024-11-08T00:01:32.285887883Z node_network_transmit_bytes_total{device="dummy0",foo="bar"} = 0
2024-11-08T00:01:32.285887883Z node_network_transmit_bytes_total{device="eth0",foo="bar"} = 0
2024-11-08T00:01:32.285887883Z node_network_transmit_bytes_total{device="eth1",foo="bar"} = 0
2024-11-08T00:01:32.285887883Z node_network_transmit_bytes_total{device="eth2",foo="bar"} = 0
2024-11-08T00:01:32.285887883Z node_network_transmit_bytes_total{device="eth3",foo="bar"} = 272029198665
2024-11-08T00:01:32.285887883Z node_network_transmit_bytes_total{device="can0",foo="bar"} = 0
2024-11-08T00:01:32.285887883Z node_network_transmit_bytes_total{device="can1",foo="bar"} = 0
2024-11-08T00:01:32.285887883Z node_network_transmit_bytes_total{device="docker0",foo="bar"} = 0
2024-11-08T00:01:32.285887883Z node_network_transmit_bytes_total{device="br-4521a4325322",foo="bar"} = 47427557011
2024-11-08T00:01:32.285887883Z node_network_transmit_bytes_total{device="vethafb2f8e",foo="bar"} = 1351850694
2024-11-08T00:01:32.285887883Z node_network_transmit_bytes_total{device="vethf3f28ee",foo="bar"} = 128595123
2024-11-08T00:01:32.285887883Z node_network_transmit_bytes_total{device="veth4c03345",foo="bar"} = 882514060

Expected behavior
Expect the timestamp to update with every scrape.

Your Environment

  • Version used: 3.1.10
  • Operating System and version: Ubuntu 20.04 ARM64
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cosmo0920 cosmo0920

Labels
status: waiting-for-triage
Projects
None yet
Milestone
Fluent Bit v3.2.2
3 participants
@edsiper @cosmo0920 @GlassBil