Additional system attributes

[bhupenbisht]

Component(s)

Describe the issue you're reporting

Looking for system uptime metric . This metrics would provide useful context on the machine that is generating telemetry and would be useful for infrastructure monitoring.

1. Server Uptime(since last boot)

[github-actions]

Pinging code owners:

• processor/resourcedetection: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[bhupenbisht]

Server uptime is a crucial metrics for infrastructure observability point of view. Could anyone look into this request, as all other tool providing the same details.

[github-actions]

Pinging code owners for receiver/hostmetrics: @dmitryax @braydonk. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[andrzej-stencel]

This seems to fit into Host Metrics receiver more, so I have updated the label. It has been requested before (the issue was closed as inactive):

• Support for scraping a system "uptime" metric #14130

I continue to believe this is a reasonable addition. Would you be up for implementing it @bhupenbisht?

[bhupenbisht]

@astencel-sumo sure i would like contribute.. pls let me know, how can i..?

[andrzej-stencel]

@bhupenbisht You can prepare a pull request implementing the change. See the original issue for tips on how to implement this (a new system scraper in the Host Metrics receiver).

[kevinnoel-be]

We do have an internal receiver/scraper to gather uptime, if still interested we could push it part of the hostmetrics receiver

[andrzej-stencel]

@kevinnoel-be this sounds great. Is this code open source - can you point to it to take a look?

[kevinnoel-be]

@andrzej-stencel It is in an internal/private OTel build, so you won't be able to see it. We created a separate receiver with scraper as we cannot extend the hostmetrics receiver and we didn't want to fork it only for this metric.

I could take same time to port this, but wondering what would be an appropriate naming for this metric as I don't see much movement on open-telemetry/semantic-conventions#648
Our internal definition for it is:

metrics:
  system.uptime:
    enabled: true
    description: The time since the system started
    unit: s
    sum:
      value_type: int
      monotonic: true
      aggregation_temporality: cumulative

[andrzej-stencel]

Understood, thanks for your response @kevinnoel-be.

I believe system.uptime is a good name and I think we can implement this without waiting on a semantic convention for this.

Also see #14130 for previous discussion and considerations regarding implementing it.

[andrzej-stencel]

@kevinnoel-be do you want to prepare a PR adding the system.uptime metric to the Host Metrics receiver? If yes, I'll assign this issue to you.

[kernelpanic77]

@andrzej-stencel I'd like to contribute as well, If it's okay with @kevinnoel-be, Can I take a shot at this?

[kevinnoel-be]

@kernelpanic77 Sure.
If you cannot find the time, ping me back and I'll pick it up

[andrzej-stencel]

Sure, thanks for offering your help @kernelpanic77! Assigning this isuse to you.

[mx-psi]

I don't think we should add this metric to the hostmetrics receiver without adding it to semantic conventions. It's fine to add it under a feature flag as a PoC for the semantic conventions, but we must not risk deviating from semantic conventions here.

[kernelpanic77]

Hi @kevinnoel-be,

I understand that the repository is internal. Could you guide me on how you implemented the uptime metric, or is there a way I can take a look at the implementation in your fork?

@andrzej-stencel @mx-psi we can create a draft PR for this until the semantic conventions is approved.

[kevinnoel-be]

@kernelpanic77 Created a new uptime scraper in the hostmetrics receiver using shirou/gopsutil host.UptimeWithContext() method behind the scenes

[mx-psi]

we can create a draft PR for this until the semantic conventions is approved.

To be clear: to my knowledge, nobody is actively working on this on the semantic conventions side. I am happy to guide you through the process if you want to contribute it yourself to semantic conventions

[krantishetty]

@kevinnoel-be is there any update on uptime metrics? We are looking forward

[andrzej-stencel]

Previous work:

• Add process.uptime and system.uptime metrics to semantic conventions opentelemetry-specification#2824

[krantishetty]

This is covering only process uptime, however we are looking for server uptime which can be captured from /proc/uptime. PR 2824 not covered with server uptime

[kernelpanic77]

@kevinnoel-be I believe that the existing code for the processes scraper, is already calling gopsutils/host, so I think we can use the existing scraper to scrape host.uptimeWithContext().

[kernelpanic77]

@krantishetty are you referring to application uptime ? Could you give an example of your usecase for more clarity.

[bhupenbisht]

@kernelpanic77 i believe krantishetty is talking about otel process uptime. Here we are looking for server uptime.

[krantishetty]

@krantishetty are you referring to application uptime ? Could you give an example of your usecase for more clarity.

Im talking about the server uptime which is last reboot of the server. Suppose if i give command # uptime, it shows server uptime since last reboot.
This can read from /proc/uptime in linux machine

[bhupenbisht]

@kernelpanic77 any luck on system uptime ...?

[krantishetty]

Hi
Any luck on system uptime?

[Ruppsn]

Hey,
this is a important metric. Any news?

[andrzej-stencel]

Sure, thanks for offering your help @kernelpanic77! Assigning this isuse to you.

@kernelpanic77 will you be working on this?

[kernelpanic77]

Sure, thanks for offering your help @kernelpanic77! Assigning this isuse to you.

@kernelpanic77 will you be working on this?

Hi
Sorry I have been away for a while. I'll raise a PR by end of this week.

[kevinnoel-be]

Created open-telemetry/semantic-conventions#1507

[bhupenbisht]

@kernelpanic77 .. hope same will work for windows servers..

[kevinnoel-be]

@bhupenbisht I haven't tested Windows specifically but it should work as documented: Linux, Windows & Mac

[braydonk]

This feature will be in the v0.113.0 release! Thanks for the implementation @kevinnoel-be.

Once v0.113.0 is out, you'll be able to add system to your list of scrapers to get system.uptime.

The metric is implemented as per the guidance in open-telemetry/oteps#185. (Haven't finished the work of adapting this OTEP into official semconv guidance, but it is for all intents and purposes our plan). In that OTEP you can find guidance for how to use the metric for common dashboard/alert scenarios.

[bhupenbisht]

@kevinnoel-be Hi Kevin, I am trying to pull uptime metrics using otel 114 version but getting error "error decoding 'receivers': error reading configuration for "hostmetrics": invalid scraper key: uptime". Could you please advise.

[Ruppsn]

Scrapers name is not uptime. It is system

Example

  hostmetrics:
    collection_interval: 1m
    scrapers:
      filesystem: {}
      memory: {}
      system: {} 

[bhupenbisht]

@Ruppsn Thanks a lot .. actually got confused, we there is nothing mentioned about it. Now i can see the uptime data.

[bhupenbisht]

@kevinnoel-be .. Thank you so much for adding the uptime feature. It’s a crucial addition. We truly appreciate your effort and hard work in making this happen.