Monitoring: allow specifying /proc or hostfs path.

[adriansr]

Discuss thread: https://discuss.elastic.co/t/metricbeat-k8s-error-getting-group-status-open-proc-pid-cgroup/255371

Since #21113 monitoring includes cgroup stats. However, those are hardcoded to read from /proc. This clashes with use cases where proc is available in a different location and causes the following error to be printed to the logs every 30s:

ERROR instance/metrics.go:285 error getting group status: open /proc/33861/cgroup: no such file or directory

Metricbeat's system module has a command-line argument, -system.hostfs, that allows to specify an alternate path to the root filesystem for these metrics.

We could have a similar flag / configuration option for the monitoring metrics.

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[fearful-symmetry]

This definitely seems like a bug. Might be an easy fix, gonna poke around.

[schans]

This is where it is hardcoded to "" (aka /):
https://github.com/elastic/beats/pull/21113/files#diff-2141c2342ca93ab304c2efd9425679f0f7ef598b9cdb9115213591671bc9c1a8R274

The first parameter of NewReader in "cgroups, err := cgroup.NewReader("", true)" should be confgiruable.

See also: https://github.com/elastic/gosigar/blob/master/cgroup/reader.go#L61

[fearful-symmetry]

Yep. Sorry for the delay, been distracted with 7.11, and then the Holidays happened. We can probably just use hostfs for the cgroup reader here.

[fearful-symmetry]

So, it looks like I actually approved a PR here a few weeks ago to fix this: #22879

@adriansr can you confirm that this addresses the issue?

[schans]

[Outdated, please see next comment.]

Hi,

#22879 has fix version 7.11.0. I have updated to that version but still get the error. I have also tried to set the env var LIBBEAT_MONITORING_CGROUPS_HIERARCHY_OVERRIDE to "/hostfs" but that doesn't seem to make a difference.

Metricebeat gets started in the container with "metricbeat -c /etc/metricbeat.yml -e -system.hostfs=/hostfs"

So I'm not sure what's going on here. Is it not part of the release or have a missed some configuration option?

I did notice I now get the updated error message "error getting cgroup .." instead of "error getting group .."

Thanks
Bart

[schans]

Ok. Did a bit more digging and got some new insights so please ignore above.

It does seem to pick up the system.hostfs setting. So far so good.

The error:
{"log.level":"error","@timestamp":"2021-02-17T18:41:27.209Z","log.origin":{"file.name":"metrics/metrics.go","file.line":301},"message":"error getting cgroup stats: open /proc/23615/cgroup: no such file or directory","ecs.version":"1.6.0"}

Now if I check in the container I notice that /proc/23615/cgroup does not exist but /hostfs/proc/23615/cgroup does. And furthermore the pid 23615 is metricbeat self:

# cat /hostfs/proc/23615/cmdline
./metricbeat -N -v -c /etc/metricbeat.yml --path.data/tmp/bla -e -system.hostfs=/hostfs

So it looks like there is a bug with the monitoring of its own process which doesn't take the system.hostfs setting into account?

Bart

[fearful-symmetry]

@schans so the error is only happening with metricbeat's own pid?

[schans]

Indeed. It is only happening for metricbeat's own pid (as seen from the host).

[schans]

From what I can find:

The reportBeatCgroups function in
https://github.com/elastic/beats/blob/7.11.0/libbeat/cmd/instance/metrics/metrics.go#L277

tries to get the cgoup stats of its own pid (process.GetSelfPid()) by calling cgroups.GetStatsForProcess(pid):

https://github.com/elastic/beats/blob/7.11.0/libbeat/cmd/instance/metrics/metrics.go#L299

It uses the Reader options set in that method with NewReaderOptions:

https://github.com/elastic/beats/blob/7.11.0/libbeat/cmd/instance/metrics/metrics.go#L287

which is missing the RootfsMountpoint .

Then gosigar tries to get the stats by calling ProcessCgroupPaths with the rootfsMountpoint which defaults to "/" if not set:

https://github.com/elastic/gosigar/blob/master/cgroup/reader.go#L97

The actual open call is in the util file:

https://github.com/elastic/gosigar/blob/master/cgroup/util.go#L235

which constructs the path based on the rootfsMountpoint and pid.

I don't know (yet) what the best option is to pass the command line flag value of system.hostfs so that it is available in reportBeatCgroups().

HTH

[schans]

Maybe getting it through the system module is an option? Like in https://github.com/elastic/beats/blob/master/metricbeat/module/system/process/process.go#L87

[fearful-symmetry]

Yep, beat me to it. We'll need to modify the cmd/instance startup to look for the rootfs variable.

[schans]

Hi all!

Any progress on this? Any way I can help?

[fearful-symmetry]

Assuming nothing gets in the way, I'd like to get this done this week, since it should be a fairly simple fix.

[fearful-symmetry]

Fix here: #24334

Took a bit longer that expected, as fixing it in a not-ugly way took a bit of time.

[Irindul]

Hi ! I seem to be having a similar issue here.
I'm using filebeat v7.15.0 inside GKE v1.21.5-gke.1302.

I have this error in my container :

error determining cgroups version: error reading /hostfs/proc/7/cgroup: open /hostfs/proc/7/cgroup: no such file or directory

PID 7 is metricbeat. Metricbeat is started in the container by
metricbeat -e -E http.enabled=true --system.hostfs=/hostfs

When I run a shell in the container, I can see that /hostfs/proc exists but I don't have any folder for PID 7. I do have other PIDs.
However, I can find /proc/7/cgroup. It seems the --system.hostfs option is still not used. I saw that you merged #24334 but this issue is still open.

Is there a way to fix this ?

[fearful-symmetry]

@Irindul There's currently a handful of inconsistencies in how metricbeat handles hostfs values. There should be a fix in an upcoming release.

[ulrichwinter]

I would like to add that setting monitoring.enabled: false as suggested, does not avoid this error message.

metricbeat version used: 7.12.1

[nugmanovtimur]

Is there any update on this? Seems like it is still an issue on 7.16.2.

[fearful-symmetry]

@nugmanovtimur Are you getting the same error message as the original issue? What platform is metricbeat running on? What OS distro and version?

[n0ur-sh]

Hello @fearful-symmetry I'm getting the below error while using version 7.16.2 over a GKE cluster and I still get the below error, along with some other errors.

ERROR   metrics/metrics.go:304  error determining cgroups version: error reading /proc/<PID>/cgroup: open /proc/1079196/cgroup: no such file or directory

2022-01-04T08:54:50.832Z        ERROR   [kubernetes.state_deployment]   state_deployment/state_deployment.go:106        unexpected status code 400 from server
2022-01-04T08:54:51.095Z        ERROR   [kubernetes.state_replicaset]   state_replicaset/state_replicaset.go:106        unexpected status code 400 from server
2022-01-04T08:54:51.239Z        ERROR   [kubernetes.state_statefulset]  state_statefulset/state_statefulset.go:106      unexpected status code 400 from server
2022-01-04T08:54:51.554Z        ERROR   [kubernetes.state_job]  state_job/state_job.go:124      unexpected status code 400 from server
2022-01-04T08:54:51.659Z        ERROR   [kubernetes.state_daemonset]    state_daemonset/state_daemonset.go:105  unexpected status code 400 from server
2022-01-04T08:54:52.179Z        ERROR   [kubernetes.state_service]      state_service/state_service.go:100      unexpected status code 400 from server
2022-01-04T08:54:52.289Z        ERROR   module/wrapper.go:259   Error fetching data for metricset kubernetes.state_cronjob: error getting family metrics: unexpected status code 400 from server
2022-01-04T08:54:53.649Z        ERROR   [kubernetes.state_resourcequota]        state_resourcequota/state_resourcequota.go:81   unexpected status code 400 from server
2022-01-04T08:54:58.408Z        ERROR   module/wrapper.go:259   Error fetching data for metricset kubernetes.state_container: error getting families: unexpected status code 400 from server
2022-01-04T08:54:58.425Z        ERROR   [kubernetes.state_pod]  state_pod/state_pod.go:108      unexpected status code 400 from server
2022-01-04T08:55:00.838Z        ERROR   [kubernetes.state_deployment]   state_deployment/state_deployment.go:106        unexpected status code 400 from server
2022-01-04T08:55:01.097Z        ERROR   [kubernetes.state_replicaset]   state_replicaset/state_replicaset.go:106        unexpected status code 400 from server
2022-01-04T08:55:01.240Z        ERROR   [kubernetes.state_statefulset]  state_statefulset/state_statefulset.go:106      unexpected status code 400 from server
2022-01-04T08:55:01.555Z        ERROR   [kubernetes.state_job]  state_job/state_job.go:124      unexpected status code 400 from server
2022-01-04T08:55:01.660Z        ERROR   [kubernetes.state_daemonset]    state_daemonset/state_daemonset.go:105  unexpected status code 400 from server
2022-01-04T08:55:02.180Z        ERROR   [kubernetes.state_service]      state_service/state_service.go:100      unexpected status code 400 from server
2022-01-04T08:55:02.289Z        ERROR   module/wrapper.go:259   Error fetching data for metricset kubernetes.state_cronjob: error getting family metrics: unexpected status code 400 from server
2022-01-04T08:55:03.649Z        ERROR   [kubernetes.state_resourcequota]        state_resourcequota/state_resourcequota.go:81   unexpected status code 400 from server
2022-01-04T08:55:08.409Z        ERROR   module/wrapper.go:259   Error fetching data for metricset kubernetes.state_container: error getting families: unexpected status code 400 from server
2022-01-04T08:55:08.426Z        ERROR   [kubernetes.state_pod]  state_pod/state_pod.go:108      unexpected status code 400 from server
2022-01-04T08:55:10.833Z        ERROR   [kubernetes.state_deployment]   state_deployment/state_deployment.go:106        unexpected status code 400 from server
2022-01-04T08:55:11.098Z        ERROR   [kubernetes.state_replicaset]   state_replicaset/state_replicaset.go:106        unexpected status code 400 from server
2022-01-04T08:55:11.239Z        ERROR   [kubernetes.state_statefulset]  state_statefulset/state_statefulset.go:106      unexpected status code 400 from server
2022-01-04T08:55:11.554Z        ERROR   [kubernetes.state_job]  state_job/state_job.go:124      unexpected status code 400 from server
2022-01-04T08:55:11.660Z        ERROR   [kubernetes.state_daemonset]    state_daemonset/state_daemonset.go:105  unexpected status code 400 from server
2022-01-04T08:55:12.180Z        ERROR   [kubernetes.state_service]      state_service/state_service.go:100      unexpected status code 400 from server
2022-01-04T08:55:12.290Z        ERROR   module/wrapper.go:259   Error fetching data for metricset kubernetes.state_cronjob: error getting family metrics: unexpected status code 400 from server
2022-01-04T08:55:13.650Z        ERROR   [kubernetes.state_resourcequota]        state_resourcequota/state_resourcequota.go:81   unexpected status code 400 from server

I tried commenting out the volume and volume mount from the yaml file to get over it but still got the same error.

[fearful-symmetry]

@n0ur-sh Are you seeing the error determining cgroups version error with any other processes besides 1079196? This error is from metricbeat's self-monitoring, and not from the actual metricbeat reporting.

[n0ur-sh]

@fearful-symmetry no the PID remains the same throughout the errors if that's what you mean. I'm still trying to find a proper workaround solution but in the best-case scenario, I still get the cgroup error. I'd be grateful if you recommended any workaround that would still let me have visibility until this issue is fully resolved.

I still have two points that I'd wish to validate:

1. Is this issue actually affecting the way the metricbeat agent work (in terms of monitoring...)?
2. Is this specific to cloud-managed K8s clusters, in other words, will having this deployment in an on-prem K8s deployment resolve the issue in terms of accessing the addressed directory?

Thanks.

UPDATE

@fearful-symmetry I tried testing the issue on the GKE node itself and contrary to what the ERROR says the file and directory exist and are accessible as shown below

POD Log

% kubectl get pods -owide --namespace=kube-system -l app=metricbeat-1641320608-metricbeat
NAME                                     READY   STATUS    RESTARTS   AGE   IP           NODE                                      NOMINATED NODE   READINESS GATES
metricbeat-1641320608-metricbeat-6rnt8   1/1     Running   1          66m   10.48.0.15   gke-elk-test-default-pool-870ca581-0r0l 

% kubectl logs metricbeat-1641320608-metricbeat-6rnt8 --namespace=kube-system | grep  -e cgroup 

2022-01-04T19:29:37.389Z        ERROR   metrics/metrics.go:304  error determining cgroups version: error reading /proc/3157431/cgroup: open /proc/3157431/cgroup: no such file or directory

PID is accurate

gke-elk-test-default-pool-870ca581-0r0l /proc # ps -aux | grep 3157431
root     3157431  2.7  4.3 1336796 174512 ?      Sl   19:29   0:03 metricbeat -e -E http.enabled=true --system.hostfs=/hostfs

The file actually exists

gke-elk-test-default-pool-870ca581-0r0l /proc # ls -l /proc/3157431/cgroup 
-r--r--r-- 1 root root 0 Jan  4 19:29 /proc/3157431/cgroup

gke-elk-test-default-pool-870ca581-0r0l /proc # cat /proc/3157431/cgroup
12:hugetlb:/kubepods/burstable/pode80e2033-8e44-4eb1-b67a-8bea308b1b7f/746cfc01bb8171d590ca5c5b98a33c3ae793de8c364adf357899df7c603cb2e3
11:devices:/kubepods/burstable/pode80e2033-8e44-4eb1-b67a-8bea308b1b7f/746cfc01bb8171d590ca5c5b98a33c3ae793de8c364adf357899df7c603cb2e3
10:freezer:/kubepods/burstable/pode80e2033-8e44-4eb1-b67a-8bea308b1b7f/746cfc01bb8171d590ca5c5b98a33c3ae793de8c364adf357899df7c603cb2e3
9:perf_event:/kubepods/burstable/pode80e2033-8e44-4eb1-b67a-8bea308b1b7f/746cfc01bb8171d590ca5c5b98a33c3ae793de8c364adf357899df7c603cb2e3
8:blkio:/kubepods/burstable/pode80e2033-8e44-4eb1-b67a-8bea308b1b7f/746cfc01bb8171d590ca5c5b98a33c3ae793de8c364adf357899df7c603cb2e3
7:cpuset:/kubepods/burstable/pode80e2033-8e44-4eb1-b67a-8bea308b1b7f/746cfc01bb8171d590ca5c5b98a33c3ae793de8c364adf357899df7c603cb2e3
6:memory:/kubepods/burstable/pode80e2033-8e44-4eb1-b67a-8bea308b1b7f/746cfc01bb8171d590ca5c5b98a33c3ae793de8c364adf357899df7c603cb2e3
5:pids:/kubepods/burstable/pode80e2033-8e44-4eb1-b67a-8bea308b1b7f/746cfc01bb8171d590ca5c5b98a33c3ae793de8c364adf357899df7c603cb2e3
4:net_cls,net_prio:/kubepods/burstable/pode80e2033-8e44-4eb1-b67a-8bea308b1b7f/746cfc01bb8171d590ca5c5b98a33c3ae793de8c364adf357899df7c603cb2e3
3:rdma:/
2:cpu,cpuacct:/kubepods/burstable/pode80e2033-8e44-4eb1-b67a-8bea308b1b7f/746cfc01bb8171d590ca5c5b98a33c3ae793de8c364adf357899df7c603cb2e3
1:name=systemd:/kubepods/burstable/pode80e2033-8e44-4eb1-b67a-8bea308b1b7f/746cfc01bb8171d590ca5c5b98a33c3ae793de8c364adf357899df7c603cb2e3
0::/system.slice/containerd.service

I hope this helps out

[fearful-symmetry]

@n0ur-sh Based on that one error you showed me, the cgroups issue is limited to shelf-monitoring, and shouldn't effect metrics; I believe those 400 errors are unrelated. The hostfs flag currently isn't visible to metricbeat's internal metrics, which will take a bit of time to fix.

[keifgwinn]

I'm seeing this, with 7.16.2
2022-01-27T11:04:11.798Z ERROR metrics/metrics.go:304 error determining cgroups version: error reading /proc/28703/cgroup: open /proc/28703/cgroup: no such file or directory

ami-0d1872e4dda33c569 is amazon-eks-node-1.19-v20210519
EKS Kubernetes Worker AMI with AmazonLinux2 image, (k8s: 1.19.6, docker: 19.03.13ce-1.amzn2, containerd: 1.4.1-2.amzn2)

installed metric beats with the published helm chart for it

[kbujold]

This issue is also seen in 7.16.3. A workaround until resolved would be great.

[ayushmathur86]

ERROR metrics/metrics.go:304 error determining cgroups version: error reading /proc/3759719/cgroup: open /proc/3759719/cgroup: no such file or directory
Cloud Provider: AWS
Version: Metricbeat 7.16.3
Environment: Red Hat OpenShift Container Platform 4.8.
However, it happens only with daemonset metricbeat, but no such error is logged by singleton pod which also has internal monitoring configured.

DS YAML:
monitoring: enabled: true cluster_uuid: "${CLUSTER_UUID}" elasticsearch: protocol: https hosts: ['${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}'] username: ${ELASTICSEARCH_USERNAME} password: ${ELASTICSEARCH_PASSWORD} ssl: enabled: true certificate_authorities: [ "metricbeat-ca.crt" ] certificate: metricbeat.crt key:metricbeat.key

Singleton YAML:
monitoring: enabled: true cluster_uuid: "${CLUSTER_UUID}" elasticsearch: protocol: https hosts: ['${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}'] username: ${ELASTICSEARCH_USERNAME} password: ${ELASTICSEARCH_PASSWORD} ssl: enabled: true certificate_authorities: [ "metricbeat-ca.crt" ] certificate: metricbeat.crt key:metricbeat.key

[fearful-symmetry]

@ayushmathur86 @kbujold @keifgwinn , a fix was merged here: #30228

Just to clarify, this is limited to preventing libbeat's self-monitoring from reading cgroups info, and should have no impact on actual upstream reported metrics.

I'm a little baffled by the actual reported issues here, though. I was able to reproduce this in k8s with kind, and the actual issue wasn't that libbeat wasn't using hostfs, the issue is that it was using system.hostfs. A container's internal PIDs live in their own namespace, and the PID a containerized process sees will not be the same PID the container host has. The "fix" that I tested was to have libbeat's cgroup monitoring just ignore /hostfs. @adriansr does this make sense? Is there another layer to this I'm missing?

[framsouza]

@fearful-symmetry does it mean that it's necessary to change the startup arguments for metricbeats? cc @ayushmathur86

[fearful-symmetry]

@framsouza the fix shouldn't require any change to config.

[ayushmathur86]

@fearful-symmetry that's a good news. May I know in which version can we expect the fix to be available ?
cc: @framsouza

[fearful-symmetry]

@ayushmathur86 this'll be in 8.2.

[kbujold]

@fearful-symmetry can the fix be merged into 7.16? We are using 7.16.3 and we would like to have a solution since these errors continuously flood the logs.

[fearful-symmetry]

@kbujold as far as I'm aware, there won't be any other 7.16 releases, but we could try to backport into a future 7.17 release. Otherwise the change will be in 8.2.

[kbujold]

@fearful-symmetry Backport to 7.17 would help thanks!

[kbujold]

We have upgraded to 7.17 is there any chance a fix could be backport into 7.17? Even if its not released we could just patch it.

[fearful-symmetry]

We were initially hesitant to backport this, but I've made a conservative patch for 7.17 here: #31002

[jlind23]

@fearful-symmetry can I close this issue then as you backported your PR?

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[fearful-symmetry]

@jlind23 oop, missed your ping. Yep, closing this.

[abstract-entity]

+1 got this bug in 8.1.2, waiting for 8.2

[abstract-entity]

Between the lines i've seen:
{"log.level":"warn","@timestamp":"2022-04-08T07:58:04.214Z","log.logger":"cfgwarn","log.origin":{"file.name":"sysinit/init.go","file.line":79},"message":"DEPRECATED: The --system.hostfs flag will be removed in the future and replaced by a config value. Will be removed in version: 8.0.0","service.name":"metricbeat","ecs.version":"1.6.0"}

I've added an env variable in place of command line flag:

    - name: SYSTEM_HOSTFS
      value: /hostfs

It seems to work for me in 8.1.2, i'm no longer spammed in log by this message

[azhinu]

With 8.1.3 adding following to metricbeat config file will cause ignoring SYSTEM_HOSTFS env and -system.hostfs flag.

metricbeat.modules:
- module: system
  hostfs: "/hostfs"

With this config option, I'm getting error determining cgroups version: error reading /proc/<PID>/cgroup.

[jsoule6]

conservative patch

Hello,

We are running 7.17.0 and still seeing the error flood the logs:

2022-06-03T15:25:22.977Z ERROR metrics/metrics.go:304 error determining cgroups version: error reading /proc/2395793/cgroup: open /proc/2395793/cgroup: no such file or directory

Was that patch made available yet?

[xpatux]

Same problem here with 8.1.0:

Service config in docker:

services:
  metricbeat-es:
    image: docker.elastic.co/beats/metricbeat:8.1.0
    command: ["-system.hostfs=/hostfs", "-e", "--environment=container"]
    user: root
    cap_add:
      - SYS_ADMIN
      - NET_ADMIN
      - SYS_PTRACE
      - DAC_READ_SEARCH 
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /proc:/hostfs/proc:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /:/hostfs:ro

Metricbeat config:

- module: system
  metricsets:
    - cpu             # CPU usage
    - load            # CPU load averages
    - memory          # Memory usage
    - network         # Network IO
    - process         # Per process metrics
    - process_summary # Process summary
    - uptime          # System Uptime
    - socket_summary  # Socket summary
    - filesystem
  filesystem.ignore_types: [nfs, smbfs, autofs]
  enabled: true
  period: 30s
  processes: ['.*']
  hostfs: "/hostfs"  

Inside container the proccess running:

| root | 925524 | 925442 | 3 | 13:13 | ? | 00:03:33 | metricbeat -system.hostfs=/hostfs -e --environment=container|

If I look for the process on /hostfs/proc/ I can found it, but not if I look on /proc.

The logs:

{

    "log.level":"error",
    "@timestamp":"2022-06-23T18:40:38.175Z",
    "log.origin":{
        "file.name":"metrics/metrics.go",
        "file.line":306
    },
    "message":"error determining cgroups version: error reading /proc/3788379/cgroup: open /proc/3788379/cgroup: no such file or directory",
    "service.name":"metricbeat",
    "ecs.version":"1.6.0"

}