[kubernetes plugin] Volume are not filtered by storage type

[nsteinmetz]

Relevant telegraf.conf:

Default configuraiton of the telegraf-ds chart except for outputs which targets my 1.x and 2.x influxdb instances.

cf https://github.com/influxdata/helm-charts/blob/master/charts/telegraf-ds/templates/configmap.yaml#L18-L40

System info:

Telegraf 1.13.4
Kubernetes 1.17.0 @ OVH Managed K8S Service

Steps to reproduce:

1. Deply telegraf-ds chart
2. Explore data to see the issue

Expected behavior:

See only volumes from PVC and not configmaps or secrets.

Actual behavior:

See all volumes mounted in pods : configmaps, volumes, secrets

Also, for secrets and configmaps, the capacity_bytes match the total capacity bytes on the local node.

Additional info:

To see all volumes, do a flux function as:

from(bucket: "crntk8s")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "kubernetes_pod_volume")
  |> filter(fn: (r) => r._field == "used_bytes" or r._field == "capacity_bytes")
  |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")
  |> group(columns: [])

You will see all mounted volumes and the wrong capacity bytes for configmaps and secrets.

[nsteinmetz]

I'm not a go dev but seems the code is there and I don't see any filtering.

https://github.com/influxdata/telegraf/blob/master/plugins/inputs/kubernetes/kubernetes.go#L280-L292

and seems you cannot do filtering based on output of the api endpoint:

   "volume": [
    {
     "time": "2020-01-30T17:28:33Z",
     "availableBytes": 3566764032,
     "capacityBytes": 3566776320,
     "usedBytes": 12288,
     "inodesFree": 870786,
     "inodes": 870795,
     "inodesUsed": 9,
     "name": "default-token-6fl2w"
    }

Searching for volumes in the outputs, I'm not even sure they are in as I can't find their name.

If I look at a podref object for which I know there is a PVC attached, it's not present as I find no capacityBytes that could match the size of the PVC

[nsteinmetz]

Seems PVC are no longer in kubelet since 1.12 - cf kubernetes/kubernetes#68522

[nsteinmetz]

Ok and seems csi-cinder driver does not report metrics yet :'(

• [cinder-csi-plugin] metrics are not accessible kubernetes/cloud-provider-openstack#913
• Implement GET_VOLUME_STATS node capability  kubernetes/cloud-provider-openstack#669

Will try on gke then for further tests.

[MyaLongmire]

Did you make any headway on this @nsteinmetz?

Is there a pattern (like volume names or mount option) that we can ignore?

[nsteinmetz]

Hello @MyaLongmire,

Currently I no longer use kubernetes for my current project so no idea if anything changed since but I didn't progress on this on my side.

[MyaLongmire]

@nsteinmetz Thanks for getting back to us! I am going to close this for now then. If anyone needs it reopened, let me know :)

[dsvk]

@MyaLongmire I would indeed appreciate a way to predictably filter out all non-PVC entries - which IMHO could even be the default plugin behavior (similar to how disk excludes some less relevant filesystems by default) but I'm fine with a non-default option or a type-style tag to make reliable filtering/tagdrop feasible too.

As for how to get the data to differentiate these volume types I'm not sure. kube_inventory plugin queries the apiserver to get the list of PVCs so either kubelet also provides that data somewhere (not familiar with it enough to tell) or the plugin might need to query both API endpoints and combine the data to extend the current measurement.

Volume names are user-defined for both PVCs and configmaps/secrets so I don't think any pattern filtering is feasible there.

For additional context, #8546 and #8565 are another issues that somewhat similarly suffer from the current two-plugin split and trying to combine kubernetes metrics with kube_inventory metadata, also #9451 (conflicting measurement name). Merging both plugins was already proposed earlier but that's apparently unlikely to happen anytime soon (<2.x), if at all.