Linux IOCounters() performance with many disks

[danielnelson]

On Linux systems with many entries in /proc/diskstats, calling disk.IOCounters() can take upwards of 10 seconds to complete. Since IOCounters returns stats for all disks, they all get checked even if you only want to read counters for a single drive.

The results of testing by @beheerderdag has indicated that it is primarily the call to udevadm to obtain the serial number. (influxdata/telegraf#2197)

I'm available to work on this issue if we can determine the right fix.

[danielnelson]

The current API does not provide a way to request a subset of disks, or to lazily get the serial number. @shirou Do you have any suggestion for the best course of action?

[shirou]

I think it happens passing specific device path to udevadm. So, how about invoke udevadm info --export-db and parse it?
It returns all devices information at once. By using this, invoke cost is reduced to always once.

[danielnelson]

This seems like it could work. @beheerderdag, could you time this call on your monster drive system?

[beheerderdag]

Around 2 seconds.

root# time udevadm info --export-db
[....]
P: /devices/virtual/vtconsole/vtcon1
E: UDEV_LOG=3
E: DEVPATH=/devices/virtual/vtconsole/vtcon1
E: SUBSYSTEM=vtconsole

P: /devices/virtual/wmi/0E7AF9F2-44A1-4C6F-A4B0-A7678480DA61
E: UDEV_LOG=3
E: DEVPATH=/devices/virtual/wmi/0E7AF9F2-44A1-4C6F-A4B0-A7678480DA61
E: MODALIAS=wmi:0E7AF9F2-44A1-4C6F-A4B0-A7678480DA61
E: SUBSYSTEM=wmi


real    0m2.102s
user    0m0.320s
sys     0m1.096s

[danielnelson]

Another proposal is to directly read udev data files using the major,minor numbers. I assume this is what --export-db is doing:

$ ll /dev/sda
brw-rw---- 1 root disk 8, 0 Mar 30 10:43 /dev/sda
$ cat /run/udev/data/b8:0 | grep SERIAL_SHORT
E:ID_SERIAL_SHORT=S2XANX0J116376

This makes me wonder if we can pull the value directly from sysfs, best I have found is:

$ cat /sys/block/sda/device/wwid
t10.ATA     SAMSUNG MZNLN512HMJP-000L7              S2XANX0J116376

I really have no idea if how reliable this is, I know it does not work with partitions.

[shirou]

@beheerderdag Could you tell me how it takes when you invoke with specific device path such as udevadm info --query=property /dev/sda ?
If it is quite fast, I think adding another method which takes path args is a better idea.

[beheerderdag]

It is quite fast:

root# time udevadm info --query=property --name=/dev/sda
UDEV_LOG=3
DEVPATH=/devices/pci0000:80/0000:80:02.2/0000:89:00.0/host6/target6:1:0/6:1:0:0/block/sda
MAJOR=8
MINOR=0
DEVNAME=/dev/sda

[...]

real    0m0.002s
user    0m0.000s
sys     0m0.000s

[shirou]

Okey, so disk.IOCounter(path string) (or something good func name) is necessary. I would appreciate if someone make a PR 😄

[shirou]

I think it is fixed by #347. If you some problem, feel free to re-open or create new issue. Thanks