Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Metricbeat][Docker] Enrich statistics about blkio #13798

Closed
unixsurfer opened this issue Sep 25, 2019 · 2 comments
Closed

[Metricbeat][Docker] Enrich statistics about blkio #13798

unixsurfer opened this issue Sep 25, 2019 · 2 comments
Labels
enhancement Metricbeat Metricbeat Team:Integrations Label for the Integrations team

Comments

@unixsurfer
Copy link

Describe the enhancement:
Docker API stats exposes most of the metric that cgroup blkio controller provides but metricbeat exposes only read/write rate/throughtput. I would like to see the following metrics(copied from cgroup blkio controller documentation) exposed per container:

  • blkio.io_service_time
    Total amount of time between request dispatch and request completion
    for the IOs done by this cgroup. This is in nanoseconds to make it
    meaningful for flash devices too. For devices with queue depth of 1,
    this time represents the actual service time. When queue_depth > 1,
    that is no longer true as requests may be served out of order. This
    may cause the service time for a given IO to include the service time
    of multiple IOs when served out of order which may result in total
    io_service_time > actual time elapsed. This time is further divided by
    the type of operation - read or write, sync or async. First two fields
    specify the major and minor number of the device, third field
    specifies the operation type and the fourth field specifies the
    io_service_time in ns.

  • blkio.io_wait_time
    Total amount of time the IOs for this cgroup spent waiting in the
    scheduler queues for service. This can be greater than the total time
    elapsed since it is cumulative io_wait_time for all IOs. It is not a
    measure of total time the cgroup spent waiting but rather a measure of
    the wait_time for its individual IOs. For devices with queue_depth > 1
    this metric does not include the time spent waiting for service once
    the IO is dispatched to the device but till it actually gets serviced
    (there might be a time lag here due to re-ordering of requests by the
    device). This is in nanoseconds to make it meaningful for flash
    devices too. This time is further divided by the type of operation -
    read or write, sync or async. First two fields specify the major and
    minor number of the device, third field specifies the operation type
    and the fourth field specifies the io_wait_time in ns.

  • blkio.io_queued
    Total number of requests queued up at any given instant for this
    cgroup. This is further divided by the type of operation - read or
    write, sync or async.

all above are available in the Docker Stats API, see below:

% curl -s --unix-socket /var/run/docker.sock http:/containers/c2/stats?stream=false|jq '.blkio_stats|keys'
[
  "io_merged_recursive",
  "io_queue_recursive",
  "io_service_bytes_recursive",
  "io_service_time_recursive",
  "io_serviced_recursive",
  "io_time_recursive",
  "io_wait_time_recursive",
  "sectors_recursive"
]

Describe a specific use case for the enhancement or feature:
Knowing service and wait time together with the size of the queue will provide a better view on the performance of the container.

@andresrc
Copy link
Contributor

Anything else to do here, or can we close to issue?

@unixsurfer
Copy link
Author

Anything else to do here, or can we close to issue?

Shall we wait for the release to come out before we close this one?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Metricbeat Metricbeat Team:Integrations Label for the Integrations team
Projects
None yet
Development

No branches or pull requests

3 participants