Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker Input Plugin Measure Thin Pool Minimum Free Space #6084

Merged
merged 10 commits into from
Jul 15, 2019
Merged

Docker Input Plugin Measure Thin Pool Minimum Free Space #6084

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
676a022
Add missing tests for docker storage metrics
GeorgeMac Jul 8, 2019
16e478e
Add failing test case for docker input plugin thinpool minimum free s…
GeorgeMac Jul 8, 2019
20cf74c
Measure docker devicemapper thin pool minimum free space in docker in…
GeorgeMac Jul 8, 2019
207c09e
Add failing check to docker input plugin tests to replace docker_thin…
GeorgeMac Jul 9, 2019
b38d6ad
Add measurement docker_devicemapper to docker input plugin
GeorgeMac Jul 9, 2019
e4461d0
Add docker_devicemapper measurement to docker input plugin README
GeorgeMac Jul 9, 2019
f2beb17
Add failing case for devicemapper base device size (docker input plugin)
GeorgeMac Jul 10, 2019
117e7a5
Measure base device size of devicemapper storage driver (docker input…
GeorgeMac Jul 10, 2019
e944107
Update docker input plugin devicemapper README list formatting
GeorgeMac Jul 10, 2019
eca9d20
Further docker input plugin devicemapper README list formatting
GeorgeMac Jul 10, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 28 additions & 11 deletions plugins/inputs/docker/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ may prefer to exclude them:
- unit
- engine_host
- server_version
- fields:
+ fields:
- n_used_file_descriptors
- n_cpus
- n_containers
Expand All @@ -117,40 +117,57 @@ may prefer to exclude them:
- n_goroutines
- n_listener_events
- memory_total
- pool_blocksize (requires devicemapper storage driver)
- pool_blocksize (requires devicemapper storage driver) (deprecated see: `docker_devicemapper`)

The `docker_data` and `docker_metadata` measurements are available only for
some storage drivers such as devicemapper.

- docker_data
+ docker_data (deprecated see: `docker_devicemapper`)
- tags:
- unit
- engine_host
- server_version
- fields:
+ fields:
- available
- total
- used

- docker_metadata
- docker_metadata (deprecated see: `docker_devicemapper`)
- tags:
- unit
- engine_host
- server_version
- fields:
+ fields:
- available
- total
- used

- docker_container_mem
The above measurements for the devicemapper storage driver can now be found in the new `docker_devicemapper` measurement

- docker_devicemapper
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need to fix this, but this is good opportunity to show an obscure README issue. In Markdown, or maybe it's only Github Markdown, if you have multiple nested lists you will need to alter the first list character when repeating them, otherwise you get a weird bit of inconsistent vertical whitespace. To see this, in the current plugin README compare the vertical whitespace difference between the docker measurement and the docker_data. We usually use - and +.

Bad:

- foo
  - bar
- foo2
  - bar2
- foo3
  - bar3
- foo4
  - bar4

Good:

- foo
  - bar
+ foo2
  - bar2
- foo3
  - bar3
+ foo4
  - bar4

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah that is weird. Nice I did not know this!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What have you done. Now I can't unsee the whitespace 😂 :trollface:

- tags:
- engine_host
- server_version
- pool_name
+ fields:
- pool_blocksize_bytes
- data_space_used_bytes
- data_space_total_bytes
- data_space_available_bytes
- metadata_space_used_bytes
- metadata_space_total_bytes
- metadata_space_available_bytes
- thin_pool_minimum_free_space_bytes

+ docker_container_mem
- tags:
- engine_host
- server_version
- container_image
- container_name
- container_status
- container_version
- fields:
+ fields:
- total_pgmafault
- cache
- mapped_file
Expand Down Expand Up @@ -195,7 +212,7 @@ some storage drivers such as devicemapper.
- container_status
- container_version
- cpu
- fields:
+ fields:
- throttling_periods
- throttling_throttled_periods
- throttling_throttled_time
Expand All @@ -206,7 +223,7 @@ some storage drivers such as devicemapper.
- usage_percent
- container_id

- docker_container_net
+ docker_container_net
- tags:
- engine_host
- server_version
Expand All @@ -215,7 +232,7 @@ some storage drivers such as devicemapper.
- container_status
- container_version
- network
- fields:
+ fields:
- rx_dropped
- rx_bytes
- rx_errors
Expand Down
47 changes: 46 additions & 1 deletion plugins/inputs/docker/docker.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -322,21 +322,50 @@ func (d *Docker) gatherInfo(acc telegraf.Accumulator) error {
"n_goroutines": info.NGoroutines,
"n_listener_events": info.NEventsListener,
}

// Add metrics
acc.AddFields("docker", fields, tags, now)
acc.AddFields("docker",
map[string]interface{}{"memory_total": info.MemTotal},
tags,
now)

// Get storage metrics
tags["unit"] = "bytes"

var (
// "docker_devicemapper" measurement fields
poolName string
deviceMapperFields = map[string]interface{}{}
)

for _, rawData := range info.DriverStatus {
name := strings.ToLower(strings.Replace(rawData[0], " ", "_", -1))
if name == "pool_name" {
poolName = rawData[1]
continue
}

// Try to convert string to int (bytes)
value, err := parseSize(rawData[1])
if err != nil {
continue
}
name := strings.ToLower(strings.Replace(rawData[0], " ", "_", -1))

switch name {
case "pool_blocksize",
"base_device_size",
"data_space_used",
"data_space_total",
"data_space_available",
"metadata_space_used",
"metadata_space_total",
"metadata_space_available",
"thin_pool_minimum_free_space":
deviceMapperFields[name+"_bytes"] = value
}

// Legacy devicemapper measurements
if name == "pool_blocksize" {
// pool blocksize
acc.AddFields("docker",
Expand All @@ -353,12 +382,28 @@ func (d *Docker) gatherInfo(acc telegraf.Accumulator) error {
metadataFields[fieldName] = value
}
}

if len(dataFields) > 0 {
acc.AddFields("docker_data", dataFields, tags, now)
}

if len(metadataFields) > 0 {
acc.AddFields("docker_metadata", metadataFields, tags, now)
}

if len(deviceMapperFields) > 0 {
tags := map[string]string{
"engine_host": d.engineHost,
"server_version": d.serverVersion,
}

if poolName != "" {
tags["pool_name"] = poolName
}

acc.AddFields("docker_devicemapper", deviceMapperFields, tags, now)
}

return nil
}

Expand Down
58 changes: 58 additions & 0 deletions plugins/inputs/docker/docker_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -702,6 +702,29 @@ func TestDockerGatherInfo(t *testing.T) {
},
)

acc.AssertContainsTaggedFields(t,
"docker",
map[string]interface{}{
"memory_total": int64(3840757760),
},
map[string]string{
"engine_host": "absol",
"server_version": "17.09.0-ce",
},
)

acc.AssertContainsTaggedFields(t,
"docker",
map[string]interface{}{
"pool_blocksize": int64(65540),
},
map[string]string{
"engine_host": "absol",
"server_version": "17.09.0-ce",
"unit": "bytes",
},
)

acc.AssertContainsTaggedFields(t,
"docker_data",
map[string]interface{}{
Expand All @@ -710,11 +733,46 @@ func TestDockerGatherInfo(t *testing.T) {
"available": int64(36530000000),
},
map[string]string{
"engine_host": "absol",
"server_version": "17.09.0-ce",
"unit": "bytes",
},
)

acc.AssertContainsTaggedFields(t,
"docker_metadata",
map[string]interface{}{
"used": int64(20970000),
"total": int64(2146999999),
"available": int64(2126999999),
},
map[string]string{
"engine_host": "absol",
"server_version": "17.09.0-ce",
"unit": "bytes",
},
)

acc.AssertContainsTaggedFields(t,
"docker_devicemapper",
map[string]interface{}{
"base_device_size_bytes": int64(10740000000),
"pool_blocksize_bytes": int64(65540),
"data_space_used_bytes": int64(17300000000),
"data_space_total_bytes": int64(107400000000),
"data_space_available_bytes": int64(36530000000),
"metadata_space_used_bytes": int64(20970000),
"metadata_space_total_bytes": int64(2146999999),
"metadata_space_available_bytes": int64(2126999999),
"thin_pool_minimum_free_space_bytes": int64(10740000000),
},
map[string]string{
"engine_host": "absol",
"server_version": "17.09.0-ce",
"pool_name": "docker-8:1-1182287-pool",
},
)

acc.AssertContainsTaggedFields(t,
"docker_container_cpu",
map[string]interface{}{
Expand Down
2 changes: 1 addition & 1 deletion plugins/inputs/docker/docker_testdata.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ var info = types.Info{
HTTPSProxy: "",
Labels: []string{},
MemoryLimit: false,
DriverStatus: [][2]string{{"Pool Name", "docker-8:1-1182287-pool"}, {"Pool Blocksize", "65.54 kB"}, {"Backing Filesystem", "extfs"}, {"Data file", "/dev/loop0"}, {"Metadata file", "/dev/loop1"}, {"Data Space Used", "17.3 GB"}, {"Data Space Total", "107.4 GB"}, {"Data Space Available", "36.53 GB"}, {"Metadata Space Used", "20.97 MB"}, {"Metadata Space Total", "2.147 GB"}, {"Metadata Space Available", "2.127 GB"}, {"Udev Sync Supported", "true"}, {"Deferred Removal Enabled", "false"}, {"Data loop file", "/var/lib/docker/devicemapper/devicemapper/data"}, {"Metadata loop file", "/var/lib/docker/devicemapper/devicemapper/metadata"}, {"Library Version", "1.02.115 (2016-01-25)"}},
DriverStatus: [][2]string{{"Pool Name", "docker-8:1-1182287-pool"}, {"Base Device Size", "10.74 GB"}, {"Pool Blocksize", "65.54 kB"}, {"Backing Filesystem", "extfs"}, {"Data file", "/dev/loop0"}, {"Metadata file", "/dev/loop1"}, {"Data Space Used", "17.3 GB"}, {"Data Space Total", "107.4 GB"}, {"Data Space Available", "36.53 GB"}, {"Metadata Space Used", "20.97 MB"}, {"Metadata Space Total", "2.147 GB"}, {"Metadata Space Available", "2.127 GB"}, {"Udev Sync Supported", "true"}, {"Deferred Removal Enabled", "false"}, {"Data loop file", "/var/lib/docker/devicemapper/devicemapper/data"}, {"Metadata loop file", "/var/lib/docker/devicemapper/devicemapper/metadata"}, {"Library Version", "1.02.115 (2016-01-25)"}, {"Thin Pool Minimum Free Space", "10.74GB"}},
NFd: 19,
HTTPProxy: "",
Driver: "devicemapper",
Expand Down