Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some improvment in mesos input plugin, #1572

Merged
merged 7 commits into from
Aug 30, 2016
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 18 additions & 12 deletions plugins/inputs/mesos/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ Mesos tasks metric groups
- executor_name
- framework_id
- source
- statistics (all metrics below will have `statistics_` prefix included in their names
- statistics
- cpus_limit
- cpus_system_time_secs
- cpus_user_time_secs
Expand All @@ -266,14 +266,20 @@ Mesos tasks metric groups
- server
- role (master/slave)

- Tasks measurements have the following tags:
- All master measurements have the extra tags:
- state (leader/follower)

- Tasks measurements have the following tags:
- server
- framework_id
- task_id

### Example Output:
```
$ telegraf -config ~/mesos.conf -input-filter mesos -test
* Plugin: mesos, Collection 1
mesos,host=172.17.8.102,server=172.17.8.101 allocator/event_queue_dispatches=0,master/cpus_percent=0,
mesos,role=master,state=leader,host=172.17.8.102,server=172.17.8.101
allocator/event_queue_dispatches=0,master/cpus_percent=0,
master/cpus_revocable_percent=0,master/cpus_revocable_total=0,
master/cpus_revocable_used=0,master/cpus_total=2,
master/cpus_used=0,master/disk_percent=0,master/disk_revocable_percent=0,
Expand All @@ -293,13 +299,13 @@ master/messages_deactivate_framework=0 ...

Meoso tasks metrics (if enabled):
```
mesos-tasks,host=172.17.8.102,server=172.17.8.101,task_id=hello-world.e4b5b497-2ccd-11e6-a659-0242fb222ce2
statistics_cpus_limit=0.2,statistics_cpus_system_time_secs=142.49,statistics_cpus_user_time_secs=388.14,
statistics_mem_anon_bytes=359129088,statistics_mem_cache_bytes=3964928,
statistics_mem_critical_pressure_counter=0,statistics_mem_file_bytes=3964928,
statistics_mem_limit_bytes=767557632,statistics_mem_low_pressure_counter=0,
statistics_mem_mapped_file_bytes=114688,statistics_mem_medium_pressure_counter=0,
statistics_mem_rss_bytes=359129088,statistics_mem_swap_bytes=0,statistics_mem_total_bytes=363094016,
statistics_mem_total_memsw_bytes=363094016,statistics_mem_unevictable_bytes=0,
statistics_timestamp=1465486052.70525 1465486053052811792...
mesos-tasks,host=172.17.8.102,server=172.17.8.101,framework_id=e3060235-c4ed-4765-9d36-784e3beca07f-0000,task_id=hello-world.e4b5b497-2ccd-11e6-a659-0242fb222ce2
cpus_limit=0.2,statistics_cpus_system_time_secs=142.49,statistics_cpus_user_time_secs=388.14,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here and below there are still some statistics_ prefixes left intact.

mem_anon_bytes=359129088,statistics_mem_cache_bytes=3964928,
mem_critical_pressure_counter=0,statistics_mem_file_bytes=3964928,
mem_limit_bytes=767557632,statistics_mem_low_pressure_counter=0,
mem_mapped_file_bytes=114688,statistics_mem_medium_pressure_counter=0,
mem_rss_bytes=359129088,statistics_mem_swap_bytes=0,statistics_mem_total_bytes=363094016,
mem_total_memsw_bytes=363094016,statistics_mem_unevictable_bytes=0,
timestamp=1465486052.70525 1465486053052811792...
```
24 changes: 20 additions & 4 deletions plugins/inputs/mesos/mesos.go
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ func (m *Mesos) Gather(acc telegraf.Accumulator) error {
for _, v := range m.Slaves {
wg.Add(1)
go func(c string) {
errorChannel <- m.gatherMainMetrics(c, ":5051", MASTER, acc)
errorChannel <- m.gatherMainMetrics(c, ":5051", SLAVE, acc)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! 👍

wg.Done()
return
}(v)
Expand Down Expand Up @@ -420,8 +420,15 @@ var client = &http.Client{
Timeout: time.Duration(4 * time.Second),
}

// TaskStats struct for JSON API output /monitor/statistics
type TaskStats struct {
ExecutorID string `json:"executor_id"`
FrameWorkID string `json:"framework_id"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think FrameworkID would look better since framework is a single word

Statistics map[string]interface{} `json:"statistics"`
}

func (m *Mesos) gatherSlaveTaskMetrics(address string, defaultPort string, acc telegraf.Accumulator) error {
var metrics []map[string]interface{}
var metrics []TaskStats

host, _, err := net.SplitHostPort(address)
if err != nil {
Expand Down Expand Up @@ -452,10 +459,11 @@ func (m *Mesos) gatherSlaveTaskMetrics(address string, defaultPort string, acc t
}

for _, task := range metrics {
tags["task_id"] = task["executor_id"].(string)
tags["task_id"] = task.ExecutorID
tags["framework_id"] = task.FrameWorkID

jf := jsonparser.JSONFlattener{}
err = jf.FlattenJSON("", task)
err = jf.FlattenJSON("", task.Statistics)

if err != nil {
return err
Expand Down Expand Up @@ -510,6 +518,14 @@ func (m *Mesos) gatherMainMetrics(a string, defaultPort string, role Role, acc t
return err
}

if role == MASTER {
if jf.Fields["master/cpus_total"] != 0.0 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use master/elected instead? Above smells more like a tribal knowledge.

tags["state"] = "leader"
} else {
tags["state"] = "follower"
}
}

acc.AddFields("mesos", jf.Fields, tags)
Copy link
Contributor

@harnash harnash Aug 3, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just noticed that slave metrics have timestamp which we can use to pass as fourth parameter to AddFields. It's an extra - you can ignore it if you want. I will try to add this later if you won't.


return nil
Expand Down
5 changes: 4 additions & 1 deletion plugins/inputs/mesos/mesos_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -345,7 +345,10 @@ func TestMesosSlave(t *testing.T) {
t.Errorf(err.Error())
}

acc.AssertContainsFields(t, "mesos-tasks", jf.Fields)
acc.AssertContainsFields(
t,
"mesos-tasks",
slaveTaskMetrics["statistics"].(map[string]interface{}))
}

func TestSlaveFilter(t *testing.T) {
Expand Down