Add Prom/Grafana dashboard for M3DB as well as docker-compose for local development of m3-stack #939

richardartoul · 2018-09-25T15:56:32Z

JSON for M3DB Grafana Dashboard
M3 stack "in a box", one line command to: start 3 m3db nodes (including namespace/topology initializing), an m3coordinator node, a prometheus node, and a grafana node with scraping and remote read/write setup. Grafana prometheus source and M3DB dashboard also auto-populate.

arnikola · 2018-09-25T16:17:01Z

github_token

@@ -0,0 +1 @@
+f6192b8d35518098fbb645ffd807a53749ffb00e


Don't think this should be here, might be a good idea to regenerate your tokens too

Yeah I deleted the access token, thanks.

Was actually deleted before I accidentally pushed it here, but I'll reach out to GitHub support as well

OK just verified the access token doesn't work by running:

curl -H "Authorization: token f6192b8d35518098fbb645ffd807a53749ffb00e" https://api.github.com

which returned

{ "message": "Bad credentials", "documentation_url": "https://developer.github.com/v3" }

So I must have deleted the token awhile back before it ever made it into this P.R

We're good 👍

Yeah just make sure you delete the token in GH settings and should be good

arnikola · 2018-09-25T17:31:58Z

src/dbnode/client/session.go

@@ -281,15 +281,15 @@ func newSession(opts Options) (clientSession, error) {
 	s.pools.writeAttempt.Init()

 	fetchAttemptPoolOpts := pool.NewObjectPoolOptions().
-		SetSize(opts.FetchBatchOpPoolSize()).
+		SetSize(1).


What's the idea here?

unintentional

codecov · 2018-09-25T18:00:45Z

Codecov Report

Merging #939 into master will decrease coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #939      +/-   ##
==========================================
- Coverage   77.96%   77.92%   -0.04%     
==========================================
  Files         411      411              
  Lines       34406    34406              
==========================================
- Hits        26824    26812      -12     
- Misses       5739     5749      +10     
- Partials     1843     1845       +2

Flag	Coverage Δ
#dbnode	`81.48% <ø> (-0.05%)`	⬇️
#m3ninx	`75.21% <ø> (ø)`	⬆️
#query	`64.3% <ø> (-0.02%)`	⬇️
#x	`84.72% <ø> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eef049a...edd73a8. Read the comment docs.

prateek · 2018-09-25T18:06:40Z

github_token

@@ -0,0 +1 @@
+f6192b8d35518098fbb645ffd807a53749ffb00e


wanna delete this file now?

prateek · 2018-09-25T18:07:02Z

scripts/development/m3_stack/README.md

+
+This docker-compose file will setup the following environment:
+
+1. 3 M3DB nodes with a single node acting as an EtcD seed


nit: ETCD

prateek · 2018-09-25T18:07:30Z

.gitignore

@@ -46,3 +46,6 @@ site/
 !m3db.io/**/vendor
 # Automatically populated from asset sources
 m3db.io/openapi
+
+# GitHub API token
+github_token


do you need this change once you delete the file?

This is where goreleaser expects it to be so might save someone else the trouble

meh separate PRs mate =P

fiiiiiine haha

prateek · 2018-09-25T18:13:20Z

scripts/development/m3_stack/dbnode_config.yml

@@ -0,0 +1,212 @@
+coordinator:


re: dbnode_config.yaml and coordinator config here - i'm a little concerned about the duplication of configs - it's one more thing to keep in sync. Anyway we can merge with others we already have?

Already sync'd with @robskillington and this is the best we can do for now until we can get some kind of templating system, I already did some kind of weird stuff to avoid having 3 different m3db configs

prateek · 2018-09-25T18:14:15Z

scripts/integration-tests/prometheus/prometheus-integration-test.sh

@@ -9,7 +9,7 @@ PARAM_TEST_BUILD="${TEST_BUILD:-true}"
 PARAM_TEST_VERIFY="${TEST_VERIFY:-true}"
 PARAM_TEST_TEARDOWN="${TEST_TEARDOWN:-true}"

-if [ $PARAM_TEST_BUILD != "true" ]; then
+if [ "$PARAM_TEST_BUILD" != "true" ]; then


maybe rewire this test to just use your new m3_stack stuff?

Also synced with @robskillington and want to keep them separate to make things easier, but also so the default configuration for the local stack can be a lot heavier

prateek · 2018-09-25T18:14:53Z

scripts/development/m3_stack/start.sh

+    }
+  }
+}'
+echo "Done initializing namespace"


nit: steal the validation that paul does in the integration test - i.e. the curl ... | jq ... bits

prateek · 2018-09-25T18:19:55Z

integrations/grafana/m3db_dashboard.json

+          "intervalFactor": 1,
+          "key": 0.08874521907599053,
+          "refId": "A",
+          "target": "fetch service:statsdex_m3dbnode name:database.bootstrapped host:$servers | <1 |  removeEmpty | count"


remove all target etc

prateek · 2018-09-25T18:21:17Z

integrations/grafana/m3db_dashboard.json

+      "renderer": "flot",
+      "seriesOverrides": [
+        {
+          "alias": "divideSeries(sumSeries(perSecond(statsdex.metrics.query.mquery01-sjc1.storage.m3dbshadow.mismatches)),sumSeries(perSecond(statsdex.metrics.query.mquery01-sjc1.storage.m3dbshadow.matches)))",


this shit too

prateek · 2018-09-25T18:22:11Z

integrations/grafana/m3db_dashboard.json

+          "renderer": "flot",
+          "seriesOverrides": [
+            {
+              "alias": "divideSeries(sumSeries(perSecond(statsdex.metrics.query.mquery01-sjc1.storage.m3dbshadow.mismatches)),sumSeries(perSecond(statsdex.metrics.query.mquery01-sjc1.storage.m3dbshadow.matches)))",


arnikola

Mostly just checking prom expressions, lgtm, but may need to run the configs by the others

arnikola · 2018-09-25T18:48:42Z

integrations/grafana/m3db_dashboard.json

+      "tableColumn": "Value",
+      "targets": [
+        {
+          "expr": "sum(database_bootstrapped{instance=~\"$instance\"} == 1)",


... == bool 1) may be better?

arnikola · 2018-09-25T18:52:13Z

integrations/grafana/m3db_dashboard.json

+      "steppedLine": false,
+      "targets": [
+        {
+          "expr": "",


Can probably delete this, unless it's for formatting

its a "not implemented" graph

arnikola · 2018-09-25T18:56:44Z

integrations/grafana/m3db_dashboard.json

+            "sort": 0,
+            "value_type": "cumulative"
+          },
+          "type":


Can get rid of this unless this is some formatting stuff it adds

same comment

arnikola · 2018-09-25T18:58:44Z

scripts/development/m3_stack/README.md

+
+## Usage
+
+Use the `start.sh` and `stop.sh` scripts


nit: Maybe give these scripts more descriptive names?

prateek · 2018-09-25T21:00:41Z

docker/grafana/Dockerfile

+COPY ./docker/grafana/dashboards.yaml /etc/grafana/provisioning/dashboards/all.yaml
+COPY ./integrations/grafana/m3db_dashboard.json /var/lib/grafana/dashboards/m3db_dashboard.json
+# Need to replace datasource template variable with name of actual data source so auto-import
+# JustWorksTM. Need to switch users because the /var/lib/grafana/dashboards directory is


prateek · 2018-09-25T21:02:21Z

scripts/development/m3_stack/README.md

@@ -0,0 +1,28 @@
+# Local Development


mind linking a new section in https://github.com/m3db/m3/blob/master/DEVELOPER.md to this page

prateek · 2018-09-25T21:03:44Z

scripts/development/m3_stack/docker-compose.yml

+    ports:
+      - "0.0.0.0:7201:7201"
+      - "0.0.0.0:7203:7203"
+      - "0.0.0.0:7208:7208"


prateek · 2018-09-25T21:03:48Z

scripts/development/m3_stack/docker-compose.yml

+    expose:
+      - "7201"
+      - "7203"
+      - "7208"


prateek

LGTM w/ last couple of nits

Richard Artoul added 5 commits September 25, 2018 12:02

interim commit

e0ffcda

Interim commit

6e8a359

Add github_token to .gitignore

302c6a2

switch docker-compose file version

37b0bb4

Undo change

377f3db

richardartoul force-pushed the ra/prom branch from c9381ec to 377f3db Compare September 25, 2018 16:04

arnikola reviewed Sep 25, 2018

View reviewed changes

Richard Artoul added 3 commits September 25, 2018 13:59

Add m3_stack

1508061

Reset docker-compose

e3aa713

Reset prometheus config

c599d70

Richard Artoul added 4 commits September 25, 2018 14:02

Remove unused files

de0f3f3

Update readme

644561c

Update prometheus yaml

9aa8a09

undo more stuff

e2d5f55

prateek reviewed Sep 25, 2018

View reviewed changes

github_token Outdated

@@ -0,0 +1 @@

f6192b8d35518098fbb645ffd807a53749ffb00e

Copy link

Collaborator

prateek Sep 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wanna delete this file now?

Undo more changes

89c6342

prateek reviewed Sep 25, 2018

View reviewed changes

Richard Artoul added 4 commits September 25, 2018 14:07

Fix config

75fc4f7

Remove github_token file

83c77b6

Update README

8a7c218

Add grafana dashboard for m3db

37b1f22

prateek reviewed Sep 25, 2018

View reviewed changes

richardartoul changed the title ~~[WIP] Add Prom/Grafana dashboard for M3DB~~ Add Prom/Grafana dashboard for M3DB as well as docker-compose for local development of m3-stack Sep 25, 2018

prateek reviewed Sep 25, 2018

View reviewed changes

Add validations

ef26fa0

prateek reviewed Sep 25, 2018

View reviewed changes

Richard Artoul added 2 commits September 25, 2018 14:24

Remove statsdex stuff from grafana

1948889

More dashboard JSON cleanup

36504df

arnikola approved these changes Sep 25, 2018

View reviewed changes

Richard Artoul added 4 commits September 25, 2018 15:28

Improvements and fix auto-import dashboard

57c480d

Remove gitingore change

295f106

Improve prom query

1eabbf5

Rename scripts

72ab0e0

prateek reviewed Sep 25, 2018

View reviewed changes

scripts/development/m3_stack/docker-compose.yml Outdated

expose:

- "7201"

- "7203"

- "7208"

Copy link

Collaborator

prateek Sep 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

prateek approved these changes Sep 25, 2018

View reviewed changes

Richard Artoul added 2 commits September 25, 2018 17:08

Remove unused ports

82fddcf

Update DEVELOPER.md

edd73a8

richardartoul merged commit 6aa47dd into master Sep 25, 2018

prateek deleted the ra/prom branch September 29, 2018 18:08


		This docker-compose file will setup the following environment:

		1. 3 M3DB nodes with a single node acting as an EtcD seed

Add Prom/Grafana dashboard for M3DB as well as docker-compose for local development of m3-stack #939

Add Prom/Grafana dashboard for M3DB as well as docker-compose for local development of m3-stack #939

Conversation

richardartoul commented Sep 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 25, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnikola left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prateek left a comment

Choose a reason for hiding this comment

richardartoul commented Sep 25, 2018 •

edited

Loading

codecov bot commented Sep 25, 2018 •

edited

Loading