Design Metrics (HTTP) server for serving metrics data for Grafana dashboard

[Alan-Cha]

The Grafana dashboard, currently being developed to display metrics data for A/B/n, needs a way to obtain metrics data.

Consider creating a separate service that is dedicated to serving metrics for Grafana dashboard.

Under the covers, this server should invoke the GetMetrics() function (see here).

Part of epic #1501

[kalantar]

Grafana requires the following from HTTP service:

1. get all applications. Expect that appliciation can be specified in dashboard URL. If not, need this. future requirement
2. get all versions for a given application - returns list of versions. future requirement
3. get all metrics data for a given app, list of versions - expects result from latest signatures - returns 2 lists for each requested version: all metrics over all users and all metrics over transactions. Internally, this calls storage.GetMetrics(app, version); if list of versions is empty, return for all versions
4. get summary metrics for a given app, list of versions - expects result from latest signatures - returns 2 lists for each requested version: all metrics over all users and all metrics over transactions. Internally, this calls storage.GetSummaryMetrics(app, version); if list of versions is empty, return for all versions

Notes/Questions

• storage.GetMetrics() api defined in GetMetrics() uses app, user, trans #1511
• storage.GetSummaryMetrics() api defined in

  iter8/controllers/storageclient/badgerdb/simple.go

  Line 183 in 6a9bde9

  func (cl Client) GetSummaryMetrics(applicationName string, version int, signature string) (*storageclient.VersionMetricSummary, error) {

• Should storage.GetSummaryMetrics() continue to compute the summary metrics or move to http service? leave it for now
• Result of rest calls should mirror result of internal implementation as closely as possible.

Next Steps

• define REST APIs
• learn how to define HTTP service

[kalantar]

See below for updated design
Grafana dashboard requests data from Iter8 (HTTP) Metrics Service.
Data source is Iter8 Metrics service endpoint.

Metrics Service APIs:

1. GET /metrics?application=app-name

Result (Content-Type: application/json):

{
  "my-app": {
    "0": {
      "numUsers": k,
      "metric1": {
        "metricsOverTransactions": [],
        "metricsOverUsers": [],
        "summaryOverTransactions": {"count": n, "mean" mean, "stddev"": stddev, "min": min, "max": max}
        "summaryOverUsers": {"count": n, "mean" mean, "stddev"": stddev, "min": min, "max": max}
      },
      "metric2": {}, ...
    },
    "1": {}, ...
  }
}

Response codes:

• 200 - ok
• 400 - request not valid JSON
• 404 - no such application
• 405 - method not allowed
• 500 - error processing request

Implementation:

Find routemap from application name
For each version in routemap
  if version in request list (or request list is empty)
    call storage.GetMetrics(application, version, version.Signature)
    call stroage.GetSummaryMetrics(application, version, version.Signature)
Combine results

Comments

1. Computation of summary metrics still in storage client -- plan leave it for now (alternative is to move it to metrics service)

[kalantar]

There are a number of http multiplexers compatible with `net/http'. We have a very small number of APIs. Per https://www.alexedwards.net/blog/which-go-router-should-i-use, plan to use default multiplexer.

[kalantar]

Here is some partial sample output for /metrics/backend. Data generated from sample A/B application.

% curl localhost:8080/summarymetrics -d '{"application": "default/backend"}'
{
   "0": {
      "NumUsers": 17,
      "MetricSummaries": {
         "sample_metric": {
            "SummaryOverTransactions": {
               "Count": 203,
               "Mean": 47.82011996059115,
               "StdDev": 28.835028215240904,
               "Min": 0.051382,
               "Max": 99.827381
            },
            "SummaryOverUsers": {
               "Count": 17,
               "Mean": 571.0284912941178,
               "StdDev": 1456.1383944589172,
               "Min": 15.18434,
               "Max": 6314.74141
            }
         }
      }
   },
   "1": {
      "NumUsers": 24,
      "MetricSummaries": {
         "sample_metric": {
            "SummaryOverTransactions": {
               "Count": 86,
               "Mean": 51.59040327906978,
               "StdDev": 29.819517493656722,
               "Min": 1.282591,
               "Max": 99.915949
            },
            "SummaryOverUsers": {
               "Count": 24,
               "Mean": 184.86561175000006,
               "StdDev": 110.78605365552966,
               "Min": 23.409473,
               "Max": 433.398395
            }
         }
      }
   }
}

[kalantar]

Grafana dashboard requests data from Iter8 (HTTP) Metrics Service.
Data source is Iter8 Metrics service endpoint.

APIs

1. GET /metrics?application=app-name

Result (Content-Type: application/json):

{
  "metric1": {
    "histogramsOverTransactions": [
      {"version": v, "label": "a-b", "value": v},
      ...,
    ],
    "histogramsOverUsers": [
      {"version": v, "label": "a-b", "value": v},
      ...,
    ],
    "summaryOverTransactions": [
      {"version": v, "count": n, "mean" mean, "stddev"": stddev, "min": min, "max": max},
      ...
    ],
    "summaryOverUsers": [
      {"version": v, "count": n, "mean" mean, "stddev"": stddev, "min": min, "max": max},
      ...,
    ],
  },
  "metric2": {}, 
  ...
}

Response codes:

• 200 - ok
• 400 - request does not indicate application
• 404 - no such application
• 405 - method not allowed
• 500 - error processing request

Implementation:

find routemap from application name
initialize result
for each version in routemap
  call storage.GetMetrics(application, version, version.Signature)
  calculateHistogramMetrics(rawMetrics)
  insert in result
  calculateSummaryMetrics(rawMetrics)
  insert in result
return result

Comments

[kalantar]

Implemented in #1493