Establish network usage metrics

[cbeams]

Description

See bisq-network/bisq#3916

Citing essential data points and KPIs for desktop client from issue above

Data points to collect within a give timeframe per app version:

• Number of active nodes (nodes that are part of the network)
• Number of available offers
• Number of trades

KPIs needed:

• Ratio between Number of available offers and Number of active nodes
• Ratio between Number of trades and Number of available offers

Rationale

If we don't have a baseline and metrics in place to measure our success we'll never be able to tell if some change was a success or a big failure.

As we don't want to include general tracking within the client, which would hurt user privacy, we'll monitor public available information and try to generate as-good-as-it-gets metrics out of it.

Our most important use case is to enable users to buy/sell BTC. Following KPIs help us to measure our success:

• Number of active nodes > Number of available offers
  If this conversion number increases compared to the previous timeframe, we either have attracted more market makers or improved the client and enabled the user to create an offer more easily.
  To be able to check if we got more market makers on board we could monitor the offerbook to check by how many nodes the offers have been created with. If the ratio Number of available offers/Number of nodes stays roughly the same we can assume that the conversion increase is either by better user targeting in our growth efforts or by improvements within the client.
• Number of available offers > Number of trades
  As the trade statistics object is published already before the trade is settled (while both parties need to be online) we can't take it as a metric if we improved the trade process itself. If this conversion number increases compared to the previous timeframe, it could mean:
  • we targeted the right type of people that are happy to take existing trades
  • we improved UX that people are able to find and take existing offers more easily
  • we have enough liquidity with a reasonable spread for quick transactions

Criteria for delivery

KPIs

Within https://monitor.bisq.network/ it should be possible to select a time frame and app version (or all app versions) and display the KPIs above.

Metrics

Within https://monitor.bisq.network/ it should be possible to select a time frame to show the version spread between active nodes

Tasks

here is the tasks list

• design metrics via
• proof-of-concept via and via
• create log scraper scripts and monitoring infrastructure
• unplanned: create one-cmd-installer for general monitoring and scraper script deployment via
• unplanned: create pricenode installer via
• unplanned: adapt the monitoring scripts to work with the new pricenode installer via
• deploy data gathering infrastructure
  - [x] unplanned: fix monitoring daemon (this broke the monitor))
  - [ ] adapt monitoring daemon to collect trade rate data
• deploy new monitoring daemon sourced another metric we already have
• configure graphs at monitor.bisq.network: KPIs and version spread metrics (via)

Estimates

Based on the numbers posted in bisq-network/bisq#3916 this project will need roughly USD 2200 to deliver this project.

[ripcurlx]

@freimair Could you please provide the tasks that are needed to complete this project?
Please also the already completed ones. Thanks!

[cbeams]

I'll just add here that the purpose of this project as originally conceived is to support our goal of improving onboarding. We need these (certain, specific) metrics in place, with some historical baseline being established, in order to effectively measure whether our new onboarding workflow implementation is actually helping anything. @ripcurlx, I'd suggest transcribing that and whatever else you see fit to the Rationale section of the description.

[ripcurlx]

@ripcurlx, I'd suggest transcribing that and whatever else you see fit to the Rationale section of the description.

Transcribed the relevant parts from the issue into the Rationale section

[freimair]

here is the tasks list

• design metrics via
• proof-of-concept via and via
• create log scraper scripts and monitoring infrastructure
• unplanned: create one-cmd-installer for general monitoring and scraper script deployment via
• unplanned: create pricenode installer via
• unplanned: adapt the monitoring scripts to work with the new pricenode installer via
• deploy data gathering infrastructure
  - [x] unplanned: fix monitoring daemon (this broke the monitor))
  - [ ] adapt monitoring daemon to collect trade rate data
  - [ ] deploy new monitoring daemon sourced another metric we already have
• configure graphs at monitor.bisq.network: KPIs and version spread metrics

[ripcurlx]

here is the tasks list

• design metrics via
• proof-of-concept via and via
• create log scraper scripts and monitoring infrastructure
• unplanned: create one-cmd-installer for general monitoring and scraper script deployment via
• deploy data gathering infrastructure
• unplanned: fix monitoring daemon (this broke the monitor))
• adapt monitoring daemon to collect trade rate data
• deploy new monitoring daemon
• configure graphs at monitor.bisq.network: KPIs and version spread metrics

Thanks! Regarding the two unplanned task: Does this change the estimated budget above? If yes, by how much?

[freimair]

Thanks! Regarding the two unplanned task: Does this change the estimated budget above? If yes, by how much?

yes, these tasks require an additional 550 USD for me (and 200 for wiz i believe) already acked by @wiz as part of the opsdev budget.

[ripcurlx]

yes, these tasks require an additional 550 USD for me (and 200 for wiz i believe) already acked by @wiz as part of the opsdev budget.

So the total amount for this project is 2750 USD, correct?

[freimair]

task	estimate	receiver
create the data gathering infrastructure tools	USD 500,00	@freimair
update monitoring daemon to deliver additional metrics	USD 450,00	@freimair
create installer for general monitoring	USD 550,00	@freimair
create pricenode installer	(?) USD 200,00	@wiz
deploy the tools	5 * 150 = USD 750,00	price node operator
configure https://monitor.bisq.network	USD 500,00	@freimair

which amount belongs to which budget is for you guys to decide.

[freimair]

here is a preliminary set of graphs. Please note that the network size is only provided by one pricenode and is multiplied x8 for a rough estimation of real numbers.

We have to discuss the second KPI, ie. Number of available offers > Number of trades. The number of available offers is a snapshot of a continuous value, the number of trades needs to be accumulated over a certain time frame. What time frame shall it be?

[freimair]

Report Review Meeting 2020-03-19

• we have preliminary data coming in including the KPIs here
• scripts are ready to be deployed
• yet, in foresight to the tor hidden service update, I decided to first create the scripts for version scraping, so we can reduce resource requirements of pricenode operators to a minimum

[cbeams]

It's great to walk through the dashboard and see this data!

[cbeams]

@ripcurlx, can you provide an update on this project?

[freimair]

All the distributed deployments have some spurious issues every now and then. Need to get myself access to a live pricenode to figure out what is going on.

And I need some more budget. Depending on the issue, +550$ should do.

[freimair]

found the error, at least it seems like it. There are no more gaps in the last 30+ days (except for the time where pricenode have been upgraded and the network size data feeds have not been enabled).

@mrosseel's new pricenode does not supply data still. so add a +75 or so to the final number of nodes for numbers after 2020-08-21.

[ripcurlx]

KPIs seem to be stable now. I'll use them for the next releases to check if we made progress. Until now because of missing data every now and then I wasn't able to use them for feature/version evaluation.

I did use the version update metric a lot which is great to see how quickly our users do update.

So from my side this project is now complete.

[freimair]

here is the grafana config

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "gnetId": null,
  "graphTooltip": 0,
  "id": 13,
  "links": [],
  "panels": [
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 3,
        "w": 5,
        "x": 0,
        "y": 0
      },
      "id": 19,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "options": {},
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "hide": false,
          "refCount": 0,
          "refId": "A",
          "target": "alias(scale(sumSeries(movingWindow(servers.*.requestsPer750Seconds.*, '15min', 'last')), 0.08), 'active nodes')"
        }
      ],
      "thresholds": "",
      "timeFrom": null,
      "timeShift": null,
      "title": "estimated Network size (avg of selected range)",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "avg"
    },
    {
      "alert": {
        "conditions": [
          {
            "evaluator": {
              "params": [
                100
              ],
              "type": "lt"
            },
            "operator": {
              "type": "and"
            },
            "query": {
              "params": [
                "A",
                "5m",
                "now"
              ]
            },
            "reducer": {
              "params": [],
              "type": "avg"
            },
            "type": "query"
          }
        ],
        "executionErrorState": "alerting",
        "for": "5m",
        "frequency": "1m",
        "handler": 1,
        "name": "Estimated network size",
        "noDataState": "no_data",
        "notifications": []
      },
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 19,
        "x": 5,
        "y": 0
      },
      "id": 2,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "connected",
      "options": {},
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "hide": true,
          "refCount": 0,
          "refId": "A",
          "target": "alias(scale(sumSeries(movingWindow(servers.*.requestsPer750Seconds.*, '15min', 'last')), 0.08), 'active nodes')"
        },
        {
          "hide": true,
          "refCount": 0,
          "refId": "B",
          "target": "aliasSub(scale(movingWindow(groupByNode(servers.*.requestsPer750Seconds.*, 0, 'sum'), '15min', 'last'), 0.08), 'servers.([a-z0-9]+)\\..*gauge-(.*?),', '\\2-\\1')",
          "textEditor": false
        },
        {
          "hide": false,
          "refCount": 0,
          "refId": "C",
          "target": "aliasByNode(scale(groupByNode(movingWindow(removeEmptySeries(servers.*.requestsPer750Seconds.*), '15min', 'average'), 1, 'sum'), 0.08), 0)",
          "textEditor": false
        }
      ],
      "thresholds": [
        {
          "colorMode": "critical",
          "fill": true,
          "line": true,
          "op": "lt",
          "value": 100,
          "yaxis": "left"
        }
      ],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "estimated Network size",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 3,
        "w": 5,
        "x": 0,
        "y": 3
      },
      "id": 18,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "options": {},
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "hide": true,
          "refCount": -1,
          "refId": "A",
          "target": "scale(sumSeries(movingWindow(servers.*.requestsPer750Seconds.*, '15min', 'last')), 0.08)"
        },
        {
          "refCount": 0,
          "refId": "C",
          "target": "divideSeries(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), #A)",
          "targetFull": "divideSeries(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), scale(sumSeries(movingWindow(servers.*.requestsPer750Seconds.*, '15min', 'last')), 0.08))"
        }
      ],
      "thresholds": "",
      "timeFrom": null,
      "timeShift": null,
      "title": "offers per node (avg of selected range)",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "avg"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "format": "none",
      "gauge": {
        "maxValue": 1,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 3,
        "w": 5,
        "x": 0,
        "y": 6
      },
      "id": 17,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "options": {},
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "hide": true,
          "refCount": -1,
          "refId": "C",
          "target": "keepLastValue(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), 200)"
        },
        {
          "hide": false,
          "refCount": 0,
          "refId": "B",
          "target": "divideSeries(movingSum(sumSeries(isNonNull(bisq.MarketStats.volume.*)), '6h'), #C)",
          "targetFull": "divideSeries(movingSum(sumSeries(isNonNull(bisq.MarketStats.volume.*)), '6h'), keepLastValue(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), 200))",
          "textEditor": false
        }
      ],
      "thresholds": "",
      "timeFrom": null,
      "timeShift": null,
      "title": "trades per offer (avg over selected range)",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "avg"
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 0,
        "y": 9
      },
      "id": 4,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": false,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "connected",
      "options": {},
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "hide": true,
          "refCount": -1,
          "refId": "A",
          "target": "scale(sumSeries(movingWindow(servers.*.requestsPer750Seconds.*, '15min', 'last')), 0.08)",
          "textEditor": false
        },
        {
          "hide": false,
          "refCount": 0,
          "refId": "C",
          "target": "divideSeries(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), #A)",
          "targetFull": "divideSeries(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), scale(sumSeries(movingWindow(servers.*.requestsPer750Seconds.*, '15min', 'last')), 0.08))"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "offers per node",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 12,
        "y": 9
      },
      "id": 5,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": false,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "connected",
      "options": {},
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "hide": true,
          "refCount": -1,
          "refId": "C",
          "target": "keepLastValue(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), 200)"
        },
        {
          "hide": false,
          "refCount": 0,
          "refId": "B",
          "target": "divideSeries(movingSum(sumSeries(isNonNull(bisq.MarketStats.volume.*)), '6h'), #C)",
          "targetFull": "divideSeries(movingSum(sumSeries(isNonNull(bisq.MarketStats.volume.*)), '6h'), keepLastValue(aggregate(bisq.P2PMarketStats.wizseed7ab2gi3x267xahrp2pkndyrovczezzb46jk6quvguciuyqrid.*.*, 'sum'), 200))",
          "textEditor": false
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "trades per offer (6h avg)",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 0,
        "y": 18
      },
      "id": 16,
      "legend": {
        "alignAsTable": false,
        "avg": false,
        "current": true,
        "max": false,
        "min": false,
        "show": false,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "connected",
      "options": {},
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "refCount": 0,
          "refId": "A",
          "target": "aliasByMetric(sumSeries(removeEmptySeries(bisq.P2PMarketStats.*.*.*)))"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Open Offers",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "cacheTimeout": null,
      "dashLength": 10,
      "dashes": false,
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 12,
        "y": 18
      },
      "id": 13,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": false,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {},
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "hide": false,
          "refCount": 0,
          "refId": "A",
          "target": "movingSum(sumSeries(isNonNull(bisq.MarketStats.volume.*)), '6hours')",
          "textEditor": false
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Number of Trades (6h avg)",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "cacheTimeout": "",
      "dashLength": 10,
      "dashes": false,
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 0,
        "y": 27
      },
      "id": 3,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "rightSide": true,
        "show": true,
        "sideWidth": 65,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "connected",
      "options": {},
      "percentage": true,
      "pluginVersion": "6.2.5",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "hide": false,
          "refCount": 0,
          "refId": "A",
          "target": "aliasSub(aliasSub(exclude(aliasByNode(sortBy(asPercent(movingAverage(groupByNode(servers.*.requestsPer750Seconds.*, 3, 'sum'), '15min')), 'last', false), 0), 'vget'), '^.*(v\\d.\\d.\\d).*', '\\1'), '_', '.')",
          "textEditor": false
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "version spread",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 12,
        "y": 27
      },
      "id": 14,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "rightSide": true,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "connected",
      "options": {},
      "percentage": true,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "hide": false,
          "refCount": 0,
          "refId": "A",
          "target": "aliasSub(aliasByNode(sortBy(asPercent(movingAverage(groupByNode(servers.*.hsversionStats.*, 3, 'sum'), '15min')), 'last', false), 0), '^.*(HSv\\d)', '\\1')",
          "textEditor": false
        },
        {
          "hide": true,
          "refCount": 0,
          "refId": "B",
          "target": "groupByNode(servers.*.hsversionStats.*, 3, 'average')"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "hs version spread",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    }
  ],
  "refresh": false,
  "schemaVersion": 18,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-24h",
    "to": "now"
  },
  "timepicker": {
    "nowDelay": "",
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "",
  "title": "Key Performance Indicators",
  "uid": "dEVJDduWz",
  "version": 62
}