Investigate expvar and a modular approach to integration with each package

[ashleyvega]

In GitLab by @cdm on Jan 23, 2019, 15:07

@ashleyvega has suggested using expvar (https://golang.org/pkg/expvar/) library for instrumentation/debugging.

1. Investigate use in real-world code, understand benefits/performance impact.
2. Decide if we can plug expvar reporting into each package in trading-core with a standardised pattern.
3. Implement config settings and reusable code to do above, test with single package.
4. Evaluate and improve.
5. Add to other packages as required and document a clear process for future developers (Maks, etc)

Useful links

• https://www.mikeperham.com/2014/12/17/expvar-metrics-for-golang/
• https://orangetux.nl/post/expvar_in_action/
• Import counts into public projects: https://godoc.org/?q=expvar
  • expvar: ~5k
  • sirupsen/logrus: ~11k
  • jinzhu/configor: 50
  • protobuf/proto: ~11k
• Things we use that use expvar:
  • prometheus/client_golang:
    • https://gitlab.com/vega-protocol/trading-core/blob/develop/vendor/github.com/prometheus/client_golang/prometheus/expvar_collector.go
  • dgraph-io/badger:
    • https://gitlab.com/vega-protocol/trading-core/blob/develop/vendor/github.com/dgraph-io/badger/db.go
    • https://gitlab.com/vega-protocol/trading-core/blob/develop/vendor/github.com/dgraph-io/badger/y/metrics.go

See also

• Elasticsearch APM (app perfermance monitoring): https://www.elastic.co/guide/en/apm/get-started/6.5/index.html

[ashleyvega]

In GitLab by @ashleyvega on Jan 23, 2019, 15:47

changed the description

[ashleyvega]

In GitLab by @ashleyvega on Jan 24, 2019, 11:58

Added expvar to go-trade-bot (https://gitlab.com/vega-protocol/go-trade-bot/tree/17-investigate-expvar). Output:

{
  "cmdline": [
    "./tradingbotservice",
    "-config",
    "scripts/config/bot-multinet-config.yml"
  ],
  "memstats": {
    "Alloc": 1953000,
    "TotalAlloc": 15623768,
    "Sys": 72284408,
    "Lookups": 0,
    "Mallocs": 312417,
    "Frees": 295620,
    "HeapAlloc": 1953000,
    "HeapSys": 66256896,
    "HeapIdle": 61939712,
    "HeapInuse": 4317184,
    "HeapReleased": 0,
    "HeapObjects": 16797,
    "StackInuse": 851968,
    "StackSys": 851968,
    "MSpanInuse": 77064,
    "MSpanSys": 98304,
    "MCacheInuse": 6912,
    "MCacheSys": 16384,
    "BuckHashSys": 1450012,
    "GCSys": 2379776,
    "OtherSys": 1231068,
    "NextGC": 4194304,
    "LastGC": 1548330817118455300,
    "PauseTotalNs": 819714,
    "PauseNs": [
      276123,
      71974,
      73054,
      39447,
      67062,
      83100,
      208954,
      0,
      ## many zero entries ##
      0
    ],
    "PauseEnd": [
      1548330816555094800,
      1548330816581530000,
      1548330816593709600,
      1548330816605611500,
      1548330816616334000,
      1548330816626943000,
      1548330817118455300,
      0,
      ## many zero entries ##
      0
    ],
    "NumGC": 7,
    "NumForcedGC": 0,
    "GCCPUFraction": 0.005355909619491515,
    "EnableGC": true,
    "DebugGC": false,
    "BySize": [
      ## skip lots of entries ##
      {
        "Size": 256,
        "Mallocs": 141,
        "Frees": 101
      },
      {
        "Size": 2048,
        "Mallocs": 393,
        "Frees": 371
      },
      {
        "Size": 16384,
        "Mallocs": 0,
        "Frees": 0
      },
      {
        "Size": 19072,
        "Mallocs": 1,
        "Frees": 0
      }
    ]
  }
}

[ashleyvega]

In GitLab by @ashleyvega on Jan 24, 2019, 13:58

changed the description

[ashleyvega]

In GitLab by @ashleyvega on Jan 24, 2019, 14:50

but putting expvar directly in with gin, by using gin-contrib/expvar was a pain, and didn't work: gin-gonic/gin#1673 (comment)

[ashleyvega]

In GitLab by @ashleyvega on Jan 24, 2019, 16:04

marked this issue as related to go-trade-bot#17

[ashleyvega]

In GitLab by @ashleyvega on Jan 28, 2019, 09:53

changed the description

[ashleyvega]

In GitLab by @ashleyvega on Jan 28, 2019, 10:28

changed the description

[ashleyvega]

In GitLab by @cdm on Feb 20, 2019, 14:10

@ashleyvega would be worth chatting with @jeremyletang soon to get his thoughts on how/if we need this for the vega binaries.

[ashleyvega]

In GitLab by @ashleyvega on Feb 20, 2019, 14:12

@cdm Sounds good. In go-trade-bot it's been easy to add, and is controlled by a config file option. It shouldn't be difficult to do likewise in trading-core.

[ashleyvega]

In GitLab by @cdm on Apr 1, 2019, 10:38

changed title from {-Logging: -}Investigate expvar and a modular approach to integration with each package to Investigate expvar and a modular approach to integration with each package

[ashleyvega]

In GitLab by @cdm on Apr 1, 2019, 14:51

@jeremyletang would expvar be useful to us to add soon, just following up you adding pprof see this ticket

[ashleyvega]

In GitLab by @cdm on Apr 1, 2019, 14:51

If so I can move into 1904

[ashleyvega]

In GitLab by @jeremyletang on Apr 1, 2019, 14:52

@cdm, discussed this with @EVODelavega earlier today, I think he may have opinion about this.

[ashleyvega]

In GitLab by @cdm on Apr 1, 2019, 15:17

Ah excellent @EVODelavega if you'd like this to be added as a priority please move into 1904

[ashleyvega]

In GitLab by @ashleyvega on Mar 3, 2020, 11:26

closed