Metric Store: A Cloud-Native Time Series Database

Metric Store Release is a BOSH release for Metric Store. It provides a persistent storage layer for metrics sent through the Loggregator subsystem. It is multi-tenant aware (the auth proxy ensures that you only have access to metrics from your apps), easy to query (it is 100% compatible with the Prometheus Query API, with some exceptions listed below), and has a powerful storage engine (the InfluxDB storage engine has built-in compression and a memory-efficient series index).

Deploying

Metric Store can be deployed within Cloud Foundry. Metric Store will have to know about Loggregator.

Cloud Config

Every BOSH deployment requires a cloud config. The Metric Store deployment manifest assumes the CF-Deployment cloud config has been uploaded.

Creating and Uploading a Release

The first step in deploying Metric Store is to create a release or download it from bosh.io. Final releases are preferable, however during the development process dev releases are useful.

The following commands will create a dev release and upload it to an environment named testing.

bosh create-release --force
bosh -e testing upload-release --rebase

Cloud Foundry

Metric Store deployed within Cloud Foundry reads from the Loggregator system and registers with the GoRouter at metric-store.<system-domain>.

You can deploy Metric Store by using this operations file.

bosh -e testing -d cf \
    deploy cf-deployment.yml \
    -o add-metric-store-to-cfd.yml

Metric Store UAA Client

By Default, Metric Store uses the doppler client included with cf-deployment.

If you would like to use a custom client, it requires the uaa.resource authority:

<custom_client_id>:
    authorities: uaa.resource
    override: true
    authorized-grant-types: client_credentials
    secret: <custom_client_secret>

Using Metric Store

Storing Metrics

Metric Store ingresses all metrics (discarding logs) from the Reverse Log Proxy on Loggregator. Any metric sent to a Loggregator Agent will travel downstream into Metric Store.

Accessing Metrics in Metric Store

Authorization and Authentication

Metric Store as deployed in a Cloud Foundry deployment depends on the CF Auth Proxy job to convert your UAA provided auth token into an authorized list of source IDs for Metric Store. In Cloud Foundry terms, the source ID can either represent an application guid (e.g. cf app <app-name> --guid), or a component name (e.g. doppler).

Each request must have the Authorization header set with a UAA provided token. If the token contains the doppler.firehose scope, the request will be able to read data from any source ID. If the source ID is an app guid, the Cloud Controller is consulted to verify if the provided token has the appropriate app access.

PromQL via HTTP

Metric Store provides Prometheus Query Language (PromQL) compatible endpoints. Queries against Metric Store can be crafted with the help of the Prometheus API Documentation.

Example: GET /api/v1/query

This issues a PromQL query against Metric Store data.

curl -G "http://<metric-store-addr>:8080/api/v1/query" --data-urlencode 'query=metrics{source_id="source-id-1"}'

Response Body

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [{ "metric": {...}, "point": [...] }]
  }
}

See the official PromQL API documentation for more information.

Notes on PromQL

A valid PromQL metric name consists of the characters [a-Z][0-9], underscore, and colon. Names can begin with [a-Z], underscore, or colon. Names cannot begin with a number [0-9]. As a measure to work with existing metrics that do not comply with the above format a conversion process takes place when matching on metric names. As noted above, any character that is not in the set of valid characters is converted to an underscore before it is written to disk. For example, to match on a metric name http.latency use the name http_latency in your query.

Prometheus API Compatability

• /api/v1/query & /api/v1/query_range, fully supported except for regex matchers on __name__ (for everyone) or source_id (for non-admins)
• /api/v1/series, /api/v1/labels, /api/v1/rules, /api/v1/alerts & /api/v1/alertmanagers, fully supported for admins
• the remaining endpoints are not currently supported

Golang Clients For BOSH-Deployed Components

Interacting with Metric Store directly, circumventing the GoRouter and CF Auth Proxy, can be done using our Go ingress client library or our Go egress client library. This will require a bosh deployed component to receive metric-store bosh links for certificate sharing. The resulting client interaction has admin access.

Using Grafana to visualize metrics

See Set up Metric Store with Grafana in the docs directory.

Contributing

We'd love to hear feedback about your experiences with Metric Store. Please feel free to open up an issue, send us a pull request, or come chat with us on Cloud Foundry Slack.