[WIP][docs] Add hugo theme

docs-beta/archetypes/default.md

@@ -0,0 +1,6 @@
+---
+title: "{{ replace .Name "-" " " | title }}"
+date: {{ .Date }}
+draft: true
+---
+

docs-beta/config.toml

@@ -0,0 +1,53 @@
+# Change the default theme to be use when building the site with Hugo
+theme = "hugo-theme-learn"
+
+
+baseURL = "/"
+languageCode = "en-US"
+defaultContentLanguage = "en"
+
+title = "M3DB Documentation"
+metaDataFormat = "yaml"
+defaultContentLanguageInSubdir= true
+
+
+[params]
+  # Prefix URL to edit current page. Will display an "Edit this page" button on top right hand corner of every page. 
+  # Useful to give opportunity to people to create merge request for your doc.
+  # See the config.toml file from this documentation site to have an example.
+  editURL = ""
+  # Author of the site, will be used in meta information
+  author = ""
+  # Description of the site, will be used in meta information
+  description = ""
+  # Shows a checkmark for visited pages on the menu
+  showVisitedLinks = false
+  # Disable search function. It will hide search bar
+  disableSearch = false
+  # Javascript and CSS cache are automatically busted when new version of site is generated. 
+  # Set this to true to disable this behavior (some proxies don't handle well this optimization)
+  disableAssetsBusting = false
+  # Set this to true to disable copy-to-clipboard button for inline code.
+  disableInlineCopyToClipBoard = false
+  # A title for shortcuts in menu is set by default. Set this to true to disable it. 
+  disableShortcutsTitle = false
+  # When using mulitlingual website, disable the switch language button.
+  disableLanguageSwitchingButton = false
+  # Hide breadcrumbs in the header and only show the current page title
+  disableBreadcrumb = true
+  # Hide Next and Previous page buttons normally displayed full height beside content
+  disableNextPrev = true
+  # Order sections in menu by "weight" or "title". Default to "weight"
+  ordersectionsby = "weight"
+  # Change default color scheme with a variant one. Can be "red", "blue", "green".
+  themeVariant = ""
+  twitter = "m3db_io"
+
+[[Languages.en.menu.shortcuts]]
+name = "<i class='fab fa-fw fa-github'></i> GitHub"
+identifier = "ds"
+url = "https://github.com/m3db/m3"
+weight = 10
+
+[outputs]
+home = [ "HTML", "RSS", "JSON"]

docs-beta/content/basics/_index.md

@@ -0,0 +1,7 @@
+---
+title: "Basics"
+date: 2020-03-02T14:11:31-05:00
+# draft: true
+---
+
+asdfasdf

docs-beta/content/basics/first-content.md

@@ -0,0 +1,6 @@
+---
+title: "First Content"
+date: 2020-03-02T14:11:46-05:00
+# draft: true
+---
+

docs-beta/content/basics/second-content/_index.md

@@ -0,0 +1,6 @@
+---
+title: "Second Content"
+date: 2020-03-02T14:11:47-05:00
+# draft: true
+---
+

docs-beta/content/case_studies/index.md

@@ -0,0 +1 @@
+# Case Studies

docs-beta/content/code_assets/README.md

@@ -0,0 +1,5 @@
+# Code Assets
+
+## Overview
+
+This directory contains assets for READMEs and documentation in the source code. Some assets are kept separate from the source code in order to avoid bloating it with large / binary files.

docs-beta/content/common/headers_optional_read_write.md

@@ -0,0 +1,17 @@
+---
+title: "Basics"
+date: 2020-03-02T14:11:31-05:00
+draft: true
+---
+
+
+- `M3-Metrics-Type`:  
+ If this header is set, it determines what type of metric to store this metric value as. Otherwise by default, metrics will be stored in all namespaces that are configured. You can also disable this default behavior by setting `downsample` options to `all: false` for a namespace in the coordinator config, for more see [disabling automatic aggregation](/how_to/query.md#disabling-automatic-aggregation).<br /><br />
+ Must be one of:  
+ `unaggregated`: Write metrics directly to configured unaggregated namespace.  
+ `aggregated`: Write metrics directly to a configured aggregated namespace (bypassing any aggregation), this requires the `M3-Storage-Policy` header to be set to resolve which namespace to write metrics to.  
+<br />
+- `M3-Storage-Policy`:  
+ If this header is set, it determines which aggregated namespace to read/write metrics directly to/from (bypassing any aggregation).  
+ The value of the header must be in the format of `resolution:retention` in duration shorthand. e.g. `1m:48h` specifices 1 minute resolution and 48 hour retention. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".<br /><br />
+Here is [an example](https://github.com/m3db/m3/blob/master/scripts/docker-integration-tests/prometheus/test.sh#L126-L146) of querying metrics from a specific namespace. 

docs-beta/content/common/headers_placement_namespace.md

@@ -0,0 +1,12 @@
+---
+title: "Basics"
+date: 2020-03-02T14:11:31-05:00
+draft: true
+---
+
+
+- `Cluster-Environment-Name`:  
+ This header is used to specify the cluster environment name. If not set, the default `default_env` is used. 
+<br />
+- `Cluster-Zone-Name`:  
+ This header is used to specify the cluster zone name. If not set, the default `embedded` is used. 

docs-beta/content/community/index.md

@@ -0,0 +1,13 @@
+# Community
+
+## How to contact the M3 team?
+
+Feel free to contact us through any of the following channels:
+
+1. Posting on the [M3 Google group](https://groups.google.com/forum/#!forum/m3db)
+2. Opening issues on the [M3 GitHub page](https://github.com/m3db/m3/issues)
+3. Chatting us on the official [M3 Gitter channel](https://gitter.im/m3db/Lobby)
+
+## GitHub/OSS
+
+Our official GitHub page can be [found here](https://github.com/m3db/m3).

docs-beta/content/coordinator/api/remote.md

@@ -0,0 +1,102 @@
+# API
+
+The M3 Coordinator implements the Prometheus Remote Read and Write HTTP endpoints, they also can be used however as general purpose metrics write and read APIs. Any metrics that are written to the remote write API can be queried using PromQL through the query APIs as well as being able to be read back by the Prometheus Remote Read endpoint.
+
+## Remote Write
+
+Write a Prometheus Remote write query to M3.
+
+### URL
+
+`/api/v1/prom/remote/write`
+
+### Method
+
+`POST`
+
+### URL Params
+
+None.
+
+### Header Params
+
+#### Optional
+
+--8<--
+docs/common/headers_optional_read_write.md
+--8<--
+
+### Data Params
+
+Binary [snappy compressed](http://google.github.io/snappy/) Prometheus [WriteRequest protobuf message](https://github.com/prometheus/prometheus/blob/10444e8b1dc69ffcddab93f09ba8dfa6a4a2fddb/prompb/remote.proto#L22-L24).
+
+### Available Tuning Params 
+
+Refer [here](https://prometheus.io/docs/practices/remote_write/) for an up to date list of remote tuning parameters. 
+
+### Sample Call
+
+There isn't a straightforward way to Snappy compress and marshal a Prometheus WriteRequest protobuf message using just shell, so this example uses a specific command line utility instead. 
+
+This sample call is made using `promremotecli` which is a command line tool that uses a [Go client](https://github.com/m3db/prometheus_remote_client_golang) to Prometheus Remote endpoints. For more information visit the [GitHub repository](https://github.com/m3db/prometheus_remote_client_golang).
+
+There is also a [Java client](https://github.com/m3dbx/prometheus_remote_client_java) that can be used to make requests to the endpoint.
+
+Each `-t` parameter specifies a label (dimension) to add to the metric.
+
+The `-h` parameter can be used as many times as necessary to add headers to the outgoing request in the form of "Header-Name: HeaderValue".
+
+Here is an example of writing the datapoint at the current unix timestamp with value 123.456:
+
+<!-- 
+Note: keep this example similar to the one found in query API 
+documentation for consistency/ease of readers.
+-->
+```bash
+docker run -it --rm                                            \
+  quay.io/m3db/prometheus_remote_client_golang:latest          \
+  -u http://host.docker.internal:7201/api/v1/prom/remote/write \
+  -t __name__:http_requests_total                              \
+  -t code:200                                                  \
+  -t handler:graph                                             \
+  -t method:get                                                \
+  -d $(date +"%s"),123.456
+promremotecli_log 2019/06/25 04:13:56 writing datapoint [2019-06-25 04:13:55 +0000 UTC 123.456]
+promremotecli_log 2019/06/25 04:13:56 labelled [[__name__ http_requests_total] [code 200] [handler graph] [method get]]
+promremotecli_log 2019/06/25 04:13:56 writing to http://host.docker.internal:7201/api/v1/prom/remote/write
+{"success":true,"statusCode":200}
+promremotecli_log 2019/06/25 04:13:56 write success
+
+# If you are paranoid about image tags being hijacked/replaced with nefarious code, you can use this SHA256 tag:
+# quay.io/m3db/prometheus_remote_client_golang@sha256:fc56df819bff9a5a087484804acf3a584dd4a78c68900c31a28896ed66ca7e7b
+```
+
+For more details on querying data in PromQL that was written using this endpoint, see the [query API documentation](../../query_engine/api/).
+
+## Remote Read
+
+Read Prometheus metrics from M3.
+
+### URL
+
+`/api/v1/prom/remote/read`
+
+### Method
+
+`POST`
+
+### URL Params
+
+None.
+
+### Header Params
+
+#### Optional
+
+--8<--
+docs/common/headers_optional_read_write.md
+--8<--
+
+### Data Params
+
+Binary [snappy compressed](http://google.github.io/snappy/) Prometheus [WriteRequest protobuf message](https://github.com/prometheus/prometheus/blob/10444e8b1dc69ffcddab93f09ba8dfa6a4a2fddb/prompb/remote.proto#L26-L28).

docs-beta/content/coordinator/index.md

@@ -0,0 +1,7 @@
+# M3 Coordinator, API for reading/writing metrics and M3 management
+
+M3 Coordinator is a service that coordinates reads and writes between upstream systems, such as Prometheus, and downstream systems, such as M3DB. 
+
+It also provides management APIs to setup and configure different parts of M3.
+
+The coordinator is generally a bridge for read and writing different types of metrics formats and a management layer for M3.

docs-beta/content/docs/Dockerfile

@@ -0,0 +1,17 @@
+# Dockerfile for building docs is stored in a separate dir from the docs,
+# otherwise the generated site will unnecessarily contain the Dockerfile
+
+FROM python:3.5-alpine
+LABEL maintainer="The M3DB Authors <[email protected]>"
+
+WORKDIR /m3db
+EXPOSE 8000
+
+# mkdocs needs git-fast-import which was stripped from the default git package
+# by default to reduce size
+RUN pip install \
+    mkdocs==0.17.3 \
+    pymdown-extensions==6.0 \
+    mkdocs-material==2.7.3
+RUN apk add --no-cache git-fast-import openssh-client
+ENTRYPOINT [ "/bin/ash", "-c" ]

docs-beta/content/docs/index.md

@@ -0,0 +1,39 @@
+# M3
+
+## About
+
+After using open-source metrics solutions and finding issues with them at scale – such as reliability, cost, and
+operational complexity – [M3](https://github.com/m3db/m3) was created from the ground up to provide Uber with a
+native, distributed time series database, a highly-dynamic and performant aggregation service, a query engine, and
+other supporting infrastructure.
+
+## Key Features
+
+M3 has several features, provided as discrete components, which make it an ideal platform for time series data at scale:
+
+* A distributed time series database, [M3DB](m3db/index.md), that provides scalable storage for time series data and a reverse index.
+* A sidecar process, [M3Coordinator](integrations/prometheus.md), that allows M3DB to act as the long-term storage for Prometheus.
+* A distributed query engine, [M3Query](query_engine/index.md), with native support for PromQL and Graphite (M3QL coming soon).
+<!-- Add M3Aggregator link -->
+* An aggregation tier, M3Aggregator, that runs as a dedicated metrics aggregator/downsampler allowing metrics to be stored at various retentions at different resolutions.
+
+## Getting Started
+
+**Note:** Make sure to read our [Operational Guides](operational_guide/index.md) before running in production!
+
+Getting started with M3 is as easy as following one of the How-To guides.
+
+* [Single M3DB node deployment](how_to/single_node.md)
+* [Clustered M3DB deployment](how_to/cluster_hard_way.md)
+* [M3DB on Kubernetes](how_to/kubernetes.md)
+* [Isolated M3Query on deployment](how_to/query.md)
+
+## Support
+
+For support with any issues, questions about M3 or its operation, or to leave any comments, the team can be
+reached in a variety of ways:
+
+* [Slack (main chat channel)](http://bit.ly/m3slack)
+* [Gitter (old chat channel)](https://gitter.im/m3db/Lobby)
+* [Email](https://groups.google.com/forum/#!forum/m3db)
+* [Github issues](https://github.com/m3db/m3/issues)

docs-beta/content/ecosystem/index.md

@@ -0,0 +1 @@
+# Ecosystem

docs-beta/content/faqs/index.md

@@ -0,0 +1,76 @@
+# FAQs
+
+- **Is there a way to disable M3DB embedded `etcd` and just use an external `etcd` cluster?**
+Yes, you can definitely do that. It's all just about setting the etcd endpoints in config as etcd hosts instead of M3DB hosts. See [these docs](https://m3db.github.io/m3/operational_guide/etcd/#configuring-an-external-etcd-cluster) for more information.
+
+- **Is there a client that lets me send metrics to m3coordinator without going through Prometheus?**
+Yes, you can use the [Prometheus remote write client](https://github.com/m3db/prometheus_remote_client_golang/).
+
+- **Why does my dbnode keep OOM’ing?**
+Refer to the [troubleshooting guide](../troubleshooting/index.md).
+
+- **Do you support PromQL?**
+Yes, M3Query and M3Coordinator both support PromQL.
+
+- **Do you support Graphite?**
+Yes, M3Query and M3Coordinator both support Graphite.
+
+- **Does M3DB store both data and (tag) metadata on the same node?**
+Yes it stores the data (i.e. the timeseries datapoints) as well as the tags since it has an embedded index. Make sure you have `IndexEnabled` set to `true` in your namespace configuration
+
+- **How are writes handled and how is the data kept consistent within M3DB?**
+M3 uses quorum/majority consistency to ensure data is written to replicas in a way that can be read back consistently. 
+For example, if you have a replication factor of 3 and your set your write and read consistencies to quorum, then all writes will only succeed if they make it to at least 2 of the 3 replicas, and reads will only succeed if they get results back from at least 2 of the 3 replicas
+
+- **Do I need to restart M3DB if I add a namespace?**
+If you’re adding namespaces, the m3dbnode process will pickup the new namespace without a restart.
+
+- **Do I need to restart M3DB if I change or delete a namespace?**
+If you’re removing or modifying an existing namespace, you’ll need to restart the m3dbnode process in order to complete the namespace deletion/modification process. It is recommended to restart one node at a time and wait for a node to be completely bootstrapped before restarting another node.
+
+- **How do I set up aggregation in the coordinator?**
+Refer to the [Aggregation section](../how_to/query.md) of the M3Query how-to guide.
+
+- **How do I set up aggregation using a separate aggregation tier?**
+See this [WIP documentation](https://github.com/m3db/m3/pull/1741/files#diff-0a1009f86783ca8fd4499418e556c6f5).
+
+- **Can you delete metrics from M3DB?**
+Not yet, but that functionality is currently being worked on.
+
+- **How can you tell if a node is snapshotting?**
+You can check if your nodes are snapshotting by looking at the `Background tasks` tab in the [M3DB Grafana dashboard](https://grafana.com/dashboards/8126).
+
+- **How do you list all available API endpoints?**
+See [M3DB openhttps://m3db.io/openapi
+
+- **What is the recommended way to upgrade my M3 stack?**
+TBA
+
+- **When graphing my Prometheus data in Grafana, I see gaps. How do I resolve this?**
+This is due to M3 having a concept of `null` datapoints whereas Prometheus does not. To resolve this, change `Stacking & Null value` to `Connected` under the `Visualization` tab of your graph.
+
+- **I am receiving the error `"could not create etcd watch","error":"etcd watch create timed out after 10s for key: _sd.placement/default_env/m3db"`**
+This is due to the fact that M3DB, M3Coordinator, etc. could not connect to the `etcd` server. Make sure that the endpoints listed under in the following config section are correct AND the correct configuration file is being used.
+```
+etcdClusters:
+  - zone: embedded
+    endpoints:
+      - HOST1_STATIC_IP_ADDRESS:2379
+      - HOST2_STATIC_IP_ADDRESS:2379
+      - HOST3_STATIC_IP_ADDRESS:2379
+``` 
+
+- **How can I get a heap dump, cpu profile, etc.**
+See our docs on the [/debug/dump api](../troubleshooting/index.md)
+
+- **How much memory utilization should I run M3DB at?**
+We recommend not going above 50%.
+
+- **What is the recommended hardware to run on?**
+TBA
+
+- **What is the recommended way to create a new namespace?**
+Refer to the [Namespace configuration guide](../operational_guide/namespace_configuration.md).
+
+- **How can I see the cardinality of my metrics?**
+Currently, the best way is to go to the [M3DB Node Details Dashboard](https://grafana.com/grafana/dashboards/8126) and look at the `Ticking` panel. However, this is not entirely accurate because of the way data is stored in M3DB -- time series are stored inside time-based blocks that you configure. In actuality, the `Ticking` graph shows you how many unique series there are for the most recent block that has persisted. In the future, we plan to introduce easier ways to determine the number of unique time series.