[DOCS] Configuration and component section overhaul (#3324)

* Begin annotated config examples Signed-off-by: ChrisChinchilla <[email protected]> * Restructure into components Signed-off-by: ChrisChinchilla <[email protected]> * More Signed-off-by: ChrisChinchilla <[email protected]> * Add more Signed-off-by: ChrisChinchilla <[email protected]> * downsample section Signed-off-by: ChrisChinchilla <[email protected]> * wip Signed-off-by: ChrisChinchilla <[email protected]> * Finish first draft of query and coordinate config file Signed-off-by: ChrisChinchilla <[email protected]> * reorg Signed-off-by: ChrisChinchilla <[email protected]> * dbnode annotated config Signed-off-by: ChrisChinchilla <[email protected]> * Moving to final updates Signed-off-by: ChrisChinchilla <[email protected]> * Rename section Signed-off-by: ChrisChinchilla <[email protected]> * Add links and redirects Signed-off-by: ChrisChinchilla <[email protected]> * Fix broken links Signed-off-by: ChrisChinchilla <[email protected]> * Remove aggregator config for now Signed-off-by: ChrisChinchilla <[email protected]> * Act on todos Signed-off-by: ChrisChinchilla <[email protected]> * Update m3dbnode config to use new syntax Signed-off-by: ChrisChinchilla <[email protected]> * Update query config Signed-off-by: ChrisChinchilla <[email protected]> * Change path to reference Signed-off-by: ChrisChinchilla <[email protected]> * Reorganise components into architecture and reference sections Signed-off-by: ChrisChinchilla <[email protected]> * Update theme Signed-off-by: ChrisChinchilla <[email protected]> * Fix links Signed-off-by: ChrisChinchilla <[email protected]> * Update generated file Signed-off-by: ChrisChinchilla <[email protected]>
m3db · Apr 6, 2021 · 5a5635d · 5a5635d
1 parent 1843061
commit 5a5635d
Show file tree

Hide file tree

Showing 72 changed files with 1,383 additions and 444 deletions.
diff --git a/README.md b/README.md
@@ -4,7 +4,7 @@
 
 <p align="center"><img src="site/static/images/logo-square.png" alt="M3 Logo" width="256" height="270"></p>
 
-[Distributed TSDB](https://m3db.io/docs/m3db/) and [Query Engine](https://m3db.io/docs/how_to/query/), [Prometheus Sidecar](https://m3db.io/docs/integrations/prometheus/), [Metrics Aggregator](https://m3db.io/docs/overview/components/#m3-aggregator), and more such as [Graphite storage and query engine](https://m3db.io/docs/integrations/graphite/).
+[Distributed TSDB](https://m3db.io/docs/reference/m3db/) and [Query Engine](https://m3db.io/docs/how_to/query/), [Prometheus Sidecar](https://m3db.io/docs/integrations/prometheus/), [Metrics Aggregator](https://m3db.io/docs/overview/reference/#m3-aggregator), and more such as [Graphite storage and query engine](https://m3db.io/docs/integrations/graphite/).
 
 ## Table of Contents
 

diff --git a/m3query_intro.md b/m3query_intro.md
@@ -0,0 +1,3 @@
+M3 Query is a distributed query engine for querying realtime and historical data stored in M3DB nodes, supporting several query languages. It is designed to support both low latency realtime queries and queries that can take longer to execute, aggregating over larger datasets for analytical use cases.
+
+For example, if you are using the Prometheus remote write endpoint with [M3 Coordinator](/docs/integrations/prometheus), you can use M3 Query instead of the Prometheus remote read endpoint. By doing so, you get all the benefits of M3 Query's engine such as [block processing](/docs/architecture/m3query/blocks/). As M3 Query provides a Prometheus compatible API, you can use 3rd party graphing and alerting solutions like Grafana.
diff --git a/netlify.toml b/netlify.toml
@@ -40,6 +40,14 @@
   from="/docs/quickstart/kubernetes"
   to="/docs/cluster/kubernetes_cluster"
 
+[[redirects]]
+  from="/docs/how_to/aggregator"
+  to="/docs/m3aggregator/setup"
+
+[[redirects]]
+  from="/docs/how_to/query"
+  to="/docs/m3query/setup"
+
 # TODO: Fix this with new content type
 [[redirects]]
   from = "/talks"

diff --git a/site/content/_index.md b/site/content/_index.md
@@ -15,9 +15,9 @@ other supporting infrastructure.
 
 M3 has several features, provided as discrete components, which make it an ideal platform for time series data at scale:
 
--   A distributed time series database, [M3DB](/docs/m3db/), that provides scalable storage for time series data and a reverse index.
+-   A distributed time series database, [M3DB](/docs/reference/m3db/), that provides scalable storage for time series data and a reverse index.
 -   A sidecar process, [M3Coordinator](/docs/integrations/prometheus), that allows M3DB to act as the long-term storage for Prometheus.
--   A distributed query engine, [M3Query](/docs/m3query), with native support for PromQL and Graphite (M3QL coming soon).
+-   A distributed query engine, [M3Query](/docs/reference/m3query), with native support for PromQL and Graphite (M3QL coming soon).
     <!-- Add M3Aggregator link -->
 -   An aggregation tier, M3Aggregator, that runs as a dedicated metrics aggregator/downsampler allowing metrics to be stored at various retentions at different resolutions.
 
@@ -30,7 +30,7 @@ Getting started with M3 is as easy as following one of the How-To guides.
 -   [Single M3DB node deployment](/docs/quickstart)
 -   [Clustered M3DB deployment](/docs/cluster)
 -   [M3DB on Kubernetes](/docs/operator)
--   [Isolated M3Query on deployment](/docs/how_to/query)
+-   [Isolated M3Query on deployment](/docs/how_to/m3query)
 
 ## Support
 

diff --git a/site/content/architecture/_index.md b/site/content/architecture/_index.md
@@ -0,0 +1,4 @@
+---
+title: Architecture
+weight: 20
+---
diff --git a/site/content/architecture/m3coordinator.md b/site/content/architecture/m3coordinator.md
@@ -0,0 +1,12 @@
+---
+title: "M3 Coordinator"
+weight: 3
+---
+
+M3 Coordinator is a service that coordinates reads and writes between upstream systems, such as Prometheus, and downstream systems, such as M3DB.
+
+It also provides management APIs to setup and configure different parts of M3.
+
+The coordinator is generally a bridge for read and writing different types of metrics formats and a management layer for M3.
+
+**Note**: M3DB by default includes the M3 Coordinator accessible on port 7201. For production deployments it is recommended to deploy it as a dedicated service to ensure you can scale the write coordination role separately and independently to database nodes as an isolated application separate from the M3DB database role.
diff --git a/site/content/architecture/m3db/_index.md b/site/content/architecture/m3db/_index.md
@@ -0,0 +1,5 @@
+---
+title: "M3DB, a distributed time series database"
+menuTitle: "M3DB"
+weight: 1
+---
diff --git a/site/content/m3db/architecture/caching.md → site/content/architecture/m3db/caching.md b/site/content/m3db/architecture/caching.md → site/content/architecture/m3db/caching.md
@@ -5,7 +5,7 @@ weight: 7
 
 ## Overview
 
-Blocks that are still being actively compressed / M3TSZ encoded must be kept in memory until they are sealed and flushed to disk. Blocks that have already been sealed, however, don't need to remain in-memory. In order to support efficient reads, M3DB implements various caching policies which determine which flushed blocks are kept in memory, and which are not. The "cache" itself is not a separate datastructure in memory, cached blocks are simply stored in their respective [in-memory objects](/docs/m3db/architecture/engine#in-memory-object-layout) with various different mechanisms (depending on the chosen cache policy) determining which series / blocks are evicted and which are retained.
+Blocks that are still being actively compressed / M3TSZ encoded must be kept in memory until they are sealed and flushed to disk. Blocks that have already been sealed, however, don't need to remain in-memory. In order to support efficient reads, M3DB implements various caching policies which determine which flushed blocks are kept in memory, and which are not. The "cache" itself is not a separate datastructure in memory, cached blocks are simply stored in their respective [in-memory objects](/docs/architecture/m3db/engine#in-memory-object-layout) with various different mechanisms (depending on the chosen cache policy) determining which series / blocks are evicted and which are retained.
 
 For general purpose workloads, the `lru` caching policy is reccommended.
 
@@ -21,7 +21,7 @@ The `all` cache policy is the opposite of the `none` cache policy. All blocks ar
 
 The `recently_read` cache policy keeps all blocks that are read from disk in memory for a configurable duration of time. For example, if the `recently_read` cache policy is set with a duration of 10 minutes, then everytime a block is read from disk it will be kept in memory for at least 10 minutes. This policy can be very effective if only a small portion of your overall dataset is ever read, and especially if that subset is read frequently (i.e as is common in the case of database backing an automatic alerting system), but it can cause very high memory usage during workloads that involve sequentially scanning all of the data.
 
-Data eviction from memory is triggered by the "ticking" process described in the [background processes section](/docs/m3db/architecture/engine#background-processes)
+Data eviction from memory is triggered by the "ticking" process described in the [background processes section](/docs/architecture/m3db/engine#background-processes)
 
 ## Least Recently Used (LRU) Cache Policy
 

diff --git a/site/content/m3db/architecture/commitlogs.md → site/content/architecture/m3db/commitlogs.md b/site/content/m3db/architecture/commitlogs.md → site/content/architecture/m3db/commitlogs.md
@@ -53,7 +53,7 @@ CommitLogMetadata {
 
 ### Compaction / Snapshotting
 
-Commit log files are compacted via the snapshotting proccess which (if enabled at the namespace level) will snapshot all data in memory into compressed files which have the same structure as the [fileset files](/docs/m3db/architecture/storage) but are stored in a different location. Once these snapshot files are created, then all the commit log files whose data are captured by the snapshot files can be deleted. This can result in significant disk savings for M3DB nodes running with large block sizes and high write volume where the size of the (uncompressed) commit logs can quickly get out of hand.
+Commit log files are compacted via the snapshotting proccess which (if enabled at the namespace level) will snapshot all data in memory into compressed files which have the same structure as the [fileset files](/docs/architecture/m3db/storage) but are stored in a different location. Once these snapshot files are created, then all the commit log files whose data are captured by the snapshot files can be deleted. This can result in significant disk savings for M3DB nodes running with large block sizes and high write volume where the size of the (uncompressed) commit logs can quickly get out of hand.
 
 In addition, since the snapshot files are already compressed, bootstrapping from them is much faster than bootstrapping from raw commit log files because the individual datapoints don't need to be decoded and then M3TSZ encoded. The M3DB node just needs to read the raw bytes off disk and load them into memory.
 

diff --git a/...nt/m3db/architecture/consistencylevels.md → ...nt/architecture/m3db/consistencylevels.md b/...nt/m3db/architecture/consistencylevels.md → ...nt/architecture/m3db/consistencylevels.md