diff --git a/docs/content/coordinator/api/remote.md b/docs/content/coordinator/api/remote.md index 6339594558..edaf81f39c 100644 --- a/docs/content/coordinator/api/remote.md +++ b/docs/content/coordinator/api/remote.md @@ -73,7 +73,7 @@ promremotecli_log 2019/06/25 04:13:56 write success # quay.io/m3db/prometheus_remote_client_golang@sha256:fc56df819bff9a5a087484804acf3a584dd4a78c68900c31a28896ed66ca7e7b ``` -For more details on querying data in PromQL that was written using this endpoint, see the [query API documentation](../../query_engine/api/). +For more details on querying data in PromQL that was written using this endpoint, see the [query API documentation](/../query_engine/api/). ## Remote Read diff --git a/docs/content/faqs/_index.md b/docs/content/faqs/_index.md index 29e3e6ecdd..a96998f0b4 100644 --- a/docs/content/faqs/_index.md +++ b/docs/content/faqs/_index.md @@ -11,7 +11,7 @@ Yes, you can definitely do that. It's all just about setting the etcd endpoints Yes, you can use the [Prometheus remote write client](https://github.com/m3db/prometheus_remote_client_golang/). - **Why does my dbnode keep OOM’ing?** -Refer to the [troubleshooting guide](../troubleshooting). +Refer to the [troubleshooting guide](/troubleshooting). - **Do you support PromQL?** Yes, M3Query and M3Coordinator both support PromQL. @@ -33,7 +33,7 @@ If you’re adding namespaces, the m3dbnode process will pickup the new namespac If you’re removing or modifying an existing namespace, you’ll need to restart the m3dbnode process in order to complete the namespace deletion/modification process. It is recommended to restart one node at a time and wait for a node to be completely bootstrapped before restarting another node. - **How do I set up aggregation in the coordinator?** -Refer to the [Aggregation section](../how_to/query) of the M3Query how-to guide. +Refer to the [Aggregation section](/how_to/query) of the M3Query how-to guide. - **How do I set up aggregation using a separate aggregation tier?** See this [WIP documentation](https://github.com/m3db/m3/pull/1741/files#diff-0a1009f86783ca8fd4499418e556c6f5). @@ -65,7 +65,7 @@ etcdClusters: ``` - **How can I get a heap dump, cpu profile, etc.** -See our docs on the [/debug/dump api](../troubleshooting) +See our docs on the [/debug/dump api](/troubleshooting) - **How much memory utilization should I run M3DB at?** We recommend not going above 50%. @@ -74,7 +74,7 @@ We recommend not going above 50%. TBA - **What is the recommended way to create a new namespace?** -Refer to the [Namespace configuration guide](../operational_guide/namespace_configuration). +Refer to the [Namespace configuration guide](/operational_guide/namespace_configuration). - **How can I see the cardinality of my metrics?** Currently, the best way is to go to the [M3DB Node Details Dashboard](https://grafana.com/grafana/dashboards/8126) and look at the `Ticking` panel. However, this is not entirely accurate because of the way data is stored in M3DB -- time series are stored inside time-based blocks that you configure. In actuality, the `Ticking` graph shows you how many unique series there are for the most recent block that has persisted. In the future, we plan to introduce easier ways to determine the number of unique time series. \ No newline at end of file diff --git a/docs/content/how_to/cluster_hard_way.md b/docs/content/how_to/cluster_hard_way.md index 299c2c83e7..10367b583f 100644 --- a/docs/content/how_to/cluster_hard_way.md +++ b/docs/content/how_to/cluster_hard_way.md @@ -56,7 +56,7 @@ M3DB_HOST_ID=m3db001 m3dbnode -f config.yml ### Kernel -Ensure you review our [recommended kernel configuration](../operational_guide/kernel_configuration) before running M3DB in production as M3DB may exceed the default limits for some default kernel values. +Ensure you review our [recommended kernel configuration](/operational_guide/kernel_configuration) before running M3DB in production as M3DB may exceed the default limits for some default kernel values. ## Config files @@ -107,8 +107,8 @@ m3dbnode -f The recommended way to create a namespace and initialize a topology is to use the `/api/v1/database/create` api. Below is an example. -**Note:** In order to create a more custom setup, please refer to the [namespace configuration](../operational_guide/namespace_configuration) and -[placement configuration](../operational_guide/placement_configuration) guides, though this is discouraged. +**Note:** In order to create a more custom setup, please refer to the [namespace configuration](/operational_guide/namespace_configuration) and +[placement configuration](/operational_guide/placement_configuration) guides, though this is discouraged. ```shell curl -X POST http://localhost:7201/api/v1/database/create -d '{ @@ -167,11 +167,11 @@ If you need to setup multiple namespaces, you can run the above `/api/v1/databas ### Replication factor (RF) -Recommended is RF3, where each replica is spread across failure domains such as a rack, data center or availability zone. See [Replication Factor Recommendations](../operational_guide/replication_and_deployment_in_zones) for more specifics. +Recommended is RF3, where each replica is spread across failure domains such as a rack, data center or availability zone. See [Replication Factor Recommendations](/operational_guide/replication_and_deployment_in_zones) for more specifics. ### Shards -See [placement configuration](../operational_guide/placement_configuration) to determine the appropriate number of shards to specify. +See [placement configuration](/operational_guide/placement_configuration) to determine the appropriate number of shards to specify. ## Test it out @@ -216,4 +216,4 @@ curl -sS -X POST http://localhost:9003/query -d '{ ## Integrations -[Prometheus as a long term storage remote read/write endpoint](../integrations/prometheus). +[Prometheus as a long term storage remote read/write endpoint](/integrations/prometheus). diff --git a/docs/content/how_to/kubernetes.md b/docs/content/how_to/kubernetes.md index d5c44693cf..6ea32f58eb 100644 --- a/docs/content/how_to/kubernetes.md +++ b/docs/content/how_to/kubernetes.md @@ -279,7 +279,7 @@ curl -sSf -X POST localhost:7201/api/v1/placement -d '{ ### Prometheus -As mentioned in our integrations [guide](../integrations/prometheus), M3DB can be used as a [remote read/write +As mentioned in our integrations [guide](/integrations/prometheus), M3DB can be used as a [remote read/write endpoint](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cremote_write%3E) for Prometheus. If you run Prometheus on your Kubernetes cluster you can easily point it at M3DB in your Prometheus server config: @@ -311,4 +311,4 @@ certain nodes. Specifically: choose. 2. Via `nodeAffinity` the pods prefer to run on nodes with the label `m3db.io/dedicated-m3db="true"`. -[kernel]: ../operational_guide/kernel_configuration.md +[kernel]: /operational_guide/kernel_configuration diff --git a/docs/content/how_to/query.md b/docs/content/how_to/query.md index 51d9386d36..514572c590 100644 --- a/docs/content/how_to/query.md +++ b/docs/content/how_to/query.md @@ -5,11 +5,11 @@ weight: 4 --- -m3query is used to query data that is stored in M3DB. For instance, if you are using the Prometheus remote write endpoint with [m3coordinator](../integrations/prometheus), you can use m3query instead of the Prometheus remote read endpoint. By doing so, you get all of the benefits of m3query's engine such as [block processing](http://m3db.github.io/m3/query_engine/architecture/blocks/). Furthermore, since m3query provides a Prometheus compatible API, you can use 3rd party graphing and alerting solutions like Grafana. +m3query is used to query data that is stored in M3DB. For instance, if you are using the Prometheus remote write endpoint with [m3coordinator](/integrations/prometheus), you can use m3query instead of the Prometheus remote read endpoint. By doing so, you get all of the benefits of m3query's engine such as [block processing](http://m3db.github.io/m3/query_engine/architecture/blocks/). Furthermore, since m3query provides a Prometheus compatible API, you can use 3rd party graphing and alerting solutions like Grafana. ## Configuration -Before setting up m3query, make sure that you have at least [one M3DB node running](single_node). In order to start m3query, you need to configure a `yaml` file, that will be used to connect to M3DB. Here is a link to a [sample config](https://github.com/m3db/m3/blob/master/src/query/config/m3query-local-etcd.yml) file that is used for an embedded etcd cluster within M3DB. +Before setting up m3query, make sure that you have at least [one M3DB node running](/how_to/single_node). In order to start m3query, you need to configure a `yaml` file, that will be used to connect to M3DB. Here is a link to a [sample config](https://github.com/m3db/m3/blob/master/src/query/config/m3query-local-etcd.yml) file that is used for an embedded etcd cluster within M3DB. ### Running @@ -24,11 +24,11 @@ Or you can run it with Docker using the Docker file located at `$GOPATH/src/gith ### Namespaces -All namespaces that you wish to query from must be configured when [setting up M3DB](single_node). If you wish to add or change an existing namespace, please follow the namespace operational guide [here](../operational_guide/namespace_configuration). +All namespaces that you wish to query from must be configured when [setting up M3DB](/how_to/single_node). If you wish to add or change an existing namespace, please follow the namespace operational guide [here](/operational_guide/namespace_configuration). ### etcd -The configuration file linked above uses an embedded etcd cluster, which is fine for development purposes. However, if you wish to use this in production, you will want an [external etcd](../operational_guide/etcd) cluster. +The configuration file linked above uses an embedded etcd cluster, which is fine for development purposes. However, if you wish to use this in production, you will want an [external etcd](/operational_guide/etcd) cluster. diff --git a/docs/content/how_to/single_node.md b/docs/content/how_to/single_node.md index 6a0544e0ba..9dada5fe52 100644 --- a/docs/content/how_to/single_node.md +++ b/docs/content/how_to/single_node.md @@ -21,9 +21,9 @@ docker pull quay.io/m3db/m3dbnode:latest docker run -p 7201:7201 -p 7203:7203 -p 9003:9003 --name m3db -v $(pwd)/m3db_data:/var/lib/m3db quay.io/m3db/m3dbnode:latest ``` -**Note:** For the single node case, we use this [sample config file](https://github.com/m3db/m3/blob/master/src/dbnode/config/m3dbnode-local-etcd.yml). If you inspect the file, you'll see that all the configuration is grouped by `coordinator` or `db`. That's because this setup runs `M3DB` and `M3Coordinator` as one application. While this is convenient for testing and development, you'll want to run clustered `M3DB` with a separate `M3Coordinator` in production. You can read more about that [here.](cluster_hard_way). +**Note:** For the single node case, we use this [sample config file](https://github.com/m3db/m3/blob/master/src/dbnode/config/m3dbnode-local-etcd.yml). If you inspect the file, you'll see that all the configuration is grouped by `coordinator` or `db`. That's because this setup runs `M3DB` and `M3Coordinator` as one application. While this is convenient for testing and development, you'll want to run clustered `M3DB` with a separate `M3Coordinator` in production. You can read more about that [here.](/how_to/cluster_hard_way). -Next, create an initial namespace for your metrics in the database using the cURL below. Keep in mind that the provided `namespaceName` must match the namespace in the `local` section of the `M3Coordinator` YAML configuration, and if you choose to [add any additional namespaces](../operational_guide/namespace_configuration) you'll need to add them to the `local` section of `M3Coordinator`'s YAML config as well. +Next, create an initial namespace for your metrics in the database using the cURL below. Keep in mind that the provided `namespaceName` must match the namespace in the `local` section of the `M3Coordinator` YAML configuration, and if you choose to [add any additional namespaces](/operational_guide/namespace_configuration) you'll need to add them to the `local` section of `M3Coordinator`'s YAML config as well. @@ -35,7 +35,7 @@ curl -X POST http://localhost:7201/api/v1/database/create -d '{ }' ``` -**Note**: The `api/v1/database/create` endpoint is abstraction over two concepts in M3DB called [placements](../operational_guide/placement) and [namespaces](../operational_guide/namespace_configuration). If a placement doesn't exist, it will create one based on the `type` argument, otherwise if the placement already exists, it just creates the specified namespace. For now it's enough to just understand that it creates M3DB namespaces (tables), but if you're going to run a clustered M3 setup in production, make sure you familiarize yourself with the links above. +**Note**: The `api/v1/database/create` endpoint is abstraction over two concepts in M3DB called [placements](/operational_guide/placement) and [namespaces](/operational_guide/namespace_configuration). If a placement doesn't exist, it will create one based on the `type` argument, otherwise if the placement already exists, it just creates the specified namespace. For now it's enough to just understand that it creates M3DB namespaces (tables), but if you're going to run a clustered M3 setup in production, make sure you familiarize yourself with the links above. Placement initialization may take a minute or two and you can check on the status of this by running the following: @@ -92,7 +92,7 @@ curl -sS -X POST http://localhost:9003/writetagged -d '{ **Note:** In the above example we include the tag `__name__`. This is because `__name__` is a reserved tag in Prometheus and will make querying the metric much easier. For example, if you have -[M3Query](query) setup as a Prometheus datasource in Grafana, you can then query for the metric +[M3Query](/how_to/query) setup as a Prometheus datasource in Grafana, you can then query for the metric using the following PromQL query: ```shell @@ -144,4 +144,4 @@ curl -sS -X POST http://localhost:9003/query -d '{ } ``` -Now that you've got the M3 stack up and running, take a look at the rest of our documentation to see how you can integrate with [Prometheus](../integrations/prometheus) and [Graphite](../integrations/graphite) +Now that you've got the M3 stack up and running, take a look at the rest of our documentation to see how you can integrate with [Prometheus](/integrations/prometheus) and [Graphite](/integrations/graphite) diff --git a/docs/content/how_to/use_as_tsdb.md b/docs/content/how_to/use_as_tsdb.md index dbecd2352b..90c2caba8e 100644 --- a/docs/content/how_to/use_as_tsdb.md +++ b/docs/content/how_to/use_as_tsdb.md @@ -6,13 +6,13 @@ title: Using M3DB as a general purpose time series database ## Overview -M3 has native integrations that make it particularly easy to use it as a metrics storage for [Prometheus](../integrations/prometheus) and [Graphite](../integrations/graphite). M3DB can also be used as a general purpose distributed time series database by itself. +M3 has native integrations that make it particularly easy to use it as a metrics storage for [Prometheus](/integrations/prometheus) and [Graphite](/integrations/graphite). M3DB can also be used as a general purpose distributed time series database by itself. ## Data Model ### IDs and Tags -M3DB's data model allows multiple namespaces, each of which can be [configured and tuned independently](../operational_guide/namespace_configuration). +M3DB's data model allows multiple namespaces, each of which can be [configured and tuned independently](/operational_guide/namespace_configuration). Each namespace can also be configured with its own schema (see "Schema Modeling" section below). diff --git a/docs/content/integrations/graphite.md b/docs/content/integrations/graphite.md index 39cb13f6d3..db96f9ae13 100644 --- a/docs/content/integrations/graphite.md +++ b/docs/content/integrations/graphite.md @@ -11,7 +11,7 @@ M3 supports ingesting Graphite metrics using the [Carbon plaintext protocol](htt ## Ingestion -Setting up the M3 stack to ingest carbon metrics is straightforward. First, make sure you've followed our [other documentation](../how_to/single_node) to get m3coordinator and M3DB setup. Also, familiarize yourself with how M3 [handles aggregation](../how_to/query). +Setting up the M3 stack to ingest carbon metrics is straightforward. First, make sure you've followed our [other documentation](/how_to/single_node) to get m3coordinator and M3DB setup. Also, familiarize yourself with how M3 [handles aggregation](/how_to/query). Once you have both of those services running properly, modify your m3coordinator configuration to add the following lines and restart it: @@ -21,7 +21,7 @@ carbon: listenAddress: "0.0.0.0:7204" ``` -This will enable a line-based TCP carbon ingestion server on the specified port. By default, the server will write all carbon metrics to every aggregated namespace specified in the m3coordinator [configuration file](../how_to/query) and aggregate them using a default strategy of `mean` (equivalent to Graphite's `Average`). +This will enable a line-based TCP carbon ingestion server on the specified port. By default, the server will write all carbon metrics to every aggregated namespace specified in the m3coordinator [configuration file](/how_to/query) and aggregate them using a default strategy of `mean` (equivalent to Graphite's `Average`). This default setup makes sense if your carbon metrics are unaggregated, however, if you've already aggregated your data using something like [statsite](https://github.com/statsite/statsite) then you may want to disable M3 aggregation. In that case, you can do something like the following: diff --git a/docs/content/integrations/prometheus.md b/docs/content/integrations/prometheus.md index a3cdcce742..ca2c0e580f 100644 --- a/docs/content/integrations/prometheus.md +++ b/docs/content/integrations/prometheus.md @@ -11,7 +11,7 @@ To write to a remote M3DB cluster the simplest configuration is to run `m3coordi Start by downloading the [config template](https://github.com/m3db/m3/blob/master/src/query/config/m3coordinator-cluster-template.yml). Update the `namespaces` and the `client` section for a new cluster to match your cluster's configuration. -You'll need to specify the static IPs or hostnames of your M3DB seed nodes, and the name and retention values of the namespace you set up. You can leave the namespace storage metrics type as `unaggregated` since it's required by default to have a cluster that receives all Prometheus metrics unaggregated. In the future you might also want to aggregate and downsample metrics for longer retention, and you can come back and update the config once you've setup those clusters. You can read more about our aggregation functionality [here](../how_to/query). +You'll need to specify the static IPs or hostnames of your M3DB seed nodes, and the name and retention values of the namespace you set up. You can leave the namespace storage metrics type as `unaggregated` since it's required by default to have a cluster that receives all Prometheus metrics unaggregated. In the future you might also want to aggregate and downsample metrics for longer retention, and you can come back and update the config once you've setup those clusters. You can read more about our aggregation functionality [here](/how_to/query). It should look something like: diff --git a/docs/content/operational_guides/_index.md b/docs/content/operational_guide/_index.md similarity index 100% rename from docs/content/operational_guides/_index.md rename to docs/content/operational_guide/_index.md diff --git a/docs/content/operational_guides/availability_consistency_durability.md b/docs/content/operational_guide/availability_consistency_durability.md similarity index 98% rename from docs/content/operational_guides/availability_consistency_durability.md rename to docs/content/operational_guide/availability_consistency_durability.md index d9f794a67a..5fd7b07165 100644 --- a/docs/content/operational_guides/availability_consistency_durability.md +++ b/docs/content/operational_guide/availability_consistency_durability.md @@ -20,7 +20,7 @@ While reading it, we recommend referring to [the default configuration file](htt We recommend running the client with `writeConsistencyLevel` set to `majority` and `readConsistencyLevel` set to `unstrict_majority`. This means that all write must be acknowledged by a quorums of nodes in order to be considered succesful, and that reads will attempt to achieve quorum, but will return the data from a single node if they are unable to achieve quorum. This ensures that reads will normally ensure consistency, but degraded conditions will cause reads to fail outright as long as at least a single node can satisfy the request. -You can read about the consistency levels in more detail in [the Consistency Levels section](../m3db/architecture/consistencylevels) +You can read about the consistency levels in more detail in [the Consistency Levels section](/m3db/architecture/consistencylevels) ### Commitlog Configuration @@ -88,7 +88,7 @@ This issue requires an operator with significant M3DB operational experience to The most important thing to understand is that **if you want to guarantee that you will be able to read the result of every successful write, then both writes and reads must be done with `majority` consistency.** This means that both writes _and_ reads will fail if a quorum of nodes are unavailable for a given shard. -You can read about the consistency levels in more detail in [the Consistency Levels section](../m3db/architecture/consistencylevels) +You can read about the consistency levels in more detail in [the Consistency Levels section](/m3db/architecture/consistencylevels) ### Commitlog Configuration diff --git a/docs/content/operational_guides/bootstrapping_crash_recovery.md b/docs/content/operational_guide/bootstrapping_crash_recovery.md similarity index 96% rename from docs/content/operational_guides/bootstrapping_crash_recovery.md rename to docs/content/operational_guide/bootstrapping_crash_recovery.md index d9b8cdcb37..13b2a18c2f 100644 --- a/docs/content/operational_guides/bootstrapping_crash_recovery.md +++ b/docs/content/operational_guide/bootstrapping_crash_recovery.md @@ -3,7 +3,7 @@ title: "Bootstrapping & Crash Recovery" weight: 9 --- -We recommend reading the [placement operational guide](placement) before reading the rest of this document. +We recommend reading the [placement operational guide](/operational_guide/placement) before reading the rest of this document. When an M3DB node is turned on (goes through a placement change) it needs to go through a bootstrapping process to determine the integrity of data that it has, replay writes from the commit log, and/or stream missing data from its peers. In most cases, as long as you're running with the default and recommended bootstrapper configuration of: `filesystem,commitlog,peers,uninitialized_topology` then you should not need to worry about the bootstrapping process at all and M3DB will take care of doing the right thing such that you don't lose data and consistency guarantees are met. Note that the order of the configured bootstrappers _does_ matter. @@ -28,7 +28,7 @@ For example, imagine a M3DB node that is responsible for shards 1, 5, 13, and 25 ### Filesystem Bootstrapper -The `filesystem` bootstrapper's responsibility is to determine which immutable [Fileset files](../m3db/architecture/storage) exist on disk, and if so, mark them as fulfilled. The `filesystem` bootstrapper achieves this by scanning M3DB's directory structure and determining which Fileset files exist on disk. Unlike the other bootstrappers, the `filesystem` bootstrapper does not need to load any data into memory, it simply verifies the checksums of the data on disk and other components of the M3DB node will handle reading (and caching) the data dynamically once it begins to serve reads. +The `filesystem` bootstrapper's responsibility is to determine which immutable [Fileset files](/m3db/architecture/storage) exist on disk, and if so, mark them as fulfilled. The `filesystem` bootstrapper achieves this by scanning M3DB's directory structure and determining which Fileset files exist on disk. Unlike the other bootstrappers, the `filesystem` bootstrapper does not need to load any data into memory, it simply verifies the checksums of the data on disk and other components of the M3DB node will handle reading (and caching) the data dynamically once it begins to serve reads. ### Commitlog Bootstrapper diff --git a/docs/content/operational_guides/etcd.md b/docs/content/operational_guide/etcd.md similarity index 96% rename from docs/content/operational_guides/etcd.md rename to docs/content/operational_guide/etcd.md index 3acc775ea9..e6de2e0382 100644 --- a/docs/content/operational_guides/etcd.md +++ b/docs/content/operational_guide/etcd.md @@ -79,7 +79,7 @@ config: ### Embedded etcd -If you're running `M3DB seed nodes` with embedded `etcd` (which we do not recommend for production workloads) and need to perform a node add/replace/remove then follow our [placement configuration guide](./placement_configuration) and pay special attention to follow the special instructions for `seed nodes`. +If you're running `M3DB seed nodes` with embedded `etcd` (which we do not recommend for production workloads) and need to perform a node add/replace/remove then follow our [placement configuration guide](/operational_guide/placement_configuration) and pay special attention to follow the special instructions for `seed nodes`. ### External etcd diff --git a/docs/content/operational_guides/kernel_configuration.md b/docs/content/operational_guide/kernel_configuration.md similarity index 100% rename from docs/content/operational_guides/kernel_configuration.md rename to docs/content/operational_guide/kernel_configuration.md diff --git a/docs/content/operational_guides/mapping_rollup.md b/docs/content/operational_guide/mapping_rollup.md similarity index 100% rename from docs/content/operational_guides/mapping_rollup.md rename to docs/content/operational_guide/mapping_rollup.md diff --git a/docs/content/operational_guides/monitoring.md b/docs/content/operational_guide/monitoring.md similarity index 100% rename from docs/content/operational_guides/monitoring.md rename to docs/content/operational_guide/monitoring.md diff --git a/docs/content/operational_guides/namespace_configuration.md b/docs/content/operational_guide/namespace_configuration.md similarity index 88% rename from docs/content/operational_guides/namespace_configuration.md rename to docs/content/operational_guide/namespace_configuration.md index 4d3fcdeb78..c9a33df06d 100644 --- a/docs/content/operational_guides/namespace_configuration.md +++ b/docs/content/operational_guide/namespace_configuration.md @@ -3,7 +3,7 @@ title: "Namespace Configuration" weight: 8 --- -Namespaces in M3DB are analogous to tables in other databases. Each namespace has a unique name as well as distinct configuration with regards to data retention and blocksize. For more information about namespaces and the technical details of their implementation, read our [storage engine documentation](../m3db/architecture/engine). +Namespaces in M3DB are analogous to tables in other databases. Each namespace has a unique name as well as distinct configuration with regards to data retention and blocksize. For more information about namespaces and the technical details of their implementation, read our [storage engine documentation](/m3db/architecture/engine). ## Namespace Operations @@ -93,7 +93,7 @@ in duration format as opposed to nanoseconds (default). ### bootstrapEnabled -This controls whether M3DB will attempt to [bootstrap](bootstrapping_crash_recovery) the namespace on startup. This value should always be set to `true` unless you have a very good reason to change it as setting it to `false` can cause data loss when restarting nodes. +This controls whether M3DB will attempt to [bootstrap](/operational_guide/bootstrapping_crash_recovery) the namespace on startup. This value should always be set to `true` unless you have a very good reason to change it as setting it to `false` can cause data loss when restarting nodes. Can be modified without creating a new namespace: `yes` @@ -111,7 +111,7 @@ Can be modified without creating a new namespace: `yes` ### snapshotEnabled -This controls whether M3DB will periodically write out [snapshot files](../m3db/architecture/commitlogs) for this namespace which act as compacted commitlog files. This value should always be set to `true` unless you have a very good reason to change it as setting it to `false` will increasing bootstrapping times (reading commitlog files is slower than reading snapshot files) and increase disk utilization (snapshot files are compressed but commitlog files are uncompressed). +This controls whether M3DB will periodically write out [snapshot files](/m3db/architecture/commitlogs) for this namespace which act as compacted commitlog files. This value should always be set to `true` unless you have a very good reason to change it as setting it to `false` will increasing bootstrapping times (reading commitlog files is slower than reading snapshot files) and increase disk utilization (snapshot files are compressed but commitlog files are uncompressed). Can be modified without creating a new namespace: `yes` @@ -129,7 +129,7 @@ Can be modified without creating a new namespace: `yes` #### blockSize -This is the most important value to consider when tuning the performance of an M3DB namespace. Read the [storage engine documentation](../m3db/architecture/storage) for more details, but the basic idea is that larger blockSizes will use more memory, but achieve higher compression. Similarly, smaller blockSizes will use less memory, but have worse compression. In testing, good compression occurs with blocksizes containing around 720 samples per timeseries. +This is the most important value to consider when tuning the performance of an M3DB namespace. Read the [storage engine documentation](/m3db/architecture/storage) for more details, but the basic idea is that larger blockSizes will use more memory, but achieve higher compression. Similarly, smaller blockSizes will use less memory, but have worse compression. In testing, good compression occurs with blocksizes containing around 720 samples per timeseries. Can be modified without creating a new namespace: `no` diff --git a/docs/content/operational_guides/namespace_mgmt.md b/docs/content/operational_guide/namespace_mgmt.md similarity index 100% rename from docs/content/operational_guides/namespace_mgmt.md rename to docs/content/operational_guide/namespace_mgmt.md diff --git a/docs/content/operational_guides/placement.md b/docs/content/operational_guide/placement.md similarity index 100% rename from docs/content/operational_guides/placement.md rename to docs/content/operational_guide/placement.md diff --git a/docs/content/operational_guides/placement_configuration.md b/docs/content/operational_guide/placement_configuration.md similarity index 99% rename from docs/content/operational_guides/placement_configuration.md rename to docs/content/operational_guide/placement_configuration.md index 56aa02b562..650320d998 100644 --- a/docs/content/operational_guides/placement_configuration.md +++ b/docs/content/operational_guide/placement_configuration.md @@ -5,7 +5,7 @@ weight: 7 M3DB was designed from the ground up to be a distributed (clustered) database that is availability zone or rack aware (by using isolation groups). Clusters will seamlessly scale with your data, and you can start with a small number of nodes and grow it to a size of several hundred nodes with no downtime or expensive migrations. -Before reading the rest of this document, we recommend familiarizing yourself with the [M3DB placement documentation](placement) +Before reading the rest of this document, we recommend familiarizing yourself with the [M3DB placement documentation](/operational_guide/placement) **Note**: The primary limiting factor for the maximum size of an M3DB cluster is the number of shards. Picking an appropriate number of shards is more of an art than a science, but our recommendation is as follows: diff --git a/docs/content/operational_guides/repairs.md b/docs/content/operational_guide/repairs.md similarity index 100% rename from docs/content/operational_guides/repairs.md rename to docs/content/operational_guide/repairs.md diff --git a/docs/content/operational_guides/replication_and_deployment_in_zones.md b/docs/content/operational_guide/replication_and_deployment_in_zones.md similarity index 96% rename from docs/content/operational_guides/replication_and_deployment_in_zones.md rename to docs/content/operational_guide/replication_and_deployment_in_zones.md index d2293b0c90..242f8e1272 100644 --- a/docs/content/operational_guides/replication_and_deployment_in_zones.md +++ b/docs/content/operational_guide/replication_and_deployment_in_zones.md @@ -5,11 +5,11 @@ weight: 1 M3DB supports both deploying across multiple zones in a region or deploying to a single zone with rack-level isolation. It can also be deployed across multiple regions for a global view of data, though both latency and bandwidth costs may increase as a result. -In addition, M3DB has support for automatically replicating data between isolated M3DB clusters (potentially running in different zones / regions). More details can be found in the [Replication between clusters](./replication_between_clusters) operational guide. +In addition, M3DB has support for automatically replicating data between isolated M3DB clusters (potentially running in different zones / regions). More details can be found in the [Replication between clusters](/operational_guide/replication_between_clusters) operational guide. ## Replication -A replication factor of at least 3 is highly recommended for any M3DB deployment, due to the consistency levels (for both reads and writes) that require quorum in order to complete an operation. For more information on consistency levels, see the documentation concerning [tuning availability, consistency and durability](availability_consistency_durability). +A replication factor of at least 3 is highly recommended for any M3DB deployment, due to the consistency levels (for both reads and writes) that require quorum in order to complete an operation. For more information on consistency levels, see the documentation concerning [tuning availability, consistency and durability](/operational_guide/availability_consistency_durability). M3DB will do its best to distribute shards evenly among the availability zones while still taking each individual node's weight into account, but if some of the availability zones have less available hosts than others then each host in that zone will be responsible for more shards than hosts in the other zones and will thus be subjected to heavier load. @@ -93,4 +93,4 @@ Typically, deployments have many more than four shards - this is a simple exampl [failure-tolerance]: https://github.com/etcd-io/etcd/blob/cca0d5c1bed134ac30e1354241f7655d2a118db4/Documentation/faq.md#what-is-failure-tolerance -[isolation-group]: ./placement_configuration.md#isolation-group +[isolation-group]: /operational_guide/placement_configuration.md#isolation-group diff --git a/docs/content/operational_guides/replication_between_clusters.md b/docs/content/operational_guide/replication_between_clusters.md similarity index 78% rename from docs/content/operational_guides/replication_between_clusters.md rename to docs/content/operational_guide/replication_between_clusters.md index d436788ce1..dfbe9f59fc 100644 --- a/docs/content/operational_guides/replication_between_clusters.md +++ b/docs/content/operational_guide/replication_between_clusters.md @@ -5,7 +5,7 @@ weight: 15 M3DB clusters can be configured to passively replicate data from other clusters. This feature is most commonly used when operators wish to run two (or more) regional clusters that function independently while passively replicating data from the other cluster in an eventually consistent manner. -The cross-cluster replication feature is built on-top of the [background repairs](repairs) feature. As a result, it has all the same caveats and limitations. Specifically, it does not currently work with clusters that use M3DB's indexing feature and the replication delay between two clusters will be at least (`block size` + `bufferPast`) for data written at the beginning of a block for a given namespace. For use-cases where a large replication delay is unacceptable, the current recommendation is to dual-write to both clusters in parallel and then rely upon the cross-cluster replication feature to repair any discrepancies between the clusters caused by failed dual-writes. This recommendation is likely to change in the future once support for low-latency replication is added to M3DB in the form of commitlog tailing. +The cross-cluster replication feature is built on-top of the [background repairs](/operational_guide/repairs) feature. As a result, it has all the same caveats and limitations. Specifically, it does not currently work with clusters that use M3DB's indexing feature and the replication delay between two clusters will be at least (`block size` + `bufferPast`) for data written at the beginning of a block for a given namespace. For use-cases where a large replication delay is unacceptable, the current recommendation is to dual-write to both clusters in parallel and then rely upon the cross-cluster replication feature to repair any discrepancies between the clusters caused by failed dual-writes. This recommendation is likely to change in the future once support for low-latency replication is added to M3DB in the form of commitlog tailing. While cross-cluster replication is built on top of the background repairs feature, background repairs do not need to be enabled for cross-cluster replication to be enabled. In other words, clusters can be configured such that: diff --git a/docs/content/operational_guides/replication_global.png b/docs/content/operational_guide/replication_global.png similarity index 100% rename from docs/content/operational_guides/replication_global.png rename to docs/content/operational_guide/replication_global.png diff --git a/docs/content/operational_guides/replication_region.png b/docs/content/operational_guide/replication_region.png similarity index 100% rename from docs/content/operational_guides/replication_region.png rename to docs/content/operational_guide/replication_region.png diff --git a/docs/content/operational_guides/replication_single_zone.png b/docs/content/operational_guide/replication_single_zone.png similarity index 100% rename from docs/content/operational_guides/replication_single_zone.png rename to docs/content/operational_guide/replication_single_zone.png diff --git a/docs/content/operational_guides/resource_limits.md b/docs/content/operational_guide/resource_limits.md similarity index 100% rename from docs/content/operational_guides/resource_limits.md rename to docs/content/operational_guide/resource_limits.md diff --git a/docs/content/operational_guides/upgrading_m3.md b/docs/content/operational_guide/upgrading_m3.md similarity index 100% rename from docs/content/operational_guides/upgrading_m3.md rename to docs/content/operational_guide/upgrading_m3.md diff --git a/docs/content/overview/components.md b/docs/content/overview/components.md index 1afbb85658..7a057df3c0 100644 --- a/docs/content/overview/components.md +++ b/docs/content/overview/components.md @@ -9,11 +9,11 @@ M3 Coordinator is a service that coordinates reads and writes between upstream s ## M3DB -M3DB is a distributed time series database that provides scalable storage and a reverse index of time series. It is optimized as a cost effective and reliable realtime and long term retention metrics store and index. For more details, see the [M3DB documentation](../m3db/). +M3DB is a distributed time series database that provides scalable storage and a reverse index of time series. It is optimized as a cost effective and reliable realtime and long term retention metrics store and index. For more details, see the [M3DB documentation](/m3db/). ## M3 Query -M3 Query is a service that houses a distributed query engine for querying both realtime and historical metrics, supporting several different query languages. It is designed to support both low latency realtime queries and queries that can take longer to execute, aggregating over much larger datasets, for analytical use cases. For more details, see the [query engine documentation](../query_engine/). +M3 Query is a service that houses a distributed query engine for querying both realtime and historical metrics, supporting several different query languages. It is designed to support both low latency realtime queries and queries that can take longer to execute, aggregating over much larger datasets, for analytical use cases. For more details, see the [query engine documentation](/query_engine/). ## M3 Aggregator diff --git a/docs/content/troubleshooting/_index.md b/docs/content/troubleshooting/_index.md index ba65694774..9981a2ac8e 100644 --- a/docs/content/troubleshooting/_index.md +++ b/docs/content/troubleshooting/_index.md @@ -12,20 +12,20 @@ These ports will not open until a namespace and placement have been created and ## Bootstrapping is slow -Double check your configuration against the [bootstrapping guide](../operational_guide/bootstrapping_crash_recovery). The nodes will log what bootstrapper they are using and what time range they are using it for. +Double check your configuration against the [bootstrapping guide](/operational_guide/bootstrapping_crash_recovery). The nodes will log what bootstrapper they are using and what time range they are using it for. If you're using the commitlog bootstrapper, and it seems to be slow, ensure that snapshotting is enabled for your namespace. Enabling snapshotting will require a node restart to take effect. -If an m3db node hasn't been able to snapshot for awhile, or is stuck in the commitlog bootstrapping phase for a long time due to accumulating a large number of commitlogs, consider using the peers bootstrapper. In situations where a large number of commitlogs need to be read, the peers bootstrapper will outperform the commitlog bootstrapper (faster and less memory usage) due to the fact that it will receive already-compressed data from its peers. Keep in mind that this will only work with a replication factor of 3 or larger and if the nodes peers are healthy and bootstrapped. Review the [bootstrapping guide](../operational_guide/bootstrapping_crash_recovery) for more information. +If an m3db node hasn't been able to snapshot for awhile, or is stuck in the commitlog bootstrapping phase for a long time due to accumulating a large number of commitlogs, consider using the peers bootstrapper. In situations where a large number of commitlogs need to be read, the peers bootstrapper will outperform the commitlog bootstrapper (faster and less memory usage) due to the fact that it will receive already-compressed data from its peers. Keep in mind that this will only work with a replication factor of 3 or larger and if the nodes peers are healthy and bootstrapped. Review the [bootstrapping guide](/operational_guide/bootstrapping_crash_recovery) for more information. ## Nodes a crashing with memory allocation errors, but there's plenty of available memory -Ensure you've set `vm.max_map_count` to something like 262,144 using sysctl. Find out more in the [Clustering the Hard Way](../how_to/cluster_hard_way.md#kernel) document. +Ensure you've set `vm.max_map_count` to something like 262,144 using sysctl. Find out more in the [Clustering the Hard Way](/how_to/cluster_hard_way.md#kernel) document. ## What to do if my M3DB node is OOM’ing? 1. Ensure that you are not co-locating coordinator, etcd or query nodes with your M3DB nodes. Colocation or embedded mode is fine for a development environment, but highly discouraged in production. -2. Check to make sure you are running adequate block sizes based on the retention of your namespace. See [namespace configuration](../operational_guide/namespace_configuration) for more information. +2. Check to make sure you are running adequate block sizes based on the retention of your namespace. See [namespace configuration](/operational_guide/namespace_configuration) for more information. 3. Ensure that you use at most 50-60% memory utilization in the normal running state. You want to ensure enough overhead to handle bursts of metrics, especially ones with new IDs as those will take more memory initially. 4. High cardinality metrics can also lead to OOMs especially if you are not adequately provisioned. If you have many unique timeseries such as ones containing UUIDs or timestamps as tag values, you should consider mitigating their cardinality.