Squashed commit of the following:

commit 4f3778dded83ad4d81aaeaad62608f6c7a0f9461 Author: ChrisChinchilla <[email protected]> Date: Mon Oct 5 12:53:40 2020 +0200 fix code inclusion Signed-off-by: ChrisChinchilla <[email protected]> commit daefb435243ed98041744c520d52ac4a92421694 Author: ChrisChinchilla <[email protected]> Date: Mon Oct 5 12:53:34 2020 +0200 Add redirects Signed-off-by: ChrisChinchilla <[email protected]> commit 612cfbb319b30177501530f9cd7bdd626ea6b107 Author: ChrisChinchilla <[email protected]> Date: Mon Oct 5 12:09:40 2020 +0200 Remove versioning for now Signed-off-by: ChrisChinchilla <[email protected]> commit cda48e18dce956a4e684dd4d6d7943df808e6846 Author: ChrisChinchilla <[email protected]> Date: Fri Oct 2 18:09:35 2020 +0200 Switch netlify directory Signed-off-by: ChrisChinchilla <[email protected]> commit c5d0b1decc95d042a565cbe97d25c9eda94549a9 Author: ChrisChinchilla <[email protected]> Date: Fri Oct 2 18:00:34 2020 +0200 Move theme to module Signed-off-by: ChrisChinchilla <[email protected]> commit 855d8c7af1bf45978ed180469ecb1b753ec1abdb Author: ChrisChinchilla <[email protected]> Date: Thu Oct 1 14:59:14 2020 +0200 Netlify dev Signed-off-by: ChrisChinchilla <[email protected]> commit a8a8eda0f2f3b8fa1b3aefa2316942ff692dc8e5 Author: ChrisChinchilla <[email protected]> Date: Thu Oct 1 14:03:34 2020 +0200 Update Hugo version Signed-off-by: ChrisChinchilla <[email protected]> commit f19ffadb7caad18663dad99ec5230e8357125954 Author: ChrisChinchilla <[email protected]> Date: Thu Oct 1 13:54:41 2020 +0200 Random file to fix odd git issues Signed-off-by: ChrisChinchilla <[email protected]> commit a6413db7891f2f741510e4b36300e3cb93904f1d Author: ChrisChinchilla <[email protected]> Date: Thu Oct 1 13:11:42 2020 +0200 Update versions Signed-off-by: ChrisChinchilla <[email protected]> commit 9919f004633ace0f88bb9cb721a6373416f28781 Author: ChrisChinchilla <[email protected]> Date: Thu Oct 1 12:45:25 2020 +0200 Convert docs theme to module Signed-off-by: ChrisChinchilla <[email protected]> commit 3a1e013338b6671eb601df3f5cbdd89d770ca171 Author: ChrisChinchilla <[email protected]> Date: Thu Oct 1 12:27:31 2020 +0200 Remove subtree again, not working Signed-off-by: ChrisChinchilla <[email protected]> commit 282ca2d870eb656ba15e20ade3d8aec6493523fe Author: ChrisChinchilla <[email protected]> Date: Thu Oct 1 12:20:11 2020 +0200 Add versions to config Signed-off-by: ChrisChinchilla <[email protected]> commit 73f03d34990b740d5bb1c57df1c5fbb328d0f701 Author: ChrisChinchilla <[email protected]> Date: Wed Sep 30 18:29:35 2020 +0200 Consolidate all commits commit f2ebf5c Author: teddywahle <[email protected]> Date: Mon Sep 21 12:33:37 2020 -0700 [query] Implemented the Graphite `integralByInterval` function (#2596) commit a66fb7d Author: arnikola <[email protected]> Date: Mon Sep 21 14:33:57 2020 -0400 [dbnode] Tile iterators for wide aggregations (#2646) commit 9ea5682 Author: teddywahle <[email protected]> Date: Sun Sep 20 21:50:31 2020 -0700 [query] Implemented the Graphite `divideSeriesLists` function (#2585) commit 35cac59 Author: Rob Skillington <[email protected]> Date: Mon Sep 21 00:21:58 2020 -0400 [coordinator] Update OpenAPI specs for namespace update endpoint (#2629) commit ef83ec4 Author: Rob Skillington <[email protected]> Date: Sun Sep 20 21:57:26 2020 -0400 [changelog] Add changelog for 0.15.15 (#2649) commit 091f833 Author: Rob Skillington <[email protected]> Date: Fri Sep 18 11:30:57 2020 -0400 [coordinator] Allow configuration of tag validation (#2647) commit 3476b4e Author: Gediminas Guoba <[email protected]> Date: Fri Sep 18 17:24:09 2020 +0300 [dbnode] Streaming writer (#2618) * [dbnode] Large tiles writer * minor refactorings * minor refactoring * Skip tagsEncoder, use encodedTags directly * Rename LargeTilesWriter to StreamingWriter * Add FIXME wrt stegment.Tail.Finalize * Address PR feedback Co-authored-by: Linas Medziunas <[email protected]> Co-authored-by: Linas Medžiūnas <[email protected]> commit 88164cf Author: Ryan Hall <[email protected]> Date: Thu Sep 17 16:38:04 2020 -0700 Only read the commit log once during bootstrapping (#2645) * Only read the commit log once during bootstrapping A recent refactoring of cold writes ( #2508) introduced a regression that increases the chances the commit log is read twice while bootstrapping. The referenced PR changed the commitlog bootstrapper to read all requested time ranges, even if a range had been fulfilled by a previous bootstrapper. This was necessary since the commitlog may have cold writes that were never commmited to a fileset. The fileystem bootstrapper would report a time range as fulfilled, but might be missing cold writes only in the commit log. It should be noted this bug was always theoretically possible, but unlikely since the commitlog bootstrapper typically wouldn't run in the first pass (cold time ranges) since the filesystem would fulfill all cold ranges and short circuit the first pass of the boostrapper. This change only reads the commit log on the first pass of the boostrapper and caches the result to skip reading it in subsequent passes. It doesn't actually matter which pass we read the commit log, the first was just chosen arbitrarily. Other attempts at fixing this bug attempted to disable the entire commit log bootstrapper during a pass, but that's not possible since the commit log bootstrapper is actually 2 bootstrappers in one, both the the commit log and snapshot files. To minimize the refactoring changes we still want to only read the snapshot files of the requested ranges. commit 3d2915f Author: nate <[email protected]> Date: Thu Sep 17 13:03:48 2020 -0400 [dbnode] Make caching after block retrieval a configuration option (#2613) commit 0ef7aba Author: nate <[email protected]> Date: Thu Sep 17 12:33:53 2020 -0400 [docs] Add documentation on fileset migrations (#2630) Signed-off-by: ChrisChinchilla <[email protected]>
m3db · Oct 5, 2020 · 00c8613 · 00c8613
1 parent 8a24f8b
commit 00c8613
Show file tree

Hide file tree

Showing 347 changed files with 3,319 additions and 12,053 deletions.
diff --git a/.gitignore b/.gitignore
@@ -39,8 +39,6 @@ coverage/
 # Release directory
 dist/
 
-# Docs directory
-site/
 
 # Node
 node_modules/
@@ -61,4 +59,7 @@ m3db.io/openapi
 .vagrant
 
 #docs ignores
-docs/public
+docs-beta/public
+docs/public
+docs/site
+site
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,21 @@
 # Changelog
 
+# 0.15.15
+
+## Features
+
+- **M3DB**: Add configuration to limit bytes read for historical metrics in a given time window ([#2627](https://github.com/m3db/m3/pull/2627))
+- **M3DB**: Add configuration to toggle block caching ([#2613](https://github.com/m3db/m3/pull/2613))
+- **M3Coordinator**: Add extended configuration of label and tag validation ([#2647](https://github.com/m3db/m3/pull/2647))
+
+## Performance
+
+- **M3DB**: Perform single pass when reading commit log entry and reuse result for second bootstrap phase ([#2645](https://github.com/m3db/m3/pull/2645))
+
+## Documentation
+
+- **M3DB**: Documentation for fileset migrations, the forward and backwards compatibility guarantees and configuring migrations ([#2630](https://github.com/m3db/m3/pull/2630))
+
 # 0.15.14
 
 ## Features

diff --git a/config.toml b/config.toml
@@ -0,0 +1,162 @@
+# Change the default theme to be use when building the site with Hugo
+[module]
+  [[module.imports]]
+    path = "github.com/chronosphereio/docs-theme"
+
+  [[module.mounts]]
+    source = "docs/content"
+    target = "content"
+
+  [[module.mounts]]
+    source = "docs/static"
+    target = "static"
+
+  [[module.mounts]]
+    source = "docs/layouts"
+    target = "layouts"
+
+  [[module.mounts]]
+    source = "docs/data"
+    target = "data"
+
+  [[module.mounts]]
+    source = "docs/assets"
+    target = "assets"
+
+  [[module.mounts]]
+    source = "docs/i18n"
+    target = "i18n"
+
+  [[module.mounts]]
+    source = "docs/archetypes"
+    target = "archetypes"
+
+theme = "docs-theme"
+
+# baseURL = "/"
+languageCode = "en-US"
+defaultContentLanguage = "en"
+# staticDir = ["static"]
+
+title = "M3DB Documentation"
+metaDataFormat = "yaml"
+defaultContentLanguageInSubdir= true
+
+# Highlighting config.
+pygmentsCodeFences = true
+pygmentsUseClasses = false
+# Use the new Chroma Go highlighter in Hugo.
+pygmentsUseClassic = false
+#pygmentsOptions = "linenos=table"
+# See https://help.farbox.com/pygments.html
+# pygmentsStyle = "vs"
+
+enableGitInfo = true
+
+[frontmatter]
+# date = ["date", ":filename", "publishDate", "lastmod"]
+
+# Image processing configuration.
+[imaging]
+resampleFilter = "CatmullRom"
+quality = 75
+anchor = "smart"
+
+[services]
+[services.googleAnalytics]
+# Comment out the next line to disable GA tracking. Also disables the feature described in [params.ui.feedback].
+# id = "UA-00000000-0"
+
+#fullversion = "v1.18.0"
+#version = "v1.18"
+githubbranch = "master"
+docsbranch = "master"
+deprecated = false
+#currentUrl = "https://kubernetes.io/docs/home/"
+#nextUrl = "https://kubernetes-io-vnext-staging.netlify.com/"
+githubWebsiteRepo = "https://github.com/m3db/m3"
+githubWebsiteRaw = "raw.githubusercontent.com/m3db/m3"
+
+# Enable Algolia DocSearch
+# algolia_docsearch = false
+
+# Enable Lunr.js offline search
+offlineSearch = false
+
+[params]
+  # Prefix URL to edit current page. Will display an "Edit this page" button on top right hand corner of every page. 
+  # Useful to give opportunity to people to create merge request for your doc.
+  # See the config.toml file from this documentation site to have an example.
+  editURL = "https://github.com/m3db/m3"
+  # Author of the site, will be used in meta information
+  author = "m3"
+  # Description of the site, will be used in meta information
+  description = "m3documentation"
+  # Shows a checkmark for visited pages on the menu
+  showVisitedLinks = false
+  # Disable search function. It will hide search bar
+  disableSearch = false
+  # Javascript and CSS cache are automatically busted when new version of site is generated. 
+  # Set this to true to disable this behavior (some proxies don't handle well this optimization)
+  disableAssetsBusting = false
+  # Set this to true to disable copy-to-clipboard button for inline code.
+  disableInlineCopyToClipBoard = true
+  # A title for shortcuts in menu is set by default. Set this to true to disable it. 
+  disableShortcutsTitle = false
+  # When using mulitlingual website, disable the switch language button.
+  disableLanguageSwitchingButton = false
+  # Hide breadcrumbs in the header and only show the current page title
+  disableBreadcrumb = true
+  # Hide Next and Previous page buttons normally displayed full height beside content
+  disableNextPrev = true
+  # Order sections in menu by "weight" or "title". Default to "weight"
+  ordersectionsby = "weight"
+  # Change default color scheme with a variant one. Can be "red", "blue", "green".
+  themeVariant = "blue"
+  twitter = "m3db_io"
+
+# TODO: Might need to refactor some of the K8s shortcodes later
+# Add your release versions here
+# TODO: Enable when ready
+# [[params.versions]]
+#   version = "1.0"
+#   url = "https://master.kubeflow.org"
+#   [[params.versions]]
+#   version = "0.9"
+#   url = "https://master.kubeflow.org"
+
+# version_menu = "Versions"
+
+# TODO: Do not like doing this really
+[markup]
+  [markup.goldmark]
+    [markup.goldmark.renderer]
+      unsafe = true
+    [markup.goldmark.parser]
+      attribute = true
+      autoHeadingID = true
+      autoHeadingIDType = "github"
+    [markup.tableOfContents]
+      endLevel = 3
+      ordered = false
+      startLevel = 2
+    [markup.goldmark.extensions]
+      definitionList = true
+      footnote = true
+      linkify = true
+      strikethrough = true
+      table = true
+      taskList = true
+      typographer = true
+
+[[Languages.en.menu.shortcuts]]
+name = "<i class='fab fa-fw fa-github'></i> GitHub"
+identifier = "ds"
+url = "https://github.com/m3db/m3"
+weight = 10
+
+[outputs]
+home = [ "HTML", "RSS", "JSON"]
+page = [ "HTML"]
+section = [ "HTML"]
+chapter = [ "HTML"]
diff --git a/docs/.gitignore b/docs/.gitignore
@@ -0,0 +1,2 @@
+# Local Netlify folder
+.netlify
diff --git a/docs/content/includes/headers_optional_read_all.md b/docs/content/includes/headers_optional_read_all.md
@@ -0,0 +1,11 @@
+* `M3-Restrict-By-Tags-JSON`:  
+ If this header is set it can ensure specific label matching is performed as part
+of every query including series metadata endpoints. As an example, the following 
+header would unconditionally cause `globaltag=somevalue` to be a part of all queries
+issued regardless of if they include the label or not in a query and also strip the
+"globaltag" from appearing as a label in any of the resulting timeseries:
+```
+M3-Restrict-By-Tags-JSON: '{"match":[{"name":"globaltag","type":"EQUAL","value":"somevalue"}],"strip":["globaltag"]}'
+```
+
+{{% fileinclude file="includes/headers_optional_read_limits.md" %}}
diff --git a/docs/content/includes/headers_optional_read_limits.md b/docs/content/includes/headers_optional_read_limits.md
@@ -0,0 +1,6 @@
+* `M3-Limit-Max-Series`:  
+ If this header is set it will override any configured per query time series limit. If the limit is hit, it will either return a partial result or an error based on the require exhaustive configuration set.<br />
+* `M3-Limit-Max-Docs`:  
+ If this header is set it will override any configured per query time series * blocks limit (docs limit). If the limit is hit, it will either return a partial result or an error based on the require exhaustive configuration set.<br />
+* `M3-Limit-Require-Exhaustive`:  
+ If this header is set it will override any configured require exhaustive setting. If "true" it will return an error if query hits a configured limit (such as series or docs limit) instead of a partial result. Otherwise if "false" it will return a partial result of the time series already matched with the response header `M3-Results-Limited` detailing the limit that was hit and a warning included in the response body.
diff --git a/docs/content/includes/headers_optional_read_write.md b/docs/content/includes/headers_optional_read_write.md
@@ -0,0 +1,11 @@
+*   `M3-Metrics-Type`:  
+    If this header is set, it determines what type of metric to store this metric value as. Otherwise by default, metrics will be stored in all namespaces that are configured. You can also disable this default behavior by setting `downsample` options to `all: false` for a namespace in the coordinator config, for more see [disabling automatic aggregation](/how_to/query.md#disabling-automatic-aggregation).
+
+     Must be one of:  
+    `unaggregated`: Write metrics directly to configured unaggregated namespace.  
+    `aggregated`: Write metrics directly to a configured aggregated namespace (bypassing any aggregation), this requires the `M3-Storage-Policy` header to be set to resolve which namespace to write metrics to.  
+
+*   `M3-Storage-Policy`:  
+     If this header is set, it determines which aggregated namespace to read/write metrics directly to/from (bypassing any aggregation).  
+     The value of the header must be in the format of `resolution:retention` in duration shorthand. e.g. `1m:48h` specifices 1 minute resolution and 48 hour retention. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".<br /><br />
+    Here is [an example](https://github.com/m3db/m3/blob/master/scripts/docker-integration-tests/prometheus/test.sh#L126-L146) of querying metrics from a specific namespace. 
diff --git a/docs/content/includes/headers_optional_read_write_all.md b/docs/content/includes/headers_optional_read_write_all.md
@@ -0,0 +1,9 @@
+*   `M3-Metrics-Type`:  
+    If this header is set, it determines what type of metric to store this metric value as. Otherwise by default, metrics will be stored in all namespaces that are configured. You can also disable this default behavior by setting `downsample` options to `all: false` for a namespace in the coordinator config, for more see [disabling automatic aggregation](/how_to/query.md#disabling-automatic-aggregation).
+    Must be one of:  
+    `unaggregated`: Write metrics directly to configured unaggregated namespace.  
+    `aggregated`: Write metrics directly to a configured aggregated namespace (bypassing any aggregation), this requires the `M3-Storage-Policy` header to be set to resolve which namespace to write metrics to.  
+*  `M3-Storage-Policy`:  
+     If this header is set, it determines which aggregated namespace to read/write metrics directly to/from (bypassing any aggregation).  
+     The value of the header must be in the format of `resolution:retention` in duration shorthand. e.g. `1m:48h` specifices 1 minute resolution and 48 hour retention. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
+    Here is [an example](https://github.com/m3db/m3/blob/master/scripts/docker-integration-tests/prometheus/test.sh#L126-L146) of querying metrics from a specific namespace.
diff --git a/docs/content/includes/headers_optional_write_all.md b/docs/content/includes/headers_optional_write_all.md
@@ -0,0 +1,8 @@
+* `M3-Map-Tags-JSON`:  
+ If this header is set it enables dynamically mutating tags in a Prometheus write request. See issue
+[2254](https://github.com/m3db/m3/issues/2254) for further context.
+Currently only `write` is supported. As an example, the following header would unconditionally cause
+`globaltag=somevalue` to be added to all metrics in a write request:
+```
+M3-Map-Tags-JSON: '{"tagMappers":[{"write":{"tag":"globaltag","value":"somevalue"}}]}'
+```
diff --git a/docs/content/includes/headers_placement_namespace.md b/docs/content/includes/headers_placement_namespace.md
@@ -0,0 +1,5 @@
+* `Cluster-Environment-Name`:  
+ This header is used to specify the cluster environment name. If not set, the default `default_env` is used. 
+<br />
+* `Cluster-Zone-Name`:  
+ This header is used to specify the cluster zone name. If not set, the default `embedded` is used. 
diff --git a/docs/content/includes/index.md b/docs/content/includes/index.md
@@ -0,0 +1,3 @@
+---
+headless: true
+---
diff --git a/docs/content/includes/podtemplate.json b/docs/content/includes/podtemplate.json
@@ -0,0 +1,22 @@
+{
+    "apiVersion": "v1",
+    "kind": "PodTemplate",
+    "metadata": {
+      "name": "nginx"
+    },
+    "template": {
+      "metadata": {
+        "labels": {
+          "name": "nginx"
+        },
+        "generateName": "nginx-"
+      },
+      "spec": {
+         "containers": [{
+           "name": "nginx",
+           "image": "dockerfile/nginx",
+           "ports": [{"containerPort": 80}]
+         }]
+      }
+    }
+  }
diff --git a/docs/static/quickstart/create-database.sh → ...nt/includes/quickstart/create-database.sh b/docs/static/quickstart/create-database.sh → ...nt/includes/quickstart/create-database.sh
diff --git a/docs/static/quickstart/write-metrics-1.sh → ...nt/includes/quickstart/write-metrics-1.sh b/docs/static/quickstart/write-metrics-1.sh → ...nt/includes/quickstart/write-metrics-1.sh
diff --git a/docs/static/quickstart/write-metrics-2.sh → ...nt/includes/quickstart/write-metrics-2.sh b/docs/static/quickstart/write-metrics-2.sh → ...nt/includes/quickstart/write-metrics-2.sh
diff --git a/docs/static/quickstart/write-metrics-3.sh → ...nt/includes/quickstart/write-metrics-3.sh b/docs/static/quickstart/write-metrics-3.sh → ...nt/includes/quickstart/write-metrics-3.sh
diff --git a/docs/content/m3coordinator/_index.md b/docs/content/m3coordinator/_index.md
@@ -0,0 +1,14 @@
+---
+title: "M3 Coordinator, API for reading/writing metrics and management"
+menuTitle: "M3 Coordinator"
+weight: 4
+chapter: true
+---
+
+M3 Coordinator is a service that coordinates reads and writes between upstream systems, such as Prometheus, and downstream systems, such as M3DB.
+
+It also provides management APIs to setup and configure different parts of M3.
+
+The coordinator is generally a bridge for read and writing different types of metrics formats and a management layer for M3.
+
+**Note**: M3DB by default includes the M3 Coordinator accessible on port 7201. For production deployments it is recommended to deploy it as a dedicated service to ensure you can scale the write coordination role separately and independently to database nodes as an isolated application separate from the M3DB database role.
diff --git a/docs/content/m3coordinator/api/remote.md b/docs/content/m3coordinator/api/remote.md
@@ -25,9 +25,9 @@ None.
 
 #### Optional
 
-{{% fileinclude file="/includes/headers_optional_read_write_all.md" %}}
+{{% fileinclude file="includes/headers_optional_read_write_all.md" %}}
 
-{{% fileinclude file="/includes/headers_optional_write_all.md" %}}
+{{% fileinclude file="includes/headers_optional_write_all.md" %}}
 
 ### Data Params
 
@@ -97,9 +97,9 @@ None.
 
 #### Optional
 
-{{% fileinclude file="/includes/headers_optional_read_write_all.md" %}}
+{{% fileinclude file="includes/headers_optional_read_write_all.md" %}}
 
-{{% fileinclude file="/includes/headers_optional_read_all.md" %}}
+{{% fileinclude file="includes/headers_optional_read_all.md" %}}
 
 ### Data Params
 

diff --git a/docs/content/m3query/api/query.md b/docs/content/m3query/api/query.md
@@ -37,9 +37,9 @@ Query using PromQL and returns JSON datapoints compatible with the Prometheus Gr
 
 #### Optional
 
-{{% fileinclude file="/includes/headers_optional_read_write_all.md" %}}
+{{% fileinclude file="includes/headers_optional_read_write_all.md" %}}
 
-{{% fileinclude file="/includes/headers_optional_read_all.md" %}}
+{{% fileinclude file="includes/headers_optional_read_all.md" %}}
 
 ### Data Params
 

diff --git a/docs/content/operational_guide/namespace_configuration.md b/docs/content/operational_guide/namespace_configuration.md
@@ -13,7 +13,7 @@ The operations below include sample cURLs, but you can always review the API doc
 
 Additionally, the following headers can be used in the namespace operations: 
 
-{{% fileinclude file="/includes/headers_placement_namespace.md" %}}
+{{% fileinclude file="includes/headers_placement_namespace.md" %}}
 
 ### Adding a Namespace
 

diff --git a/docs/content/operational_guide/placement_configuration.md b/docs/content/operational_guide/placement_configuration.md
@@ -80,7 +80,7 @@ The instructions below all contain sample curl commands, but you can always revi
 
 Additionally, the following headers can be used in the placement operations: 
 
-{{% fileinclude file="/includes/headers_placement_namespace.md" %}}
+{{% fileinclude file="includes/headers_placement_namespace.md" %}}
 
 #### Placement Initialization
 

diff --git a/docs/content/operational_guide/resource_limits.md b/docs/content/operational_guide/resource_limits.md
@@ -131,4 +131,4 @@ limits:
 
 The following headers can also be used to override configured limits on a per request basis (to allow for different limits dependent on caller):
 
-{{% fileinclude file="/includes/headers_optional_read_limits.md" %}}
+{{% fileinclude file="includes/headers_optional_read_limits.md" %}}
Original file line number	Diff line number	Diff line change
Expand Up		@@ -131,4 +131,4 @@ limits:

		The following headers can also be used to override configured limits on a per request basis (to allow for different limits dependent on caller):

		{{% fileinclude file="/includes/headers_optional_read_limits.md" %}}
		{{% fileinclude file="includes/headers_optional_read_limits.md" %}}