Merge remote-tracking branch 'upstream/master' into stack-overflow-fi…

…x-master

.golangci.yml

@@ -231,6 +231,8 @@ linters:
     - dogsled
     # There are functions with complexity required.
     - gocyclo
+    # Doesn't buy us much being super opinionated on how to name test functions.
+    - thelper
   disable-all: false
   presets:
     # bodyclose, errcheck, gosec, govet, scopelint, staticcheck, typecheck

CONTRIBUTING.md

@@ -17,3 +17,7 @@ For code changes that affect M3 functionality, or fix bugs, read the [developer
 ## Reporting a Bug
 
 If you have found a bug in M3, [create an issue](https://github.com/m3db/m3/issues/new) and fill in the details asked for in the issue template as much as possible.
+
+## Documentation
+
+For additions and changes to M3 documentation, read the [documentation guide](DOCUMENTATION.md).

DOCUMENTATION.md

@@ -0,0 +1,101 @@
+# Documentation Guide
+
+M3 is a large and complex project, and any help you can offer to explain it better is most welcome. If you have a suggestion for the documentation M3 welcomes it, and documentation pull requests follow the same process as [code contributions](CONTRIBUTING.MD).
+
+The rest of this document explains how to setup the documentation locally and the structure of the content.
+
+## Setup Documentation Locally
+
+### Prerequisites
+
+The M3 documentation uses [Hugo](https://gohugo.io/), a static site generator written in Go. [Find installation instructions in the Hugo documentation](https://gohugo.io/getting-started/installing/).
+
+If you also want to run and serve the M3 documentation, you need the custom theme, [Victor](https://github.com/chronosphereio/victor). Clone or download it into the same directory as the M3 codebase.
+
+To run some tests and test the production build steps (this doesn't run M3 itself), you need [Docker](https://www.docker.com).
+
+## Folder Structure
+
+Hugo hosts the M3 documentation, website, and Open API spec.
+
+### Website
+
+The website is the HTML, CSS, and JavaScript files in the _site/static_ folder. Edit those files to make any changes.
+
+### Open API Docs
+
+Hugo hosts the Open API spec from the _site/static/openapi_ folder. To make changes to the spec, edit the _spec.yml_ file in _src/query/generated/assets_, and a CI job generates the file that Hugo hosts automatically.
+
+### Configuration
+
+The _site/config_ folder has configuration for Hugo, split into three folders:
+
+-   A default  _config.toml_ file in _site/config/\_default_
+-   Overridden configuration for development and production environments in _site/config/development_ and _site/config/production_ respectively.
+
+### Theme Overrides
+
+The _site/layouts_ folder adds a several changes and overridden files to the Victor theme used by the M3 documentation.
+
+### Documentation
+
+The _site/content_ folder contains the documentation files, organized by folders that match the paths for URLs. The _includes_ folder is a special folder not served as part of the documentation and files used by other files.
+
+#### Theme
+
+The M3 documentation uses its own (open source) theme, Victor. You can read all the features that theme provides in [the repository for Victor](https://github.com/chronosphereio/victor).
+
+Victor is a theme based on Hugo modules, [read more in the Hugo docs](https://gohugo.io/hugo-modules/use-modules/) about how to use and update it.
+
+#### Shortcodes
+
+The M3 documentation adds the following extra shortcodes:
+
+-   `{{% apiendpoint %}}` - Combines the values of `Params.api.localCordinator` + `Params.api.apiEndpoint` as defined in the site configuration to output the base API endpoint for M3 running on a localhost.
+-   `{{% docker-version %}}` - Outputs the value of `Params.releases.docker` as defined in the site configuration to output the consistent current Docker release.
+-   `{{% now %}}` - Outputs the current Unix timestamp to allow for up to date timestamps in code examples.
+
+## Running Documentation
+
+As noted in the prerequisites section, if you want to run the documentation locally to see how your edits look with `hugo server`, you need to have the Victor theme in the same parent directory as the M3 codebase, as `hugo server` runs in Hugo's "development" mode (and matches _site/config/development/config.toml_).
+
+This does mean that as you make changes to the theme or documentation, it refreshes automatically in the browser preview. Sometimes Hugo doesn't refresh included files, so you may need to restart the server process.
+
+## Testing Documentation
+
+The M3 documentation has a number of tests that need to pass before a pull request is merged to master and deployed. These include:
+
+-   Link checking with htmltest
+
+To run them you need Docker installed and running, and at the top level of the M3 project, run `make clean install-vendor-m3 docs-test`. If the test is successful you see green text, if not, htmltest tells you which links in which pages you need to fix.
+
+## Building Documentation
+
+There are a couple of different ways to build the documentation depending what you want to do.
+
+- If you want to build only the most up-to-date version of the docs, you can use `hugo` to build.
+- If you want to build all versions of the docs using Hugo run in Docker (this is what CI does to test the documentation). From the top level of the project, run `make docs-build`.
+- If you want to build all versions of the docs with a system-installed version of Hugo (this is what Netlify does to build and serve the documentation). From the top level of the project, run `make site-build`.
+
+## Creating a New Documentation Version
+
+M3 releases versions with some slight changes to documentation for each one which users can access from the drop down menu under the right hand navigation.
+
+Archiving a version of the documentation is a slightly complex process as Hugo doesn't natively support versioning.
+
+1.  Add the new version and path to _site/config/production/config.toml_ in the format:
+
+    ```toml
+    [[params.versions]]
+      version = "{versionNumber}"
+      url = "/v{versionNumber}/docs"
+    ```
+2.  Add the `/v{versionNumber}` path before each `{{ .RelPermalink}}` in the two anchor tags in _site/layouts/partials/navbar.html_
+3.  Make sure the Victor theme is at least version 0.2.11
+4.  Create and tag the commits with the name "docs/v{versionNumber}"
+5.  Switch to a new branch that will add the new version to the master branch
+6.  Add the new version to _site/config/production/config.toml_
+7.  Also add the new version to _site/config/production/config.toml_ in every pre-existing docs version tag
+8.  Switch back to the branch from step 5
+9.  Test everything locally with `make site-build`
+10. Push the new tag to the remote repository

Makefile

@@ -82,6 +82,7 @@ TOOLS :=               \
 	read_data_files      \
 	read_index_files     \
 	read_index_segments  \
+	read_commitlog       \
 	query_index_segments \
 	clone_fileset        \
 	dtest                \

README.md

@@ -4,7 +4,7 @@
 
 <p align="center"><img src="site/static/images/logo-square.png" alt="M3 Logo" width="256" height="270"></p>
 
-[Distributed TSDB](https://m3db.io/docs/m3db/) and [Query Engine](https://m3db.io/docs/how_to/query/), [Prometheus Sidecar](https://m3db.io/docs/integrations/prometheus/), [Metrics Aggregator](https://m3db.io/docs/overview/components/#m3-aggregator), and more such as [Graphite storage and query engine](https://m3db.io/docs/integrations/graphite/).
+[Distributed TSDB](https://m3db.io/docs/reference/m3db/) and [Query Engine](https://m3db.io/docs/how_to/query/), [Prometheus Sidecar](https://m3db.io/docs/integrations/prometheus/), [Metrics Aggregator](https://m3db.io/docs/overview/reference/#m3-aggregator), and more such as [Graphite storage and query engine](https://m3db.io/docs/integrations/graphite/).
 
 ## Table of Contents
 

go.mod

@@ -104,7 +104,7 @@ require (
 	go.uber.org/atomic v1.6.0
 	go.uber.org/config v1.4.0
 	go.uber.org/goleak v1.1.10
-	go.uber.org/zap v1.13.0
+	go.uber.org/zap v1.16.0
 	golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb
 	golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9
 	golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4
@@ -122,8 +122,8 @@ require (
 	gotest.tools v2.2.0+incompatible
 )
 
-// branch 0.9.3-pool-read-binary-3
-replace github.com/apache/thrift => github.com/m3db/thrift v0.0.0-20190820191926-05b5a2227fe4
+// branch 0.9.3-patch
+replace github.com/apache/thrift => github.com/m3dbx/thrift v0.0.0-20210326170526-6e3eef8b4a26
 
 // NB(nate): upgrading to the latest msgpack is not backwards compatibile as msgpack will no longer attempt to automatically
 // write an integer into the smallest number of bytes it will fit in. We rely on this behavior by having helper methods