Merge pull request #13962 from serathius/contributor-docs

Add contributor docs

.github/ISSUE_TEMPLATE/bug-report.yaml

@@ -6,7 +6,7 @@ body:
 - type: markdown
   attributes:
     value: |
-      Please read https://etcd.io/docs/latest/reporting_bugs/
+      Please read https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md
       If this matter is security related, please disclose it privately via [email protected].
       Please fill the form below and provide as much information as possible.
       Not doing so may result in your bug not being addressed in a timely manner.

Documentation/README.md

@@ -1 +1,4 @@
-The etcd docs are hosted at [etcd.io](https://etcd.io/), and so most files formerly in this directory have moved to the [website](https://github.com/etcd-io/website/) repo.
+This directory includes etcd project internal documentation for new and existing contributors.
+
+For user and developer documentation please go to [etcdd.io](https://etcd.io/),
+which is developed in [website](https://github.com/etcd-io/website/) repo.

Documentation/contributor-guide/branch_management.md

@@ -0,0 +1,27 @@
+# Branch management
+
+## Guide
+
+* New development occurs on the [main branch][main].
+* Main branch should always have a green build!
+* Backwards-compatible bug fixes should target the main branch and subsequently be ported to stable branches.
+* Once the main branch is ready for release, it will be tagged and become the new stable branch.
+
+The etcd team has adopted a *rolling release model* and supports two stable versions of etcd.
+
+### Main branch
+
+The `main` branch is our development branch. All new features land here first.
+
+To try new and experimental features, pull `main` and play with it. Note that `main` may not be stable because new features may introduce bugs.
+
+Before the release of the next stable version, feature PRs will be frozen. A [release manager](../dev-internal/release/#release-management) will be assigned to major/minor version and will lead the etcd community in test, bug-fix and documentation of the release for one to two weeks.
+
+### Stable branches
+
+All branches with prefix `release-` are considered _stable_ branches.
+
+After every minor release ([semver.org](https://semver.org/)), we will have a new stable branch for that release, managed by a [patch release manager](../dev-internal/release/#release-management). We will keep fixing the backwards-compatible bugs for the latest two stable releases. A _patch_ release to each supported release branch, incorporating any bug fixes, will be once every two weeks, given any patches.
+
+[main]: https://github.com/etcd-io/etcd/tree/main
+

Documentation/contributor-guide/local_cluster.md

@@ -0,0 +1,149 @@
+# Set up local cluster
+
+For testing and development deployments, the quickest and easiest way is to configure a local cluster. For a production deployment, refer to the [clustering][clustering] section.
+
+## Local standalone cluster
+
+### Starting a cluster
+
+Run the following to deploy an etcd cluster as a standalone cluster:
+
+```
+$ ./etcd
+...
+```
+
+If the `etcd` binary is not present in the current working directory, it might be located either at `$GOPATH/bin/etcd` or at `/usr/local/bin/etcd`. Run the command appropriately.
+
+The running etcd member listens on `localhost:2379` for client requests.
+
+### Interacting with the cluster
+
+Use `etcdctl` to interact with the running cluster:
+
+1. Store an example key-value pair in the cluster:
+
+    ```
+      $ ./etcdctl put foo bar
+      OK
+    ```
+
+    If OK is printed, storing key-value pair is successful.
+
+2. Retrieve the value of `foo`:
+
+    ```
+    $ ./etcdctl get foo
+    bar
+    ```
+
+    If `bar` is returned, interaction with the etcd cluster is working as expected.
+
+## Local multi-member cluster
+
+### Starting a cluster
+
+A `Procfile` at the base of the etcd git repository is provided to easily configure a local multi-member cluster. To start a multi-member cluster, navigate to the root of the etcd source tree and perform the following:
+
+1. Install `goreman` to control Procfile-based applications:
+
+    ```
+    $ go get github.com/mattn/goreman
+    ```
+
+2. Start a cluster with `goreman` using etcd's stock Procfile:
+
+    ```
+    $ goreman -f Procfile start
+    ```
+
+    The members start running. They listen on `localhost:2379`, `localhost:22379`, and `localhost:32379` respectively for client requests.
+
+### Interacting with the cluster
+
+Use `etcdctl` to interact with the running cluster:
+
+1. Print the list of members:
+
+    ```
+    $ etcdctl --write-out=table --endpoints=localhost:2379 member list
+    ```
+    The list of etcd members are displayed as follows:
+
+    ```
+    +------------------+---------+--------+------------------------+------------------------+
+    |        ID        | STATUS  |  NAME  |       PEER ADDRS       |      CLIENT ADDRS      |
+    +------------------+---------+--------+------------------------+------------------------+
+    | 8211f1d0f64f3269 | started | infra1 | http://127.0.0.1:2380  | http://127.0.0.1:2379  |
+    | 91bc3c398fb3c146 | started | infra2 | http://127.0.0.1:22380 | http://127.0.0.1:22379 |
+    | fd422379fda50e48 | started | infra3 | http://127.0.0.1:32380 | http://127.0.0.1:32379 |
+    +------------------+---------+--------+------------------------+------------------------+
+    ```
+
+2. Store an example key-value pair in the cluster:
+
+    ```
+    $ etcdctl put foo bar
+    OK
+    ```
+
+    If OK is printed, storing key-value pair is successful.
+
+### Testing fault tolerance
+
+To exercise etcd's fault tolerance, kill a member and attempt to retrieve the key.
+
+1. Identify the process name of the member to be stopped.
+
+    The `Procfile` lists the properties of the multi-member cluster. For example, consider the member with the process name, `etcd2`.
+
+2. Stop the member:
+
+    ```
+    # kill etcd2
+    $ goreman run stop etcd2
+    ```
+
+3. Store a key:
+
+    ```
+    $ etcdctl put key hello
+    OK
+    ```
+
+4.  Retrieve the key that is stored in the previous step:
+
+    ```
+    $ etcdctl get key
+    hello
+    ```
+
+5. Retrieve a key from the stopped member:
+
+    ```
+    $ etcdctl --endpoints=localhost:22379 get key
+    ```
+
+    The command should display an error caused by connection failure:
+
+    ```
+    2017/06/18 23:07:35 grpc: Conn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 127.0.0.1:22379: getsockopt: connection refused"; Reconnecting to "localhost:22379"
+    Error:  grpc: timed out trying to connect
+    ```
+6. Restart the stopped member:
+
+    ```
+    $ goreman run restart etcd2
+    ```
+
+7. Get the key from the restarted member:
+
+    ```
+    $ etcdctl --endpoints=localhost:22379 get key
+    hello
+    ```
+
+    Restarting the member re-establish the connection. `etcdctl` will now be able to retrieve the key successfully. To learn more about interacting with etcd, read [interacting with etcd section][interacting].
+
+[clustering]: https://etcd.io/docs/latest/op-guide/clustering/
+[interacting]: https://etcd.io/docs/latest/dev-guide/interacting_v3/

Documentation/contributor-guide/logging.md

@@ -0,0 +1,29 @@
+# Logging Conventions
+
+etcd uses the [capnslog][capnslog] library for logging application output categorized into *levels*. A log message's level is determined according to these conventions:
+
+* Error: Data has been lost, a request has failed for a bad reason, or a required resource has been lost
+  * Examples:
+    * A failure to allocate disk space for WAL
+
+* Warning: (Hopefully) Temporary conditions that may cause errors, but may work fine. A replica disappearing (that may reconnect) is a warning.
+  * Examples:
+    * Failure to send raft message to a remote peer
+    * Failure to receive heartbeat message within the configured election timeout
+
+* Notice: Normal, but important (uncommon) log information.
+  * Examples:
+    * Add a new node into the cluster
+    * Add a new user into auth subsystem
+
+* Info: Normal, working log information, everything is fine, but helpful notices for auditing or common operations.
+  * Examples:
+    * Startup configuration
+    * Start to do snapshot
+
+* Debug: Everything is still fine, but even common operations may be logged, and less helpful but more quantity of notices.
+  * Examples:
+    * Send a normal message to a remote peer
+    * Write a log entry to disk
+
+[capnslog]: https://github.com/coreos/pkg/tree/master/capnslog

Documentation/contributor-guide/modules.md

@@ -0,0 +1,93 @@
+# Golang modules
+
+The etcd project (since version 3.5) is organized into multiple
+[golang modules](https://golang.org/ref/mod) hosted  in a [single repository](https://golang.org/ref/mod#vcs-dir).
+
+![modules graph](modules.svg)
+
+There are following modules:
+
+  - **go.etcd.io/etcd/api/v3** - contains API definitions
+  (like protos & proto-generated libraries) that defines communication protocol
+  between etcd clients and server.
+
+  - **go.etcd.io/etcd/pkg/v3** - collection of utility packages used by etcd
+  without being specific to etcd itself. A package belongs here
+  only if it could possibly be moved out into its own repository in the future.
+  Please avoid adding here code that has a lot of dependencies on its own, as
+  they automatically becoming dependencies of the client library
+  (that we want to keep lightweight).
+
+  - **go.etcd.io/etcd/client/v3** - client library used to contact etcd over
+  the network (grpc). Recommended for all new usage of etcd.
+
+  - **go.etcd.io/etcd/client/v2** - legacy client library used to contact etcd
+  over HTTP protocol. Deprecated. All new usage should depend on /v3 library.
+
+  - **go.etcd.io/etcd/raft/v3** - implementation of distributed consensus
+  protocol. Should have no etcd specific code.
+
+  - **go.etcd.io/etcd/server/v3** - etcd implementation.
+  The code in this package is etcd internal and should not be consumed
+  by external projects. The package layout and API can change within the minor versions.
+
+  - **go.etcd.io/etcd/etcdctl/v3** - a command line tool to access and manage etcd.
+
+  - **go.etcd.io/etcd/tests/v3** - a module that contains all integration tests of etcd.
+    Notice: All unit-tests (fast and not requiring cross-module dependencies)
+    should be kept in the local modules to the code under the test.
+
+  - **go.etcd.io/bbolt** - implementation of persistent b-tree.
+    Hosted in a separate repository: https://github.com/etcd-io/bbolt.
+
+
+### Operations
+
+1. All etcd modules should be released in the same versions, e.g.
+   `go.etcd.io/etcd/client/[email protected]` must depend on `go.etcd.io/etcd/api/[email protected]`.
+
+   The consistent updating of versions can by performed using:
+   ```shell script
+   % DRY_RUN=false TARGET_VERSION="v3.5.10" ./scripts/release_mod.sh update_versions
+   ```
+2. The released modules should be tagged according to https://golang.org/ref/mod#vcs-version rules,
+   i.e. each module should get its own tag.
+   The tagging can be performed using:
+   ```shell script
+   % DRY_RUN=false REMOTE_REPO="origin" ./scripts/release_mod.sh push_mod_tags
+   ```
+
+3. All etcd modules should depend on the same versions of underlying dependencies.
+   This can be verified using:
+   ```shell script
+   % PASSES="dep" ./test.sh
+   ```
+
+4. The go.mod files must not contain dependencies not being used and must
+   conform to `go mod tidy` format.
+   This is being verified by:
+   ```
+   % PASSES="mod_tidy" ./test.sh
+   ```
+
+5. To trigger actions across all modules (e.g. auto-format all files), please
+   use/expand the following script:
+   ```shell script
+   % ./scripts/fix.sh
+   ```
+
+### Future
+
+As a North Star, we would like to evaluate etcd modules towards following model:
+
+![modules graph](modules-future.svg)
+
+This assumes:
+  - Splitting etcdmigrate/etcdadm out of etcdctl binary.
+    Thanks to this etcdctl would become clearly a command-line wrapper
+    around network client API,
+    while etcdmigrate/etcdadm would support direct physical operations on the
+    etcd storage files.
+  - Splitting etcd-proxy out of ./etcd binary, as it contains more experimental code
+    so carries additional risk & dependencies.
+  - Deprecation of support for v2 protocol.

Documentation/contributor-guide/reporting_bugs.md

@@ -0,0 +1,45 @@
+# Reporting bugs
+
+If any part of the etcd project has bugs or documentation mistakes, please let us know by [opening an issue][etcd-issue]. We treat bugs and mistakes very seriously and believe no issue is too small. Before creating a bug report, please check that an issue reporting the same problem does not already exist.
+
+To make the bug report accurate and easy to understand, please try to create bug reports that are:
+
+- Specific. Include as much details as possible: which version, what environment, what configuration, etc. If the bug is related to running the etcd server, please attach the etcd log (the starting log with etcd configuration is especially important).
+
+- Reproducible. Include the steps to reproduce the problem. We understand some issues might be hard to reproduce, please includes the steps that might lead to the problem. If possible, please attach the affected etcd data dir and stack strace to the bug report.
+
+- Isolated. Please try to isolate and reproduce the bug with minimum dependencies. It would significantly slow down the speed to fix a bug if too many dependencies are involved in a bug report. Debugging external systems that rely on etcd is out of scope, but we are happy to provide guidance in the right direction or help with using etcd itself.
+
+- Unique. Do not duplicate existing bug report.
+
+- Scoped. One bug per report. Do not follow up with another bug inside one report.
+
+It may be worthwhile to read [Elika Etemad’s article on filing good bug reports][filing-good-bugs] before creating a bug report.
+
+We might ask for further information to locate a bug. A duplicated bug report will be closed.
+
+## Frequently asked questions
+
+### How to get a stack trace
+
+``` bash
+$ kill -QUIT $PID
+```
+
+### How to get etcd version
+
+``` bash
+$ etcd --version
+```
+
+### How to get etcd configuration and log when it runs as systemd service ‘etcd2.service’
+
+``` bash
+$ sudo systemctl cat etcd2
+$ sudo journalctl -u etcd2
+```
+
+Due to an upstream systemd bug, journald may miss the last few log lines when its processes exit. If journalctl says etcd stopped without fatal or panic message, try `sudo journalctl -f -t etcd2` to get full log.
+
+[etcd-issue]: https://github.com/etcd-io/etcd/issues/new
+[filing-good-bugs]: http://fantasai.inkedblade.net/style/talks/filing-good-bugs/