Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update rolling upgrade docs with node drain, etc. #7542

Merged
merged 10 commits into from
Aug 6, 2020
8 changes: 5 additions & 3 deletions _includes/v20.1/prod-deployment/node-shutdown.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
- If the node was started with a process manager like [systemd](https://www.freedesktop.org/wiki/Software/systemd/), stop the node using the process manager. The process manager should be configured to send `SIGTERM` and then, after about 1 minute, `SIGKILL`.
- If the node was started using [`cockroach start`](cockroach-start.html) and is running in the foreground, press `ctrl-c` in the terminal.
- If the node was started using [`cockroach start`](cockroach-start.html) and the `--background` and `--pid-file` flags, run `kill <pid>`, where `<pid>` is the process ID of the node.
<ul>
<li>If the node was started with a process manager, stop the node by sending <code>SIGTERM</code> with the process manager. If the node is not shutting down after 1 minute, send <code>SIGKILL</code>. When using <code><a href="https://www.freedesktop.org/wiki/Software/systemd/" target="_blank">systemd</a></code>, for example, set <code>TimeoutStopSecs=60</code> in your configuration template and run <code>systemctl stop &lt;systemd config filename&gt;</code> to stop the node without <code>systemd</code> restarting it.</li>
<li>If the node was started using <a href="cockroach-start.html"><code>cockroach start</code></a> and is running in the foreground, press <code>ctrl-c</code> in the terminal.</li>
<li>If the node was started using <a href="cockroach-start.html"><code>cockroach start</code></a> and the <code>--background</code> and <code>--pid-file</code> flags, run <code>kill &lt;pid&gt;</code>, where <code>&lt;pid&gt;</code> is the process ID of the node.</li>
</ul>
8 changes: 5 additions & 3 deletions _includes/v20.2/prod-deployment/node-shutdown.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
- If the node was started with a process manager like [systemd](https://www.freedesktop.org/wiki/Software/systemd/), stop the node using the process manager. The process manager should be configured to send `SIGTERM` and then, after about 1 minute, `SIGKILL`.
- If the node was started using [`cockroach start`](cockroach-start.html) and is running in the foreground, press `ctrl-c` in the terminal.
- If the node was started using [`cockroach start`](cockroach-start.html) and the `--background` and `--pid-file` flags, run `kill <pid>`, where `<pid>` is the process ID of the node.
<ul>
<li>If the node was started with a process manager, stop the node by sending <code>SIGTERM</code> with the process manager. If the node is not shutting down after 1 minute, send <code>SIGKILL</code>. When using <code><a href="https://www.freedesktop.org/wiki/Software/systemd/" target="_blank">systemd</a></code>, for example, set <code>TimeoutStopSecs=60</code> in your configuration template and run <code>systemctl stop &lt;systemd config filename&gt;</code> to stop the node without <code>systemd</code> restarting it.</li>
<li>If the node was started using <a href="cockroach-start.html"><code>cockroach start</code></a> and is running in the foreground, press <code>ctrl-c</code> in the terminal.</li>
<li>If the node was started using <a href="cockroach-start.html"><code>cockroach start</code></a> and the <code>--background</code> and <code>--pid-file</code> flags, run <code>kill &lt;pid&gt;</code>, where <code>&lt;pid&gt;</code> is the process ID of the node.</li>
</ul>
2 changes: 1 addition & 1 deletion v20.1/cockroach-quit.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ key: stop-a-node.html
---

{{site.data.alerts.callout_danger}}
`cockroach quit` is no longer recommended, and will be deprecated in v20.2. To stop a node, it's best to first run [`cockroach node drain`](cockroach-node.html) and then do one of the following:
`cockroach quit` is no longer recommended, and will be deprecated in v20.2. To stop a node, it's best to first run [`cockroach node drain`](cockroach-node.html), wait for the node to be completely drained, and then do one of the following:

{% include {{ page.version.version }}/prod-deployment/node-shutdown.md %}
{{site.data.alerts.end}}
Expand Down
34 changes: 17 additions & 17 deletions v20.1/upgrade-cockroach-version.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,25 +121,25 @@ Note that this behavior is specific to upgrades from v19.2 to v20.1; it does not
We recommend creating scripts to perform these steps instead of performing them manually. Also, if you are running CockroachDB on Kubernetes, see our documentation on [single-cluster](orchestrate-cockroachdb-with-kubernetes.html#upgrade-the-cluster) and/or [multi-cluster](orchestrate-cockroachdb-with-kubernetes-multi-cluster.html#upgrade-the-cluster) orchestrated deployments for upgrade guidance instead.
{{site.data.alerts.end}}

1. Connect to the node.

2. Stop the `cockroach` process.

Without a process manager like `systemd`, use this command:
1. Drain the node of SQL clients and [distributed SQL](architecture/sql-layer.html#distsql) queries:

{% include copy-clipboard.html %}
~~~ shell
$ pkill cockroach
cockroach node drain --certs-dir=certs --host=<address of node to drain>
~~~

If you are using `systemd` as the process manager, use this command to stop a node without `systemd` restarting it:
Once the node has been drained, you'll see a confirmation:

{% include copy-clipboard.html %}
~~~ shell
$ systemctl stop <systemd config filename>
~~~
node is draining... remaining: 0 (complete)
ok
~~~

1. After the node is completely drained, stop the node:

{% include {{ page.version.version }}/prod-deployment/node-shutdown.md %}

Then verify that the process has stopped:
Verify that the process has stopped:

{% include copy-clipboard.html %}
~~~ shell
Expand All @@ -148,7 +148,7 @@ We recommend creating scripts to perform these steps instead of performing them

Alternately, you can check the node's logs for the message `server drained and shutdown completed`.

3. Download and install the CockroachDB binary you want to use:
1. Download and install the CockroachDB binary you want to use:

<div class="filters clearfix">
<button style="width: 15%" class="filter-button" data-scope="mac">Mac</button>
Expand Down Expand Up @@ -180,7 +180,7 @@ We recommend creating scripts to perform these steps instead of performing them
~~~
</div>

4. If you use `cockroach` in your `$PATH`, rename the outdated `cockroach` binary, and then move the new one into its place:
1. If you use `cockroach` in your `$PATH`, rename the outdated `cockroach` binary, and then move the new one into its place:

<div class="filters clearfix">
<button style="width: 15%" class="filter-button" data-scope="mac">Mac</button>
Expand Down Expand Up @@ -212,7 +212,7 @@ We recommend creating scripts to perform these steps instead of performing them
~~~
</div>

5. Start the node to have it rejoin the cluster.
1. Start the node to have it rejoin the cluster.

Without a process manager like `systemd`, re-run the [`cockroach start`](cockroach-start.html) command that you used to start the node initially, for example:

Expand All @@ -231,13 +231,13 @@ We recommend creating scripts to perform these steps instead of performing them
$ systemctl start <systemd config filename>
~~~

6. Verify the node has rejoined the cluster through its output to `stdout` or through the [Admin UI](admin-ui-overview.html).
1. Verify the node has rejoined the cluster through its output to `stdout` or through the [Admin UI](admin-ui-overview.html).

{{site.data.alerts.callout_info}}
To access the Admin UI for a secure cluster, [create a user with a password](create-user.html#create-a-user-with-a-password). Then open a browser and go to `https://<any node's external IP address>:8080`. On accessing the Admin UI, you will see a Login screen, where you will need to enter your username and password.
{{site.data.alerts.end}}

7. If you use `cockroach` in your `$PATH`, you can remove the old binary:
1. If you use `cockroach` in your `$PATH`, you can remove the old binary:

{% include copy-clipboard.html %}
~~~ shell
Expand All @@ -246,7 +246,7 @@ We recommend creating scripts to perform these steps instead of performing them

If you leave versioned binaries on your servers, you do not need to do anything.

8. Wait at least one minute after the node has rejoined the cluster, and then repeat these steps for the next node.
1. Wait at least one minute after the node has rejoined the cluster, and then repeat these steps for the next node.

## Step 5. Finish the upgrade

Expand Down
2 changes: 1 addition & 1 deletion v20.2/cockroach-quit.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ key: stop-a-node.html
---

{{site.data.alerts.callout_danger}}
`cockroach quit` is deprecated. To stop a node, it's best to first run [`cockroach node drain`](cockroach-node.html) and then do one of the following:
`cockroach quit` is deprecated. To stop a node, it's best to first run [`cockroach node drain`](cockroach-node.html), wait for the node to be completely drained, and then do one of the following:

{% include {{ page.version.version }}/prod-deployment/node-shutdown.md %}
{{site.data.alerts.end}}
Expand Down
34 changes: 17 additions & 17 deletions v20.2/upgrade-cockroach-version.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,25 +81,25 @@ For each node in your cluster, complete the following steps. Be sure to upgrade
We recommend creating scripts to perform these steps instead of performing them manually. Also, if you are running CockroachDB on Kubernetes, see our documentation on [single-cluster](orchestrate-cockroachdb-with-kubernetes.html#upgrade-the-cluster) and/or [multi-cluster](orchestrate-cockroachdb-with-kubernetes-multi-cluster.html#upgrade-the-cluster) orchestrated deployments for upgrade guidance instead.
{{site.data.alerts.end}}

1. Connect to the node.

2. Stop the `cockroach` process.

Without a process manager like `systemd`, use this command:
1. Drain the node of SQL clients and [distributed SQL](architecture/sql-layer.html#distsql) queries:

{% include copy-clipboard.html %}
~~~ shell
$ pkill cockroach
cockroach node drain --certs-dir=certs --host=<address of node to drain>
~~~

If you are using `systemd` as the process manager, use this command to stop a node without `systemd` restarting it:
Once the node has been drained, you'll see a confirmation:

{% include copy-clipboard.html %}
~~~ shell
$ systemctl stop <systemd config filename>
~~~
node is draining... remaining: 0 (complete)
ok
~~~

1. After the node is drained, stop the node:

{% include {{ page.version.version }}/prod-deployment/node-shutdown.md %}

Then verify that the process has stopped:
Verify that the process has stopped:

{% include copy-clipboard.html %}
~~~ shell
Expand All @@ -108,7 +108,7 @@ We recommend creating scripts to perform these steps instead of performing them

Alternately, you can check the node's logs for the message `server drained and shutdown completed`.

3. Download and install the CockroachDB binary you want to use:
1. Download and install the CockroachDB binary you want to use:

<div class="filters clearfix">
<button style="width: 15%" class="filter-button" data-scope="mac">Mac</button>
Expand Down Expand Up @@ -140,7 +140,7 @@ We recommend creating scripts to perform these steps instead of performing them
~~~
</div>

4. If you use `cockroach` in your `$PATH`, rename the outdated `cockroach` binary, and then move the new one into its place:
1. If you use `cockroach` in your `$PATH`, rename the outdated `cockroach` binary, and then move the new one into its place:

<div class="filters clearfix">
<button style="width: 15%" class="filter-button" data-scope="mac">Mac</button>
Expand Down Expand Up @@ -172,7 +172,7 @@ We recommend creating scripts to perform these steps instead of performing them
~~~
</div>

5. Start the node to have it rejoin the cluster.
1. Start the node to have it rejoin the cluster.

Without a process manager like `systemd`, re-run the [`cockroach start`](cockroach-start.html) command that you used to start the node initially, for example:

Expand All @@ -191,13 +191,13 @@ We recommend creating scripts to perform these steps instead of performing them
$ systemctl start <systemd config filename>
~~~

6. Verify the node has rejoined the cluster through its output to `stdout` or through the [Admin UI](admin-ui-overview.html).
1. Verify the node has rejoined the cluster through its output to `stdout` or through the [Admin UI](admin-ui-overview.html).

{{site.data.alerts.callout_info}}
To access the Admin UI for a secure cluster, [create a user with a password](create-user.html#create-a-user-with-a-password). Then open a browser and go to `https://<any node's external IP address>:8080`. On accessing the Admin UI, you will see a Login screen, where you will need to enter your username and password.
{{site.data.alerts.end}}

7. If you use `cockroach` in your `$PATH`, you can remove the old binary:
1. If you use `cockroach` in your `$PATH`, you can remove the old binary:

{% include copy-clipboard.html %}
~~~ shell
Expand All @@ -206,7 +206,7 @@ We recommend creating scripts to perform these steps instead of performing them

If you leave versioned binaries on your servers, you do not need to do anything.

8. Wait at least one minute after the node has rejoined the cluster, and then repeat these steps for the next node.
1. Wait at least one minute after the node has rejoined the cluster, and then repeat these steps for the next node.

## Step 5. Finish the upgrade

Expand Down