Merge pull request #2340 from cockroachdb/bulk_io_small_issues

Documented Bulk IO issues

v2.0/backup.md

@@ -113,15 +113,26 @@ The URL for your backup's destination/locations must use the following format:
 
 `[path to backup]` must be unique for each backup, but the other values depend on where you want to store the backup.
 
+<style>
+	table td:nth-child(2) {
+	    min-width: 130px;
+	}
+</style>
+
 | Backup Location | scheme | host | parameters |
 |-----------------|--------|------|------------|
 | Amazon S3 | `s3` | Bucket name | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` |
 | Azure | `azure` | Container name | `AZURE_ACCOUNT_KEY`, `AZURE_ACCOUNT_NAME` |
 | Google Cloud Storage | `gs` | Bucket name | None––currently only supports instance auth, but we can build non-instance auth at a customer's request |
 | HTTP | `http` | Remote host | N/A |
-| NFS | `nodelocal` | File system location | N/A |
+| NFS [<sup>1</sup>](#considerations) | `nodelocal` | File system location [<sup>2</sup>](#considerations) | N/A |
+| S3-compatible services | `s3` | Bucket name | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, `AWS_ENDPOINT` |
+
+#### Considerations
 
-{{site.data.alerts.callout_info}}Because CockroachDB is a distributed system, you cannot meaningfully store backups "locally" on nodes. The entire backup file must be stored in a single location, so attempts to store backups locally must point to an NFS drive to be useful.{{site.data.alerts.end}}
+- The backup location parameters often contain special characters that need to be URI-encoded. Use Javascript's [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) function or Go language's [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape) function to URI-encode the parameters. Other languages provide similar functions to URI-encode special characters.
+- [<sup>1</sup>](#backup-file-urls) Because CockroachDB is a distributed system, you cannot meaningfully store backups "locally" on nodes. The entire backup file must be stored in a single location, so attempts to store backups locally must point to an NFS drive to be useful.
+- [<sup>2</sup>](#backup-file-urls) The file system backup location on a local node or NFS drive is relative to the path specified by the `--external-io-dir` flag set while [starting the node](start-a-node.html). If the `--external-io-dir` flag is set to `disabled`, then backups to local directories and NFS drives are disabled. By default, the flag is set to the `extern` subdirectory of the first configured [`store`](start-a-node.html#store). To set the `--external-io-dir` flag to the locations you want to use without needing to restart nodes, create symlinks to the desired locations from within the `extern` directory.
 
 ## Examples
 

v2.0/import.md

@@ -169,17 +169,20 @@ URLs for the file you want to import and your temp directory must use the follow
 |----------|--------|------|------------|
 | Amazon S3 | `s3` | Bucket name | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` |
 | Azure | `azure` | Container name | `AZURE_ACCOUNT_KEY`, `AZURE_ACCOUNT_NAME` |
-| Google Cloud [<sup>1</sup>](#notes) | `gs` | Bucket name | None |
-| HTTP [<sup>2</sup>](#notes) | `http` | Remote host | N/A |
-| NFS/Local [<sup>3</sup>](#notes) | `nodelocal` | File system location | N/A |
+| Google Cloud [<sup>1</sup>](#considerations) | `gs` | Bucket name | None |
+| HTTP [<sup>2</sup>](#considerations) | `http` | Remote host | N/A |
+| NFS/Local [<sup>3</sup>](#considerations) | `nodelocal` | File system location | N/A |
+| S3-compatible services | `s3` | Bucket name | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, `AWS_ENDPOINT` |
 
-#### Notes
+#### Considerations
 
-[<sup>1</sup>](#import-file-temp-directory-urls) Only supports instance auth.
+- [<sup>1</sup>](#import-file-temp-directory-urls) Only supports instance auth.
 
-[<sup>2</sup>](#import-file-temp-directory-urls) You can easily create your own HTTP server with [Caddy or nginx](create-a-file-server.html).
+- [<sup>2</sup>](#import-file-temp-directory-urls) You can easily create your own HTTP server with [Caddy or nginx](create-a-file-server.html).
 
-[<sup>3</sup>](#import-file-temp-directory-urls) If using NFS for your temp directory, each node in the cluster must have access to the NFS using the same URL.
+- [<sup>3</sup>](#import-file-temp-directory-urls) The file system backup location on the NFS drive is relative to the path specified by the `--external-io-dir` flag set while [starting the node](start-a-node.html). If the flag is set to `disabled`, then imports from local directories and NFS drives are disabled. 
+
+- The location parameters often contain special characters that need to be URI-encoded. Use Javascript's [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) function or Go language's [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape) function to URI-encode the parameters. Other languages provide similar functions to URI-encode special characters.
 
 ### Import Options
 

v2.0/restore.md

@@ -109,8 +109,13 @@ The URL for your backup's locations must use the following format:
 | Google Cloud Storage | `gs` | Bucket name | None––currently only supports instance auth, but we can build non-instance auth at a customer's request |
 | HTTP | `http` | Remote host | N/A |
 | NFS | `nodelocal` | File system location | N/A |
+| S3-compatible services | `s3` | Bucket name | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, `AWS_ENDPOINT` |
 
-{{site.data.alerts.callout_info}}Backups stored on NFSes work only if each node in the cluster accesses the drive in the same way.{{site.data.alerts.end}}
+#### Considerations
+
+- The backup location parameters often contain special characters that need to be URI-encoded. Use Javascript's [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) function or Go language's [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape) function to URI-encode the parameters. Other languages provide similar functions to URI-encode special characters.
+- Backups stored on NFSes work only if each node in the cluster accesses the drive in the same way.
+- The file system backup location on a local node or NFS drive is relative to the path specified by the `--external-io-dir` flag set while [starting the node](start-a-node.html). If the flag is set to `disabled`, then backups and restores from local directories and NFS drives are disabled. 
 
 ### Restore Option List
 

v2.0/start-a-node.md

@@ -42,6 +42,7 @@ Flag | Description
 `--background` | Set this to start the node in the background. This is better than appending `&` to the command because control is returned to the shell only once the node is ready to accept requests.
 `--cache` | The total size for caches, shared evenly if there are multiple storage devices. This can be a percentage or any bytes-based unit, for example: <br><br>`--cache=25%`<br>`--cache=1000000000 ----> 1000000000 bytes`<br>`--cache=1GB ----> 1000000000 bytes`<br>`--cache=1GiB ----> 1073741824 bytes` <br><br><strong>Note:</strong> If you enter the cache size as a percentage, you might need to escape the `%` sign, for instance, while configuring CockroachDB through systemd service files.<br><br><span class="version-tag">Changed in v1.1: </span>**Default:** `128MiB`<br><br>The default cache size is reasonable for local development clusters. For production deployments, this should be increased to 25% or higher. See [Recommended Production Settings](recommended-production-settings.html#cache-and-sql-memory-size-changed-in-v1-1) for more details.
 `--certs-dir` | The path to the [certificate directory](create-security-certificates.html). The directory must contain valid certificates if running in secure mode.<br><br>**Default:** `${HOME}/.cockroach-certs/`
+`--external-io-dir` | <span class="version-tag">New in v2.0:</span> The path of the external IO directory with which the local file access paths are prefixed while performing backup and restore operations using local node directories or NFS drives. If set to `disabled`, backups and restores using local node directories and NFS drives are disabled.<br><br>**Default:** `extern` subdirectory of the first configured [`store`](#store).<br><br>To set the `--external-io-dir` flag to the locations you want to use without needing to restart nodes, create symlinks to the desired locations from within the `extern` directory. 
 `--host` | The hostname or IP address to listen on for intra-cluster and client communication. The node will also advertise itself to other nodes using this address only if `--advertise-host` is not specified; in this case, if it is a hostname, it must be resolvable from all nodes, and if it is an IP address, it must be routable from all nodes.<br><br>**Default:** Listen on all interfaces, but this flag can be set to listen on an external address
 `--http-host` | The hostname or IP address to listen on for Admin UI HTTP requests. <br><br>**Default:** same as `--host`
 `--http-port` | The port to bind to for Admin UI HTTP requests. <br><br>**Default:** `8080`