From 273e0381e5d749aa576ff74ce1f87df7b0e80d27 Mon Sep 17 00:00:00 2001 From: Stef Nestor Date: Wed, 1 Sep 2021 15:45:00 -0600 Subject: [PATCH 1/7] [DOC] Update Persist Keystore via Docker From feedback from ES Devs summarized in [^1], I believe this needs to reflect a directory mount rather than file mount to not error. Also adding in the two common mounting errors, but not sure if this is the right place for them. [^1] https://discuss.elastic.co/t/persist-elasticsearch-kibana-keystores-with-docker/283099 --- docs/reference/setup/install/docker.asciidoc | 32 ++++++++++++++++++-- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/docs/reference/setup/install/docker.asciidoc b/docs/reference/setup/install/docker.asciidoc index 7bfc4120ef956..4d569029c7a74 100644 --- a/docs/reference/setup/install/docker.asciidoc +++ b/docs/reference/setup/install/docker.asciidoc @@ -404,15 +404,41 @@ file is obfuscated but not encrypted. If you want to encrypt your bind-mount it to the container as `/usr/share/elasticsearch/config/elasticsearch.keystore`. In order to provide the Docker container with the password at startup, set the Docker environment -value `KEYSTORE_PASSWORD` to the value of your password. For example, a `docker -run` command might have the following options: +value `KEYSTORE_PASSWORD` to the value of your password. For example, a `docker run` +command might have the following options: [source, sh] -------------------------------------------- --v full_path_to/elasticsearch.keystore:/usr/share/elasticsearch/config/elasticsearch.keystore +-v full_path_to/config:/usr/share/elasticsearch/config -E KEYSTORE_PASSWORD=mypassword -------------------------------------------- +If the keystore is mounted incorrectly, it will induce example Docker errors from attempting to + +- add keystore to running container without full reference + + [source,sh] + -------------------------------------------- + Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy + -------------------------------------------- + +- mount direct file rather than parent directory + + [source,sh] + -------------------------------------------- + Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.io.IOException: Is a directory: SimpleFSIndexInput(path="/usr/share/elasticsearch/config/elasticsearch.keystore") + Likely root cause: java.io.IOException: Is a directory + -------------------------------------------- + +Versus a working example keystore mounting with update example would be + + +[source,sh] +-------------------------------------------- +docker run -it --rm -v /amex/elasticsearch/config:/usr/share/elasticsearch/config dockerproxy.aexp.com/elasticsearch:6.8.3 bin/elasticsearch-keystore create +docker run -it --rm -v /amex/elasticsearch/config:/usr/share/elasticsearch/config dockerproxy.aexp.com/elasticsearch:6.8.3 bin/elasticsearch-keystore add xpack.ssl.key_passphrase +-------------------------------------------- + [[_c_customized_image]] ===== Using custom Docker images In some environments, it might make more sense to prepare a custom image that contains From 5035b081764fb863223f94a43ad6e7271128b22e Mon Sep 17 00:00:00 2001 From: Stef Nestor Date: Mon, 6 Sep 2021 13:13:12 -0600 Subject: [PATCH 2/7] feedback --- docs/reference/setup/install/docker.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/setup/install/docker.asciidoc b/docs/reference/setup/install/docker.asciidoc index 4d569029c7a74..ea17996b46fb8 100644 --- a/docs/reference/setup/install/docker.asciidoc +++ b/docs/reference/setup/install/docker.asciidoc @@ -435,8 +435,8 @@ Versus a working example keystore mounting with update example would be [source,sh] -------------------------------------------- -docker run -it --rm -v /amex/elasticsearch/config:/usr/share/elasticsearch/config dockerproxy.aexp.com/elasticsearch:6.8.3 bin/elasticsearch-keystore create -docker run -it --rm -v /amex/elasticsearch/config:/usr/share/elasticsearch/config dockerproxy.aexp.com/elasticsearch:6.8.3 bin/elasticsearch-keystore add xpack.ssl.key_passphrase +docker run -it --rm -v /Users/me/elasticsearch/config:/usr/share/elasticsearch/config docker.elastic.co/elasticsearch/elasticsearch:7.14.0 bin/elasticsearch-keystore create +docker run -it --rm -v /Users/me/elasticsearch/config:/usr/share/elasticsearch/config docker.elastic.co/elasticsearch/elasticsearch:7.14.0 bin/elasticsearch-keystore add test_keystore_setting -------------------------------------------- [[_c_customized_image]] From d32eefddabfe9e9b0f5b9b049921c3e48646b30c Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 8 Sep 2021 21:15:31 -0400 Subject: [PATCH 3/7] Reorganize --- docs/reference/setup/install/docker.asciidoc | 115 +++++++++++++------ 1 file changed, 79 insertions(+), 36 deletions(-) diff --git a/docs/reference/setup/install/docker.asciidoc b/docs/reference/setup/install/docker.asciidoc index ea17996b46fb8..45e2fa5452dff 100644 --- a/docs/reference/setup/install/docker.asciidoc +++ b/docs/reference/setup/install/docker.asciidoc @@ -395,49 +395,48 @@ uid:gid `1000:0`**. Bind mounted host directories and files must be accessible b and the data and log directories must be writable by this user. [[docker-keystore-bind-mount]] -===== Mounting an {es} keystore +===== Create an encrypted {es} keystore -By default, {es} will auto-generate a keystore file for secure settings. This -file is obfuscated but not encrypted. If you want to encrypt your -<> with a password, you must use the -`elasticsearch-keystore` utility to create a password-protected keystore and -bind-mount it to the container as -`/usr/share/elasticsearch/config/elasticsearch.keystore`. In order to provide -the Docker container with the password at startup, set the Docker environment -value `KEYSTORE_PASSWORD` to the value of your password. For example, a `docker run` -command might have the following options: +By default, {es} will auto-generate a keystore file for <>. This file is obfuscated but not encrypted. -[source, sh] --------------------------------------------- --v full_path_to/config:/usr/share/elasticsearch/config --E KEYSTORE_PASSWORD=mypassword --------------------------------------------- - -If the keystore is mounted incorrectly, it will induce example Docker errors from attempting to +To encrypt your secure settings with a password and have them persist outside +the container, use a `docker run` command to manually create the keystore +instead. The command must: -- add keystore to running container without full reference +* Bind-mount the `config` directory. The command will create an + `elasticsearch.keystore` file in this directory. To avoid errors, do + not directly bind-mount the `elasticsearch.keystore` file's path. +* Use the `elasticsearch-keystore` tool with the `create` option. +* Provide a keystore password using the `KEYSTORE_PASSWORD` or + `KEYSTORE_PASSWORD_FILE` environment variables. Alternatively, you can use + `elasticsearch-keystore` tool's `-p` option to use a password prompt. - [source,sh] - -------------------------------------------- - Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy - -------------------------------------------- - -- mount direct file rather than parent directory - - [source,sh] - -------------------------------------------- - Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.io.IOException: Is a directory: SimpleFSIndexInput(path="/usr/share/elasticsearch/config/elasticsearch.keystore") - Likely root cause: java.io.IOException: Is a directory - -------------------------------------------- +ifeval::["{release-state}"!="unreleased"] +For example: -Versus a working example keystore mounting with update example would be +[source,sh,subs="attributes"] +---- +docker run -it --rm \ +-v full_path_to/config:/usr/share/elasticsearch/config \ +docker.elastic.co/elasticsearch/elasticsearch:{version} \ +bin/elasticsearch-keystore create \ +-E KEYSTORE_PASSWORD=mypassword +---- +You can also use a `docker run` command to add or update secure settings in the +keystore. You'll receive a prompt to enter setting values. -[source,sh] --------------------------------------------- -docker run -it --rm -v /Users/me/elasticsearch/config:/usr/share/elasticsearch/config docker.elastic.co/elasticsearch/elasticsearch:7.14.0 bin/elasticsearch-keystore create -docker run -it --rm -v /Users/me/elasticsearch/config:/usr/share/elasticsearch/config docker.elastic.co/elasticsearch/elasticsearch:7.14.0 bin/elasticsearch-keystore add test_keystore_setting --------------------------------------------- +[source,sh,subs="attributes"] +---- +docker run -it --rm \ +-v full_path_to/config:/usr/share/elasticsearch/config \ +docker.elastic.co/elasticsearch/elasticsearch:{version} \ +bin/elasticsearch-keystore \ +add my.secure.setting \ +my.other.secure.setting \ +---- +endif::[] [[_c_customized_image]] ===== Using custom Docker images @@ -486,4 +485,48 @@ You should use `centos:8` as a base in order to avoid incompatibilities. Use http://man7.org/linux/man-pages/man1/ldd.1.html[`ldd`] to list the shared libraries required by a utility. +[[troubleshoot-docker-errors]] +==== Troubleshoot Docker errors for {es} + +Here’s how to resolve common errors when running {es} with Docker. + +===== elasticsearch.keystore is a directory + +[source,txt] +---- +Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.io.IOException: Is a directory: SimpleFSIndexInput(path="/usr/share/elasticsearch/config/elasticsearch.keystore") Likely root cause: java.io.IOException: Is a directory +---- + +A <> `docker run` command attempted +to directly bind-mount an `elasticsearch.keystore` file that doesn't exist. If +you use the `-v` or `--volume` flag to mount a file that doesn't exist, Docker +instead creates a directory with the same name. + +To resolve this error: + +. Delete the `elasticsearch.keystore` directory in the `config` directory. +. Update the `-v` or `--volume` flag to point to the `config` directory path + rather than the keystore file's path. For an example, see + <<<>. +. Retry the command. + +===== elasticsearch.keystore: Device or resource busy + +[source,txt] +---- +Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy +---- + +A <> `docker run` command attempted +to directly bind-mount the `elasticsearch.keystore` file. To update the +keystore, the container requires access to other files in the `config` +directory, such as `keystore.tmp`. + +To resolve this error: + +. Update the `-v` or `--volume` flag to point to the `config` directory + path rather than the keystore file's path. For an example, see + <<<>. +. Retry the command. + include::next-steps.asciidoc[] From fe5f967662f40e2a7e20c5ab0b3cb75c10731c01 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Thu, 9 Sep 2021 09:24:16 -0400 Subject: [PATCH 4/7] reword --- docs/reference/setup/install/docker.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/setup/install/docker.asciidoc b/docs/reference/setup/install/docker.asciidoc index 45e2fa5452dff..680ecbbf19929 100644 --- a/docs/reference/setup/install/docker.asciidoc +++ b/docs/reference/setup/install/docker.asciidoc @@ -406,7 +406,7 @@ instead. The command must: * Bind-mount the `config` directory. The command will create an `elasticsearch.keystore` file in this directory. To avoid errors, do - not directly bind-mount the `elasticsearch.keystore` file's path. + not directly bind-mount the `elasticsearch.keystore` file. * Use the `elasticsearch-keystore` tool with the `create` option. * Provide a keystore password using the `KEYSTORE_PASSWORD` or `KEYSTORE_PASSWORD_FILE` environment variables. Alternatively, you can use From 26d80ccf0516156ca97320870bb9f56735776091 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Thu, 9 Sep 2021 09:39:26 -0400 Subject: [PATCH 5/7] fix formatting --- docs/reference/setup/install/docker.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/setup/install/docker.asciidoc b/docs/reference/setup/install/docker.asciidoc index 680ecbbf19929..bc84d5234a0df 100644 --- a/docs/reference/setup/install/docker.asciidoc +++ b/docs/reference/setup/install/docker.asciidoc @@ -507,7 +507,7 @@ To resolve this error: . Delete the `elasticsearch.keystore` directory in the `config` directory. . Update the `-v` or `--volume` flag to point to the `config` directory path rather than the keystore file's path. For an example, see - <<<>. + <>. . Retry the command. ===== elasticsearch.keystore: Device or resource busy @@ -526,7 +526,7 @@ To resolve this error: . Update the `-v` or `--volume` flag to point to the `config` directory path rather than the keystore file's path. For an example, see - <<<>. + <>. . Retry the command. include::next-steps.asciidoc[] From 21708a4b33e20e953401c214377bbc30866cc579 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Fri, 10 Sep 2021 10:24:16 -0400 Subject: [PATCH 6/7] address review feedback --- docs/reference/setup/install/docker.asciidoc | 30 ++++++++++++++------ 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/docs/reference/setup/install/docker.asciidoc b/docs/reference/setup/install/docker.asciidoc index bc84d5234a0df..024e5b353c709 100644 --- a/docs/reference/setup/install/docker.asciidoc +++ b/docs/reference/setup/install/docker.asciidoc @@ -407,10 +407,8 @@ instead. The command must: * Bind-mount the `config` directory. The command will create an `elasticsearch.keystore` file in this directory. To avoid errors, do not directly bind-mount the `elasticsearch.keystore` file. -* Use the `elasticsearch-keystore` tool with the `create` option. -* Provide a keystore password using the `KEYSTORE_PASSWORD` or - `KEYSTORE_PASSWORD_FILE` environment variables. Alternatively, you can use - `elasticsearch-keystore` tool's `-p` option to use a password prompt. +* Use the `elasticsearch-keystore` tool with the `create -p` option. You'll be + prompted to enter a password for the keystore. ifeval::["{release-state}"!="unreleased"] For example: @@ -420,12 +418,12 @@ For example: docker run -it --rm \ -v full_path_to/config:/usr/share/elasticsearch/config \ docker.elastic.co/elasticsearch/elasticsearch:{version} \ -bin/elasticsearch-keystore create \ --E KEYSTORE_PASSWORD=mypassword +bin/elasticsearch-keystore create -p ---- You can also use a `docker run` command to add or update secure settings in the -keystore. You'll receive a prompt to enter setting values. +keystore. You'll be prompted to enter the setting values. If the keystore is +encrypted, you'll also be prompted to enter the keystore password. [source,sh,subs="attributes"] ---- @@ -434,10 +432,26 @@ docker run -it --rm \ docker.elastic.co/elasticsearch/elasticsearch:{version} \ bin/elasticsearch-keystore \ add my.secure.setting \ -my.other.secure.setting \ +my.other.secure.setting ---- endif::[] +If you've already created the keystore and don't need to update it, you can +bind-mount the `elasticsearch.keystore` file directly. For example, you can +add the following to `docker-compose.yml`: + +[source,yaml] +---- +elasticsearch: +... + volumes: + ... + - type: bind + source: full_path_to/config/elasticsearch.keystore + target: /usr/share/elasticsearch/config/elasticsearch.keystore +---- + + [[_c_customized_image]] ===== Using custom Docker images In some environments, it might make more sense to prepare a custom image that contains From 00517f4426af0cb066ca7645d4ce00e03c512bc6 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Fri, 10 Sep 2021 10:33:23 -0400 Subject: [PATCH 7/7] remove extra whitespace --- docs/reference/setup/install/docker.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/setup/install/docker.asciidoc b/docs/reference/setup/install/docker.asciidoc index 024e5b353c709..ebad11e123111 100644 --- a/docs/reference/setup/install/docker.asciidoc +++ b/docs/reference/setup/install/docker.asciidoc @@ -451,7 +451,6 @@ elasticsearch: target: /usr/share/elasticsearch/config/elasticsearch.keystore ---- - [[_c_customized_image]] ===== Using custom Docker images In some environments, it might make more sense to prepare a custom image that contains