Garbage collection of plugins container causes crashed pod

[mikelorant]

Kubernetes garbage collects containers after a period of time. If the init container is garbage collected, this can cause the pod to lose the tracking information about its status.

Kubernetes documentation explains why this occurs:

We currently keep dead containers around for several reasons:
To keep container logs and other data around
To keep a record of the container state

This leads to an edge case of the following:

Because init containers can be restarted, retried, or re-executed, init container code should be idempotent. In particular, code that writes to files on EmptyDirs should be prepared for the possibility that an output file already exists.

This is tracked as an issue with Kubernetes: kubernetes/kubernetes#62362

However, this is easily triggered when running Elasticsearch with an install-plugins container. It is important to make sure there are some plugins installed.

On a node which has the Elasticsearch pod, identify how long ago the init container ran. Then run the following command:

docker system prune -a -f --filter "until=3h"

Replace 3h with a value less than the age of the install-plugins container.

This will cause the Elasticsearch pod to restart the install-plugins container and attempt to install the plugins again. However, since the elasticsearch container is still running, these plugins are already installed on the emptyDir volume. The volume is not empty as expected and plugins fail to install since they already exist.

Recommend to always check if the plugins are installed first before attempting to install due to this case.

[sebgl]

Relates #3294.
We could adapt the plugins doc to make the plugin install idempotent. For example, write the list of plugins already installed in a text file, and check the content of this file before installing the plugins again?

[paschdan]

Could we not just do this in the plugin docs:

instead of

spec:
  nodeSets:
  - name: default
    count: 3
    podTemplate:
      spec:
        initContainers:
        - name: install-plugins
          command:
          - sh
          - -c
          - |
            bin/elasticsearch-plugin install --batch repository-gcs

check for existance

spec:
  nodeSets:
  - name: default
    count: 3
    podTemplate:
      spec:
        initContainers:
        - name: install-plugins
          command:
          - sh
          - -c
          - |
            /bin/test -d /usr/share/elasticsearch/plugins/repository-gcs || bin/elasticsearch-plugin install --batch repository-gcs

the question is, how to do it efficently for multiple plugins.

[mikelorant]

I think the solution should be part of elasticsearch-plugin itself. It needs a --skip or --overwrite option that either ignores if the plugin exists or overwrites an existing plugin it if it does.

[anyasabo]

I didn't see anything open already so I opened a feature request in the ES repo elastic/elasticsearch#61385

[pebrc]

Closing this as the most popular plugins are bundled now. And #5145 should make things easier.