Cannot run elasticsearch with security best practice readOnlyRootFilesystem: true

[ebuildy]

Bug Report

What did you do?

As a good practice, we configure security context (here readOnlyRootFilesystem: true):

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
spec:
  nodeSets:
    - 
      count: 3
      name: master
      podTemplate:
        spec:
          containers:
            - 
              securityContext: &securityContext
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                    - ALL
                privileged: false
                readOnlyRootFilesystem: true
                runAsNonRoot: true
                runAsUser: 1000
          initContainers:
            - name: elastic-internal-init-filesystem
              securityContext: 
                <<: *securityContext
            - name: elastic-internal-suspend
              securityContext:
                <<: *securityContext
            - name: elastic-internal-init-keystore
              securityContext:
                <<: *securityContext

What did you expect to see?

Operator create the STS, create pods, create container and init container can run fine.

What did you see instead? Under which circumstances?

The first initContainer cannot run because a folder is not writable. Diging the source code, we can see:

prepare-fs.sh script:

...
echo "Linking /mnt/elastic-internal/xpack-file-realm/users to /usr/share/elasticsearch/config/users"
		ln -sf /mnt/elastic-internal/xpack-file-realm/users /usr/share/elasticsearch/config/users
...

where the directory /usr/share/elasticsearch/config is not a volume, hence read-only.

We cant find any workaround, except changing polaris or OPA rules.

As a clean solution, the prepare-fs.sh script should use a temporary directory, mounted as emptyDir.

References:

• Elasticsearch pod cannot be created when using with PSP #2108
• Cannot create Kibana with the pod security policy "readOnlyRootFilesystem=true" #4022

[barkbay]

The way Elasticsearch starts requires the /tmp directory to be writable, other than that I think it can be started with readOnlyRootFilesystem: true:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-sample
spec:
  version: 8.5.0
  nodeSets:
  - name: default
    config:
      node.store.allow_mmap: false
    podTemplate:
      spec:
        volumes:
          - name: tmp-volume
            emptyDir: { }
        containers:
        - name: elasticsearch
          securityContext:
            readOnlyRootFilesystem: true
          volumeMounts:
            - mountPath: /tmp
              name: tmp-volume
    count: 3
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-sample
spec:
  version: 8.5.0
  count: 1
  elasticsearchRef:
    name: "elasticsearch-sample"
  podTemplate:
    spec:
      containers:
        - name: kibana
          securityContext:
            readOnlyRootFilesystem: true

[ebuildy]

Thanks you,

The issue happens on initContainers, to copy the configuration, the directory /usr/share/elasticsearch/config is not writable.

We cant specify a volume, because the copy operation need files from /usr/share/elasticsearch/config.

(We run elastic stack with Helm charts, so we were able to create another initContainer to do the trick)

[pebrc]

I think we should adapt the example configuration from #6126 (comment) as the default in ECK:

• the ECK operator should always mount an emptyDir volume in /tmp
• the ECK operator should set readOnlyRootFilesystem: true