doc: add log rotation and log shipping sidecar examples

[jasonodonnell]

Add an example to the documentation that sets up Vault Helm to include a sidecar container that rotates Vault audit logs to satisfy this issue: #109.

This would use server.extraContainers to add in any other containers we may need.

References:

• Doc: https://www.vaultproject.io/docs/platform/k8s/helm.html
• Source: https://github.com/hashicorp/vault/tree/master/website/source/docs/platform/k8s

[sidewinder12s]

Are there any examples of how to setup these sidecars?

I'm running into a lot of trouble trying to get this working correctly with the number of security bits enabled that make most off the shelf sidecar containers just break because they all assume they can run as root.

[reddy9694]

Have you found any workaround for this ..?

Are there any examples of how to setup these sidecars?

I'm running into a lot of trouble trying to get this working correctly with the number of security bits enabled that make most off the shelf sidecar containers just break because they all assume they can run as root.

[sidewinder12s]

I ended up writing a custom container with logrotate & a go based cron scheduler. Due to the security settings on Vault most OSS log rotate containers just didn't work and I wish I had just started with something custom.

[lhw]

I know reviving an old issue is kind of annoying. But this is the best fitting place right now as its the first search hit for the exact issue.

I ended up writing a custom container with logrotate & a go based cron scheduler. Due to the security settings on Vault most OSS log rotate containers just didn't work and I wish I had just started with something custom.

I took this as inspiration to update this old issue with a solution for the problem at hand:
Here is the go binary and container which is specifically setup for this helm chart deployment (uid/gid): https://github.com/HanseMerkur/vault-logrotate just build it according to the dockerfile and it should work.

The values for the most recent helm charts (v0.20.1) look something like this:

vault:
  server:
    # HUP signal for logrotate
    shareProcessNamespace: true
    # Add the lograte config from a config map
    volumes:
      - name: logrotate-config
        configMap:
          name: logrotate-config
    # And finally the container
    extraContainers:
     - name: auditlog-rotator
       image: vault-logrotate:latest
       imagePullPolicy: Always
       env:
         - name: CRONTAB
           value: "*/5 * * * *"
       volumeMounts:
       - mountPath: /etc/logrotate.conf
         name: logrotate-config
         subPath: logrotate.conf
         readOnly: true
       - mountPath: /vault/audit
         name: audit

The logrotate.conf ConfigMap can look something like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: logrotate-config
data:
  logrotate.conf: |    
    /vault/audit/vault.log {
        copytruncate
        size 100M
        missingok
        nocompress

        postrotate
            pkill -HUP vault
        endscript
    }

[Jaturu]

Adding another example for a scenario that is using kubernetes, raft with integrated storage and persistent volumes that are ReadWriteOnce.

Dockerfile

There is an issue with Alpine's index for edge/community where the supercronic package is not included. In addition, it would be necessary to modify the /etc/apk/repositories file in order to pull the package so I choose to copy it in. You made need to change the arch type to match your environment.

FROM hashicorp/vault:latest

COPY logrotate.conf /etc/logrotate.conf
COPY --chown=vault:vault run.sh /tmp/run.sh
COPY --chown=vault:vault supercronic-0.2.1-r2.apk /tmp/supercronic-0.2.1-r2.apk
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh

RUN apk update && \
       apk add --no-cache \
       logrotate && \
       chmod 0755 /tmp/run.sh && \
       cd /tmp && \
       apk add --no-cache supercronic-0.2.1-r2.apk

ENTRYPOINT ["docker-entrypoint.sh"]

run.sh

#!/bin/sh
echo 'SHELL=/bin/sh' > /home/vault/vaultcrontab
echo '* * * * * /home/vault/logrotate.sh' >> /home/vault/vaultcrontab
echo '/usr/sbin/logrotate --state=/home/vault/logrotate.status /etc/logrotate.conf' >> /home/vault/logrotate.sh
chmod 0755 /home/vault/logrotate.sh

supercronic /home/vault/vaultcrontab &

docker-entrypoint.sh

I pulled the latest version and then added one line for starting supercronic using run.sh.

#!/usr/bin/dumb-init /bin/sh
set -e

# Note above that we run dumb-init as PID 1 in order to reap zombie processes
# as well as forward signals to all processes in its session. Normally, sh
# wouldn't do either of these functions so we'd leak zombies as well as do
# unclean termination of all our sub-processes.

# Prevent core dumps
ulimit -c 0

# Start supercronic
/tmp/run.sh
<snip>

logrotate.conf

/vault/audit/audit.log {
    copytruncate
    size 10M
    missingok
    nocompress
    notifempty
    rotate 7

    postrotate
        pkill -HUP vault
    endscript
}

[hajdukda]

https://github.com/HanseMerkur/vault-logrotate

If anyone decides to use it - keep in mind that dockerfile is incorrectly built, rename the go binary, pkill/pgrep will point at self and kill itself causing interrupts.

This script also does not respect graceful shutdowns. Subscribe to system calls for SIGTERM and afterwards to ctx.Done() returned by cron.

[jpenghashi]

Thank you so much for the discussions!

I made a step-by-step tutorial based on the comment #142 (comment).