[stable/postgresql] data directory "/bitnami/postgresql" has group or world access

[alphabt]

Describe the bug
postgres master container is unable to start. It failed with error below

INFO  ==> ** Starting PostgreSQL **
LOG:  skipping missing configuration file "/bitnami/postgresql/postgresql.auto.conf"
FATAL:  data directory "/bitnami/postgresql" has group or world access
DETAIL:  Permissions should be u=rwx (0700).

Version of Helm and Kubernetes:

Helm: 2.12.3
Kubernetes: 1.11.7

Which chart:
stable/postgresql v3.13.1

What happened:
postgresql master container couldn't start

What you expected to happen:
container starts normally

How to reproduce it (as minimally and precisely as possible):

requirements.yaml

dependencies:
  - name: postgresql
    version: "3.13.1"
    repository: "https://kubernetes-charts.storage.googleapis.com/"

helm install

helm upgrade --install \
  --wait \
  [...snipped]
  --set postgresql.enabled="$postgres_enabled" \
  --set postgresql.postgresqlUsername="$POSTGRES_USER" \
  --set postgresql.postgresqlPassword="$POSTGRES_PASSWORD" \
  --set postgresql.postgresqlDatabase="$POSTGRES_DB" \

Anything else we need to know:
I believe it's somehow due to chart pointing to an upgraded version of the postgres image (10.7.0-r69+), since my setup hasn't changed and it was working fine as of 2019/5/6, before 10.7.0-r69 dropped.

[Skeen]

I have the same issue, chart + this as subchart used to work, but is now broken with no change whatsoever on my end.

[juan131]

Hi @Skeen @alphabt

Does the issue disappear when setting the parameter image.tag to 10.7.0-r68?

[alphabt]

While I'm trying to repro using image.tag, I noticed this peculier log happened before the error:

LOG:  skipping missing configuration file "/bitnami/postgresql/postgresql.auto.conf"

Could this be significant or just a red herring?

[juan131]

Hi @alphabt

I think it's related with the mountPoint, which changed from /bitnami/postgresql/ to /bitnami/postgresql/data.

Using the chart version 3.18.4 and the image version 0.7.0-r68 use /bitnami/postgresql/.
Using the chart version 4.0.0 and the image version 0.7.0-r69 or newer, use /bitnami/postgresql/data.

[alphabt]

@juan131

I was able to confirm it repros with image.tag set to 0.7.0-r69, but was fine with 0.7.0-r68.

I see that to fix it on my side I would either:

• Pin image.tag to 0.7.0-r68 in my chart, or
• Upgrade chart to use 4.x

This breakage is quite unexpected since I assumed others like me are using the older chart 3.x normally, and it suddenly breaks without any change on our sides. Should we have a better upgrade / compatibility story going forward to prevent unexpected breakage like this in the future?

Thanks for the triage btw!

[juan131]

Using "rolling tags" is not recommended at all on production environment. You can find more information in the link below:

https://docs.bitnami.com/containers/how-to/understand-rolling-tags-containers/

However, I admit we should have found a way to avoid this happening.

I see that to fix it on my side I would either:

• Pin image.tag to 0.7.0-r68 in my chart, or
• Upgrade chart to use 4.x

I highly recommend you to upgrade to 4.x.x since the new image is based on Bash configuration scripts decreasing the size of the container (Node.js is not necessary anymore). More info in the link below:

https://github.com/bitnami/bitnami-docker-postgresql#notable-changes

[alphabt]

Thank you for the info on rolling tags, I was not aware of that before and I thought appVersion in postgresql\Chart.yaml was enough to pin the postgres version. I'll definitely take that into consideration now.

Much appreciated!

[juan131]

You're welcome! And sorry for the inconveniences caused with this change introduce in the image.

Please do not hesitate to report any issue you face with the PostgreSQL chart or image. We really appreciate your collaboration.

[johnmarcou]

Same issue here, using bitnami/postgresql chart v3.15.0.
Forcing the docker image to docker.io/bitnami/postgresql:10.7.0-r68 fixed the issue.

I would suggest to avoid to use imagePullPolicy: Always as a default, but IfNotPresent. This wouldn't help for fresh installed helm release, but it would "reduce" the risk to crash production environments when a Docker image is overrided.

[juan131]

Hi @johnmarcou

I highly recommend you to upgrade to chart v4.0.1 and image docker.io/bitnami/postgresql:10.7.0-r71. You should be able to upgrade the chart by running the commands below (please note the commands below assumed you named your release 'my-release'):

$ helm install stable/postgresql --name my-release \
     --version 3.15.0 \
     --set image.tag=10.7.0-r68 \
     --set postgresqlPassword=my-password \
     --set postgresqlDatabase=my-database
$ export POSTGRES_PASSWORD=$(kubectl get secret --namespace default my-release-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
$ helm upgrade my-release stable/postgresql \
     --version 4.0.1 \
     --set image.tag=10.7.0-r71 \
     --set postgresqlPassword=$POSTGRES_PASSWORD \
     --set postgresqlDatabase=my-database

[Skeen]

@juan131 Upgrading to v4.x.x yesterday fixed the issue for me.

[jmvizcainoio]

I'm using bitnami/postgresql:11.2.0 and now is crashing.

Chart version postgresql-4.0.1

INFO  ==> ** Starting PostgreSQL **
2019-05-10 10:37:09.603 GMT [1] LOG:  skipping missing configuration file "/bitnami/postgresql/postgresql.auto.conf"
2019-05-10 10:37:09.603 GMT [1] FATAL:  data directory "/bitnami/postgresql" has invalid permissions
2019-05-10 10:37:09.603 GMT [1] DETAIL:  Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).

is on a staging postgresql that is power off everyday but today doesn't startup.

[thonh]

Yeah, I have same issue with "helm install stable/kong"

[juan131]

Hi everyones,

We shouldn't have included a breaking change on a rolling tag to avoid breaking the chart. We apologize for this error and we'll work on improving the documentation so everyone understand the implications on using rolling tags.

We just released new revisions that are compatible with both 3.x.x and 4.x.x charts:

• 9.6 series: 9.6.12-r73
• 10 series: 10.7.0-r72
• 11 series: 11.2.0-r73

It should be safe from now on to continue using the rolling tags but, as it's explained in the link below, it's not a recommended option for production environments:

https://docs.bitnami.com/containers/how-to/understand-rolling-tags-containers

Please do not hesitate to let us know if you face any issue with these new images.

[jmvizcainoio]

using 11.2.0-r73 it works thank @juan131

[juan131]

That's great @jmvizcainoio !

Feel free to close this issue if you considered it solved.

[alphabt]

It's resolved, thanks @juan131

[sello4354]

Have you pushed the above containers to Azure Containers Registry ? I am getting "failed to pull image" error when i try to override the image tag with 10.7.0-r72.

[juan131]

Hi @sello4354

The containers have been published to the Azure Containers Registry too but it would take some time to be available (Azure processes).

You won't see the specific revision, but a new version 10 instead. We expect them to be available in a couple of days.

[mmailhos]

FYI the issue is not corrected with docker.io/bitnami/postgresql:9.6.12-r73

  Normal   Pulled                  9m17s (x4 over 10m)   kubelet, ip-10-110-2-162.ap-southeast-2.compute.internal  Container image "docker.io/bitnami/postgresql:9.6.12-r73" already present on machine
  Warning  BackOff                 5m59s (x29 over 10m)  kubelet, ip-10-110-2-162.ap-southeast-2.compute.internal  Back-off restarting failed container

INFO  ==> ** PostgreSQL setup finished! **
INFO  ==> ** Starting PostgreSQL **
FATAL:  data directory "/bitnami/postgresql/data" has group or world access
DETAIL:  Permissions should be u=rwx (0700).

[beltran-rubo]

This could be related to #20910 Could you please try to enable the "initContainer" for fixing the issue with permissions?

[juan131]

Hi everyone,

When we disabled the initContainer by default, the following actions were not running by default anymore:

mkdir -p /bitnami/postgresql/data
chmod 700 /bitnami/postgresql/data
find /bitnami/postgresql/data -mindepth 1 -maxdepth 1 -not -name ".snapshot" -not -name "lost+found" | \
    xargs chown -R 1001:1001

As you can see, there are two different actions done there:

• Restricting permissions on data directory to 700
• Adapting ownership of data directory

As commented in the PR were we introduced the change, we're also enabling a securityContext to address the issue with the ownership. However, the securityContext doesn't address "restricting permissions on data directory". Sorry we missed that.

To fix it, we adapted the logic in the PostgreSQL image so it restricts the permissions in its initialization logic. See https://github.com/bitnami/bitnami-docker-postgresql/blob/master/11/debian-10/rootfs/libpostgresql.sh#L654

I created a PR increasing the image revision so it uses the latest image including this fix. see #21015

[viveksinghggits]

Hey @juan131
I am still facing the issue,

FATAL:  data directory "/bitnami/postgresql/data" has group or world access
DETAIL:  Permissions should be u=rwx (0700)

I am installing chart with below details

"image.tag":              "9.6.12-r73",
version:  "8.6.4",

I tried to install "9.6.12-r73" with earlier chart version like 3.15.0 and 4.0.1, in that case the helm installation fails stating

Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "StatefulSet" in version "apps/v1beta2"

I would also like to highlight that this happens when I am restoring the data that has been backedup using pg_basebackup, by copying the backup files into data dir.

[juan131]

Hi,

Given the stable deprecation timeline, this Bitnami maintained Helm chart is now located at bitnami/charts. Please visit the bitnami/charts GitHub repository to create Issues or PRs.

In this issue, we tried to explain more carefully the reasons and motivations behind this transition, please don't hesitate to add a comment in this issue if you have any question related to the migration itself.

Regards,