nforem.web.1 container can't connect to postgres

[Surgbc]

My deployment does not run because of an error when deploying web container.

dokku deploy nforem gives the following:

-----> Deploying web (count=1)
       Attempting pre-flight checks (1.)
       Waiting for 10 seconds (1.)
230d3054d293a09da422a609526bed1582747c2a1993764f4d844bfbf04c4665
 !     App container failed to start (1.)
=====> Start of nforem container output (1.)
       Proc entrypoint 1 does not exist. Please check your Procfile
       Proc entrypoint 1 does not exist. Please check your Procfile
       Proc entrypoint 1 does not exist. Please check your Procfile
       Proc entrypoint 1 does not exist. Please check your Procfile
       Proc entrypoint 1 does not exist. Please check your Procfile
       Proc entrypoint 1 does not exist. Please check your Procfile
       Proc entrypoint 1 does not exist. Please check your Procfile
=====> End of nforem container output (1.)
parallel: This job failed:
/var/lib/dokku/plugins/available/scheduler-docker-local/bin/scheduler-deploy-process-container nforem herokuish dokku/nforem:latest  web 1 1

And docker run dokku/nforem /start web gives:

buildpack=pgbouncer at=pgbouncer-enabled
buildpack=pgbouncer at=config-gen-start
Setting DATABASE_URL_PGBOUNCER config var
buildpack=pgbouncer at=config-gen-end
buildpack=pgbouncer at=config-gen-override DATABASE_URL
buildpack=pgbouncer at=pgbouncer-launched pid=244 signal=SIGINT
buildpack=pgbouncer at=pgbouncer-start
2022-07-10 08:50:40.443 UTC [245] LOG kernel file descriptor limit: 1048576 (hard: 1048576); max_client_conn: 100, max expected fd use: 114
2022-07-10 08:50:40.443 UTC [245] LOG listening on 127.0.0.1:6000
2022-07-10 08:50:40.444 UTC [245] LOG listening on unix:/tmp/.s.PGSQL.6000
2022-07-10 08:50:40.444 UTC [245] LOG process up: PgBouncer 1.14.0, libevent 2.1.11-stable (epoll), adns: evdns2, tls: OpenSSL 1.1.1f  31 Mar 2020
buildpack=pgbouncer at=app-start
buildpack=pgbouncer at=app-launched pid=246
[247] Puma starting in cluster mode...
[247] * Puma version: 5.6.4 (ruby 3.0.2-p107) ("Birdie's Version")
[247] *  Min threads: 5
[247] *  Max threads: 5
[247] *  Environment: production
[247] *   Master PID: 247
[247] *      Workers: 2
[247] *     Restarts: (✔) hot (✖) phased
[247] * Preloading application
2022-07-10 08:50:54.134 UTC [245] LOG C-0x7f44a76e8b10: db1/[email protected]:33804 login attempt: db=db1 user=postgres tls=no
2022-07-10 08:50:54.137 UTC [245] WARNING DNS lookup failed: dokku-postgres-foremdb: result=-2
2022-07-10 08:50:54.137 UTC [245] LOG S-0x7f44a7729f50: db1/postgres@(bad-af):0 closing because: server DNS lookup failed (age=0s)
bundler: failed to load command: puma (/app/vendor/bundle/ruby/3.0.0/bin/puma)
[247] ! Unable to load application: ActiveRecord::ConnectionNotEstablished: connection to server at "127.0.0.1" (127.0.0.1), port 6000 failed: timeout expired
/app/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.2.4/lib/active_record/connection_adapters/postgresql_adapter.rb:87:in `rescue in new_client': connection to server at "127.0.0.1" (127.0.0.1), port 6000 failed: timeout expired (ActiveRecord::ConnectionNotEstablished)
...
...
...
buildpack=pgbouncer at=app-end
buildpack=pgbouncer at=exit process=app
buildpack=pgbouncer at=kill-app pid=246
buildpack=pgbouncer at=wait-app pid=246
buildpack=pgbouncer at=kill-aux name=pgbouncer pid=244 signal=SIGINT
buildpack=pgbouncer at=pgbouncer-end
2022-07-10 08:50:56.180 UTC [245] LOG Got SIGTERM, ignoring!
2022-07-10 08:50:56.180 UTC [245] LOG got SIGINT, shutting down

[Surgbc]

The hosts file here /var/lib/docker/containers/278b210ae473ed1a7b97b53136dc7f9bf296f248cdc8741d0d7db0212bb9075b/hosts contains

172.17.0.3      dokku-postgres-foremdb 595af28010ea dokku.postgres.foremdb
172.17.0.2      dokku-redis-redisdb d2db72767ddf dokku.redis.redisdb
172.17.0.5      278b210ae473

Which has the postgres container alright. But for some reason, it cannot resolve dokku-postgres-foremdb. Changing dokku-postgres-foremdb to its IP in the ENV variables solves the first problem.

dokku config:set nforem DATABASE_URL=postgres://[email protected]:5432/foremdb

But now a new problem all together:

ERROR S-0x7f9a1c529d50: db1/[email protected]:5432 cannot do SCRAM authentication: wrong password type

[Surgbc]

The current script will fail because of at least two errors:

1. DNS resolution error.

2022-07-10 08:50:54.137 UTC [245] WARNING DNS lookup failed: dokku-postgres-foremdb: result=-2
2022-07-10 08:50:54.137 UTC [245] LOG S-0x7f44a7729f50: db1/postgres@(bad-af):0 closing because: server DNS lookup failed (age=0s)

2. postgres 14.x server in dokku.postgres.foremdb refusing to authenticate client in nforem.web.1 because it requires SCRAM-SHA-256 while client only understands md5.

Quick Solutions

1. Change domain name in hosts file to IP, by:

Getting IP of docker postgres container

docker ps 
# get the container-ID 
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <<container-ID>>
# get the IP-address
dokku config:set nforem DATABASE_URL=postgres://postgres:currentpwd@<<IP-address>>:5432/foremdb

2. Downgrade postgres

dokku postgres:clone foremdb foremdbBac
dokku postgres:unlink foremdb nforem
dokku postgres:destroy foremdb
export POSTGRES_IMAGE="postgres"
export POSTGRES_IMAGE_VERSION="13.3"
dokku postgres:create foremdb
// mark the password shown during this process. It prints out the dsn.
dokku postgres:link foremdb nforem
// check ip of new container. May be the same as the old container...
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <<container-ID>>
dokku config:set nforem DATABASE_URL=postgres://postgres:newPwd@<<IP-address>>:5432/foremdb
docker run dokku/nforem /start web

[Surgbc]

The authentication Type is set in config/dir/pg_hba.conf you can equally change scram-sha-256 to md5 in this file rather than downgrading.

dokku postgres:info foremdb --config-dir
cd /var/lib/dokku/services/postgres/foremdb/data
cat pg_hba.conf