Postgrex.Protocol failed to connect: ** (DBConnection.ConnectionError) tcp connect (tmate-db:5432): timeout

[anantSwajal]

The server was running fine for 2 months before it suddenly stopped working. Whenever I try to run the container, after initializing I get the following error.

tmate-master      | 09:41:22.932 [error] Postgrex.Protocol (#PID<0.2044.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (tmate-db:5432): timeout
tmate-master      | 09:41:39.181 [error] Postgrex.Protocol (#PID<0.2045.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (tmate-db:5432): timeout
tmate-master      | 09:41:56.813 [error] Postgrex.Protocol (#PID<0.2041.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (tmate-db:5432): timeout

These errors come a while after the postgres db is accepting connections, so its not that either. It keeps retrying and failing continuously.

I have tried a lot of things, most of them point to the fact that its the tmate-master container which is unable to make the connection, the db is hosted fine.

The following is my docker-compose.yml Most of it is irrelevant I think, except for tmate-master and PG_URI

version: "3.5"

services:
  tmate-ssh-server:    
    image: tmate/tmate-ssh-server:latest    
    container_name: tmate-ssh-server    
    restart: unless-stopped    
    cap_add:    
    - SYS_ADMIN    
    ports:    
    - "2200:22"    
    environment:      
      WEBSOCKET_HOSTNAME: tmate-websocket      
      SSH_KEYS_PATH: "/etc/tmate-keys"      
      SSH_HOSTNAME: tmate.example.com      
      SSH_PORT_LISTEN: 22      
      SSH_PORT_ADVERTISE: 2200    
    volumes:    
    - ssh-keys:/etc/tmate-keys    
    - unix-sockets:/tmp/tmate/sessions    
    networks:      
      bridge:

  tmate-websocket:    
    image: tmate/tmate-websocket:latest    
    container_name: tmate-websocket    
    restart: unless-stopped    
    environment:      
      VIRTUAL_HOST: tmate-ws.example.com      
      VIRTUAL_PORT: 4001      
      CERT_NAME: example.com      
      WEBSOCKET_BASE_URL: wss://tmate-ws.example.com/      
      MASTER_BASE_URL: https://tmate.example.com/      
      USER_FACING_BASE_URL: https://tmate.example.com/      
      ERL_NODE_NAME: tmate.example.com      
      ERL_COOKIE: myerlcookie      
      DAEMON_HMAC_KEY: mydaemonhmackey      
      INTERNAL_API_AUTH_TOKEN: myinternalapiauthtoken    
    volumes:    
    - unix-sockets:/tmp/tmate/sessions    
    networks:      
      bridge:

  tmate-master:    
    image: tmate/tmate-master:latest    
    container_name: tmate-master    
    # https://github.com/tmate-io/tmate-master/issues/13#issuecomment-787470427    
    hostname: tmate-1    
    domainname: example.com    
    restart: unless-stopped    
    expose:    
    - 4000    
    environment:      
      VIRTUAL_HOST: tmate.example.com      
      VIRTUAL_PORT: 4000      
      CERT_NAME: example.com      
      MASTER_BASE_URL: 
      https://tmate.example.com/      
      ERL_COOKIE: myerlcookie      
      # Must match <hostname>.<domainname>      
      ERL_NODE_NAME: tmate-1.example.com      
      # ** (ArgumentError) cookie store expects conn.secret_key_base to be at least 64 bytes      
      SECRET_KEY_BASE: 0123456789abcdefghijklmnopqrstuvwxyz=0123456789abcdefghijklmnopqrstuvwxyz      
      INTERNAL_API_AUTH_TOKEN: myinternalapiauthtoken      
      PG_URI: postgres://tmate:postgrespassword@tmate-db:5432/tmate      
      # Muste be >= 2      
      PG_POOLSIZE: 5    
    networks:      
      bridge:
      internal:  
      
  tmate-db:    
    image: postgres:12-alpine    
    container_name: tmate-db    
    restart: unless-stopped    
    environment:      
      POSTGRES_USER: tmate      
      POSTGRES_PASSWORD: postgrespassword      
      POSTGRES_DB: tmate    
    volumes:
    - pgdata:/var/lib/postgresql/data
    networks:      
      internal:
      
volumes:  
  ssh-keys:  
  unix-sockets:  
  pgdata:
  
networks:  
  bridge:  
  internal:    
    internal: true

If anyone has any ideas as to why the server is failing the connection, please help.

I have tried a lot of changes

• I have tried different versions of postgres for tmate-db. Everything from 11-alpine3.15 to 15. None of them work, all result in the same timeout
• revert docker to an older snap. Same error still
• Run the server from an old instance backup, still same error
• Host a new postgres db on AWS RDS and connect to that instead. I still get the timeout, the server was looking in the right place since I got nxdomain after deleting that server
• Connect to tmate-db through psql on host machine. Connection was made properly without issues
• Ping tmate-db from tmate-master. Ping returned without issues
• change networks on the docker compose. No changes still timed out

I am currently all out of ideas except for forking the original repo and trying to make a build myself. Which would be pretty challenging since I have no experience with Elixer. The project was working fine and suddenly stopped working, so I am very unsure on how to proceed.

Any ideas about even what to look into will be appreciated.

[btclinux]

Hi! You can connect your tmate-master container to local postgres instance.

1. docker-compose down
2. Install postgres 12 and create user/database in psql
3. Install Elixir
4. export PG_URI=postgres://tmate:[email protected]:5432/tmate
5. export MIX_ENV=prod
6. Clone tmate-master repo and migrate to your postgres with: mix ecto.migrate
7. Bind your postgres instance to docker0 interface, in my case it is 172.17.0.1, look in your ifconfig. You can do it in postgres.conf
8. Allow tmate user access to tmate database from 172.0.0.0/8 network. Config file is pg_hba.conf
9. Restart postgres service
10. Remove tmate-db section from your docker-compose.yml
11. Change PG_URI to postgres://tmate:[email protected]:5432/tmate
12. docker-compose up