Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pi-hole Adminpage shows Error 503 (Service unavailable) #1299

Closed
6 tasks done
madnuttah opened this issue Jan 23, 2023 · 26 comments
Closed
6 tasks done

Pi-hole Adminpage shows Error 503 (Service unavailable) #1299

madnuttah opened this issue Jan 23, 2023 · 26 comments

Comments

@madnuttah
Copy link

madnuttah commented Jan 23, 2023 •
edited
Loading

This is a: Bug

Details

After upgrading to 2023.01.8 the Adminpage fails with Error 503, switching back to 2023.01.6 makes the page appear again.

Related Issues

  • I have searched this repository/Pi-hole forums for existing issues and pull requests that look similar

https://github.com/pi-hole/pi-hole/issues/5131

How to reproduce the issue

  1. Environment data
  • Operating System: Docker on Alpine 3.17.1
  • Hardware: PC
  • Kernel Architecture: x86/amd64
  • Docker Install Info and version:
    • Software source: official docker-ce version 20.10.21
    • Supplimentary Software: portainer-ce
  • Hardware architecture: x86/amd64
  1. My docker-compose.yaml:

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    hostname: pihole
    domainname: mydomain.lan
    mac_address: BA:BE:CA:FF:EE:AA
    depends_on:
      - unbound
    cap_add:
      #- NET_ADMIN
      - NET_BIND_SERVICE
    networks:
      docker:
        ipv4_address: 172.16.0.253
      docker-bridge:
        ipv4_address: 192.168.0.253
    dns:
      - 172.16.0.252@5335
    ports:
      - 53:53/tcp
      - 53:53/udp
    environment:
      FTLCONF_LOCAL_IPV4: 172.16.0.1
      VIRTUAL_HOST: pihole.mydomain.lan
      WEBPASSWORD: password 
      WEBUIBOXEDLAYOUT: boxed
      WEBTHEME: default-light
      TZ: Europe/Amsterdam
      CORS_HOSTS: mydomain.lan
      PIHOLE_DNS_1: 172.16.0.252#5335
      IPv6: "false"    
      DNS_BOGUS_PRIV: "true"
      DNS_FQDN_REQUIRED: "true"
      REV_SERVER: "true"
      REV_SERVER_TARGET: 172.16.0.254
      REV_SERVER_DOMAIN: mydomain.lan
      REV_SERVER_CIDR: 172.16.0.0/24
      DNSMASQ_LISTENING: all
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./pihole/etc/dnsmasq.d/:/etc/dnsmasq.d/:z
      - ./pihole/etc/lighttpd/pihole.pem:/etc/lighttpd/pihole.pem:z
      - ./pihole/etc/lighttpd/pihole.key:/etc/lighttpd/pihole.key:z
      - ./pihole/etc/lighttpd/root_ca.pem:/etc/lighttpd/root_ca.pem:z
      - ./pihole/etc/lighttpd/external.conf:/etc/lighttpd/conf-enabled/external.conf:z
      - ./pihole/etc/pihole/:/etc/pihole/:z
      - ./pihole/etc/hosts.list:/etc/hosts:z
      - ./pihole/etc/resolv.conf:/etc/resolv.conf:z
      - ./pihole/etc/hostname:/etc/hostname:z
    restart: unless-stopped

My external.conf:

server.modules += ("mod_redirect")
server.modules += ("mod_setenv")
server.modules += ("mod_openssl")
#server.port = 80
$SERVER["socket"] == "[::]:80" { }

$HTTP["scheme"] == "http" {
    url.redirect = ("" => "https://${url.authority}${url.path}${qsa}")
}

$HTTP["scheme"] == "https" {
    # HTTP Strict Transport Security (63072000 seconds)
    setenv.add-response-header = (
        "Strict-Transport-Security" => "max-age=63072000"
    )
}

$SERVER["socket"] ==     ":443" { ssl.engine = "enable" }
#$SERVER["socket"] == "[::]:443" { ssl.engine = "enable" }
ssl.privkey = "/etc/lighttpd/pihole.key"
ssl.pemfile = "/etc/lighttpd/pihole.pem"
ssl.ca-file = "/etc/lighttpd/root_ca.pem"
ssl.openssl.ssl-conf-cmd = ("MinProtocol" => "TLSv1.3")
ssl.openssl.ssl-conf-cmd += ("Options" => "-ServerPreference")
  1. any additional info to help reproduce

cat error-pihole.log

2023-01-23 13:30:26: server.c.1513) server started (lighttpd/1.4.59)
2023-01-23 13:30:31: gw_backend.c.503) bind failed for: unix:/run/lighttpd/pihole-php-fastcgi.socket-0: No such file or directory
2023-01-23 13:30:31: gw_backend.c.670) ERROR: spawning backend failed.
2023-01-23 13:30:31: gw_backend.c.970) all handlers for /admin/index.php? on .php are down.

I've checked Pi-hole discrourse too but I couldn't find a solution for my particular problem.

These common fixes didn't work for my issue

  • I have tried a newer or older version of Docker Pi-hole (depending what version the issue started in for me) - Version 2023.01.6 is working
  • I have tried removing/destroying my container, and re-creating a new container
  • I have tried fresh volume data by backing up and moving/removing the old volume data
  • I have tried running the stock docker run example(s) in the readme (removing any customizations I added)
  • I have tried running without my volume data mounts to eliminate volumes as the cause

Thank you.
@PromoFaux
Copy link
Member

I'm beginning to regret ever having touched the lighttpd config.

Inside the container can you run the following please?

ls -lah /run
ls -lah /run/lighttpd

@PromoFaux
Copy link
Member

Can you double check that the image you are running is in fact 2023.01.8? I'm just wondering if you downloaded 2023.01.7 (which did have this issue - but has since been removed) Please also provide a debug token

@madnuttah
Copy link
Author 

root@pihole:/# ls -lah /run
total 48K
drwxr-xr-x 1 root   root   4.0K Jan 23 13:30 .
drwxr-xr-x 1 root   root   4.0K Jan 23 13:30 ..
-rw-r--r-- 1 root   root      3 Jan 23 13:30 crond.pid
---------- 1 root   root      0 Jan 23 13:30 crond.reboot
-rw-r--r-- 1 root   root      4 Jan 23 13:30 lighttpd.pid
drwxrwxrwt 2 root   root   4.0K Jan  9 01:00 lock
drwxr-xr-x 1 pihole pihole 4.0K Jan 23 13:30 pihole
-rw-r--r-- 1 pihole pihole    3 Jan 23 13:30 pihole-FTL.pid
drwxr-xr-x 5 root   root   4.0K Jan 23 13:30 s6
drwxr-xr-x 2 root   root   4.0K Jan 23 13:30 s6-linux-init-container-results
lrwxrwxrwx 1 root   root     23 Jan 23 13:30 s6-rc -> s6-rc:s6-rc-init:DHDobG
drwxr-xr-x 3 root   root   4.0K Jan 23 13:30 s6-rc:s6-rc-init:DHDobG
drwxr-xr-x 4 root   root   4.0K Jan 23 13:30 service
-rw-rw-r-- 1 root   utmp      0 Jan  9 01:00 utmp

root@pihole:/# ls -lah /run/lighttpd
ls: cannot access '/run/lighttpd': No such file or directory

root@pihole:/# pihole -v
  Pi-hole version is v5.15.1 (Latest: v5.15.1)
  AdminLTE version is v5.18.2 (Latest: v5.18.2)
  FTL version is v5.20.1 (Latest: v5.20.1)

Portainer

org.opencontainers.image.version 2023.01.8

@PromoFaux
Copy link
Member

Are you able to run a debug token from inside the container too, please? (pihole -d)

Also on the host system:

 docker inspect pihole/pihole

@madnuttah
Copy link
Author

madnuttah commented Jan 23, 2023 •
edited
Loading

Token has been uploaded, here's the inspect result:

Blahblah:~# docker inspect pihole/pihole
[
    {
        "Id": "sha256:f5fcdb2e357456af30e2c4e3a531d77f97e30aaadbf208a7d76c2dca4163487a",
        "RepoTags": [
            "pihole/pihole:latest"
        ],
        "RepoDigests": [
            "pihole/pihole@sha256:a93460b997838f6f8df38e6bdbdb869ae4c57e6c4811b7dab7ae7847eff5d2ab"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2023-01-22T17:56:54.417032024Z",
        "Container": "",
        "ContainerConfig": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "53/tcp": {},
                "53/udp": {},
                "67/udp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "phpver=php",
                "PHP_ERROR_LOG=/var/log/lighttpd/error-pihole.log",
                "IPv6=True",
                "S6_KEEP_ENV=1",
                "S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
                "S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0",
                "FTLCONF_LOCAL_IPV4=0.0.0.0",
                "VIRTUAL_HOST=",
                "FTL_CMD=no-daemon",
                "DNSMASQ_USER=pihole"
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "dig +short +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
                ]
            },
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/s6-init"
            ],
            "OnBuild": null,
            "Labels": {
                "org.opencontainers.image.created": "2023-01-22T17:56:09.578Z",
                "org.opencontainers.image.description": "Pi-hole in a docker container",
                "org.opencontainers.image.licenses": "",
                "org.opencontainers.image.revision": "4256d62a1e166f20079841cca6ef072da480da52",
                "org.opencontainers.image.source": "https://github.com/pi-hole/docker-pi-hole",
                "org.opencontainers.image.title": "docker-pi-hole",
                "org.opencontainers.image.url": "https://github.com/pi-hole/docker-pi-hole",
                "org.opencontainers.image.version": "2023.01.8"
            },
            "Shell": [
                "/bin/bash",
                "-c"
            ]
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 321212763,
        "VirtualSize": 321212763,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/f9f92557aa14fe9bb25ab66982827cc45eb082513c6c0c9042efaadb3414c3be/diff:/var/lib/docker/overlay2/185acca5bbf71954dc1b11a807eb3e24d75271dcc0ec055596ebd710501b1206/diff:/var/lib/docker/overlay2/56801f80b1f508882cf12e1f02fac708d17c267daafeaed3610b551ec8d110b1/diff:/var/lib/docker/overlay2/c82d75c940faf61cb0faaddb740833dacda806616cf55ee1a1cc4585108bc68b/diff:/var/lib/docker/overlay2/0e71d24ddf747fa3a6e427336cd8be22c2e80dd4b133f40179e47ae2eab2824e/diff:/var/lib/docker/overlay2/97ffaafbd2f11f342b27d00f71b517b3aa049ed578588d79dbcb996254cf6426/diff:/var/lib/docker/overlay2/cb0bedb0a9b09bb8ac95ce9eb7731b7433fa1ba1cb8380de89f0112dda90b1be/diff:/var/lib/docker/overlay2/e2c553ffad790ae65684ac97573f38aefdc9c7fd659f2e91fd9f6071a70f0589/diff",
                "MergedDir": "/var/lib/docker/overlay2/403628f040b09718ffefe2e4e8805255b40224b8f85451d99792f754586ea716/merged",
                "UpperDir": "/var/lib/docker/overlay2/403628f040b09718ffefe2e4e8805255b40224b8f85451d99792f754586ea716/diff",
                "WorkDir": "/var/lib/docker/overlay2/403628f040b09718ffefe2e4e8805255b40224b8f85451d99792f754586ea716/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:67a4178b7d47beb6a1f697a593bd0c6841c67eb0da00f2badefb05fd30671490",
                "sha256:fe643b01915a0855a6dac972ce69862020d45ecfd557fa0211034158a68fc1a9",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
                "sha256:b9b4031b44a03856838552a7f45d640d947bcda734ae39faf468281e01079c9c",
                "sha256:dcf40b6ae2a2446019b9a9fb8fe5dc44f5623d49448517ac02c02753ee4b4375",
                "sha256:b957746b11cd1338cc2d54c93f0c50ab1230f363d09fd2deb6842ae642576932",
                "sha256:58a6e5f2a99ee17103bd1fec86a5d91c64bab194d42bb1a67f0756062d8c5da7",
                "sha256:930ef68d1d851ad8cb848ab21ba292de281cd4f5771323becd2cc63f0f544734",
                "sha256:2f5121acf1322ff581ca753f022221a7470c177b8168e3e5679258bd66cd0252"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

@PromoFaux
Copy link
Member

Token has been uploaded

I need the token ID else It would be like finding a needle in a haystack :) Should have been printed to the console at the end of the process

@madnuttah
Copy link
Author

madnuttah commented Jan 23, 2023 •
edited
Loading

Sorry, mate.

@PromoFaux
Copy link
Member

Yeah, everything points to you being on the right image - which doesn't make sense!

#1298 was specifically created to prevent this,

It seems like these lines of code are not running on your instance:

docker-pi-hole/src/s6/debian-root/usr/local/bin/bash_functions.sh

Lines 121 to 124 in 4256d62

# Ensure that /run/lighttpd exists for the php socket, and is owned by www-data.
# Without this, the web interface will return a 503. Not sure how this used to work, as this was always the directory that was used in previous versions of the image.
mkdir -p /run/lighttpd
chown www-data:www-data /run/lighttpd

What does the startup log look like please?

@madnuttah
Copy link
Author

What log do you need exactly?

@PromoFaux
Copy link
Member

The output when the container starts.... in portainer you can click the logs button for the running container

@madnuttah
Copy link
Author

madnuttah commented Jan 23, 2023 •
edited
Loading

Ah I see. Looking good as far as I can tell:

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service cron: starting
s6-rc: info: service cron successfully started
s6-rc: info: service _uid-gid-changer: starting
s6-rc: info: service _uid-gid-changer successfully started
s6-rc: info: service _startup: starting
  [i] Starting docker specific checks & setup for docker pihole/pihole
  [i] Setting capabilities on pihole-FTL where possible
  [i] Applying the following caps to pihole-FTL:
        * CAP_CHOWN
        * CAP_NET_BIND_SERVICE
        * CAP_NET_RAW
  [i] Ensuring basic configuration by re-running select functions from basic-install.sh
  [i] Installing configs from /etc/.pihole...
  [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
  [i] Installing /etc/dnsmasq.d/01-pihole.conf...
  [✓] Installed /etc/dnsmasq.d/01-pihole.conf
  [i] Installing /etc/.pihole/advanced/06-rfc6761.conf...
  [✓] Installed /etc/dnsmasq.d/06-rfc6761.conf
  [i] Installing latest logrotate script...
	[i] Existing logrotate file found. No changes made.
ln: failed to create symbolic link '/etc/pihole/macvendor.db': Operation not supported
s6-rc: info: service _startup successfully started
s6-rc: info: service pihole-FTL: starting
s6-rc: info: service pihole-FTL successfully started
s6-rc: info: service lighttpd: starting
s6-rc: info: service lighttpd successfully started
s6-rc: info: service _postFTL: starting
s6-rc: info: service _postFTL successfully started
s6-rc: info: service legacy-services: starting
  Checking if custom gravity.db is set in /etc/pihole/pihole-FTL.conf
s6-rc: info: service legacy-services successfully started
  [✗] DNS resolution is currently unavailable
  [i] Time until retry: 120
  [✓] DNS resolution is now available
  [i] Neutrino emissions detected...

  [✓] Pulling blocklist source list into range
  [i] Preparing new gravity database...
  [✓] Preparing new gravity database
  [i] Using libz compression
  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [i] Status: Pending...
  [✓] Status: Retrieval successful

Edit: More log.

@PromoFaux
Copy link
Member

ln: failed to create symbolic link '/etc/pihole/macvendor.db': Operation not supported

There we go. It is falling over at:

docker-pi-hole/src/s6/debian-root/usr/local/bin/bash_functions.sh

Line 114 in 4256d62

ln -s /macvendor.db /etc/pihole/macvendor.db

This is unusual behaviour, which leads me to thinking that maybe it is a permission issue on your volume mount to /etc/pihole ( - ./pihole/etc/pihole/:/etc/pihole/:z)

I wonder, can you try starting without the volume attached to see if it progresses past that section?

Brief google suggests it is because you may be using a shared volume (judging by the z flag). I've not read this but:

https://forums.docker.com/t/symlinks-on-shared-volumes-not-supported/9288

@PromoFaux
Copy link
Member

(I suspect it has always fallen over there for you, but since nothing else really important happens after that line (until now!) you would not have noticed it)

@madnuttah
Copy link
Author

madnuttah commented Jan 23, 2023 •
edited
Loading

I'll check your suggestion and get back to you with my findings.

@madnuttah
Copy link
Author

Without /etc/pihole the container spins up.

@PromoFaux
Copy link
Member

Later on (i.e, when I've finished work) I will look at making the script not hard exit if it cannot create that symlink.

@madnuttah
Copy link
Author

madnuttah commented Jan 23, 2023 •
edited
Loading

Many thanks for your assistance, it's very appreciated.

@PromoFaux
Copy link
Member

I have a couple of ideas for solutions - just testing them both now

@PromoFaux PromoFaux mentioned this issue Jan 23, 2023
Merged
1 task
@PromoFaux
Copy link
Member

@madnuttah , can you please give the :dev tag a try and also paste in the container startup logs?

@madnuttah
Copy link
Author

madnuttah commented Jan 23, 2023 •
edited
Loading

There's no webpage loading at all now, fyi.

Here's the log:

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service cron: starting
s6-rc: info: service cron successfully started
s6-rc: info: service _uid-gid-changer: starting
s6-rc: info: service _uid-gid-changer successfully started
s6-rc: info: service _startup: starting
  [i] Starting docker specific checks & setup for docker pihole/pihole
  [i] Setting capabilities on pihole-FTL where possible
  [i] Applying the following caps to pihole-FTL:
        * CAP_CHOWN
        * CAP_NET_BIND_SERVICE
        * CAP_NET_RAW
  [i] Ensuring basic configuration by re-running select functions from basic-install.sh
  [i] Installing configs from /etc/.pihole...
  [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
  [i] Installing /etc/dnsmasq.d/01-pihole.conf...
  [✓] Installed /etc/dnsmasq.d/01-pihole.conf
  [i] Installing /etc/.pihole/advanced/06-rfc6761.conf...
  [✓] Installed /etc/dnsmasq.d/06-rfc6761.conf
  [i] Installing latest logrotate script...
	[i] Existing logrotate file found. No changes made.
  [i] Assigning password defined by Environment Variable
  [✓] New password set
  [i] Setting Web Theme based on WEBTHEME variable, using value default-light
  [i] Added ENV to php:
                    "TZ" => "Europe/Amsterdam",
                    "PIHOLE_DOCKER_TAG" => "",
                    "PHP_ERROR_LOG" => "/var/log/lighttpd/error-pihole.log",
                    "CORS_HOSTS" => "mydomain.lan",
                    "VIRTUAL_HOST" => "pihole.mydomain.lan",
  [i] Using IPv4
  [i] Preexisting ad list /etc/pihole/adlists.list detected (exiting setup_blocklists early)
  [i] Existing DNS servers detected in setupVars.conf. Leaving them alone
  [i] Applying pihole-FTL.conf setting LOCAL_IPV4=172.16.0.1
  [i] FTL binding to default interface: eth0
  [i] Enabling Query Logging
  [i] Testing lighttpd config: Syntax OK
  [i] All config checks passed, cleared for startup ...
  [i] Docker start setup complete
  [i] pihole-FTL (no-daemon) will be started as pihole
s6-rc: info: service _startup successfully started
s6-rc: info: service pihole-FTL: starting
s6-rc: info: service pihole-FTL successfully started
s6-rc: info: service lighttpd: starting
s6-rc: info: service lighttpd successfully started
s6-rc: info: service _postFTL: starting
s6-rc: info: service _postFTL successfully started
s6-rc: info: service legacy-services: starting
  Checking if custom gravity.db is set in /etc/pihole/pihole-FTL.conf
s6-rc: info: service legacy-services successfully started
  [i] Neutrino emissions detected...

@PromoFaux
Copy link
Member

Hmmm. Make sure you've downloaded the latest dev tag (updated 14 mins ago)

adam@adam-pc:~/play$ docker pull pihole/pihole:dev
dev: Pulling from pihole/pihole
Digest: sha256:d70884e3d3bae81559f061eeac17ffaa4d449a5d365c0de61350abe9f1e23b5e
Status: Image is up to date for pihole/pihole:dev
docker.io/pihole/pihole:dev

image

@madnuttah
Copy link
Author

madnuttah commented Jan 23, 2023 •
edited
Loading

Edit: sorry. I checked the IP of pihole and the GUI loads. I used fqdn:443 which doesn't work right now.

@PromoFaux
Copy link
Member

So the image as-provided is now working, but your SSL configurations are not at this moment, correct?

@madnuttah
Copy link
Author

That's correct.

@PromoFaux
Copy link
Member

Super, thanks for that feedback. I will tag a new version.

I'm afraid I can't be much help with the SSL configuration - I normally just run a reverse proxy container (traefik, though considering a move to caddy) and terminate there - haven't noticed any adverse affect of doing that 🤷‍♂️

@madnuttah
Copy link
Author

Thanks for your efforts! I'll try to get my SSL config working again.

@StrikeAgainst StrikeAgainst mentioned this issue Jan 28, 2023
Closed
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@PromoFaux @madnuttah