Some directories return 404 with file-server --browse

[lonble]

When I use file-server, I find that caddy always returns 404 when I access the root "/" of the disk or the user's home directory, even i run it as root. The documentation doesn't explain the reason for this, or even say anything about auto-hiding these directories. This is already causing a lot of trouble for my actual use case and I need some option to disable this implicit feature.

[mholt]

Can you give us a config to reproduce this?

[lonble]

I just run this command in terminal:

sudo caddy file-server --browse --root /

and this is chrome's screen shot

Additional, when I visit /home everything works fine, but when I visit /home/lonble I get 404 again.

Sorry I closed the issue due to wrong use of shortcut keys, I open it again

[mholt]

Ah, the file-server command.

It's pretty bare-bones, I don't think it should be hiding anything by defaut:

caddy/modules/caddyhttp/fileserver/command.go

Line 81 in bf380d0

handler := FileServer{Root: root}

Let me try to reproduce it real quick.

What happens when you run sudo caddy environ? What is $HOME?

[mholt]

So I just tried the same command on my Linux machine and it works for me.

I'm guessing it has something to do with your system setup. PROTECT_HOME=true? Something else funky might be going on.

I recommend troubleshooting with curl -v by the way. Browsers do weird stuff.

[mholt]

Oh, try enabling --access-log. Anything interesting?

Also try a caddyfile:

{
    debug
}

:80

root * /
file_server

[lonble]

So I just tried the same command on my Linux machine and it works for me.

I'm guessing it has something to do with your system setup. PROTECT_HOME=true? Something else funky might be going on.

I recommend troubleshooting with curl -v by the way. Browsers do weird stuff.

$ sudo caddy environ 
caddy.HomeDir=/root
caddy.AppDataDir=/root/.local/share/caddy
caddy.AppConfigDir=/root/.config/caddy
caddy.ConfigAutosavePath=/root/.local/share/caddy/autosave.json
caddy.Version=v2.4.6
runtime.GOOS=linux
runtime.GOARCH=amd64
runtime.Compiler=gc
runtime.NumCPU=8
runtime.GOMAXPROCS=8
runtime.Version=go1.17.3
os.Getwd=/home/lonble

COLORTERM=truecolor
XAUTHORITY=/run/user/1000/gdm/Xauthority
LANG=en_US.UTF-8
TERM=xterm-256color
DISPLAY=:0
PATH=/home/lonble/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/lonble/Programs
MAIL=/var/mail/root
LOGNAME=root
USER=root
HOME=/root
SHELL=/bin/bash
SUDO_COMMAND=/usr/bin/caddy environ
SUDO_USER=lonble
SUDO_UID=1000
SUDO_GID=1000

$ curl -v 127.0.0.1
*   Trying 127.0.0.1:80...
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Server: Caddy
< Date: Sat, 22 Jan 2022 08:27:21 GMT
< Content-Length: 0
< 
* Connection #0 to host 127.0.0.1 left intact

[lonble]

run with --access-log

2022/01/22 08:31:46.695	ERROR	http.log.access	handled request	{"request": {"remote_addr": "127.0.0.1:52314", "proto": "HTTP/1.1", "method": "GET", "host": "127.0.0.1", "uri": "/", "headers": {"Sec-Ch-Ua": ["\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"97\", \"Chromium\";v=\"97\""], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-Dest": ["document"], "Sec-Ch-Ua-Mobile": ["?0"], "Connection": ["keep-alive"], "Sec-Ch-Ua-Platform": ["\"Linux\""], "Dnt": ["1"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-User": ["?1"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"], "Accept-Encoding": ["gzip, deflate, br"], "Accept-Language": ["en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7"]}}, "common_log": "127.0.0.1 - - [22/Jan/2022:16:31:46 +0800] \"GET / HTTP/1.1\" 404 0", "user_id": "", "duration": 0.000201815, "size": 0, "status": 404, "resp_headers": {"Server": ["Caddy"]}}

run with the Caddyfile:

{
    debug
}

:80

root * /
file_server browse

I accessed /, /home/, /home/lonble/

2022/01/22 08:38:14.441	DEBUG	http.handlers.file_server	sanitized path join	{"site_root": "/", "request_path": "/", "result": "/"}
2022/01/22 08:38:14.441	DEBUG	http.handlers.file_server	no index file in directory	{"path": "/", "index_filenames": ["index.html", "index.txt"]}
2022/01/22 08:38:14.441	DEBUG	http.handlers.file_server	browse enabled; listing directory contents	{"path": "/", "root": "/"}
2022/01/22 08:38:42.108	DEBUG	http.handlers.file_server	sanitized path join	{"site_root": "/", "request_path": "/home/lonble", "result": "/home/lonble"}
2022/01/22 08:38:42.108	DEBUG	http.handlers.file_server	no index file in directory	{"path": "/home/lonble", "index_filenames": ["index.html", "index.txt"]}
2022/01/22 08:38:42.108	DEBUG	http.handlers.file_server	browse enabled; listing directory contents	{"path": "/home/lonble", "root": "/"}
2022/01/22 08:38:42.108	DEBUG	http.handlers.file_server	redirecting to trailing slash to preserve hrefs	{"request_path": "/home/lonble"}
2022/01/22 08:38:42.114	DEBUG	http.handlers.file_server	sanitized path join	{"site_root": "/", "request_path": "/home/lonble/", "result": "/home/lonble/"}
2022/01/22 08:38:42.114	DEBUG	http.handlers.file_server	no index file in directory	{"path": "/home/lonble/", "index_filenames": ["index.html", "index.txt"]}
2022/01/22 08:38:42.114	DEBUG	http.handlers.file_server	browse enabled; listing directory contents	{"path": "/home/lonble/", "root": "/"}
2022/01/22 08:39:10.489	DEBUG	http.handlers.file_server	sanitized path join	{"site_root": "/", "request_path": "/home/", "result": "/home/"}
2022/01/22 08:39:10.489	DEBUG	http.handlers.file_server	no index file in directory	{"path": "/home/", "index_filenames": ["index.html", "index.txt"]}
2022/01/22 08:39:10.489	DEBUG	http.handlers.file_server	browse enabled; listing directory contents	{"path": "/home/", "root": "/"}

[lonble]

I can access / normally on my VPS, but I can't access directories such as /bin, /lib, /etc, which is strange.
All I found that I couldn't access were directories, and all the files I tried were accessible, including /etc/shadow. When I try to access a directory that does exist, caddy rewrites the uri, like adding / to the end of the directory, but then it shows a 404. I'm guessing this may be related to the implementation of --browse.

$ curl -IL 127.0.0.1/usr/include
HTTP/1.1 301 Moved Permanently
Connection: close
Content-Type: text/html; charset=utf-8
Date: Sat, 22 Jan 2022 09:25:47 GMT
Location: /usr/include/
Server: Caddy

HTTP/1.1 404 Not Found
Connection: close
Date: Sat, 22 Jan 2022 09:25:47 GMT
Server: Caddy

$ curl -IL 127.0.0.1/usr/include/stdio.h
HTTP/1.1 200 OK
Content-Length: 30002
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: text/x-chdr; charset=utf-8
Date: Sat, 22 Jan 2022 09:25:52 GMT
Etag: "qt0dn4n5e"
Keep-Alive: timeout=4
Last-Modified: Wed, 12 May 2021 19:18:40 GMT
Proxy-Connection: keep-alive
Server: Caddy

The title of this Issue doesn't reflect the problem very accurately, I'll update it.

[francislavoie]

Could you try building from the browse-errors branch and try again, making sure to have debug logging enabled?

master...browse-errors

At this point this is the 3rd report in a couple weeks about 404s from browse so I feel like there might be somekind of issue, but we haven't been able to reproduce it.

[lonble]

I built caddy from the browse-errors branch and ran the file server, everything just worked fine, nothing new was found in the log. I also built caddy from master branch and it worked fine too. But when I run the caddy packaged by archlinux based on release 2.4.6, the 404 appears again. So I downloaded the source code of version 2.4.6 from the release page and built caddy with it and I got a 404.
If you confirm this bug exists in the old release and has been fixed in the latest code, I will close this issue.

[gutenye]

For me, symbolic link cause 404, not sure if it's same as the author

reproduce steps

touch a.txt
ln -s a.txt b.txt

[lonble]

For me, symbolic link cause 404, not sure if it's same as the author

reproduce steps

touch a.txt
ln -s a.txt b.txt

For me, not related to symbolic links. Both /usr/bin/ and /bin/ are inaccessible.
My issue is only on the directory, other files are fine.

[francislavoie]

@gutenye maybe you should open a new issue and in detail show the behaviour you're seeing. Please enable the debug global option, use curl -v to make the request and share that output, and share Caddy's logs after the request. Please try with both the browse-errors branch and v2.4.6.

@lonble thanks for checking! We did make some changes since v2.4.6 was released in the fileserver package but I didn't think it was related to 404s. We've had five commits in that package since the last release. I think the last one I the most likely to have been the fix (but you could confirm by building against those specific commits to find where it was resolved):

https://github.com/caddyserver/caddy/commits/master/modules/caddyhttp/fileserver

[lonble]

After checking several other issues about reporting 404, I found the problem. As described by @bones-was-here in #4411 ,

It seems that if a symlink exists in a directory, that directory's contents can't be viewed with the browser. The error is 404. It happens with valid and broken symlinks. 2.4.5 does not have this issue.

After my testing, it is confirmed that it has been fixed in #4415 .
So I think this issue is a duplicate, thanks for your reply and help.

[dgw]

I can also confirm that a basic caddy file-server --root /path/to/folder --browse --listen 0.0.0.0:8080 returned 404 if the root contained a broken symlink, and worked perfectly once I deleted the symlink.

$ caddy version
v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=

With the (broken) symlink in place, I tried both the release caddy command and one built from master (741b050) using Go 1.17. Results were the same; v2.4.6 emitted a 404, and master showed the directory listing as expected.

When might we see the next release containing the patch/fix from #4415?

[francislavoie]

When might we see the next release?

When we've merged in all the changes we have in the v2.5.0 milestone, probably. For now, it's fine to use a build from the master branch if you need this fix right away.

[glowinthedark]

Looks like until the fix is released the workaround is to use the previous stable branch (assuming go 1.18+):

go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest

xcaddy build v2.4.5 

Another option that worked is building from the commit hash of the linked pull request (this will produce v2.4.7...):

xcaddy build c8b5a81607c9c2799f46f42154ee311884723aa4 

[Unrelated to this issue]: master (b4f1a71) throws an error (both with go v1.17.6 and v1.18)

Output of xcaddy build master:

cloud.google.com/go/kms/apiv1: ambiguous import: found package cloud.google.com/go/kms/apiv1 in multiple modules:
	cloud.google.com/go v0.83.0 (/Users/vio/projects/go/pkg/mod/cloud.google.com/[email protected]/kms/apiv1)
	cloud.google.com/go/kms v1.1.0 (/Users/vio/projects/go/pkg/mod/cloud.google.com/go/[email protected]/apiv1)

[francislavoie]

The fix is released in v2.5.0-beta.1, you can use that.