Fail to add or download anything from netfilter.org

[jsquyres]

Description

Attempting to grabit add ... any URL from netfilter.org seems to result in a 403 failure, despite the fact that I can successfully wget the same URL with no errors.

Expected Behavior

I expect grabit to be able to successfully grabit add ... each of the following URLs:

• https://www.netfilter.org/pub/libmnl/libmnl-1.0.5.tar.bz2
• https://www.netfilter.org/pub/libnfnetlink/libnfnetlink-1.0.2.tar.bz2
• https://www.netfilter.org/pub/conntrack-tools/conntrack-tools-1.4.8.tar.xz

Actual Behavior

grabit returns a 403 error, and fails to add an entry to the lockfile. Even if I manually add a proper entry to the lockfile and grabit download, grabit returns the same 403 error and fails to download the file.

Affected Version

The tip of main.

Steps to Reproduce

$ grabit add https://www.netfilter.org/pub/libmnl/libmnl-1.0.5.tar.bz2
11:00PM FTL failed to get url: failed to download 'https://www.netfilter.org/pub/libmnl/libmnl-1.0.5.tar.bz2': ErrValidator: response error for https://www.netfilter.org/pub/libmnl/libmnl-1.0.5.tar.bz2: unexpected status: 403

However:

$ wget https://www.netfilter.org/pub/libmnl/libmnl-1.0.5.tar.bz2
--2024-05-06 23:00:30--  https://www.netfilter.org/pub/libmnl/libmnl-1.0.5.tar.bz2
Resolving www.netfilter.org (www.netfilter.org)... 92.243.18.11, 2001:4b98:dc0:43:216:3eff:fe87:a456
Connecting to www.netfilter.org (www.netfilter.org)|92.243.18.11|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 314791 (307K) [application/x-bzip2]
Saving to: ‘libmnl-1.0.5.tar.bz2’

libmnl-1.0.5.tar.bz2                100%[==================================================================>] 307.41K   491KB/s    in 0.6s    

2024-05-06 23:00:38 (491 KB/s) - ‘libmnl-1.0.5.tar.bz2’ saved [314791/314791]

Checklist

• I have read the contributing guidelines
• I have verified this does not duplicate an existing issue

[jsquyres]

The code is failing in the requests call to download the file:

grabit/internal/resource.go

Lines 52 to 55 in afaa354

	err = requests.
	URL(u).
	ToFile(fileName).
	Fetch(ctx)

If I add the following, it all works -- but this is clearly not the Right solution:

        err = requests.
                URL(u).
+               AddValidator(nil).
                ToFile(fileName).
                Fetch(ctx)

I've run out of time to investigate further at the moment; filing this issue in the hope that someone else recognizes what's happening here.

[rabadin]

Interestingly, it appears that netfilter.org is quite picky about the request headers:

curl -v -H 'Accept:' -o /dev/null https://netfilter.org/projects/libnftnl/files/libnftnl-1.2.5.tar.xz
[...]
< HTTP/1.1 403 Forbidden
[...]

Or even:

# Setting the user-agent to empty
curl -v -A '' -o /dev/null https://netfilter.org/projects/libnftnl/files/libnftnl-1.2.5.tar.xz[...]
< HTTP/1.1 403 Forbidden
[...]