Mirror update breaks if optional content-length header is missing for files with 0 size

[norg]

If you run mirror update against a destination that provides files with 0 size and don't use the optional content-length header, aptly stops the update.

Detailed Description

This can quite easily be reproduced:

aptly mirror create  -force-architectures -with-udebs -with-installer -with-sources bullseye-bugtest https://debian.inf.tu-dresden.de/debian bullseye main contrib non-free
aptly mirror update bullseye-bugtest
Downloading https://debian.inf.tu-dresden.de/debian/dists/bullseye/Release...
Downloading & parsing package files...
Downloading https://debian.inf.tu-dresden.de/debian/dists/bullseye/main/binary-all/Packages.gz...
Downloading https://debian.inf.tu-dresden.de/debian/dists/bullseye/main/debian-installer/binary-all/Packages.gz...
Downloading https://debian.inf.tu-dresden.de/debian/dists/bullseye/main/installer-all/current/images/SHA256SUMS...
Downloading https://debian.inf.tu-dresden.de/debian/dists/bullseye/main/binary-amd64/Packages.gz...
Downloading https://debian.inf.tu-dresden.de/debian/dists/bullseye/main/debian-installer/binary-amd64/Packages.gz...
Downloading https://debian.inf.tu-dresden.de/debian/dists/bullseye/main/installer-amd64/current/images/SHA256SUMS...
ERROR: unable to update: could not determine length of https://debian.inf.tu-dresden.de/debian/dists/bullseye/main/installer-amd64/current/images/netboot/debian-installer/amd64/grub/x86_64-efi/fdt.lst

While one workaround is to use another mirror that uses nginx instead of apache, I would still argue that it should be fixed within aptly. The header is only optional, see https://www.rfc-editor.org/rfc/rfc7231#section-4.3.2 for example.
Why is a mirror specifially used in the first place? Because the ftp.debian.org just redirects to different mirrors and thus the tls check fails.

Context

Since this breaks syncing to some mirrors it's a fix that should help several users.

Possible Implementation

In the code one could set the content-length to default 0 if the header is missing and no other indicator for the length is provided. Since the code works for all other files, that should be okay or mabye as an option in the config "skip content lenght check" or so.

The part of the code would be https://github.com/aptly-dev/aptly/blob/master/http/download.go#L113 where the check could improved.
I tried a test with setting the default to 0 if the return value was below 0 in that line and this solved it for me in a test environment.

Your Environment

Current 1.4.0 aplty release on debian but also archlinux can reproduce it.

[ron-slc]

Hello, Andi (@norg)

I can confirm this Aptly issue report, for files served by APACHE repo servers. Nginx repo servers are indeed successful.

Confirmed on: Debian 10 and Ubuntu 20.04, building the current 1.4.0 aptly.

As an easy work-around I have updated my aptly mirror URL to a repository served by Nginx.

I am relatively new to GO/golang - so I am not confident in supplying a patch which digs into response headers, to conclude if a file is truly 0 size versus a download communication error, or missing file, etc..

So effectively - I am confirming and bumping this issue report.