Download file with redirect using unidici stream issue

[nocquidant]

Hi, i can't download https://download.geofabrik.de/europe-latest.osm.pbf using unidici stream.
I am not quite sure if it is an issue with unidici, but curl has no problem with it...

Even if the file is 29.1GB, it fails immediately with the following error.

Response received {
  statusCode: 404,
  headers: {
    date: 'Wed, 02 Oct 2024 17:37:22 GMT',
    server: 'Apache',
    'content-length': '196',
    'content-type': 'text/html; charset=iso-8859-1',
    'x-cache': 'MISS from download-proxy7',
    'x-cache-lookup': 'MISS from download-proxy7:80',
    via: '1.1 download-proxy7 (squid/5.9)',
    connection: 'close'
  }
}

The code is below:

// 4.3 GB -> https://download.geofabrik.de/europe/france-latest.osm.pbf
// 29.1GB -> https://download.geofabrik.de/europe-latest.osm.pbf
const sourceFailure = "https://download.geofabrik.de/europe-latest.osm.pbf";
const sourceSuccess = "https://download.geofabrik.de/europe/france-latest.osm.pbf";

const source = sourceFailure;

describe("Unit tests for big file download", () => {
	it("fetches successfully", async () => {
		const sourceURL = new URL(source);
		const highWaterMark = 1024 * 1024 * 1;

		// set up stream
		const passthrough = new PassThrough({ highWaterMark });

		// calculate fileSize in bytes
		let fileSize = 0;
		passthrough.on("data", (chunk) => {
			fileSize += chunk.byteLength;
		});

		// Abort requests after 15 minutes
		const timeout = 15 * 60 * 1000;
		const abortTimeout = AbortSignal.timeout(timeout);

		const httpClient = new undici.Client(sourceURL.origin);
		const requestStream = httpClient.stream(
			{
				path: sourceURL.pathname,
				method: "GET",
				opaque: passthrough,
				signal: abortTimeout,
				maxRedirections: 5,
				// throwOnError: false,
				// reset: true,
				highWaterMark,
			},
			({ statusCode, headers }) => {
				console.log("Response received", { statusCode, headers });
				if (statusCode !== 200) {
					throw new Error(`Failed to download ${source}, status: ${statusCode}`);
				}
				return passthrough;
			},
		);

		await Promise.all([
			requestStream.catch(async (err) => {
				throw err;
			}),
		]);

		await httpClient.close();

		console.log(`Read file of size: ${fileSize}`);
	});
});

Note that I have no problem with https://download.geofabrik.de/europe/france-latest.osm.pbf.

Could it be a unidici option that I missed?

Thans for your help!
--nick

[nocquidant]

Maybe it's because there is a redirect (even though I have 404?).
But i thougth unidici would follow the redirect with the option: maxRedirections: 5?

curl -L -r 0-99 -v https://download.geofabrik.de/europe-latest.osm.pbf -o europe-latest.osm.pbf
[...]
* Connection #0 to host download.geofabrik.de left intact
* Issue another request to this URL: 'https://ftp5.gwdg.de/pub/misc/openstreetmap/download.geofabrik.de/europe-latest.osm.pbf'
*   Trying 134.76.12.6:443...
* Connected to ftp5.gwdg.de (134.76.12.6) port 443 (#1)
[...]

[metcoder95]

Not really, sure; I cannot see a redirect here and using the code you share returns 200 for me:

Response received {
  statusCode: 200,
  headers: {
    'content-length': '4645925613',
    'content-type': 'application/octet-stream',
    date: 'Thu, 03 Oct 2024 06:42:56 GMT',
    server: 'Apache',
    'last-modified': 'Thu, 03 Oct 2024 02:15:41 GMT',
    etag: '"114eb32ed-6238921926d40"',
    'accept-ranges': 'bytes',
    age: '0',
    'x-cache': 'HIT from download-proxy8',
    'x-cache-lookup': 'HIT from download-proxy8:80',
    via: '1.1 download-proxy8 (squid/5.9)',
    connection: 'keep-alive'
  }
}

[nocquidant]

Thank you for your response.
Yes, everything is working fine this morning, and curl no longer displays Issue another request to this URL: ... again...

So trying to go forward, is my assumption correct: with the "maxRedirections" option set, the redirections should be followed?
Also, am I missing an optimization option for downloading very large files?
Thanks again!

[nocquidant]

404 error reappears...

Response received {
  statusCode: 404,
  headers: {
    date: 'Thu, 03 Oct 2024 09:10:57 GMT',
    server: 'Apache',
    'content-length': '196',
    'content-type': 'text/html; charset=iso-8859-1',
    'x-cache': 'MISS from download-proxy7',
    'x-cache-lookup': 'MISS from download-proxy7:80',
    via: '1.1 download-proxy7 (squid/5.9)',
    connection: 'keep-alive'
  }
}

[metcoder95]

So trying to go forward, is my assumption correct: with the "maxRedirections" option set, the redirections should be followed?

Yes, it would.

404 error reappears...

This seems to me mostly a server issue; it could be with some network/proxy checks made by the origin. Don't believe the issue comes from undici

[joelschou]

I'm not sure if what I've discovered is related (I'm in the process of searching before filing a bug), but it seems the redirect-handler has a bug.

In my situation, I'm seeing a bunch of headers passed to the original request disappear from the redirected request. I see examples of Response headers in your comments, @nocquidant, but have you dug into the Request headers across the redirection(s)?

Here's my bug filing: #3773