Download large files as parallel chunks using HTTP Range Requests in Go.
Chonker works on Go 1.19, oldstable, and stable releases.
Chonker speeds up downloads from cloud services like Amazon S3 & CloudFront. It does this in two ways.
- Download small pieces of a file (a.k.a a chunk) using HTTP Range requests.
- Download chunks in parallel.
Chonker allows CDN services to cache and serve files even if the entire file is bigger than the individual object cache limit.
It also overcomes the per-connection limit that blob storage services often have by opening connections in parallel.
The Go standard library HTTP Client downloads files as buffered streams of bytes. The Client fetches bytes into a request buffer as fast as it can, and you read bytes from the buffer as fast as you can.
Why is this a problem?
This works great when one beefy connection to an origin server can use the entire available network bandwidth between you and it. Blob file services like Amazon S3 and caching CDNs like Amazon CloudFront impose per-connection limits, but support an almost unlimited number of connections from each client.
If you are downloading a large file from S3, its almost always better to download the file in chunks, using parallel connections.
See the S3 Developer Guide and the CloudFront Developer Guide for more information on cache sizes and parallel GETs.
Use chonker.Do to fetch a response for a request, or create a "chonky" http.Transport
that fetches requests using HTTP Range sub-requests.
Chonker integrates well with Go download libraries.
Grab and other download managers
can use a http.Client with a "chonky" http.Transport.
In turn, Chonker functions accept HTTP clients that could provide automatic retries
or detailed logs. See
Heimdall or go-retryablehttp
for more.
Chonk is a Go program that uses the chonker library to download a URL into a local
file. Run chonk -h for usage details.
go build -o chonk ./cmd/chonk
./chonk https://example.comtest.sh is a BASH shell script that exercises the chonk program with
a list of files of varying sizes. Run test.sh -h for usage details.
The Chonker cat illustration is from Freepik
Chonker is available under the terms of the MIT license.
See LICENSE for the full license text.