Download only as needed

[zontreck]

This PR adds a small change that makes prefill do a HEAD request, and check if X-Cache-Status is set to HIT. If it is, then it can safely skip and move onto the next chunk. Avoids unnecessary downloads from the cache server.

[zontreck]

This is useful, for instance, when a download is interrupted. As some modern games can be as large as 100+ GB, and while steam has many, many chunks and URLs to check, this still speeds up the process of resuming the download, or skipping games that the cache already has.

[tpill90]

Thanks for the pr. I'll probably be able to take a look at this in the next few days or so. Will let you know any feedback then

[zontreck]

After some updates have released, i have confirmed my patch holds up to those as well, only downloading the changed stuff, and skipping by the rest that the cache already has. Hope this makes it in.

[tpill90]

I've finally been table to take some time to check this out, and test it out a bit. It does work and certainly speeds up things by a lot! However I have two reservations about including these changes:

1. This change breaks the benchmark command's functionality. Since clients would no longer be pulling the actual download over LAN, then running a benchmark as a client will produce incorrect results. Running this as a client on my desktop achieved speeds of over 450gbit/s which would never be possible by a single client. Keeping the benchmark functionality is very important as it is depended on by the community to identify bottlenecks in their setups.

2. I am also a bit cautious to change the download behavior to be different than what the Steam client itself does. SteamPrefill has always been intended to match the real client's behavior as much as possible, so that there can be the confidence that downloading via Steam/SteamPrefill will have the exact same result.

[zontreck]

Hmm perhaps put this behind a command line toggle then to not break the default behavior of other commands?

[zontreck]

I can modify this to put it behind a command line toggle, or just fix it so that benchmark and other such commands will not utilize the HEAD request check.

[tpill90]

Are you on discord? Would you maybe be interested in joining the lancache discord and chatting about this a bit there?

[zontreck]

Yea i'm on discord.

[zontreck]

I updated the files in the PR. I added a command line toggle to turn on skipping chunks that have already been downloaded. Updated it to be compatible with upstream/master. By default, this will use the current behavior of not skipping chunks.

I wasn't sure where the best place for this toggle would've been. Nor was i sure what the best nickname for the flag was.