add hf_transfer to requirements.txt

[RonanKMcGovern]

This will allow users to set 'HF_HUB_ENABLE_HF_TRANSFER=True' and greatly speed up weight download speed from HF.

[WoosukKwon]

According to the introduction, it seems like hf_transfer is not at a stable stage as of now?

This library is a power user tool, to go beyond ~500MB/s on very high bandwidth network, where Python cannot cap out the available bandwidth.
This is not meant to be a general usability tool. It purposefully lacks progressbars and comes generally as-is.

Since using the library only requires installing hf_transfer and setting the env variable, I believe we can leave this as optional for users. Instead, maybe we can mention it in our docs it so that the feature is more visible. WDYT? @RonanKMcGovern

[simon-mo]

@WoosukKwon I think it is actually pretty stable. The main problem is just progress bar which is fine to ignore for product use case.

[WoosukKwon]

I think it is actually pretty stable.

@simon-mo Could you provide more background on this?

[simon-mo]

It is actively maintained and I also spent quite some time in the past benchmarking/reading its code.

[WoosukKwon]

I see. Still, my impression is that this library is experimental.

Even if we'd like to use it for vLLM, I believe we should set the env variable like os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"? Otherwise, I don't think we have to include it as vLLM's dependency.

[Yard1]

Agreed, we should not make optional packages into core requirements.

[chenxu2048]

Agreed, we should not make optional packages into core requirements.

Maybe we could add a guide in Quickstart instead of an optional package in requirements.txt.

[RonanKMcGovern]

That’s understandable on not wanting to include an unstable library. I hadn’t appreciated that. Putting it in quick start is an improvement, although it means that the docker image couldn’t be used straight up on a service like runpod. (Unless I’m missing something).

…

On Tue, Feb 27, 2024 at 3:45 AM 陈序 ***@***.***> wrote: Agreed, we should not make optional packages into core requirements. Maybe we could add a guide in Quickstart instead of an optional package in requirements.txt. — Reply to this email directly, view it on GitHub <#3031 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASVG6CQGE6GYRCEZBS4X62LYVVJD5AVCNFSM6AAAAABDZCPB6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRVG4ZTGNRZGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[RonanKMcGovern]

FWIW hf_transfer is installed in the TGI docker image

[Yard1]

Perhaps we could include it in the vllm docker image, but not in the requirements?

[RonanKMcGovern]

Perhaps we could include it in the vllm docker image, but not in the requirements?

That would fit my needs anyway.

Shall I put it under "# install build and runtime dependencies"

[simon-mo]

This would be a good place to put it

vllm/Dockerfile

Line 99 in 3b7178c

pip install accelerate

[RonanKMcGovern]

This would be a good place to put it

vllm/Dockerfile

Line 99 in 3b7178c

pip install accelerate

Thanks! I've done that now and pulled it out from where I had added it to reqs

[WoosukKwon]

Shouldn't we also add something like os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" to the dockerfile?

[simon-mo]

@WoosukKwon I think that can be optional? User can enable it by docker run -e HF_HUB_ENABLE_HF_TRANSFER=1 or setting the relevant variable in their configuration.

[WoosukKwon]

does not it require installing rust for this package to work?

@simon-mo Is this true? Otherwise, I feel we can merge the PR?

[simon-mo]

It had prebuilt package. Rust is not required.

[WoosukKwon]

@simon-mo Got it. Let's merge the PR then?

[RonanKMcGovern]

thanks for the merge, this is great

[RonanKMcGovern]

does the docker image need to be re-built to incorporate this? @simon-mo

[simon-mo]

Yes. It will be there for next release. Thank you for the patience.

[RonanKMcGovern]

Thanks @simon-mo , I think it's worth pushing asap because the docker image is much more usable for drbx if hf_transfer is there. Otherwise it takes an age to start model.