Feature request: Enable activation of hf_transfer if available.

[michaelfeil]

A suggestion is to accelerate the usage of from huggingface_hub import snapshot_download by using pip install hf_transfer https://github.com/huggingface/hf_transfer

hf_transfer is a pretty lightweight binary, that uses rust to download files where python is compule / gil bound while downloading files.

Benefits: download from 200Mbit -> 2GBit (my experience in gcp, us-central) + nice to have if you have to do multiple concurrent downloads (lora)

try:
    # enable hf hub transfer if available
    import hf_transfer  # type: ignore # noqa
    import huggingface_hub.constants  # type: ignore

    # can also be set  via env  var HF_HUB_ENABLE_HF_TRANSFER 
    # I would not suggest doing so, as its unclear if any venv will have 
    # pip install hf_transfer
    huggingface_hub.constants.HF_HUB_ENABLE_HF_TRANSFER = True
except ImportError:
    pass

Let me know if this proposal is acceptable, if so, I can open a PR.

[robertgshaw2-neuralmagic]

Are there any negatives to hf_transfer over snapshot_download?

[michaelfeil]

https://github.com/huggingface/huggingface_hub/blob/f386b2ae74bf18443836936941ae8bd1bfd40903/docs/source/en/guides/download.md?plain=1#L191

@rib-2 Its a feature flag of snapshot_download

Faster downloads

If you are running on a machine with high bandwidth,
you can increase your download speed with hf_transfer,
a Rust-based library developed to speed up file transfers with the Hub.
To enable it:

1. Specify the hf_transfer extra when installing huggingface_hub
   (e.g. pip install huggingface_hub[hf_transfer]).
2. Set HF_HUB_ENABLE_HF_TRANSFER=1 as an environment variable.

hf_transfer is a power user tool!
It is tested and production-ready,
but it lacks user-friendly features like advanced error handling or proxies.
For more details, please take a look at this section.

[michaelfeil]

@robertgshaw2-neuralmagic Should I open a PR?

[michaelfeil]

Update, hf_transfer has been added to #3031 #3008 - does it make sense to use it if available? Using the hook above?