How to connect Huggingface datasets as a dvc remote

[osbm]

I need to share my data with my friends for a small project. I wanted to use Huggingface datasets for storage, and DVC for versioning. Is this even possible?

I know it basically becomes storing data in another git repository but the huggingface has the storage capacity for our needs. :)

[daavoo]

Interesting question @osbm !

If I understand correctly, you are referring to datasets hosted on the Huggingface Hub, right?

I am asking because the datasets library also supports access to custom cloud storage

[osbm]

yes, i am talking about datasets hosted on Huggingface hub.

Thank you for your reply :)

[daavoo]

Got it.

If I remember correctly, the Hub relies on git-lfs for handling the big files.

So, I think it might not be straightforward to plug in a DVC remote on top of the Hub unless either DVC and/or Hub work on deeper integration. Usually, DVC is used as an alternative to git-lfs (https://dvc.org/doc/user-guide/related-technologies#git-lfs-large-file-storage)

Take this with a grain of salt, as I might be missing something. I would need to take a closer look at how the Hub works to give a proper answer.

[osbm]

Thank you for your answer, i think integration between huggingface and DVC would be awesome.