Add British Library books dataset

[davanstrien]

This pull request adds a dataset of text from digitised (primarily 19th Century) books from the British Library. This collection has previously been used for training language models, e.g. https://github.com/dbmdz/clef-hipe/blob/main/hlms.md. It would be nice to make this dataset more accessible for others to use through datasets.

This is still a WIP but I wanted to get some initial feedback in particular; I wanted to check:

• I am handling the use of iter_archive correctly - I intend to ensure that dl_manager.download gets the complete list of URLs to download upfront, so the progress bar knows how much is left to download and then to pass through the gen_kwargs a list of downloaded zip archives wrapped in iter_archive. I am unsure if there is a more elegant approach for this?
• the number of configs: I have aimed to keep this limited - there are a lot of URLs covering the entire dataset, but I have tried to base the configs on what I believe the majority of people will want to they are not presented with too many options - I am happy to hear suggestions for changing this

If there are other glaring omissions or mistakes, I'd be happy to hear them. If this approach seems sensible in general, I will finish all the remaining TODOs, generate dummy_data, etc.

[lhoestq]

Nice ! Thanks a lot for adding this dataset :)

The dataset card is awesome ! And you did well regarding iter_archive and the configurations, feel free to continue in this direction !

[lhoestq]

Awesome thank you !

The only thing missing are the dummy data we use to test the dataset script regularly. You can use the datasets-cli dummy_data ./datasets/blbooks command to have the instructions to generate them.

Since the dataset has a very specific structure it might not be that easy so feel free to ping me if you have questions or if I can help !

[davanstrien]

Thanks for all the help and suggestions

Since the dataset has a very specific structure it might not be that easy so feel free to ping me if you have questions or if I can help !

I did get a little stuck here! So far I have created directories for each config i.e:

datasets/datasets/blbooks/dummy/1700_1799/1.0.2/dummy_data.zip

I have then added two examples of the jsonl.gz files that are in the underlying dataset to each dummy_data directory.This fails the test using local files.

Since

def _generate_examples(self, data_dirs):

takes as input data_dirs which is a list of iter_dirs do I need to put the dummy files inside another directory? i.e.

datasets/datasets/blbooks/dummy/1700_1799/1.0.2/dummy_data/1700/00.jsonl.gz

[lhoestq]

I think I managed to create the dummy data :)

I think everything is good now, if you don't have other changes to do, please mark your PR as "ready for review" and ping me!

[davanstrien]

I think I managed to create the dummy data :)

Thanks so much for that!

I think everything is good now, if you don't have other changes to do, please mark your PR as "ready for review" and ping me!

Think it is ready to merge from my end @lhoestq.

[lhoestq]

Nice thanks :)

[lhoestq]

The CI failure on windows is unrelated to your PR and fixed on master, we can ignore it