Update datasets to 2.21.0

[albertvillanova]

Update datasets to 2.21.0.

Fix #3024.

[albertvillanova]

The are errors in the CI tests.

I guess at least some of them are related to the change in the Hub in relation with gated datasets:

• ConnectionError for gated datasets and unauthenticated users datasets#7109

I am updating huggingface-hub.

[albertvillanova]

I have decided to update first to datasets-2.20.0 and check all CI tests pass:

• Update datasets to 2.20.0 #3040

Afterwards, we can merge main to this PR and see the CI tests: if there is a bug in datasets-2.21.0, we could make a patch release and update to datasets-2.21.1 instead.

[albertvillanova]

I am resolving the conflicts of this PR.

I think we could merge it and then I could update to dataset 3.0 and see if something new is broken...

[albertvillanova]

The current failing test is unrelated:

ERROR tests/job_runners/split/test_descriptive_statistics.py::test_polars_struct_thread_panic_error - huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-66f560ea-0d3bd15934a45d7838ac6bb3;f1b1e4ce-2768-4f41-a5d8-f1c1c129addc)

Repository Not Found for url: https://hub-ci.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/test_polars_panic_error/resolve/main/test_polars_panic_error.parquet.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.

See:

• CI worker test_polars_struct_thread_panic_error is broken #3069

[lhoestq]

ok !