2.3.1

Bug fixes

• Fix patching module that doesn't exist by @lhoestq in #4495
  • fix bug when importing the lib when scipy is not installed
• Re-add download_manager module in utils by @lhoestq in #4497
  • fix moved imports of DownloadConfig, DownloadMode, DownloadManager
• Support streaming UDHR dataset by @albertvillanova in #4487

Full Changelog: 2.3.0...2.3.1

2.3.0

Datasets Changes

• New: ImageNet-Sketch by @nateraw in #4301
• New: Biwi Kinect Head Pose by @dnaveenr in #3903
• New: enwik8 by @HallerPatrick in #4321
• New: LCCC dataset by @silverriver in #4416
• New: TruthfulQA by @jon-tow in #4159
• New: BIG-bench by @andersjohanandreassen in #4125
• New: QuickDraw by @mariosasko in #3592
• New: SST-2 by @albertvillanova in #4473
• Update: imagenet-1k - remove manual download by @mariosasko in #4299
  • ImageNet can now be loaded in python with load_dataset without requiring a manual download !
  • It also supports streaming mode with load_dataset("imagenet-1k", streaming=True)
• Update: spider - Remove Google Drive URL by @albertvillanova in #4410
• Update: blended_skill_talk - add missing columns to by @mariosasko in #4437
• Update: multi-news - Use newer version with fixes by @JohnGiorgi in #4451
• Update: fever - update data URLs by @albertvillanova in https://github.com/huggingface/datasets/pull/44554459
• Update: udhr - Add and fix language tags by @albertvillanova in https://github.com/huggingface/datasets/pull/
• Update: udhr - update metadata by @leondz in #4362
• Update: wider_face - Replace data URLs once hosted on the Hub by @albertvillanova in #4469
• Update: PASS - update dataset version by @mariosasko in #4488
• Fix: GEM - fix bug in wiki_auto_asset_turk config by @albertvillanova in #4389
• Fix: GEM - fix URL for totto config by @albertvillanova in #4396
• Fix: timit_asr - fix DuplicatedKeysError by @albertvillanova in #4424
• Fix: timit_asr - Make extensions case-insensitive by @albertvillanova in #4425
• Fix: timit_asr - Fix directory names for LDC data by @albertvillanova in #4436
• Fix: iwslt2017 by @lhoestq in #4481

Dataset Features

• to_tf_dataset rewrite by @Rocketknight1 in #4170
  • see more in the documentation
• Support DataLoader with num_workers > 0 in streaming mode by @lhoestq in #4375
  • see more in the documentation
• Added stratify option to train_test_split by @nandwalritik in #4322
• Re-add support for Apache Beam functionality by @albertvillanova in #4328
• Resume push_to_hub: skip identical files in push_to_hub instead of overwriting by @mariosasko in #4402
• Support nested/complex feature types as features in packaged loaders by @mariosasko in #4364
• Optimize contiguous shard and select by @lhoestq in #4466

Dataset Cards

• Minor fixes/improvements in scene_parse_150 card by @mariosasko in #4447
• Tidy up license metadata for google_wellformed_query, newspop, sick by @leondz in #4378
• Fix example in opus_ubuntu, Add license info by @leondz in #4360
• Update README.md of fquad by @lhoestq in #4450

Documentation

• Add API code examples for loading methods by @stevhliu in #4300
• Add API code examples for remaining main classes by @stevhliu in #4292
• Generalize tutorials for audio and vision by @stevhliu in #4468
• [Docs] How to use with PyTorch page by @lhoestq in #4474
• First draft of the docs for TF + Datasets by @Rocketknight1 in #4457

Other improvements and bug fixes

• Update CI deprecated legacy image by @albertvillanova in #4393
• remove int documentation from logging docs by @lvwerra in #4392
• Fix docstring in DatasetDict::shuffle by @felixdivo in #4344
• Fix Version equality by @albertvillanova in #4359
• Set builder name from module instead of class by @albertvillanova in #4388
• Test dill by @albertvillanova in #4385
• Refactor download by @albertvillanova in #4384
• Fix dependency on dill version by @albertvillanova in #4397
• Support remote cache_dir by @albertvillanova in #4347
• Update imagenet gate by @lhoestq in #4408
• Fix dataset builder default version by @albertvillanova in #4356
• Uncomment logging deactivation for ArrowBasedBuilder by @thomasw21 in #4403
• Rename DatasetBuilder config_name by @albertvillanova in #4414
• Fix metadata validation by @albertvillanova in #4390
• Add HF.co for PRs/Issues for specific datasets by @lhoestq in #4427
• Fix type hint and documentation for new_fingerprint by @fxmarty in #4326
• Skip hidden files/directories in data files resolution and iter_files by @mariosasko in #4412
• Fix docstring of inspect_dataset by @albertvillanova in #4438
• Fix builder docstring by @albertvillanova in #4432
• Fix kwargs in docstrings by @albertvillanova in #4444
• Fix missing args in docstring of load_dataset_builder by @albertvillanova in #4445
• Add missing kwargs to docstrings by @albertvillanova in #4446
• Add extractor for bzip2-compressed files by @asivokon in #4421
• Fix dummy dataset generation script for handling nested types of _URLs by @silverriver in #4434
• Update dataset_infos.json with new split info in Dataset.push_to_hub to avoid verification error by @mariosasko in #4415
• Update builder docstring for deprecated/added arguments by @albertvillanova in #4429
• Extend support for streaming datasets that use xml.dom.minidom.parse by @albertvillanova in #4464
• Fix script fetching and local path handling in inspect_dataset and inspect_metric by @mariosasko in #4433
• Fix bigbench config names by @lhoestq in #4465
• Fix 401 error for unauthticated requests to non-existing repos by @lhoestq in #4472
• Reorder returned validation/test splits in script template by @albertvillanova in #4470
• Better ImportError message when a dataset script dependency is missing by @lhoestq in #4484
• Fix cast to null by @lhoestq in #4485
• Update _format_columns in remove_columns by @alvarobartt in #4411
• Fix wrong map parameter name in cache docs by @h4iku in #4293
• Pin the revision in imagenet download links by @lhoestq in #4492
• Refactor column mappings for question answering datasets by @lewtun in #4391

New Contributors

• @leondz made their first contribution in #4378
• @felixdivo made their first contribution in #4344
• @nandwalritik made their first contribution in #4322
• @fxmarty made their first contribution in #4326
• @HallerPatrick made their first contribution in #4321
• @silverriver made their first contribution in #4416
• @asivokon made their first contribution in #4421
• @andersjohanandreassen made their first contribution in #4125

Full Changelog: https://git...

2.2.2

Datasets fixes

• Fix: irc_disentangle - fix checksum and bug dataset by @albertvillanova in #4377
• Fix: CC-Aligned - fix invalid url by @juntang-zhuang in #4231
• Fix: multi_news - don't strip proceeding hyphen by @JohnGiorgi in #4353

Bug fixes

• Support lists of multi-dimensional numpy arrays by @albertvillanova in #4194
• Check if dataset features match before push in DatasetDict.push_to_hub by @mariosasko in #4372
• Pin dill by @albertvillanova in #4380
  • dill 0.3.5 has some issues in transformers - pinning the version to <0.3.5 for now

Dataset Cards

• Adding eval metadata for ade v2 by @sashavor in #4319
• Adding eval metadata for AG News by @sashavor in #4329
• Adding eval metadata to Allociné dataset by @sashavor in #4330
• Adding eval metadata to Amazon Polarity by @sashavor in #4331
• Adding eval metadata for arabic speech corpus by @sashavor in #4332
• Adding eval metadata for Banking 77 by @sashavor in #4333
• Eval metadata Batch 4: Tweet Eval, Tweets Hate Speech Detection, VCTK, Weibo NER, Wisesight Sentiment, XSum, Yahoo Answers Topics, Yelp Polarity, Yelp Review Full by @sashavor in #4338
• Eval metadata batch 3: Reddit, Rotten Tomatoes, SemEval 2010, Sentiment 140, SMS Spam, Snips, SQuAD, SQuAD v2, Timit ASR by @sashavor in #4337
• Eval metadata batch 1: BillSum, CoNLL2003, CoNLLPP, CUAD, Emotion, GigaWord, GLUE, Hate Speech 18, Hate Speech by @sashavor in #4335
• Eval metadata batch 2 : Health Fact, Jigsaw Toxicity, LIAR, LJ Speech, MSRA NER, Multi News, NCBI Disease, Poem Sentiment by @sashavor in #4336

Docs

• Add API code examples for Builder classes by @stevhliu in #4313
• Add redirect to dataset script in the repo structure page by @lhoestq in #4369

Other improvements and bug fixes

• Fix failing CI on Windows for sari and wiki_split metrics by @albertvillanova in #4342
• Fix never ending GH Action to build documentation by @albertvillanova in #4345
• Fix warning in upload_file by @albertvillanova in #4355
• Fix warning in push_to_hub by @albertvillanova in #4357
• Remove config names as yaml keys by @lhoestq in #4367
• Add missing language tags for udhr dataset by @albertvillanova in #4371
• Remove links in docs to old dataset viewer by @mariosasko in #4373

New Contributors

• @JohnGiorgi made their first contribution in #4353
• @juntang-zhuang made their first contribution in #4231

Full Changelog: 2.2.1...2.2.2

2.2.1

Datasets bug fixes

• Fix cnn_dailymail (dm stories were ignored) by @lhoestq in #4317
  • datasets 2.2.0 introduced a bug in cnn_dailymail and some examples were missing in the dataset

General improvements and bug fixes

• Fix: Add missing comma by @mrm8488 in #4303
• Catch pull error when mirroring by @lhoestq in #4314
• Remove unused multiprocessing args from test CLI by @albertvillanova in #4308
• Fix CLI run_beam namespace by @albertvillanova in #4315
• Support passing config_kwargs to CLI run_beam by @albertvillanova in #4316
• Don't check f.loc in _get_extraction_protocol_with_magic_number by @lhoestq in #4318

New Contributors

• @mrm8488 made their first contribution in #4303

Full Changelog: 2.2.0...2.2.1

2.2.0

Dataset Changes

• New: ImageNet by @apsdehal in #4178
  • Manual download only for now
• New: Google Conceptual Captions by @abhishekkrthakur in #1459
• New: Conceptual 12M by @thomasw21 in #4162
• New: Visual Genome by @thomasw21 in #4161
• New: RVL-CDIP by @dnaveenr in #4050
• New: Text-based NP Enrichment (TNE) by @yanaiela in #4153
• New: TextVQA by @apsdehal in #3967
• New: ETT time series dataset by @kashif in #4213
• Update: assin2 - update metadata by @lhoestq in #4172
• Update: Librispeech - Add 'all' config by @patrickvonplaten in #4184
• Update: XGLUE - Support streaming dataset by @albertvillanova in #4249
• Update: crd3 - group all the turns in one example by @shanyas10 in #4240
• Update: pubmed_qa - Remove google drive URL by @lhoestq in #4255
• Update: SAMSum - Replace data URL dataset and support streaming by @albertvillanova in #4254
• Update: SAMSum - Replace data URL dataset within the same repository by @albertvillanova in #4267
• Update: big_patent - Replace data URL in dataset and support streaming by @albertvillanova in #4236
• Update: openbookqa - Add missing features for additional config by @albertvillanova in #4278
• Update: commonsense_qa - Add missing features by @albertvillanova in #4280
• Fix: Common Voice - Make sure bytes are correctly deleted if path exists by @patrickvonplaten in #4212
• Fix: openbookqa - fix bug in choices labels by @manandey in #4259
• Fix: openbookqa - fix style in openbookqa dataset by @albertvillanova in #4270

Dataset Features

• Add support for metadata files to imagefolder by @mariosasko in #4069
  • load a folder of images and metadata stored in metadata.jsonl, more info in the documentation on how to load an image dataset
• Infer splits from the data_dir parameter when loading datasets without script by @polinaeterna in #4144
  • splits are inferred from the directory and file names, see more info in the documentation on how to structure your repository
• Enable label alignment for token classification datasets by @lewtun in #4277
• Add drop_last_batch to IterableDataset.map by @mariosasko in #4215
• Load dataset with TSV files by @albertvillanova in #4246

Dataset Cards

• Autoeval config by @NRajani in #4234
  • Add train-deval-index metadata to automate evaluation on your datasets based on their tasks
• Adding license information for Openbookcorpus by @meg-huggingface in #3525
• Make code for image downloading from image urls cacheable by @mariosasko in #4218
• Fix description links in dataset cards by @albertvillanova in #4222
• Add YAML tags to Dataset Card rotten tomatoes by @mo6zes in #4262
• Remove a copy-paste sentence in dataset cards by @albertvillanova in #4281
• Update LexGLUE README.md by @iliaschalkidis in #4285
• leadboard info added for TNE by @yanaiela in #4273
• Add Lahnda language tag by @mariosasko in #4286
• Add license and point of contact to big_patent dataset by @albertvillanova in #4269
• Add HF Speech Bench to Librispeech Dataset Card by @sanchit-gandhi in #4266

Metrics Changes

• Perplexity Speedup by @emibaylor in #4108
• Add AUC ROC Metric by @emibaylor in #4158
• Small fixes in ROC AUC docs by @wschella in #4239
• Fix/start token mask issue and update documentation by @TristanThrush in #4258
• Add pearsonr mc, update functionality to match the original docs by @emibaylor in #4226

Metric Cards

• Metric card for the XTREME-S dataset by @sashavor in #4251
• Creating metric card for MAE by @sashavor in #4252
• Create metric cards for mean IOU by @sashavor in #4253
• Create metric card for Mahalanobis Distance by @sashavor in #4257
• Create metric card for MSE by @sashavor in #4256
• Fix exact match by @emibaylor in #4166
• Fix google bleu typos, examples by @emibaylor in #4165
• Add f1 metric card, update docstring in py file by @emibaylor in #4227
• Add Recall Metric Card by @emibaylor in #4204
• Matthews Correlation Metric Card by @emibaylor in #4110
• Add Precision Metric Card by @emibaylor in #4203
• Add Accuracy Metric Card by @emibaylor in #4223
• Add Spearmanr Metric Card by @emibaylor in #4109
• Metric card template by @emibaylor in #3915

Documentation

• Document save_to_disk and push_to_hub on images and audio files by @lhoestq in #4193
• Add to docs how to load from local script by @albertvillanova in #4200
• Add code examples to API docs by @stevhliu in #4168
• Add code examples for DatasetDict by @stevhliu in #4245
• Add API code examples for IterableDataset by @stevhliu in #4274
• Add packaged builder configs to the documentation by @lhoestq in #4307
• [Imagefolder] Docs + Don't infer labels from file names when there are metadata + Error messages when metadata and images aren't linked correctly by @lhoestq in #4311

General improvements and bug fixes

• Generate tasks.json taxonomy from huggingface_hub by @julien-c in #4154
• Fix when map function modifies input in-place by @thomasw21 in #4174
• Support streaming cnn_dailymail dataset by @albertvillanova in #4188
• Don't duplicate data when encoding audio or image by @lhoestq in #4187
• Fix outdated docstring about default dataset config by @lhoestq in #4186
• Deprecate shard_size in push_to_hub in favor of max_shard_size by @mariosasko in #4190
• Fix some type annotation in doc by @thomasw21 in #4202
• Update GH template for dataset viewer issues by @albertvillanova in #4201
• Update auth when mirroring datasets on the hub by @lhoestq in #4242
• Rename imagenet2012 -> imagenet-1k by @lhoestq in #4263
• Skip checksum computation in Imagefolder by default by @mariosasko in #4214
• Fix convert_file_size_to_int for kilobits and megabits by @mariosasko in #4205
• Fix typo in logging docs by @stevhliu in #4272
• Bump PyArrow Version to 6 by @dnaveenr in #4250
• task id update by @NRajani in #4244
• Avoid recursion error in map if example is returned as dict value by @mariosasko in #4216
• Update minimal PyArrow version warning by @mariosasko in #4279
• [Minor edit] Fix typo in class name by @cakiki in #4207
• Stream private zipped images by @lhoestq in #4173
• Fix filesystem docstring by @stevhliu in https://github.com/huggingface/datasets/pul...

2.1.0

Datasets Changes

• New: initial monash time series forecasting by @kashif in #3743
• New: Roman Urdu Hate Speech dataset by @bp-high in #3972
• New: Adversarial GLUE by @jxmorris12 in #3849
• New: MetaShift by @dnaveenr in #3900
• New: GSM8K by @jon-tow in #4103
• New: SBU Captions Photo by @thomasw21 in #4130
• Deprecated: Multilingual Librispeech - deprecate dataset in favor of facebook/multilingual_librispeechby @polinaeterna in #4060
• Update (BREAKING): TIMIT - Redirect users to download data manually from LDC by @lhoestq in #4145
• Update: Wikipedia by @albertvillanova in #3821 and #3989
• Update: conll2012_ontonotesv5 - Support streaming by @albertvillanova in #4002
• Update: daily_dialog - Support streaming by @albertvillanova in #4008
• Update: id_clickbait - Support streaming by @albertvillanova in #4014
• Update: blimp - Support streaming by @albertvillanova in #4016
• Update: scan - Support streaming by @albertvillanova in #4017
• Update: yelp_review_full - Replace data url by @lhoestq in #4018
• Update: yelp_polarity - Support streaming by @lhoestq in #4019
• Update: amazon_polarity - Replace data URL by @lhoestq in #4020
• Update: dbpedia_14 - Replace data url by @lhoestq in #4022
• Update: xtreme - Support streaming dataset for bucc18 config by @albertvillanova in #4026
• Update: yahoo_answers_topics - Replace data url by @lhoestq in #4023* Update: ASSIN 2 dataset - replace broken Google Drive URLS by links on github by @ruanchaves in #4004
• Update: xcopa - Support streaming by @albertvillanova in #4039
• Update: medical_dialog - Add configs with processed data by @albertvillanova in #4127
• Update: xtreme - Support streaming for udpos config by @albertvillanova in #4131
• Update: xtreme - Support streaming for PAWS-X config by @albertvillanova in #4132
• Update: xtreme - Support streaming for PAN-X config by @albertvillanova in #4135
• Update: SQuAD v2 - Use a constant for the articles regex by @bryant1410 in #4030
• Update: HANS - Support streaming by @mariosasko in #4155
• Fix: cats_vs_dogs - fix checksum error dataset by @albertvillanova in #4033
• Fix: xcopa - fix null checksum by @albertvillanova in #4034
• Fix: amazon_us_reviews - fix metadata - 4/4/2022 by @trentonstrong in #4092

Dataset Cards

• Updated annotations for nli_tr dataset by @e-budur in #4058
• Add missing label for emotion description by @lijiazheng99 in #4151
• Remove unncessary 'pylint disable' message in ReadMe by @Datta0 in #3955
• Improve RedCaps dataset card by @mariosasko in #4100
• Fix duplicate key in multi_news by @lhoestq in #4164

Datasets Tags and Search on the Hugging Face Hub

• Tasks alignment with models by @lhoestq in #4066
• Update datasets task tags to align tags with models by @lhoestq in #4067

Metrics Changes

• Xtreme-S Metrics by @patrickvonplaten in #3799
• Fix xtreme s metrics by @patrickvonplaten in #3957
• Avoid info log messages from transformers in FrugalScore metric by @albertvillanova in #3938
• Add exact match metric by @emibaylor in #3899
• Fix comet metric by @lhoestq in #3945
• Add zero_division argument to precision and recall metrics by @albertvillanova in #4035
• Support float data types in pearsonr/spearmanr metrics by @albertvillanova in #4054
• Remove GLEU metric by @emibaylor in #3949

Metric Cards

• Perplexity Metric Card by @emibaylor in #3905
• Create README.md by @sashavor in #3917
• Create README.md for CER metric by @sashavor in #3911
• Create README.md by @sashavor in #3944
• Update README.md by @sashavor in #3933
• Create SARI metric card by @sashavor in #3932
• Create MAUVE metric card by @sashavor in #3934
• Create CoVAL metric card by @sashavor in #3940
• Google BLEU Metric Card by @emibaylor in #3948
• Create metric card for BERTScore by @sashavor in #3966
• Rename wer to cer by @pmgautam in #4012
• Create metric card for XNLI by @sashavor in #4046
• Create metric card for the Code Eval metric by @sashavor in #4049
• Add TER metric card by @emibaylor in #3981
• BLEU metric card by @emibaylor in #3947
• Create metric card for CUAD by @sashavor in #4043
• Create metric card for METEOR by @sashavor in #4065
• Create a metric card for Competition MATH by @sashavor in #4073
• Create metric card for seqeval by @sashavor in #4070
• Create README.md by @sashavor in #3930
• Create metric card for Frugal Score by @sashavor in #4089
• Updating FrugalScore metric card by @sashavor in #4097
• Proposing WikiSplit metric card by @sashavor in #4098
• Fix formatting in BLEU metric card by @mariosasko in #4157

Documentation

• Doc maintenance by @stevhliu in #3926
• [Doc] Don't use v for version tags on GitHub by @sgugger in #3943
• Use templates for doc-builidng jobs by @sgugger in #3914
• Add align_labels_with_mapping docs by @stevhliu in #3931
• Add tip on how to speed up loading with ImageFolder by @mariosasko in #3980
• Fix main_classes docs index by @lhoestq in #3925
• More consistent references in docs by @mariosasko in #3988
• Docs maintenance by @stevhliu in #3999
• Add ROUGE Metric Card by @emibaylor in #4076
• Add chrF(++) Metric Card by @emibaylor in #4082
• Add SacreBLEU Metric Card by @emibaylor in #4083

General improvements and bug fixes

• Fix flatten of complex feature types by @mariosasko in #3723
• Fix flatten of Sequence feature type by @lhoestq in #3962
• Exclude Google Drive tests of the CI by @lhoestq in #3982
• Close PIL.Image file handler in Image.decode_example by @mariosasko in #3995
• Fix Faiss custom_index device by @albertvillanova in #3987
• Fix None issue with Sequence of dict by @lhoestq in #4010
• Update main readme by @lhoestq in #3927
• Fix map remove_columns on empty dataset by @lhoestq in #4021
• Fix Audio.encode_example() when writing an array by @polinaeterna in #3998
• Use audio feature in ASR task template by @lhoestq in #4006
• I...

2.0.0

🤗 Datasets 2.0.0

We're happy to announce that our new documentation is available at hf.co/docs/datasets !

Dataset Features

• Load a folder of images using the imagefolder dataset loader:
  • Add imagefolder dataset by @nateraw in #2830
  • Faster ImageFolder + add option to drop labels by @mariosasko in #3887
• Push your image and audio datasets on the Hugging Face Hub with push_to_hub:
  • Add support for Audio and Image feature in push_to_hub by @mariosasko in #3685
• New processing methods for streaming datasets:
  • Add IterableDataset.filter by @lhoestq in #3826
  • Manipulate columns on IterableDataset (rename columns, cast, etc.) by @lhoestq in #3862
  • Add the new methods to IterableDatasetDict by @lhoestq in #3923
• And more:
  • Add more compression types for to_json by @bhavitvyamalik in #3551
  • Multi-GPU support for FaissIndex by @rentruewang in #3721

Breaking changes

• API changes for map and shuffle for datasets loaded in streaming mode:
  • Align map when streaming: update instead of overwrite + add missing parameters by @lhoestq in #3801
  • Align IterableDataset.shuffle with Dataset.shuffle by @lhoestq in #3842
• Rename GenerateMode to DownloadMode by @albertvillanova in #3759
• Remove deprecated methods/params (preparation for v2.0) by @mariosasko in #3803
• Remove deprecated remove_columns param in filter by @mariosasko in #3827
• Module namespace cleanup for v2.0 by @mariosasko in #3875

Dataset Changes

• New: CFPB Consumer Complaints by @kayvane1 in #3617
• New: told-br (brazilian hate speech) by @JAugusto97 in #3683
• New: electricity load diagram by @kashif in #3722
• New: MIT Scene Parsing Benchmark by @mariosasko in #3607
• New: ElkarHizketak v1.0 by @antxa in #3780
• New: wikitablequestions by @SivilTaram in #3870
• New: ontonotes_conll by @richarddwang in #3853
• Update: BnL Historical Newspapers - make the dataset streamable by @albertvillanova in #3616
• Update: Common voice - add validated partition by @shalymin-amzn in #3669
• Update: Common Voice - add local paths to audio files by @lhoestq in #3736
• Update: Common Voice - simplify code by @lhoestq in #3817
• Update: Natural Questions - add dev-only configuration by @albertvillanova in #3699
• Update: pubmed - update data url by @albertvillanova in #3692
• Update: pubmed - make the dataset streamable by @abhi-mosaic in #3740
• Update: RedCaps - make the dataset streamable by @mariosasko in #3737
• Update: cats_vs_dogs - update metadata by @albertvillanova in #3752
• Update: newsroom - update manual download url by @albertvillanova in #3779
• Update: xcopa - update to new version by @albertvillanova in #3810
• Update: cats_vs_dogs size by @mariosasko in #3878
• Fix: sem_eval_2018_task_1 - fix download location by @maxpel in #3643
• Fix: newsqa - fix unique keys by @albertvillanova in #3696
• Fix: The Pile datasets - fix host urls by @albertvillanova in #3627
• Fix: Evidence Infer Treatment - fix dataset script by @albertvillanova in #3718
• Fix: NewsQA - fix dataset script by @albertvillanova in #3734
• Fix: head_qa - fix data url by @albertvillanova in #3766
• Fix: msr_sqa - fix unique keys by @albertvillanova in #3771
• Fix: reddit_tifu - fix data url by @albertvillanova in #3774
• Fix: wiki_lingua - fix spanish data file url by @albertvillanova in #3806
• Fix: beans - fix data urls by @mariosasko in #3890
• Fix: CRD3 - fix NonMatchingChecksumError by @albertvillanova in #3921
• Fix: MultiWOZ 2.2 - fix NonMatchingChecksumError by @albertvillanova in #3922

Dataset cards

• Add code example in wikipedia card by @lhoestq in #3678
• Fix Multi-News dataset metadata and card by @albertvillanova in #3731
• Reddit dataset card additions by @anna-kay in #3781
• Update gigaword card and info by @mariosasko in #3775
• Reddit dataset card contribution by @anna-kay in #3797

Metric Changes

• New: FrugalScore by @moussaKam in #3674
• New: Mahalanobis distance by @JoaoLages in #3794
• New: mIoU by @NielsRogge in #3745
• New: MSE and MAE - V2 by @dnaveenr in #3874
• Fix: METEOR - fix bug due to nltk version by @albertvillanova in #3884

Metric cards

• Add perplexity to metrics by @emibaylor in #3757
• Create SQuAD metric README.md by @sashavor in #3873
• SQuAD v2 metric: create README.md by @sashavor in #3879
• Update README.md for SQuAD v2 metric by @sashavor in #3908
• Update README.md for SQuAD metric by @sashavor in #3907
• Create README.md for WER metric by @sashavor in #3898
• Create README.md for GLUE by @sashavor in #3916

New documentation

• Update docs to new frontend/UI by @mishig25 in #3690
• Image process doc by @stevhliu in #3882

General improvements and bug fixes

• Better TQDM output by @mariosasko in #3654
• Prioritize module.builder_kwargs over defaults in TestCommand by @lvwerra in #3672
• Extend support for streaming datasets that use os.path.relpath by @albertvillanova in #3623
• Add Fon language tag by @albertvillanova in #3620
• Remove unnecessary 'r' arg in by @bryant1410 in #3661
• Fix TestCommand to copy dataset_infos to local dir with only data files by @albertvillanova in #3680
• Upgrade black to version ~=22.0 by @LysandreJik in #3691
• Fix streaming for servers not supporting HTTP range requests by @albertvillanova in #3689
• Pin ElasticSearch by @lhoestq in #3701
• Raise informative error when loading a save_to_disk dataset by @albertvillanova in #3705
• Fix ClassLabel to/from dict when passed names_file by @albertvillanova in #3695
• Fix CI code quality issue by @albertvillanova in #3710
• Check if indices values in Dataset.select are within bounds by @mariosasko in #3719
• Pin pandas to avoid bug in streaming mode by @albertvillanova in #3725
• Use config pandas version in CSV dataset builder by @albertvillanova in #3726
• Set base path to hub url for canonical datasets by @lhoestq in #3709
• Fix ValueError message formatting in int2str by @akulchik in #3742
• Patch all module attributes in its namespace by @albertvillanova in #3727
• Fix typo in train split name by @albertvillanova in #3751
• feat: 🎸 generate info if dataset_infos.json does not exist by @severo in https://github.com/huggingface/datase...

1.18.4

Bug fixes

• Prioritize module.builder_kwargs over defaults in TestCommand #3672 (@lvwerra)
• Fix TestCommand to copy dataset_infos to local dir with only data files #3680 (@albertvillanova)
• Upgrade black to version ~=22.0 #3691 (@LysandreJik)
• Fix streaming for servers not supporting HTTP range requests #3689 (@albertvillanova)
• Pin ElasticSearch #3701 (@lhoestq)
• Fix ClassLabel to/from dict when passed names_file #3695 (@albertvillanova)
• Fix CI code quality issue #3710 (@albertvillanova)
• Check if indices values in Dataset.select are within bounds #3719 (@mariosasko)
• Pin pandas to avoid bug in streaming mode #3725 (@albertvillanova)
• Use config pandas version in CSV dataset builder #3726 (@albertvillanova)
• Fix dataset mirroring (@lhoestq)
• Fix ValueError message formatting in int2str #3742 (@akulchik)
• Patch all module attributes in its namespace #3727 (@albertvillanova)
• Fix HfFileSystem docstring #3768 (@lhoestq)
• Fix: dataset name is stored in keys #3772 (@thomasw21)
• Fix Google Drive URL to avoid Virus scan warning #3787 (@albertvillanova)
• Fix error message in CSV loader for newer Pandas versions #3798 (@mariosasko)
• Pin responses to fix CI for Windows #3840 (@albertvillanova)

Full Changelog: 1.18.3...1.18.4

1.18.3

Bug fixes

• Fix MP3 resampling when a dataset's audio files have different sampling rates by @lhoestq in #3665
• Extend dataset builder for streaming in get_dataset_split_names by @mariosasko in #3657

Dataset changes

• New: Turkic X-WMT evaluation set for machine translation by @mirzakhalov in #3605
• New: British Library books dataset by @davanstrien in #3603
• Fix: wiki_bio - Update link by @jxmorris12 in #3651

Other improvements

• sp. Columbia => Colombia by @serapio in #3652
• Run pyupgrade for Python 3.6+ by @bryant1410 in #3560

New Contributors

• @serapio made their first contribution in #3652
• @mirzakhalov made their first contribution in #3605

Full Changelog: 1.18.2...1.18.3

1.18.2

Bug fixes

• Fix streaming datasets that are not reset correctly by @lhoestq in #3646
• Fix numpy rngs when shuffling with seed=None by @mariosasko in #3641
• Fix dataset slicing with negative bounds when indices mapping is not None by @mariosasko in #3642
• Fix add_column on datasets with indices mapping by @mariosasko in #3647

Other improvements

• Update index.rst by @VioletteLepercq in #3636
• Fix Windows CI: bump python to 3.7 by @lhoestq in #3648

New Contributors

• @VioletteLepercq made their first contribution in #3636

Full Changelog: 1.18.1...1.18.2