DEP: Update dependency datasets to >=5.1.0,<5.2.0

[renovate]

This PR contains the following updates:

Package	Change	Age	Confidence
datasets	>=5.0.0,<5.1.0 → >=5.1.0,<5.2.0

---

Release Notes

huggingface/datasets (datasets)

v5.1.0

Compare Source

New Formats

• Harbor ⚓: RL environments:
  • Add harbor benchmarks by @​lhoestq in #​8729
• Vortex 🌀: High performance columnar data
  • Add support for datasets backed by vortex file format by @​robert3005 in #​8502
• Bio formats 🧬: store biological data like DNA sequences, protein chains, and 3D molecular structures
  • feat(fasta): add lightweight FASTA file format support by @​behroozazarkhalili in #​7923
  • Decode BioSequence and BioStructure records with universal newlines by @​behroozazarkhalili in #​8600
  • Add lightweight FASTQ file format support by @​behroozazarkhalili in #​7924
  • feat: Add GenBank file format support for biological sequence data by @​behroozazarkhalili in #​7951
  • Add lightweight PDB (Protein Data Bank) file support by @​behroozazarkhalili in #​7926
  • feat: Add mmCIF file support for macromolecular structures by @​behroozazarkhalili in #​7925
  • feat(features): add BioSequence and BioStructure decodable features by @​behroozazarkhalili in #​8542

Dataset Features

• Add a strategy argument to split_dataset_by_node for iterable datasets by @​behroozazarkhalili in #​8565
• Allow IterableDataset.push_to_hub num_shards above dataset.num_shards by @​behroozazarkhalili in #​8716

Other improvements and bug fixes

• Fix fixed size binary by @​Alex-PLACET in #​8281
• Fix offline simulation with huggingface_hub 1.27 by @​hanouticelina in #​8414
• Fix info from data_dir/data_files or malformed yaml by @​lhoestq in #​8462
• Forward iter_arrow, is_typed and features from RepeatExamplesIterable by @​ebarkhordar in #​8395
• Fix archive path check allowing sibling-directory escapes by @​icn5381 in #​8490
• disallow hdf5 external links by @​lhoestq in #​8523
• Fix deepcopy bottleneck in state_dict() by @​yuxin00j in #​8387
• Optimize PyArrow block shuffling with dynamic rebatched chunk interleaving by @​yuxin00j in #​8407
• Fix Pdf.decode_example raising TypeError on remote non-Hub paths by @​behroozazarkhalili in #​8566
• Fix dead PDF processing link in README by @​aarushkandukoori in #​8512
• Forward max_buffer_input_shards from IterableDatasetDict.shuffle by @​vineethsaivs in #​8594
• Preserve storage options across URL hops by @​tandede in #​8496
• Better error when a column written as null gets values in a later batch by @​ebarkhordar in #​8591
• Advance the user-supplied generator in Dataset.shuffle even on cache hits by @​BenYang12 in #​8589
• Bump fsspec upper bound to 2026.7.0 by @​roshanjrajan in #​8561
• Shard the sources when skipping a distributed IterableDataset by @​ebarkhordar in #​8569
• Fix infinite repetition of empty iterable datasets by @​Jokasa7 in #​8577
• Reject a non-callable generator in GeneratorConfig by @​LeSingh1 in #​8585
• Fix async map checkpoint accounting across repeated resumes by @​SBrouck in #​8575
• [httpx migration] Import httpx from huggingface_hub, require huggingface_hub>=1.31.0 by @​Wauplin in #​8598
• chore: enable Dependabot weekly GitHub Actions bumps by @​hf-dependantbot-rollout[bot] in #​8221
• [docs] Fix legacy hf CLI references (datasets) by @​Wauplin in #​8657
• Scope GITHUB_TOKEN permissions per job by @​hf-security-analysis[bot] in #​8672
• Pin GitHub Actions to commit SHAs by @​hf-security-analysis[bot] in #​8673
• Fix push_to_hub for IterableDataset with unknown features by @​davanstrien in #​8658
• Vectorize interleave_datasets index generation (probabilities + first/all_exhausted) by @​saforem2 in #​8318
• Fix torchcodec docs links by @​pratikgx in #​8700
• [docs] Fix HF_HOME example in cache guide by @​ZainnQureshii in #​8694
• Fix broken internal anchor link in share.mdx by @​aarushkandukoori in #​8666
• Keep Json fields inside a struct from being double-encoded when some struct rows are null by @​breken-ai in #​8712
• Don't import aiohttp when importing datasets by @​kuldeeepy in #​8680
• Fix inverted span for minus-strand multi-exon GenBank locations by @​Linxiushen in #​8677
• Key parquet column options by leaf path so nested columns get compressed by @​jayzuccarelli in #​8674
• Implement arrow_array on PandasArrayExtensionArray so ArrayXD columns can go back from pandas to Arrow by @​JoeyTan21 in #​8701
• Batch in arrow when the source is arrow-backed, and let _consolidate type numpy scalars by @​Yonghui-Lee in #​8665
• Fix Json.cast_storage double-encoding large_string and string_view columns by @​SulimanAbdulrazzaq in #​8682
• Fix audio array handling in preprocess_function by @​causius0 in #​8731
• Decode remote NIfTI files through the bytes path by @​KennyMcSimpson in #​8737
• Keep take/skip counts when there are more DataLoader workers than shards by @​yupengtang in #​8695
• Fix epoch iteration after skipping or taking a streaming subset by @​galleonli in #​8707
• Fix IterableDataset.filter with arrow, pandas and polars formats by @​felipebridge in #​8725
• Bump huggingface_hub upper bound to <3.0 (datasets) by @​Wauplin in #​8678
• Preserve source shards when shuffling iterable datasets by @​yuchenwang3 in #​8670
• Speed up JSON export of indexed datasets by @​aprylewu in #​8662
• Fix repeated Arrow rebatch checkpoint restoration by @​tryu-ai in #​8727
• fix default iterable dataset dict and polars exports by @​cwklurks in #​8632
• harbor: fix paths on windows by @​lhoestq in #​8739

New Contributors

• @​Alex-PLACET made their first contribution in #​8281
• @​robert3005 made their first contribution in #​8502
• @​icn5381 made their first contribution in #​8490
• @​yuxin00j made their first contribution in #​8387
• @​behroozazarkhalili made their first contribution in #​8565
• @​aarushkandukoori made their first contribution in #​8512
• @​tandede made their first contribution in #​8496
• @​BenYang12 made their first contribution in #​8589
• @​roshanjrajan made their first contribution in #​8561
• @​Jokasa7 made their first contribution in #​8577
• @​LeSingh1 made their first contribution in #​8585
• @​SBrouck made their first contribution in #​8575
• @​hf-dependantbot-rollout[bot] made their first contribution in #​8221
• @​saforem2 made their first contribution in #​8318
• @​pratikgx made their first contribution in #​8700
• @​ZainnQureshii made their first contribution in #​8694
• @​breken-ai made their first contribution in #​8712
• @​kuldeeepy made their first contribution in #​8680
• @​Linxiushen made their first contribution in #​8677
• @​JoeyTan21 made their first contribution in #​8701
• @​Yonghui-Lee made their first contribution in #​8665
• @​SulimanAbdulrazzaq made their first contribution in #​8682
• @​causius0 made their first contribution in #​8731
• @​KennyMcSimpson made their first contribution in #​8737
• @​yupengtang made their first contribution in #​8695
• @​galleonli made their first contribution in #​8707
• @​felipebridge made their first contribution in #​8725
• @​yuchenwang3 made their first contribution in #​8670
• @​aprylewu made their first contribution in #​8662
• @​tryu-ai made their first contribution in #​8727
• @​cwklurks made their first contribution in #​8632

Full Changelog: huggingface/datasets@5.0.1...5.1.0

---

Configuration

📅 Schedule: (in timezone America/New_York)

• Branch creation
  • At any time (no schedule defined)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[github-actions]

Documentation Preview

The rendered documentation for PR #66 is available at View the Netlify preview.

Preview updated for commit c89408a.