Add support for iterate_over_all for the CombinedDataset
#122
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tchaton
changed the title
iterate_over_all for the CombinedDataset.iterate_over_all for the CombinedDataset
justusschock
approved these changes
May 7, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before submitting
This PR aims at improving the behaviour of the combined dataset for more traditional use case.
Before this PR, the CombinedDataset would stop iterating as soon as one of its wrapped dataset would trigger a stop iteration. This is a great for LLM pre-training but less practical for more traditional use case.
This PR introduces the argument
iterate_over_allto enable the combined dataset to see all the samples from all the datasets. The sampling is still random but only the last items will be un-sampled.What does this PR do?
Fixes #112
PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃