MTF train script #295
Merged
MTF train script #295
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Muennighoff
reviewed
Jul 2, 2022
Muennighoff
reviewed
Jul 3, 2022
| # Option 1 of data loading using --data-path | ||
| # For T0, data has to be provided in the form --data-path input-data target-data input-data2 target-data2 ... | ||
| if args.data_path: | ||
| # TODO: Not yet compatible with dataset weights (Will break at prefixes, weights = analyze_data_prefix(args.data_path)) |
There was a problem hiding this comment.
I think the question is:
- Do we want separate indexed datasets for each T0 dataset?
-
Easy to remove unwanted indexed datasets (e.g. if we the bias filtering reveals something)
-
- Do we want to combine them all into one giant indexed dataset?
-
Probably faster to get started, as no need to worry about weights; small datasets with too few items to make a batch; BlendableDataset compatibility
-
There was a problem hiding this comment.
- data_path accepts multiple arguments (haven't tested but should work, it's just the weighting mechanism that we should have IMO)
- works as well. I don't really know how much we're going to be able to experiment with different filtered version of T0.
Muennighoff
reviewed
Jul 3, 2022
Muennighoff
reviewed
Jul 4, 2022
Co-authored-by: Niklas Muennighoff <[email protected]>
- Remove unecessary code from MTFDataset - Create size API for MTF dataset - Use new size API to build packed index much faster
thomasw21
commented
Jul 4, 2022
Comment on lines
+487
to
+489
| # TODO @thomasw21 handle the case where a single sample cannot fit inside a row. We can | ||
| # - silently skip that value [currently implemented] | ||
| # - truncate to `seq_length`, and keep the right part |
There was a problem hiding this comment.
I'm wondering is we actually should warn someone about it? Like add a warning or something. I don't know.
There was a problem hiding this comment.
I think it should almost never happen for T0 & seq len 2048, right?
So having a warning would be good imo
There was a problem hiding this comment.
Yeah ... let me add that.
|
Too late for those that wanted to review. |
younesbelkada
pushed a commit
to younesbelkada/Megatron-DeepSpeed
that referenced
this pull request
Sep 28, 2022
Co-authored-by: Lintang Sutawika <[email protected]> Co-authored-by: Lintang Sutawika <[email protected]> Co-authored-by: Muennighoff <[email protected]>
adammoody
pushed a commit
to adammoody/Megatron-DeepSpeed
that referenced
this pull request
Dec 18, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.