FIX: Small numerical discrepancy for p-tuning after loading the model #2047
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There is a small numerical discrepancy between the outputs of a p-tuning model before and after loading. Even though it is small, it can still affect generations, so this PR eliminates it. As an example, without the fix, this is the difference in logits for opt-125m: > torch.testing.assert_close(output_loaded, output_peft) E AssertionError: Tensor-likes are not close! E E Mismatched elements: 30 / 10557120 (0.0%) E Greatest absolute difference: 1.1086463928222656e-05 at index (0, 9, 9314) (up to 1e-05 allowed) E Greatest relative difference: 0.00021288332936819643 at index (0, 9, 9314) (up to 1.3e-06 allowed) Details about how this comes about are explained here: huggingface#2043 (comment) The gist of it is that if we take a single sample, repeat it X times, and then forward it through a model (which is the training path in p-tuning), we would expect the same output as if we forwarded this sample only once and repeated the output X times (the inference path for p-tuning). However, for sufficiently large models, the two approaches can have tiny differences. With the fixed approach, there is no difference between training and inference code paths when it comes to this. The new code should also be slightly more compute efficient, but in practice will not make a noticeable difference.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
sayakpaul
reviewed
Sep 3, 2024
tests/test_gpu_examples.py
Outdated
| torch.cuda.empty_cache() | ||
| gc.collect() | ||
|
|
||
| torch.manual_seed(0) |
There was a problem hiding this comment.
Even if we're loading from a pre-trained model, we need to set a seed?
sayakpaul
reviewed
Sep 3, 2024
| model_id = "facebook/opt-125m" | ||
| inputs = torch.arange(10).view(-1, 1).to(self.device) | ||
|
|
||
| torch.manual_seed(0) |
There was a problem hiding this comment.
Even if we're loading from a pre-trained model, we need to set a seed?
There was a problem hiding this comment.
It's more of a routine I have to ensure reproducibility in tests. Since we call get_peft_model, which initializes a random model, it makes sense to set a seed here. The seed below is indeed not required, as there is only model loading, so I removed it.
sayakpaul
approved these changes
Sep 3, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There is a small numerical discrepancy between the outputs of a p-tuning model before and after loading. Even though it is small, it can still affect generations, so this PR eliminates it.
As an example, without the fix, this is the difference in logits for opt-125m:
Details about how this comes about are explained here:
#2043 (comment)
The gist of it is that if we take a single sample, repeat it X times, and then forward it through a model (which is the training path in p-tuning), we would expect the same output as if we forwarded this sample only once and repeated the output X times (the inference path for p-tuning). However, for sufficiently large models, the two approaches can have tiny differences.
With the fixed approach, there is no difference between training and inference code paths when it comes to this. The new code should also be slightly more compute efficient, but in practice will not make a noticeable difference.