Make sure random seed persists beyond storage #2671
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D61479553 |
|
This pull request was exported from Phabricator. Differential Revision: D61479553 |
saitcakmak
added a commit
to saitcakmak/Ax
that referenced
this pull request
Aug 19, 2024
Summary: Pull Request resolved: facebook#2671 When the random seed is not specified in `model_kwargs`, it is set to a fixed value in `__init__`. If we do not store this generated `seed` and reload the experiment / GS, we end up continuing the random generation using a different seed. Storing `seed` in `model_state` will ensure it is stored (in last GR) and reused when the GS is reloaded. Model state is extracted from last GR in `GS._fit_current_model`: https://www.internalfb.com/code/fbsource/[4d9fa225216d]/fbcode/ax/modelbridge/generation_strategy.py?lines=856 This gets passed down to `ModelSpec.fit` as `**model_kwargs`, which takes precedence over `ModelSpec.model_kwargs`: https://www.internalfb.com/code/fbsource/[4d9fa225216d]/fbcode/ax/modelbridge/model_spec.py?lines=131, which will ensure any `"seed": None` kwarg will get overwritten by the generated seed from last GR. Also removed the `if not self.deduplicate` block from `_get_state`. Whether we save the state should not depend on `deduplicate`, as it is taken into account during sampling. Not saving the seed would just lead to additional unpredictability in the generator behavior. Reviewed By: bernardbeckerman Differential Revision: D61479553
saitcakmak
force-pushed
the
export-D61479553
branch
from
fc90c42
to
f464ceb
Compare
Summary: Pull Request resolved: facebook#2671 When the random seed is not specified in `model_kwargs`, it is set to a fixed value in `__init__`. If we do not store this generated `seed` and reload the experiment / GS, we end up continuing the random generation using a different seed. Storing `seed` in `model_state` will ensure it is stored (in last GR) and reused when the GS is reloaded. Model state is extracted from last GR in `GS._fit_current_model`: https://www.internalfb.com/code/fbsource/[4d9fa225216d]/fbcode/ax/modelbridge/generation_strategy.py?lines=856 This gets passed down to `ModelSpec.fit` as `**model_kwargs`, which takes precedence over `ModelSpec.model_kwargs`: https://www.internalfb.com/code/fbsource/[4d9fa225216d]/fbcode/ax/modelbridge/model_spec.py?lines=131, which will ensure any `"seed": None` kwarg will get overwritten by the generated seed from last GR. Also removed the `if not self.deduplicate` block from `_get_state`. Whether we save the state should not depend on `deduplicate`, as it is taken into account during sampling. Not saving the seed would just lead to additional unpredictability in the generator behavior. Reviewed By: bernardbeckerman Differential Revision: D61479553
|
This pull request was exported from Phabricator. Differential Revision: D61479553 |
saitcakmak
force-pushed
the
export-D61479553
branch
from
f464ceb
to
2a9ef9c
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2671 +/- ##
==========================================
- Coverage 95.29% 95.29% -0.01%
==========================================
Files 495 495
Lines 47766 47773 +7
==========================================
+ Hits 45517 45523 +6
- Misses 2249 2250 +1 ☔ View full report in Codecov by Sentry. |
|
This pull request has been merged in 40c8417. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
When the random seed is not specified in
model_kwargs, it is set to a fixed value in__init__. If we do not store this generatedseedand reload the experiment / GS, we end up continuing the random generation using a different seed. Storingseedinmodel_statewill ensure it is stored (in last GR) and reused when the GS is reloaded.Model state is extracted from last GR in
GS._fit_current_model: https://www.internalfb.com/code/fbsource/[4d9fa225216d]/fbcode/ax/modelbridge/generation_strategy.py?lines=856This gets passed down to
ModelSpec.fitas**model_kwargs, which takes precedence overModelSpec.model_kwargs: https://www.internalfb.com/code/fbsource/[4d9fa225216d]/fbcode/ax/modelbridge/model_spec.py?lines=131, which will ensure any"seed": Nonekwarg will get overwritten by the generated seed from last GR.Differential Revision: D61479553