Fix param randomization, generating repeating values #785
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
uri-granta
approved these changes
Sep 19, 2023
| @@ -68,7 +68,7 @@ def randomize_hyperparameters(object: gpflow.Module) -> None: | |||
| param.assign(sample) | |||
| elif param.prior is not None: | |||
| # handle constant priors for multi-dimensional parameters | |||
| if param.prior.batch_shape == param.prior.event_shape == [] and tf.rank(param) == 1: | |||
| if param.prior.batch_shape == param.prior.event_shape == [] and len(param.shape) == 1: | |||
There was a problem hiding this comment.
maybe add a comment about why this should not be changed to tf.rank? (though now that there's a test at least we'd catch it)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related issue(s)/PRs: None
Summary
This PR workarounds an issue where
randomize_hyperparametersgenerated same repeating values for model hyperparameters when the global seed was set. The issue only occurred whentf.functioncompilation was enabled.The issue seems to be related to the following documented behaviour of tensorflow:
When the function being compiled has a dynamic conditional (i.e.
tf.cond) and the branches contain randomization calls, it seems internally tensorflow acts like "... re-run of a program". This is likely related to the fact that AutoGraph executes both branches during tracing. This could potentially be a tensorflow bug, but requires more investigation.This PR simply removes the
tf.Tensorcondition expression (which is converted totf.condvia AutoGraph) to a static python expression. Also added a unit test to catch the issue, which fails on previous version of the code.Fully backwards compatible: yes
PR checklist