Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RF: weight parametrization #1578

Merged
merged 4 commits into from
Jul 11, 2024
Merged

Conversation

albertz
Copy link
Member

@albertz albertz commented Jul 11, 2024

Fix #1518.

  • Add rf.gradient_checkpoint_scope.
  • Add rf.register_parametrization, similarly as in PyTorch.
  • Add rf.weight_dropout and rf.weight_noise.

@albertz albertz requested a review from NeoLegends July 11, 2024 16:03
@albertz
Copy link
Member Author

albertz commented Jul 11, 2024

Still WIP. Well, the parametrization API is basically already ready. But I want to add some tests for it. Also I want to add weight dropout and variational noise already. Maybe also weight norm. Similar as here:
https://pytorch.org/docs/stable/_modules/torch/nn/utils/parametrizations.html
https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrizations.weight_norm.html#torch.nn.utils.parametrizations.weight_norm

I'm probably not squashing all commits together but leave them as separate commits. This PR here is mostly just for easier reviewing, and maybe also discussions. (Although the main design questions are already being discussed in #1518.)

@albertz albertz force-pushed the albert-rf-weight-parametrize-1518 branch from 0ce986f to 262a06b Compare July 11, 2024 20:25
@albertz albertz force-pushed the albert-rf-weight-parametrize-1518 branch from 262a06b to 8a2dda9 Compare July 11, 2024 21:24
@albertz albertz marked this pull request as ready for review July 11, 2024 21:24
@albertz albertz requested a review from a team as a code owner July 11, 2024 21:24
@albertz
Copy link
Member Author

albertz commented Jul 11, 2024

Ok, all done, at least weight_dropout and weight_noise. I think this is enough for this PR.

@albertz albertz force-pushed the albert-rf-weight-parametrize-1518 branch from 8a2dda9 to 6d21af0 Compare July 11, 2024 22:11
@albertz albertz merged commit f0a1a4f into master Jul 11, 2024
57 checks passed
@albertz albertz deleted the albert-rf-weight-parametrize-1518 branch July 11, 2024 22:34
@albertz
Copy link
Member Author

albertz commented Jul 11, 2024

I already merged this because I want to use this in some experiments, and I think we can just directly fix any issues in the code in follow-up commits/PRs. I think we even can still do changes to the API for a short while, as long as no-one else is using this.

So, please anyway review/comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

RF weight dropout and variational noise
1 participant