[Feature] Add serveral differential privacy algorithms #369
Conversation
| """Returns the l2 norm of weight | ||
|
|
||
| Arguments: | ||
| p (int): The order of norm. |
There was a problem hiding this comment.
modified accordingly
There was a problem hiding this comment.
In the funcational mechanism original paper, the noise addition process is performed only once for the objective function, while your implementation performs noise addition in each iteration, is this supported by the theory?
|
|
||
| def forward(self, ctx): | ||
| l2_norm = 0. | ||
| for param in ctx.model.parameters(): |
There was a problem hiding this comment.
An interesting discussion about whether l2 regularization should be applied to beta and gamma of BN layers: https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994.
IMO, we can keep the implementation same as that in torch.norm
There was a problem hiding this comment.
The current torch.norm will calculate the l2 norm for all parameters including bn weight and bias.
For now, we use the parameter skip_bn to aovid calculating l2-norm for bn layer(with 'bn' in its name).
There was a problem hiding this comment.
Conflicts are solved
# Conflicts: # federatedscope/core/auxiliaries/optimizer_builder.py # federatedscope/register.py
Main modifications:
fmtrainer