ParameterAveragingOptimizer: support scheduler

[justheuristic]

• switch self.averager from DecentralizedAverager to TrainingAverager
  ( see parameters

  hivemind/hivemind/client/averaging/training.py

  Lines 24 to 26 in 27ea94e

  	:param average_parameters: whether or not to average model parameters in self.step(...)
  	:param average_gradients: whether or not to average model gradients in self.step(...)
  	:param average_opt_statistics: if specified, average optimizer statistics with corresponding names in statedict

  )
• add .epoch paramter to track local progress
• add optional scheduler parameter
• make a background thread triggered by event on each .step (see progress_reporter for example)
  • AFTER each step, add +1 to local steps and trigger progress event for progress reporter
  • this thread should report (current steps, epoch) to collaboration
    • use f"{self.prefix}.progress" as key and self.averager.endpoint as subkey
  • fetch steps and epoch from all the peers (see fetch_collaboration_state)
  • if some peer has higher epoch than us, set to his epoch
  • if peers on latest epoch collectively accumulated T steps, self.epoch += 1
• update scheduler to current epoch
• save/load epoch with state_dict/local_state_dict
• if remote epoch was above current epoch by more than M (parameter, e.g. 3), load_state_from_peers
• optionally change name from simple.py to something more appropriate, e.g. asgd.py averaged_opt.py, **your ideas

[justheuristic]

The functionality was originally implemented by @xtinkt in #252 , then reintegrated by me in #400

The exact behavior of ParamterAveragingOptimizer can be achieved as such:

hivemind.Optimizer(..., use_local_updates=True, target_batch_size=AVERAGE_EVERY_THIS_MANY_SAMPLES)

For more detailed explanation, please refer to

• https://learning-at-home.readthedocs.io/en/latest/modules/optim.html
• quickstart on optimizer - https://learning-at-home.readthedocs.io/en/latest/user/quickstart.html