Protect training progress and metrics with signatures and DHT schema validation

[borzunov]

This PR follows #219 and implements:

1. Validating DHT schema for training progress and metrics
2. Signing the training progress and the metrics (so a peer can't change the stats of someone else)
3. Using the local public key as a replacement for the peer's UUID
4. Using type-validated data structures for the stats in the code instead of plain lists

I have tested the following:

1. Training (run_first_peer.py and run_trainer.py) works without errors.
2. When connecting to the DHT from the Python interactive console, the DHT does not accept progress/metrics records that:
   • Doesn't match the schema.
   • Doesn't have a signature or have an invalid one.

Along the way, it improves the validator system:

1. Makes RSASignatureValidator and SchemaValidator picklable because their instances may be sent to the DHT in another process (this happens when we add validators to the existing DHT process after its creation).
2. Implements caching of the private key while creating RSASignatureValidator, so it is not generated again for each DHT-using entity inside one process (saves time because the generation takes ~100 ms).
3. Implements the BytesWithPublicKey type for DHT schemas.
4. Adds prefix parameter to SchemaValidator (if present, adds prefix + '_' to all field names) and moves them from the bytes to str keys (since the rest of code uses str keys).
5. Refactors RSASignatureValidator (e.g. better naming like signature_validator.ownership_marker -> signature_validator.local_public_key).

[mryab]

Will it generate a new key pair after restarting an experiment on the same peer?

[borzunov]

Yes. However, the public key here replaces the trainer UUID that was also randomly regenerated after each restart.

Do you think having a constant key pair (e.g. for one machine) may be useful? I imagine cases like implementing a leaderboard, however it may be a better idea to aggregate peers with different key pairs by endpoint/login at this stage (since one user may want to run several trainers that will have different IDs = different key pairs).

[mryab]

I think it's reasonable to assume that a single user shouldn't have to generate new key pairs for each run just for convenience. However, we can probably leave it as is for now if nobody objects