Fix random freezes in averager.step, improve error handling

[justheuristic]

This patch introduces three changes:

• fix a heisenbug where DecentralizedAverager would randomly hang on pytorch ops: Deadlock with multiprocessing (using fork) and OpenMP / PyTorch should warn after OMP and fork that multithreading may be broken pytorch/pytorch#17199
• tweak "sanity check failed" clause in DecentralizedAverager._step so that it is no longer triggered by sanctioned retries
• tweak Matchmaking.request_join_group to handle RPC errors instead of cancelling the entire matchmaking

[justheuristic]

So, here's how this might explain the error reported earlier by @nevec :

• initializing small tensors with ones/zeros -> no hang
• initializing large tensors with ones/zeros -> averager hangs
• initializing large tensors with empty -> no hang
• initializing large tensors with randn -> no hang

Maybe

• large ones/zeros are assigned with multiple threads, which triggers OMP -> subsequent fork -> pytorch ops in fork will hang
• small ones/zeros contend with a single thread -> no OMP -> no hang
• empty needs no item assignment, just malloc -> no OMP -> no hang
• randn uses RNG which is is inherently sequential -> no OMP -> no hang

[mryab]

Suggested change

	logger.warning(f"{self} - failed to request potential leader {leader}: {e}")
	logger.exception(f"{self} - failed to request potential leader {leader}: {e}")

[justheuristic]

fixed it, thanks