PyTorch without distributed support not supported

[adamjstewart]

If PyTorch isn't built with distributed support (commonly the case on macOS), many features of lightly crash with an error. For example:

lib/python3.10/site-packages/lightly/loss/ntx_ent_loss.py:164: in forward
    labels = labels + dist.rank() * batch_size
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def rank() -> int:
        """Returns the rank of the current process."""
>       return dist.get_rank() if dist.is_initialized() else 0
E       AttributeError: module 'torch.distributed' has no attribute 'is_initialized'

Any time torch.distributed is used, it should be wrapped by:

if torch.distributed.is_available():

Lightning themselves have introduced and fixed this same bug several times:

• Cannot import name 'default_pg_timeout' from 'torch.distributed'  Lightning-AI/pytorch-lightning#16637
• Get AttributeError: type object 'object' has no attribute 'SUM' Lightning-AI/pytorch-lightning#15494
• Module 'torch.distributed' has no attribute 'ProcessGroup' when importing PyTorch Lightning Lightning-AI/pytorch-lightning#10348

[guarin]

Ah, thanks for bringing this up! We use dist in multiple places and should definitely check for it.

[guarin]

This should be fixed with #1180.