ValueError __main__

[jacksonw765]

Got this working twice but cannot get it to run using CUDA. Running into this error when calling train.

       INFO     [17:56:38] Loaded checkpoint 'logs\44k\G_0.pth' (iteration 0)                           utils.py:257
       INFO     [17:56:38] Loaded checkpoint 'logs\44k\D_0.pth' (iteration 0)                           utils.py:257

INFO: Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2
INFO [17:56:38] Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2 distributed.py:244
[W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [kubernetes.docker.internal]:49979 (system error: 10049 - The requested address is not valid in its context.).
so-vits-svc-fork\venv\Scripts\python.exe: Error while finding module specification for 'main' (ValueError: main.spec is None)

[SouljaVR]

Was getting the same issue, happens for me if I have CUDA enabled on more than one GPU on the python process sovits is using. I made sure it was only using 1 GPU for CUDA in the Nvidia settings and it's fine now. I can't get multi GPU working though.

[jacksonw765]

Great catch, that worked!

[34j]

https://github.com/Lightning-AI/lightning/blob/master/src/lightning/fabric/utilities/distributed.py#L257-L257