Standalone Lite: DDP Spawn Strategy Family

[awaelchli]

What does this PR do?

Adds the DDP Spawn family of strategies supported in Lite.
Very similar to #14670 , but using the spawn launchers.

Note: In the future, the spawn strategies will merge together with their non-spawn versions, as most logic related to process creation has been factored out to the launchers already.

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

[carmocca]

@awaelchli I believe this is not correct, as this no longer sets XLAStrategy._launched=True and then #14926 fails with

  File "/home/runner/work/lightning/tests/tests_lite/strategies/test_xla.py", line 17, in broadcast_on_tpu_fn
    result = strategy.broadcast(obj)
  File "/home/runner/work/lightning/src/lightning_lite/strategies/xla.py", line 146, in broadcast
    data_tensor = torch.tensor(data, device=self.root_device, dtype=torch.float)
  File "/home/runner/work/lightning/src/lightning_lite/strategies/xla.py", line 72, in root_device
    raise RuntimeError("Accessing the XLA device before processes have spawned is not allowed.")

I'm a bit confused about what's the best way to do this.

Do strategy.setup_environment()?

If yes:

• This does not set the local rank. Should it? or do we still manually set the local rank?
• Should this also be done for the other Lite launchers?

Also, why did Lite remove _worker_setup? The logic being different between PL and Lite strategies is confusing.

This blocks #14926

[awaelchli]

This was done in reaction to your comment: #11073 (comment)
I think it is correct, otherwise many many tests for ddp spawn would fail, and tpu spawn is not fundamentally different regarding this local rank business.

I commented on #14926 that maybe all that is missing in the test is a strategy.setup_environment().

[awaelchli]

My motivation with #11073 was always to simplify these things so that these questions wouldn't come up in the first place. But nobody wants to merge it lol, already posted 3x times in waiting pr over the last 5 months or so.

This does not set the local rank. Should it? or do we still manually set the local rank?

For the multiprocessing launcher, the information of local rank can only come from the launcher directly. So the answer here is no.

Should this also be done for the other Lite launchers?

If #11073 lands both codes would be identical in this regard.

Also, why did Lite remove _worker_setup? The logic being different between PL and Lite strategies is confusing.

If #11073 lands both codes would be identical in this regard.