Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deprecate the FairScale integration #16353

Merged
merged 10 commits into from
Jan 17, 2023
Merged

Deprecate the FairScale integration #16353

merged 10 commits into from
Jan 17, 2023

Conversation

carmocca
Copy link
Contributor

@carmocca carmocca commented Jan 13, 2023 •
edited by github-actions bot
Loading

What does this PR do?

See title.

I haven't added deprecation tests as they are extensively covered in the sharded tests.
All deprecations use the exact same message, so there's no need to check whether it has been shown already. (warnings do this by default)

Does your PR introduce any breaking changes? If yes, please list them.

None

cc @Borda @tchaton @awaelchli

@carmocca carmocca added deprecation Includes a deprecation strategy: fairscale sharded (removed) Sharded Data Parallel pl Generic label for PyTorch Lightning package labels Jan 13, 2023
@carmocca carmocca added this to the v1.9 milestone Jan 13, 2023
@carmocca carmocca self-assigned this Jan 13, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Jan 13, 2023 •
edited
Loading

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow
Check ID Status
pl-cpu (macOS-11, pytorch, 3.8, 1.11) success ✅
pl-cpu (macOS-11, pytorch, 3.9, 1.12) success ✅
pl-cpu (macOS-11, pytorch, 3.10, 1.13) success ✅
pl-cpu (macOS-11, pytorch, 3.8, 1.10, oldest) success ✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.10) success ✅
pl-cpu (ubuntu-20.04, pytorch, 3.9, 1.11) success ✅
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.12) success ✅
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.13) success ✅
pl-cpu (ubuntu-20.04, pytorch, 3.7, 1.10, oldest) success ✅
pl-cpu (windows-2022, pytorch, 3.9, 1.11) success ✅
pl-cpu (windows-2022, pytorch, 3.10, 1.12) success ✅
pl-cpu (windows-2022, pytorch, 3.10, 1.13) success ✅
pl-cpu (windows-2022, pytorch, 3.7, 1.10, oldest) success ✅
pl-cpu (slow, macOS-11, pytorch, 3.7, 1.11) success ✅
pl-cpu (slow, ubuntu-20.04, pytorch, 3.7, 1.11) success ✅
pl-cpu (slow, windows-2022, pytorch, 3.7, 1.11) success ✅
pl-cpu (macOS-11, lightning, 3.8, 1.13) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.13) success ✅
pl-cpu (windows-2022, lightning, 3.8, 1.13) success ✅

These checks are required after the changes to src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/fully_sharded_native_amp.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, src/pytorch_lightning/strategies/strategy.py, tests/tests_pytorch/callbacks/test_stochastic_weight_avg.py, tests/tests_pytorch/deprecated_api/test_remove_2-0.py, tests/tests_pytorch/plugins/precision/test_sharded_precision.py, tests/tests_pytorch/plugins/test_cluster_integration.py, tests/tests_pytorch/strategies/test_ddp_fully_sharded_with_full_state_dict.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_registry.py, tests/tests_pytorch/strategies/test_sharded_strategy.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Azure GPU
Check ID Status
pytorch-lightning (GPUs) success ✅

These checks are required after the changes to src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/fully_sharded_native_amp.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, src/pytorch_lightning/strategies/strategy.py, tests/tests_pytorch/callbacks/test_stochastic_weight_avg.py, tests/tests_pytorch/deprecated_api/test_remove_2-0.py, tests/tests_pytorch/plugins/precision/test_sharded_precision.py, tests/tests_pytorch/plugins/test_cluster_integration.py, tests/tests_pytorch/strategies/test_ddp_fully_sharded_with_full_state_dict.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_registry.py, tests/tests_pytorch/strategies/test_sharded_strategy.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Azure HPU
Check ID Status
pytorch-lightning (HPUs) success ✅

These checks are required after the changes to src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/fully_sharded_native_amp.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, src/pytorch_lightning/strategies/strategy.py, tests/tests_pytorch/callbacks/test_stochastic_weight_avg.py, tests/tests_pytorch/deprecated_api/test_remove_2-0.py, tests/tests_pytorch/plugins/precision/test_sharded_precision.py, tests/tests_pytorch/plugins/test_cluster_integration.py, tests/tests_pytorch/strategies/test_ddp_fully_sharded_with_full_state_dict.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_registry.py, tests/tests_pytorch/strategies/test_sharded_strategy.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Azure IPU
Check ID Status
pytorch-lightning (IPUs) success ✅

These checks are required after the changes to src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/fully_sharded_native_amp.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, src/pytorch_lightning/strategies/strategy.py, tests/tests_pytorch/callbacks/test_stochastic_weight_avg.py, tests/tests_pytorch/deprecated_api/test_remove_2-0.py, tests/tests_pytorch/plugins/precision/test_sharded_precision.py, tests/tests_pytorch/plugins/test_cluster_integration.py, tests/tests_pytorch/strategies/test_ddp_fully_sharded_with_full_state_dict.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_registry.py, tests/tests_pytorch/strategies/test_sharded_strategy.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Docs
Check ID Status
make-doctest (pytorch) success ✅
make-html (pytorch) success ✅

These checks are required after the changes to src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/fully_sharded_native_amp.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, src/pytorch_lightning/strategies/strategy.py, docs/source-pytorch/advanced/model_parallel.rst, docs/source-pytorch/extensions/strategy.rst.

🟢 mypy
Check ID Status
mypy success ✅

These checks are required after the changes to src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/fully_sharded_native_amp.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, src/pytorch_lightning/strategies/strategy.py.

🟢 install
Check ID Status
install-pkg (ubuntu-22.04, app, 3.7) success ✅
install-pkg (ubuntu-22.04, app, 3.10) success ✅
install-pkg (ubuntu-22.04, fabric, 3.7) success ✅
install-pkg (ubuntu-22.04, fabric, 3.10) success ✅
install-pkg (ubuntu-22.04, pytorch, 3.7) success ✅
install-pkg (ubuntu-22.04, pytorch, 3.10) success ✅
install-pkg (ubuntu-22.04, lightning, 3.7) success ✅
install-pkg (ubuntu-22.04, lightning, 3.10) success ✅
install-pkg (ubuntu-22.04, notset, 3.7) success ✅
install-pkg (ubuntu-22.04, notset, 3.10) success ✅
install-pkg (macOS-12, app, 3.7) success ✅
install-pkg (macOS-12, app, 3.10) success ✅
install-pkg (macOS-12, fabric, 3.7) success ✅
install-pkg (macOS-12, fabric, 3.10) success ✅
install-pkg (macOS-12, pytorch, 3.7) success ✅
install-pkg (macOS-12, pytorch, 3.10) success ✅
install-pkg (macOS-12, lightning, 3.7) success ✅
install-pkg (macOS-12, lightning, 3.10) success ✅
install-pkg (macOS-12, notset, 3.7) success ✅
install-pkg (macOS-12, notset, 3.10) success ✅
install-pkg (windows-2022, app, 3.7) success ✅
install-pkg (windows-2022, app, 3.10) success ✅
install-pkg (windows-2022, fabric, 3.7) success ✅
install-pkg (windows-2022, fabric, 3.10) success ✅
install-pkg (windows-2022, pytorch, 3.7) success ✅
install-pkg (windows-2022, pytorch, 3.10) success ✅
install-pkg (windows-2022, lightning, 3.7) success ✅
install-pkg (windows-2022, lightning, 3.10) success ✅
install-pkg (windows-2022, notset, 3.7) success ✅
install-pkg (windows-2022, notset, 3.10) success ✅

These checks are required after the changes to src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/fully_sharded_native_amp.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, src/pytorch_lightning/strategies/strategy.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

@mergify mergify bot added the ready PRs ready to be merged label Jan 13, 2023
@Borda Borda enabled auto-merge (squash) January 13, 2023 16:28
@carmocca
Copy link
Contributor Author

There's an issue with standalone tests hanging, marking as a draft...

@carmocca carmocca marked this pull request as draft January 13, 2023 18:37
auto-merge was automatically disabled January 13, 2023 18:37

Pull request was converted to draft

@mergify mergify bot removed the ready PRs ready to be merged label Jan 13, 2023
@carmocca carmocca force-pushed the deprecate/fairscale branch from b9599d1 to defaea4 Compare January 13, 2023 20:46
@Borda Borda modified the milestones: v1.9, v1.9.x Jan 16, 2023
@carmocca
Copy link
Contributor Author

carmocca commented Jan 17, 2023 •
edited
Loading

The hang is caused by

with pytest.deprecated_call():
    trainer = Trainer()
trainer.fit()

when multiple processes are used

because the deprecation message is only shown on rank 0, the other ranks fail the deprecate_call and then fit hangs after because some processes didn't make it.

@carmocca carmocca marked this pull request as ready for review January 17, 2023 00:53
@carmocca carmocca force-pushed the deprecate/fairscale branch from 996de14 to 48c789a Compare January 17, 2023 00:53
@mergify mergify bot added the ready PRs ready to be merged label Jan 17, 2023
@carmocca carmocca merged commit 0f4f809 into master Jan 17, 2023
@carmocca carmocca deleted the deprecate/fairscale branch January 17, 2023 01:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Borda Borda Borda approved these changes

@awaelchli awaelchli awaelchli approved these changes

@justusschock justusschock Awaiting requested review from justusschock justusschock is a code owner

@williamFalcon williamFalcon Awaiting requested review from williamFalcon

@edenlightning edenlightning Awaiting requested review from edenlightning

@tchaton tchaton Awaiting requested review from tchaton tchaton is a code owner

Assignees

@carmocca carmocca

Labels
deprecation Includes a deprecation pl Generic label for PyTorch Lightning package ready PRs ready to be merged strategy: fairscale sharded (removed) Sharded Data Parallel
Projects
None yet
Milestone
v1.9
Development

Successfully merging this pull request may close these issues.
3 participants
@carmocca @awaelchli @Borda