Update self-push workflow

[ydshieh]

What does this PR do?

Update self-push CI workflow file:

• tests_fetcher.py is updated to output a json file, containing a dictionary mapping test categories to the identified test files (which is used by the updated push CI below)
• Reorganize the tests into models (e.g. models/bert, models/gpt2, etc.) and modeling categories (pipeline, tokenization), same as in scheduled CI
• notification_service.py and self-scheduled.yml are updated to use [single/multi]-gpu as artifact name prefixes (i.e. no more -docker at the end): with this minimal change, notification_service.py could be reused

Some workflow runs:

• push CI
• scheduled CI

Some tests failed intentionally (to verify their reports). The reports could be found on transformers-ci-feedback-tests channel.

TODO:

• create new report channel and add the channel ID to the workflow file

I added some reviews that contain some of my questions.

@sgugger Maybe you could have a look for the changes in test_fetcher.py?
@stas00 Maybe for the changes regarding DeepSpeed and multi-gpu configurations?

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[stas00]

I find it difficult to make sense of some of the changes, due to diff being hard to follow, so I'm totally fine with you going ahead with the deepspeed side of things and sort it out if something breaks afterwards. I test on my machine as well, so if it's broken for a day or two on CI it's no big deal.

[ydshieh]

I find it difficult to make sense of some of the changes, due to diff being hard to follow

OK @stas00 , the big diff is probably because I removed unused blocks. No real big change - I just copied from scheduled CI.
See below if you would like to have a quick look.

My only questions are

• why we use options: --gpus 0 previously in run_tests_torch_cuda_extensions_multi_gpu
• could we use options: --gpus all for single gpu case as well as multi gpu?

prev.

image: nvcr.io/nvidia/pytorch:21.03-py3
options: --gpus 0

now.

huggingface/transformers-pytorch-deepspeed-latest-gpu
options: --gpus all

and

prev.

      - name: Install dependencies
        run: |
          apt -y update && apt install -y libaio-dev
          pip install --upgrade pip
          pip install .[deepspeed-testing]

now

      - name: Re-compile DeepSpeed
        working-directory: /workspace
        run: |
          pip install deepspeed # installs the deps correctly
          rm -rf DeepSpeed
          git clone https://github.com/microsoft/DeepSpeed && cd DeepSpeed && rm -rf build
          DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 DS_BUILD_UTILS=1 python3 -m pip install -e . --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check

[stas00]

thank you for highlighting the changes @ydshieh - that's super helpful.

1. So for gpus:

• multi-gpu should be --gpus all (needs 2 gpus)
• single-gpu should be --gpus 0 (must have only one gpu)

so looking at the diff the original seems to be correct. perhaps not everywhere?

2. For dependencies the original is correct.

Have a look at what it signifies:

extras["deepspeed-testing"] = extras["deepspeed"] + extras["testing"] + extras["optuna"]

so the change is missing important dependencies install.

and the new instructions aren't correct.

We only want the bleed edge (your now) install only for nightly build. self-push should use the released deepspeed version, that pip install .[deepspeed-testing] takes care of (but which of course can be moved into the docker if it's running via the docker image). If it's already there, then there is no need for that last pip call either.

Bottom line - no change from the original in either case logically.

If I missed something please let me know.

[ydshieh]

so looking at the diff the original seems to be correct. perhaps not everywhere?

The current main branch has a job run_tests_torch_cuda_extensions_multi_gpu in self-push.yml which has --gpus 0.
In the latest commit in this PR, I reverted to the original version regarding DeepSpeed parts, but set --gpus all for multi-gpu job.

Remark 1: some places in self-scheduled.yml have to be fixed.)

Remarks 2: I checked this doc expose-gpus-for-use, and think we can still use --gpus all even if the host machine has only 1 GPU. --gpus 0 is necessary only if the host has multiple GPUs but we want to use only 1 of them.

2. For dependencies the original is correct.
   self-push should use the released deepspeed version, that pip install .[deepspeed-testing]

I will change back to the original version for this part (Done), thank you.

[stas00]

As long as the tests are run with CUDA_VISIBLE_DEVICES=0 for run_tests_single_gpu jobs it indeed doesn't matter if more than 1 gpu is available.

But it's critical we ensure that it is set correctly, otherwise tests requiring a single gpu will get skipped.

Thank you for fixing where the setting are incorrect, @ydshieh!

[LysandreJik]

This looks great, thanks for working on it @ydshieh!

The notification_service_deprecated.py can likely be removed now.

Before merging it, could you do a test run when modifying the setup.py to ensure that all tests are run correctly? Thank you!

[LysandreJik]

Love that!

[LysandreJik]

Smart!

[ydshieh]

Before merging it, could you do a test run when modifying the setup.py to ensure that all tests are run correctly? Thank you!

I had to fix a bug (i.e. when the test list is tests, i.e. when setup.py is changed).
A full test workflow run is here.
After looking some failures, I am convinced that this PR is ready to be merged (the failures are the same as in scheduled CI runs).

Thank you for the reviews!