Automated Testing for Lightning EcoSystem Projects

---

Automate issue discovery for your projects against Lightning nightly and releases.
You get CPUs, Multi-GPUs testing for free, and Slack notification alerts if issues arise!

How do I add my own Project?

Pre-requisites

Here are pre-requisites for your project before adding to the Lightning EcoSystem CI:

• Your project already includes some Python tests with PyTorch Lightning as a dependency
• You'll be a contact/responsible person to resolve any issues that the CI finds in the future for your project

Adding your own project config

1. First, fork this project (with CLI or in browser) to be able to create a new Pull Request, and work within a specific branch.

   gh repo fork Lightning-AI/ecosystem-ci
   cd ecosystem-ci/

2. Copy the template file in configs folder and call it <my_project_name>.yaml.

   cp configs/template.yaml configs/<my_project_name>.yaml
   

3. At the minimum, modify the HTTPS variable to point to your repository. See Configuring my project for more options.

   source_repository:
     HTTPS: https://github.com/MyUsername/MyProject.git
   ...

   If your project tests multiple configurations or you'd like to test against multiple Lightning versions such as master and release branches, create a config file for each one of them. As an example, have a look at metrics master and metrics release CI files.
4. Define your runtimes (OS and Python version) in your config file to be executed on CPU and/or add the config filename in the Azure GPU CI file.
   • For CPU integration, specify the OS and Python version combinations inside your config file:

     runtimes:
       - {os: "ubuntu-20.04", python: "3.9"}
       - {os: "macOS-12", python: "3.7"}
       - {os: "windows-2019", python: "3.8"}
     ...

   • For GPU integration, add your config filename in the Azure GPU CI file file:

     ...
     jobs:
     - template: testing-template.yml
       parameters:
         configs:
         - "Lightning-AI/metrics_pl-develop.yaml"
         - "Lightning-AI/metrics_pl-release.yaml"
         - "MyUsername/my_project-master.yaml"

5. Add the responsible person(s) to CODEOWNERS for your organization folder or just the project.

   # MyProject
   /configs/Myusername/MyProject*    @Myusername
   

6. Finally, create a draft PR to the repo!

Additional suggestions and engagement rules

• To qualify for GPU machines we require your project to have 100+ GitHub stars (please note that this is for capacity reasons and may change in the future)
• (Optional) Join our Slack channel #alerts-ecosystem-ci to be notified if your project is breaking
• (Kind request) include Lightning badge in your readme:

  [![Lightning](https://img.shields.io/badge/-Lightning-792ee5?logo=pytorchlightning&logoColor=white)](https://lightning.ai)

Configuring my project

The config include a few different sections:

• source_repository include your project
• env (optional) define any environment variables required when running tests
• dependencies listing all dependencies which are taken outside pip
• testing defines specific pytest arguments and what folders shall be tested

All dependencies as well as the target repository is sharing the same template with the only required field HTTPS and all others are optional:

source_repository:
  HTTPS: https://github.com/Lightning-AI/metrics.git
  username: my-nick  # Optional, used when checking out private/protected repo
  password: dont-tell-anyone # Optional, used when checking out private/protected repo
  token: authentication-token # Optional, overrides the user/pass when checking out private/protected repo
  checkout: master # Optional, checkout a particular branch or a tag
  install_extras: all # Refers to standard pip option to install some additional dependencies defined with setuptools, typically used as `<my-package>[<install_extras>]`.

# Optional, if any installation/tests require some env variables
env:
   MY_ENV_VARIABLE: "VAR"

copy_tests:
    - integrations # copied folder from the original repo into the running test directory
    # this is copied as we use the helpers inside integrations as regular python package
    - tests/__init__.py
    - tests/helpers

# Optional, additional pytest arguments and control which directory to test on
testing:
  dirs:
    - integrations
  pytest_args: --strict

Note: If you define some files as done above, and they are using internal-cross imports, you need to copy the __init__.py files from each particular package level.

The testing section provides access to the pytest run args and command.

testing:
  # by default pytest is called on all copied items/tests
  dirs:
    - integrations
  # OPTIONAL, additional pytest arguments
  pytest_args: --strict