chore(ci): define a pipeline with multiple gates

[mdelapenya]

What does this PR do?

This PR refactors all the CI infrastructure, having one big workflow file instead of multiple ones. After these changes, the ci.yml file is the single source of truth in terms of CI pipelines, defining multiple what I call "quality gates", which is nothing than different gates in the pipeline that will have more importance depending on when they are run in the pipeline: those to the left in the diagram will run earlier because they have more priority.

In this big workflow we will define dependency relationships, leveraging the needs keyword from GH actions, therefore one "stage" o GH job, won't be triggered until its needed job has finished with success. The proposed pipeline defines:

flowchart TD
    Zero( Start )
    Zero --> Core
    Zero --> ModuleGen
    Core[Core Library] -->|ubuntu - macos\nGo 1.19.x, 1.20.x| B[Go Modules]
    Core[Core Library] -->|ubuntu\nGo 1.19.x, 1.20.x| Ryuk[Ryuk Disabled]
    Core[Core Library] -->|ubuntu\nGo 1.19.x, 1.20.x| Rootless[Docker Rootless]
    B[Core Library] --> |ubuntu - macos\nGo 1.19.x, 1.20.x| Examples[Example Modules]
    ModuleGen[Module Generator]
    ModuleGen --> |ubuntu - macos\nGo 1.19.x, 1.20.x| End( End )
    Ryuk --> |ubuntu\nGo 1.19.x, 1.20.x| End( End )
    Rootless --> |ubuntu\nGo 1.19.x, 1.20.x| End( End )
    Examples --> |ubuntu - macos\nGo 1.19.x, 1.20.x| End( End )

Loading

At the same time, we have created a reusable workflow to run the Go tests (see ci-test-go.yml file) to run the Go tests. It will receive certain input parameters to configure the execution:

Input param	Required	Type	Default	Description
go-version	true	string		The version of Go to use for the test.
platform	true	string		The platform to run the test on.
project-directory	true	string	"."	The directory where the Go project is located.
rootless-docker	false	boolean	false	Run the test with rootless docker, adding steps to configure rootless Docker.
ryuk-disabled	false	boolean	false	Disable the ryuk container for the test. Will set the TESTCONTAINERS_RYUK_DISABLED environment variable

An example usage of this reusable module:

  test-module-generator:
    strategy:
      matrix:
        go-version: [1.19.x, 1.x]
        platform: [ubuntu-latest, macos-latest]
    uses: ./.github/workflows/ci-test-go.yml
    with:
      go-version: ${{ matrix.go-version }}
      platform: ${{ matrix.platform }}
      project-directory: "modulegen"
      rootless-docker: false
      ryuk-disabled: false

As a result, we will be able to run the tests for the core library, the different Docker configs (e.g. rootless), tests for each Go module, with a single reusable workflow, which will define a proper visual name, based on the input parameters used to call it.

Finally, and because creating a new module using the module generator tool created a workflow for each module, we modified the code generator to generate the new pipeline file, adding a list of modules/examples to the matrix of modules/examples.

Why is it important?

We have experimented many situations where our CI jobs take a lot of time to be processed, mainly after an updates wave from @dependabot. This happens basically because the jobs get queued because the number of allocated CI workers is passing the limit of workers in the entire @testcontainers organisation!! AND THIS IS AN ISSUE AS WE WERE IMPACTING OTHER TC-LANG PROJECTS.

With this new approach, we will reduce the number of workers needed by each "gate", reducing the workers that are needed at the same time. From an initial ramp up of +90 workers, we will have:

• a first gate with 8 workers
• a second gate with 40-50
• a third gate with 40-50

allowing more granularity when new PRs (in any repo of the org) come to grab a free worker.

How to test this PR

The result in the GH actions UI will be: https://github.com/testcontainers/testcontainers-go/actions/runs/5750784979?pr=1414

[netlify]

✅ Deploy Preview for testcontainers-go ready!

Name	Link
🔨 Latest commit	6767ce2
🔍 Latest deploy log	https://app.netlify.com/sites/testcontainers-go/deploys/64cc303875180d00087fa0b7
😎 Deploy Preview	https://deploy-preview-1414--testcontainers-go.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[eddumelendez]

I was wondering if we can auto discover the folders under modules to build the matrix automatically

[mdelapenya]

This file is autogenerated by the modules generator

[mdelapenya]

I need to adjust the repository settings for the gh check

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
0.0% Duplication