Allow GitHub Actions matrix exclusions to take an array

[alexedwardjones]

I have a job in one of my workflows that includes quite a large matrix with a lot of different combinations. I have an ever increasing number of exclude fields but I want to keep everything consolidated in one job.

The exclude list is getting quite long and doesn't seem to support providing an array which would shorten it significantly. Is this possible?

So...

      matrix:
        project: [projectA, projectB, projectC, projectD]
        region: [uk, france, germany]
        exclude:
          - region: uk
            project: projectA
          - region: uk
            project: projectB
          - region: france
            project: projectD
          - region: germany
            project: projectD

...would become...

      matrix:
        project: [projectA, projectB, projectC, projectD]
        region: [uk, france, germany]
        exclude:
          - region: uk
            project: [projectA, projectB]
          - region: [france, germany]
            project: projectD

[caspiano]

Ditto for include

[ethomson]

Hi @alexedwardjones - unfortunately, no, include and exclude do not support specifying an array in this manner, where it would get expanded into its component pieces. This particular syntax would not be possible, since include and exclude do support specifying an array however, because you can use dictionaries and arrays in matrix definitions.

In other words - this is legal:

strategy:
  matrix:
    a: [one, two]
    b:
    - ['1', '2']
    - ['3', '4']

which would produce a four element matrix { a=one, b=[1, 2] }, { a=one, b=[3, 4] }, { a = two, b=[1, 2] }, { a = two, b=[3, 4] }. In the { a=one, b=[1, 2] } case you can refer to b's elements as ${{ matrix.b[0] }} and ${{ matrix.b[1] }}.

This is also legal:

strategy:
  matrix:
    a: [one, two]
    b:
    - ['1', '2']
    - ['3', '4']
    exclude:
    - a: two
      b: ['3', '4']

Which will remove the { a=two, b=[3,4] } matrix entry.

I mention this not to dismiss your problem, but because these more advanced matrix scenarios might also be able to solve your problem. But if not, in the new year, I'd like to make complex workflows easier to specify and will keep your workflows in mind.

Thanks much for the feedback.

[chrispat]

Another option would be a dynamic matrix using job outputs. I thought we had an example in the docs but I can't find it right now, however, here is a helpful blog post https://michaelheap.com/dynamic-matrix-generation-github-actions/.

[xor-freenet]

I understand that it would be difficult to make the existing GitHub Actions implementation work like this.
However the implementation should follow the needs of the user, the user should not be forced to obey the needs of the implementation.

A workaround would be possible by e.g. using {} instead of [] or any different characters as replacement for the brackets, or requiring an escape character, e.g. \[. And you could probably come up with an infinite further amount of workarounds.

Even if you don't, heck, just require people to write "ACTIONS_PLEASE_PARSE_THIS_AS_ARRAY" before it, that's certainly unused and as long as it works I'm fine :)

I support this request because the lack of this is annoying in the following usecase which might be somewhat common:

matrix:
  os: [ubuntu-latest, windows-latest, ...]
  java-version: [..., 15, 16, 17, ...]

  exclude:
    - os: windows-latest
      java-version: [16, 17, ...] # Not available yet / broken

[umarcor]

To build on @chrispat's suggestion, in multiple Python projects we use dynamic matrix generation intensively, in order to work around GitHub Action's limitations to deal with global variables/env, and for complex matrix/job/task definition.

pyTooling/Actions: Reusable workflows is a pipeline (workflow) composed by 10+ reusable workflows. The first of them, Parameters.yml generates lists of jobs and params, which are passed to matrix: { include: ... } in other jobs. See how all of them are combined in ExamplePipeline.yml.

A particularly interesting feature is comparing the following three cases:

• artifact: ${{ fromJson(needs.Params.outputs.params).artifacts.unittesting }}: GitHub's built-in fromJson is used to decode the output, and then the dot notation is used to navigate the object.
• jobs: ${{ needs.Params.outputs.python_jobs }}: the output is passed to a reusable workflow without decoding it (it's an stringified json).
• matrix: { include: ${{ fromJson(inputs.jobs) }} }: inside a reusable workflow, an stringified json input is decoded with fromJson and passed as the include list of a matrix job.

umarcor/umarcor: Gource uses a similar strategy, but the stringified input is passed to a Composite Action, instead of a Reusable Wokrflow.:

• There is a YAML file which uses anchors to define complex job parameters: config.yml.
• In a workflow, a Python step is used to read the YAML file and produce JSON outputs: https://github.com/umarcor/umarcor/blob/1867b34053572660a7e5f797d5fc379214e46748/.github/workflows/Gource.yml#L84-L97.
• One of the outputs is used in the matrix: { include: ... } of a job, through fromJson; and another output is passed as-is (stringified) to a Composite Action: https://github.com/umarcor/umarcor/blob/1867b34053572660a7e5f797d5fc379214e46748/.github/workflows/Gource.yml#L113-L136.
• The Action uses the field in Python straightaway (no need to use JSON explicitly): job = ${{ inputs.job }}.

hdl/containers combines ouputs/inputs with Python scripts, JS Actions, Composite Actions, Reusable Workflows and Dispatchable workflows:

• common.yml is a Reusable Workflow with one job that receives a key and uses a Python script (pyHDLC) to process a complex YAML file (config.yml) and produce a list of jobs as an stringified JSON. That is used in the matrix: { include: ... } of a job, where a Composite Action is called.
  • The Composite Action uses a JS Action to allow using post steps with regular (bash/python) scripts/steps. See build-test-release/action.yml, pyTooling/Actions/with-post-step and Support pre and post steps in Composite Actions actions/runner#1478.
• In regular workflows, the Reusable Workflow is called, by passing the key and secrets. See e.g. boolector.yml.
• A Dispatchable Workflow (dispatch.yml) allows triggering the Reusable Workflow manually or through the API. See Homogenise inputs from callable workflows and event inputs from dispatchable workflows #9092.

---

Overall, after the enhancements introduced in recent months (Reusable Workflows and Composite Actions with conditional steps, and which can use uses), GitHub Actions is starting to be usable for complex CI setups and particularly for non-JS developers (say Python, Ruby, bash...). After some pending details such as #9050 and #9092 are solved, I would expect Reusable Workflows and Composite Actions to be merged into a single feature (similarly to workflow_dispatch replacing repository_dispatch). That would reduce the scope of GitHub's matrix interpretation to including lists, without need to exclude.

[a4z]

Just found that issue since I searched for a way of doing multiple include / exclude

I understand the reasoning that the existing keywords , include / exclude , can not be reused for having an array instead of a single string.

but what about adding new keywords, includes / excludes , that take an array ?

because reusable workflow and combined workflow might solve some problems, at some point they become ugly workaround or do ont work at all with reasonable afford.

[pboling]

This is an excellent idea