Support context/matrix variables and Composite Actions inputs in steps of type 'uses'

[umarcor]

It is currently not possible to use variables from the matrix (in a regular workflow), or inputs (in a Composite Action) to define the Actions to be used in steps of type uses. The workflow won't start because of a parser/syntax issues: the 'uses' attribute must be a path, a Docker image, or owner/repo@ref.

See https://github.com/hdl/containers/blob/GHA-MWEs/.github/workflows/MWE-1479.yml and https://github.com/hdl/containers/blob/GHA-MWEs/utils/mwe-input-var-in-uses/action.yml:

  matrix-var-in-uses-name:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        action:
          - actions/checkout
    steps:
      - uses: ${{ matrix.action }}@v2


  matrix-var-in-uses-version:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        version:
          - v1
          - v2
    steps:
      - uses: actions/checkout@${{ matrix.version }}


  matrix-var-in-uses-docker:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        img:
          - debian:bullseye-slim
          - debian:buster-slim
    steps:
     - uses: docker://${{ matrix.img }}


  matrix-var-in-uses-docker-tag:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        img:
          - bullseye-slim
          - buster-slim
    steps:
     - uses: docker://debian:${{ matrix.img }}


  matrix-var-in-uses-action:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: ./utils/mwe-input-var-in-uses

name: '[HDLC] MWE Input variable in uses'
description: 'Minimal Working Example to reproduce a bug when using an input in a uses step of a Composite Action.'

inputs:
  version:
    description: 'Version of actions/checkout to use.'
    default: 'v2'
    required: false
  image:
    description: 'Name of the container image.'
    default: 'asciidoctor/docker-asciidoctor'
    required: true

runs:
  using: 'composite'
  steps:
    - uses: actions/checkout@${{ inputs.version }}
    - uses: docker://${{ inputs.image }}
      with:
        args: doc/make.sh

Additional information

The inability to use Composite Action inputs in steps of type uses: docker:// is particularly uncomfortable, because it forces using the docker CLI explicitly. Therefore, environment variables and binds need to be manually and explicitly described. See, for instance, the following workaround to the Composite Action above: https://github.com/VUnit/vunit_action/blob/master/action.yml.

Being able to use the context would be a suitable solution to using a local action without explicitly checking out the repository, as discussed in actions/runner#646 (comment). Precisely, uses: ${{ github.repository }}/.github/myaction@${{ github.ref_name }} would be equivalent to:

- uses: actions/checkout@v2
- uses: ./.github/myaction

Refs:

• Support context/matrix variables and Composite Actions inputs in steps of type 'uses' actions/runner#1479
• Next Steps for Fully Functioning Composite Actions actions/runner#646 (comment)

[ethomson]

Thanks for the feedback - you're right that at the moment we resolve actions before we expand any of these. It's something that we can explore in the future.

One thing jumps out at me though - and it's possible that this is just an oversimplification of the problem to reproducible steps (which I appreciate!) but I wanted to mention it if not:

  matrix-var-in-uses-docker-tag:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        img:
          - bullseye-slim
          - buster-slim
    steps:
     - uses: docker://debian:${{ matrix.img }}

is not a thing that one would really do, idiomatically. You're using the uses syntax to run something within a container, but that "something" is not an action. Container based actions were the original programmatic, reusable scripts for GitHub Actions; new actions should be JavaScript or composite actions for portability.

If the goal is to run a command inside a container, then that should be done using a container-based job. (And you can use matrix elements to specify that container.)

This:

  matrix-var-in-uses-docker-tag:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        img:
          - bullseye-slim
          - buster-slim
    container: debian:${{ matrix.img }}
    steps:
     - run: echo Hello, world.

would be the preferred mechanism.

[umarcor]

@ethomson, as explained in #9053, several Actions do expect to be executed on the virtual environments provided by GitHub and will unfortunately not work on Container Jobs. In order to use them, the content or layout of the container needs to be changed. Hence, as a rule of thumb, Container Steps are more robust if other Actions are used in the same job.

A more realistic use case is:

  build-and-test-tool:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        img:
          - bullseye
          - buster
    steps:

      - name: '🧰 Checkout'
        uses: actions/checkout@v2

      - name: '🚧 Build tool on container with build dependencies only'
        uses: docker://ghcr.io/myorg/myrepo/debian/${{ matrix.img }}/build
        with:
          args: make

      - name: '📤 Upload package'
        uses: actions/upload-artifact@v2
        with:
          path: tool-artifact.tar.gz

      - name: '⛴️ Build container based on image with runtime dependencies'
        run: |
          docker build -t docker://ghcr.io/myorg/myrepo/debian/${{ matrix.img }} - <<'EOF'
          FROM docker://ghcr.io/myorg/myrepo/debian/${{ matrix.img }}/run
          ...
          EOF

      - name: '🚦 Test tool installed on container with runtime dependencies only'
        uses: docker://ghcr.io/myorg/myrepo/debian/${{ matrix.img }}
        with:
          args: make test
  
      - name: '📦 Update nightly assets'
        run: gh release upload nightly tool-artifact.tar.gz --clobber

      - name: '🛰️ Push container image to ghcr.io'
        uses: pyTooling/Actions/with-post-step@r0
        with:
          main: |
            echo '${{ github.token }}' | docker login ghcr.io -u GitHub-Actions --password-stdin
            docker push ghcr.io/myorg/myrepo/debian/${{ matrix.img }}
          post: docker logout ghcr.io

      - name: 📊 Publish Unit Test Results
        uses: dorny/test-reporter@v1
        with:
          name: Unit Test Results
          path: test-report.xml
          reporter: java-junit

As you see, we are using a single job to run three different environments: building, testing and "other CI stuff".
It is desirable to execute "🧰 Checkout", "📤 Upload package", "📦 Update nightly assets", "🛰️ Push container image to ghcr.io" and "📊 Publish Unit Test Results" outside of the containers because git, gh, docker (the client CLI), etc. are not available inside. It is also desirable to split build and runtime dependencies, in order to catch packaging problems.

Note that the example workflow above is also related to #9053. Hence, in practice, a Composite Action and the docker CLI need to be used.

You're using the uses syntax to run something within a container, but that "something" is not an action. Container based actions were the original programmatic, reusable scripts for GitHub Actions

As commented in #9053, the only difference between a Container Action and a Container Step is the usage of a configuration file (action.yml). Moreover, using pre-built container images is desirable because it reduces the workflow execution time, compared to building a Dockerfile each time the Action is used.

new actions should be JavaScript or composite actions for portability.

That is really unfortunate... Neither JavaScript Actions nor Composite Actions can provide any isolation; they are executed on the same environment as the caller. Conversely, Container Actions or Container Steps allow to significantly reduce workflow execution time by separating environment building from provisioning; along with allowing testing/building on other distributions, such as Debian, Fedora, CentOS, ...

If the goal is to run a command inside a container, then that should be done using a container-based job. (And you can use matrix elements to specify that container.)

As shown in the example workflow above, the purpose is to use multiple environments in a single job, in order to separate concerns. If Container Actions/Container Steps are considered to be deprecated, the problem is moved one level up in the hierarchy.

The workflow above can be rewritten as 4-5 jobs, which would be significantly more verbose and would generate more traffic (uploading/downloading artifacts). Nonetheless, the main caveat is it would not be scalable. As far as I am aware, matrix jobs can only depend on other Matrix jobs, not on one specific job/task in the matrix. It is neither possible to define jobs and dependencies programmatically. Therefore, 8-10 jobs would need to be hardcoded in order to achieve the same functionality as the example proposed above. In repos with dozens of items per matrix, that is not maintainable.

The least bad alternative at the moment is to use a Reusable Workflow, which encapsulates the 4-5 steps/jobs. However
that is as complex to maintain as ignoring almost all GitHub Actions features and writing a custom script from scratch.

Anyway, please don't be mislead by the container use case. This discussion is about supporting variables in field uses:, regardless of the kind of Action (JavaScript, Composite or Container). The main application is to programmatically change which version of a JavaScript or Composite Action to use.

[jsoref]

You might be able to cheat by defining a local composite action.yml on the fly having expanded the matrix manually w/ the pieces you need and then doing a uses to that local action.yml.

These days, a composite action can use (via uses) other things. Sure, this means reinventing matrix yourself, but it probably means you can at least do what you want dynamically w/o waiting for this feature to be implemented.

[umarcor]

@jsoref I believe it's not possible to use variables in field uses, no matter how you define/pass them. That's because the ${{}} replacement is not processed at all, it just chokes on the syntax. There are mechanisms to work around it, such as using run: docker run instead of uses: docker://..., but they are more verbose and redundant.

FTR:

• https://github.com/pyTooling/Actions/
• Support YAML Anchors actions/runner#1182

[jsoref]

Right, you wouldn't be using variables, you'd have to flatten each instance of what would have been a matrix entry into its evaluated value.

[jsoref]

I suppose I should sketch this out for people:

on:
  push:

jobs:
  something:
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest]
    steps:
    - uses: actions/checkout@v4
    - runs: ./build-local-action.sh
      env:
        matrix: ${{ toJSON(matrix) }}
    - uses: ./local-action

build-local-action.sh would be responsible for generating ./local-action/action.yml that fills things in the way people want.

Local actions can't possibly be eagerly evaluated, because they need to support things like uses: actions/checkout@v4 which means you have the ability to write your action with whichever stuffs you want.

[ncruces]

I'm using this with envsubst < template.yml > action.yml to expand a "template action" into an action.

[nyarly]

I return to this issue every few months (working on a couple of years!). We're trying to build out a reusable set of steps and workflows in Github Actions, and one of the stumbling blocks is that composite actions can't reasonably refer to "library" actions in the shared actions repository.

That is, specifically, we have a service repo org/service, and its build.yaml workflow refers to org/actions/build@v1. build there is a composite action, and refers to, say, the dockerize action in org/actions - at present, our options are:

• uses: ./dockerize would refer to the local version of dockerize ... but in the consuming repo, which doesn't and shouldn't have a dockerize action! I'd argue this is a misfeature - the use cases for "execute whatever action you've written in the local repo" seem, with respect, over-engineered.
• uses: org/actions/dockerize@${{ github.action_ref}} doesn't work because no context is available to uses: I can see the security implications in general - ${{ inputs }} and ${{ env }} especially seem like punt-sized footguns, but I'm also at a loss as to where else I would use github.action_ref except in a composite's uses:
• Likewise uses: ${{ github.action_path}}/dockerize
• uses: org/actions/build@v1 which works... but relies on a brittle reference - when actions is updated to v2, all the internal references need to be updated at the same time. It would be very easy to push either uses: org/actions/build@v2 in our v1 tag or uses: org/actions/build@v1 in our v2 tag. Both are headaches waiting to happen as we have dozens (and growing!) of repos consuming these actions.
• runs: ${{ github.action_path}}/lib/script.sh might work, but in several cases the library action itself has uses: lines for 3rd party actions (e.g. google-github-actions/auth@v0) and then does things with the results. If there were a way for a script to execute actions, that might work for us, but I don't know of such a thing.
• We could template an action.yaml file to inject the appropriate uses: values and then run that. I've now seen 3 different ways to do precisely that, complete with 3rd party implementations which I'm sure won't also uses: docker://bitcoin-miner@latest in my action :) Point being that templating the action.yaml is an end run around the kind of security protections the current design of uses: is trying to provide.

I think that relative uses: paths in composite actions not referring to paths relating to the "local" repo is a very strange design outcome, but I can see how changing it would risk breaking existing workflows. A flag to enable "local-ness" would be worthwhile.

Reviewing the github context, it's a very broad interface, and I can see edge cases where making it entirely available in uses: might be unfortunate - I might publish a public action with a uses: that interrogates github.event to extract workflow_dispatch parameters and then be able to run arbitrary code that way... A stretch, but maybe enough to put the kibosh on making github available to uses: - maybe an action context that exposes a safe subset of values, though? Basically github.action_*. Honestly, though, this would make composite uses: much more flexible in ways that I'm not clear that there's a use case for. Relative path in the shared repo would cover 100% of the cases I need.

There is an existing PR on actions/runner to make the GITHUB_RUNNER_PATH env available to composite actions that's been open without comment since February; that would serve my purposes very well.

[alonbl]

Hi,

It is like GitHub developers do not use GitHub actions for their development so they do not experience the difficulties of their own technology or the scale is very small... Re-usability is a fundamental concern in engineering.

Until properly resolved, a temporary (hack?) solution for an CI action library is possible.

Provided a repository ci-library1 containing actions a1 and a2 , a1 may call a2 using the following:

name: 'a1'
description: 'a1'
runs:
  using: "composite"
  steps:
    - shell: sh
      run: |
        action_path="${{ github.action_path }}"
        if [ "${action_path#${{ github.workspace }}}" = "${action_path}" ]; then
          ln -sfn "${{ github.action_path }}/.." ci-library1
        fi
    - uses: ./ci-library1/a2

Regards,

[jaapio]

As a work around to test an github action I used the following code:

    runs-on: "ubuntu-latest"
    steps:
      - name: "Checkout"
        uses: "actions/checkout@v4"

      - name: "prepare action"
        ##################################################################
        # This is a workaround for the fact that the action.yml file
        # does not support templating. We replace the image with the
        # Dockerfile, so that we can use the Dockerfile from the current
        # branch.
        ##################################################################
        run: "sed -i 's/^  image:.*$/  image: \"Dockerfile\"/g' action.yml"

      - name: "Run action"
        uses: "./"

The released action will use an image while the current pipeline will use a docker file, containing the current version of the code. We use this to do a smoke test on the action.