Sharing Docker containers between jobs in a workflow

[torkelrogstad]

Is it possible to share Docker containers between jobs in a workflow? I tried building a container and setting load to true, but that does not make it available in the next jobb (tried running docker image ls, and the images I built did not appear

[crazy-max]

@torkelrogstad

Docker containers between jobs in a workflow?

Each job is isolated in his own runner. That's why you cannot see your image in another job. But you can pass data between jobs in a workflow. You can use the actions/upload-artifact and actions/download-artifact actions to do this.

Here is a quick example to upload an image as an artifact:

name: ci

on:
  push:
    branches: master

jobs:
  upload-artifact:
    runs-on: ubuntu-latest
    steps:
      -
        name: Checkout
        uses: actions/checkout@v2
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      -
        name: Build and export
        uses: docker/build-push-action@v2
        with:
          context: .
          file: ./Dockerfile
          tags: name/app:latest
          outputs: type=oci,dest=/tmp/image.tar
      -
        name: Upload artifact
        uses: actions/upload-artifact@v2
        with:
          name: app
          path: /tmp/image.tar

[torkelrogstad]

Thank you!

[torkelrogstad]

I'm almost there now: after using actions/download-artifact@v2 and docker load --input image.tar I'm getting the image, but without the correct name and tag.

Here's my setup (simplified and slightly redacted):

      - name: Build and push frontend image
        id: docker_build
        uses: docker/build-push-action@v2
        with:
          context: ./frontend
          file: ./frontend/Dockerfile
          builder: ${{ steps.buildx.outputs.name }}
          tags: image-name
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache,mode=max
          outputs: type=oci,dest=/tmp/frontend-image.tar

      - name: Upload frontend image as artifact
        uses: actions/upload-artifact@v2
        with:
          name: frontend-image
          path: /tmp/frontend-image.tar

## in a different job in the workflow


      - name: Download artifacts (Docker images) from previous workflows
        uses: actions/download-artifact@v2

      - name: Load Docker images from previous workflows
        run: |
          docker load --input frontend-image/frontend-image.tar 

      - run: docker image ls

REPOSITORY                                           TAG                 IMAGE ID            CREATED              SIZE
<none>                                               <none>              1bd63aac8130        About a minute ago   31.5MB

This is consistent with what I'm seeing locally:

$ docker load --input frontend-image.tar
Loaded image ID: sha256:87b186615f939424d3c0937368c244891683b3692fcf0a8ba4a26c681d0483cc

$ docker inspect 87b186615f939424d3c0937368c244891683b3692fcf0a8ba4a26c681d0483cc
[
    {
        "Id": "sha256:87b186615f939424d3c0937368c244891683b3692fcf0a8ba4a26c681d0483cc",
        "RepoTags": [],
...

I'd expect to see repo tags. If I peek into the tar file, I see a file index.json:

{
    "schemaVersion": 2,
    "manifests": [
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:5e30d8d72f9ee23801194bafb0a47327b8852db0a24e8b7683c872120a55aae1",
            "size": 2122,
            "annotations": {
                "io.containerd.image.name": "redacted, but correct name",
                "org.opencontainers.image.created": "2020-11-11T10:20:52Z",
                "org.opencontainers.image.ref.name": "redacted, but correct tag"
            }
        }
    ]
}

I don't know the exact purpose or mechanics of this file, but it looks like the tag information is included, but not picked up when I'm loading. Do you know how to fix this?

Your help is much appreciated.

[crazy-max]

@torkelrogstad

I don't know the exact purpose or mechanics of this file, but it looks like the tag information is included, but not picked up when I'm loading. Do you know how to fix this?

Switch to outputs: type=docker,dest=/tmp/image.tar output:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      -
        name: Checkout
        uses: actions/checkout@v2
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      -
        name: Build and push
        uses: docker/build-push-action@v2
        with:
          context: .
          file: ./Dockerfile
          tags: myimage:latest
          outputs: type=docker,dest=/tmp/myimage.tar
      -
        name: Upload artifact
        uses: actions/upload-artifact@v2
        with:
          name: myimage
          path: /tmp/myimage.tar

  use:
    runs-on: ubuntu-latest
    needs: build
    steps:
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      -
        name: Download artifact
        uses: actions/download-artifact@v2
        with:
          name: myimage
          path: /tmp
      -
        name: Load Docker image
        run: |
          docker load --input /tmp/myimage.tar
          docker image ls -a

[electronjoe]

Just what I was looking for! Thanks for the precise example @crazy-max !

[gkiar]

Hey — Thanks for the super clear responses in the thread until now! I've been struggling with one step which follows directly from this, which is building a new image using the cached image I just loaded as its base. Would you be able to also show how the docker buildx tool could build from the loaded image? My current (failing) build is here.

[leighmcculloch]

The example above was really helpful. Is there a way to have the download artifact occur before services in the second job, so that the services can be loaded from the artifacts?

[ciaranmcnulty]

Is there a way to achieve this when doing multi-arch builds?

[cdalvaro]

Is there a way to achieve this when doing multi-arch builds?

You can achieve it by starting a registry container and pushing your multi-arch image to that container. Then, save the data of the registry into a .tar file and share it among jobs.

Here you have an example: https://github.com/cdalvaro/docker-salt-master/blob/c556e06e5527cc2dc95966039858671f5f16cebe/.github/workflows/build-and-test.yml

Although I would prefer to do it by simply using the outputs: type=oci,dest=/tmp/frontend-image.tar option

[JoshMcCullough]

IMO it's still be useful to be able to run multiple jobs on the same container. For example if I have a job that gets NPM deps installed, and a build job and a test job which I want to run in parallel. I believe, currently, I'd have to upload/download the result of the NPM install (node_modules) as artifacts, or use the upload/download cache actions to get those deps onto the build/test containers.

This also means each of my jobs need to install NodeJS.

[lgrosz]

The provided solution only seems to work when the docker commands are ran through a run key. Is it possible to get the docker-build-push actions to also see the loaded images?

[manali14]

How do you push the loaded image to the registry using docker/build-push-action?