Question - Can I use docker.sock?

[vikas027]

What I'd like:
I am using Bottlerocket on my EKS clusters and using GitHub Actions runners. Everything works fine but the problem occurs when I use a GitHub action task which tries to build a docker image using the docker socket.

Sample GitHub Action Task

- name: Commitsar check
  uses: aevea/[email protected]

Error Logs

Build container for action use: '/runner/_work/_actions/aevea/commitsar/v0.16.0/Dockerfile'.
  /usr/local/bin/docker build -t 60e226:7de2787af7e04b038ce49eb6a1a987d8 -f "/runner/_work/_actions/aevea/commitsar/v0.16.0/Dockerfile" "/runner/_work/_actions/aevea/commitsar/v0.16.0"
  Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
  Warning: Docker build failed with exit code 1, back off 8.072 seconds before retry.
  /usr/local/bin/docker build -t 60e226:7de2787af7e04b038ce49eb6a1a987d8 -f "/runner/_work/_actions/aevea/commitsar/v0.16.0/Dockerfile" "/runner/_work/_actions/aevea/commitsar/v0.16.0"
  Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
  Warning: Docker build failed with exit code 1, back off 4.07 seconds before retry.
  /usr/local/bin/docker build -t 60e226:7de2787af7e04b038ce49eb6a1a987d8 -f "/runner/_work/_actions/aevea/commitsar/v0.16.0/Dockerfile" "/runner/_work/_actions/aevea/commitsar/v0.16.0"
  Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Error: Docker build failed with exit code 1

I know this is not a Bottlerocket issue but keen to understand if there is workaround I use in Bottlerocket.

Any alternatives you've considered:
I would have to go with Amazon Linux AMIs

Related Issues

• Compatibility with containerd? actions/runner#1265
• Feature Request - Compatible with Bottlerocket (containerd) aevea/commitsar#410

[arnaldo2792]

Hello @vikas027! thanks for opening this issue we will start looking at it 👍

[arnaldo2792]

Hello @vikas027, I did some research about what you are experiencing, and I have a few pointers that might help you. First I just want to clarify that, as you already mention in the Github issues in the description, the AWS k8s variants don’t ship with docker in them. Second, I found that the Github action that you are trying to use depends on docker to work, which complicates things a little since the host won’t have any docker binary/socket available for the action to use/call.

Based on the logs that you shared, I’m assuming that you are either using a custom build of Bottlerocket or a k8s pod with the Github actions runner binary in it. If you already have the container image and the runner working in it, you could try to modify your pod spec a little to deploy dind, something similar to:

containers:
  - name: runner
    image: the-custom-image-with-the-runner-and-docker-client
    env:
      # This is the "hack" to use dind, which will be
      # running in the pod, and any docker command issued
      # by the runner should work
      - name: DOCKER_HOST
        value: tcp://localhost:2375
  - name: dind-daemon
    image: docker:<version>-dind
    securityContext:
      privileged: true
    volumeMounts:
      - name: docker-graph-storage
        mountPath: /var/lib/docker
volumes: 
  - name: docker-graph-storage 
    emptyDir: {}

I confirmed that it is possible to build images (which the aevea/commitsar Github action will try to do):

[I] kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
dind                                2/2     Running   1          4m17s

[I] kubectl exec -it dind -- /bin/sh
Defaulting container name to docker-cmds.
Use 'kubectl describe pod/dind -n default' to see all of the containers in this pod.

~ cd /tmp/

cat > Dockerfile <<EOF
FROM alpine
RUN echo This is a test
EOF

docker build -t test .
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM alpine
 ---> 021b3423115f
Step 2 : RUN echo This is a test
 ---> Running in f8e484598e7d
This is a test
 ---> 5af6e90016c6
Removing intermediate container f8e484598e7d
Successfully built 5af6e90016c6

If this doesn’t work for you, please provide us as much detail as you can about your current setup, so that we can investigate further and provide you a better workaround.

[vikas027]

Hey @arnaldo2792 ,

Thanks a lot for looking into this.

Second, I found that the Github action that you are trying to use depends on docker to work, which complicates things a little since the host won’t have any docker binary/socket available for the action to use/call.

I would like to make a small correction here. The docker socket is not only used by this GitHub action but any GitHub action (aevea/commitsar was just an example), see my github action repo as an example.

I’m assuming that you are either using a custom build of Bottlerocket or a k8s pod with the Github actions runner binary in it

No, I am not using a custom bottle rocket build, I am using an official AMI ami-05d9ed4cf385c2074

Also, thanks for the yaml. I know this path but it looks a bit hacky to me. I think, I would leave this for GitHub Actions community to solve, ideally they should allow native use of other container runtimes like containerd and/or have support for tools like buildah/img/kaniko to build docker images.

For now, I think it would easier to just have another set of k8s nodes using Amazon Linux AMIs and use them to host GitHub Actions runners.

[gthao313]

Hey @vikas027. Thanks for your comments! We are working on using Bottlerocket for running self-hosted github action runners via actions-runner-controller. We'll let you know once we have any updates. Thanks!

[gthao313]

Hello @vikas027, Thanks for your patience. I was able to use Bottlerocket for running self-hosted github action runners via actions-runner-controller. I tried to set up a GitHub action (aevea/commitsar) on my test repository and use Bottlerocket to host self-hosted GitHub action runners to execute the commitsar check action, and I had not encountered the docker socket issue that you encountered during GitHub actions.

From my research, actions-runner-controller automatically help user set up docker environment, and during my test commitsar image was able to be pulled down.

  Pull down action image 'aevea/commitsar'
  Status: Downloaded newer image for aevea/commitsar:latest
  docker.io/aevea/commitsar:latest

However, you have to deploy dind to deal with docker socket issue. For now, I recommend you to use Bottlerocket to host self-hosted GitHub action runner through docker in docker and confirm if it still doesn’t work for you. Here are some helpful resources that I was following: Commitsar, actions-runner-controller.

If this doesn’t work for you, please provide us as much detail as you can about your current setup, so that we can investigate further and provide you a better workaround.

[vikas027]

Hey @gthao313 ,

As you know, Bottlerocket AMI does not uses a docker or docker socket, it uses containerd. The same was also confirmed on a call with the AWS Bottlerocket team.

Also, I am not sure how dind (docker in docker) will help here (when there is no docker installed on the instance) unless there is some magic happening on the controller.

For now, I have pointed my runners to an Amazon Linux AMI which has docker and is working fine.

[stmcginnis]

Reading through the comments, I believe this issue has been resolved. If there are any outstanding questions or open items, please feel free to reopen this issue or file a new one. Thanks!

[vikas027]

Hey @stmcginnis - I think the issue persists with containerd runtime, please have a look at my explanation above. Thanks!

[arnaldo2792]

Hey @vikas027, unfortunately @stmcginnis isn't a maintainer of Bottlerocket anymore (:cry: ). I'll write down how DinD comes in picture here, probable a blogpost or something with clear instructions on how to use GitHub actions with Bottlerocket could help here. I'll sync with @stockholmux to check what's the best way to share this.

[plaisted]

was this writeup ever posted? I'm looking into similar use cases for running CI/CD on top of EKS/bottlerocket that requires the ability to run/build containers on demand.