GitLab Runner - SELinux Issue

[bdwyertech]

Image I'm using:
Latest v1.1.2 on EKS 1.20
Also tried going back to EKS 1.19 back to v1.0.x

What I expected to happen:
GitLab Runner should be able to create pods in my cluster.

What actually happened:
GitLab Runner was not able to create pods within my cluster.

How to reproduce the problem:
Deploy the gitlab/gitlab-runner helm chart to test.

It seems like there is an SELinux issue going on here preventing the pod from pulling some of its containers. I think it means its attempting to mount an unlabeled_t tmpfs underneath /local (of type local_t).

AVC avc:  denied  { associate } for  pid=3531 comm="containerd" name="bin" dev="nvme1n1p1" ino=398203 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:local_t:s0 tclass=filesystem permissive=0

Jul 08 23:15:20 ip-10-12-34-56.ec2.internal containerd[3531]: time="2021-07-08T23:15:20.688955107Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:runner-byxzzoom-project-9754-concurrent-0z4zg6,Uid:b7593111-1d5f-4b49-8f6d-a6fbd99caf83,Namespace:default,Attempt:0,} returns sandbox id \"f9e509c805a42f3407d4d01af96df4b40cf24308d61c1d1b2bdc2b1fde070ad6\""
Jul 08 23:15:20 ip-10-12-34-56.ec2.internal containerd[3531]: time="2021-07-08T23:15:20.693249544Z" level=info msg="PullImage \"registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-c1edb478\""
Jul 08 23:15:25 ip-10-12-34-56.ec2.internal audit[3531]: AVC avc:  denied  { associate } for  pid=3531 comm="containerd" name="bin" dev="nvme1n1p1" ino=398203 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:local_t:s0 tclass=filesystem permissive=0
Jul 08 23:15:25 ip-10-12-34-56.ec2.internal systemd[1]: local-var-lib-containerd-tmpmounts-containerd\x2dmount075234526.mount: Succeeded.
Jul 08 23:15:25 ip-10-12-34-56.ec2.internal systemd[1]: var-lib-containerd-tmpmounts-containerd\x2dmount075234526.mount: Succeeded.
Jul 08 23:15:25 ip-10-12-34-56.ec2.internal kernel: audit: type=1400 audit(1625786125.684:10): avc:  denied  { associate } for  pid=3531 comm="containerd" name="bin" dev="nvme1n1p1" ino=398203 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:local_t:s0 tclass=filesystem permissive=0
Jul 08 23:15:25 ip-10-12-34-56.ec2.internal containerd[3531]: time="2021-07-08T23:15:25.724078212Z" level=error msg="PullImage \"registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-c1edb478\" failed" error="failed to pull and unpack image \"registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-c1edb478\": failed to extract layer sha256:f126afed68c068bc47bf2c4468f4764794b6206a18a47a57789da42ca55eb1cb: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount075234526: permission denied: unknown"
Jul 08 23:15:25 ip-10-12-34-56.ec2.internal kubelet[4159]: E0708 23:15:25.724642    4159 remote_image.go:113] PullImage "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-c1edb478" from image service failed: rpc error: code = Unknown desc = failed to pull and unpack image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-c1edb478": failed to extract layer sha256:f126afed68c068bc47bf2c4468f4764794b6206a18a47a57789da42ca55eb1cb: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount075234526: permission denied: unknown
Jul 08 23:15:25 ip-10-12-34-56.ec2.internal kubelet[4159]: E0708 23:15:25.724728    4159 kuberuntime_image.go:51] Pull image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-c1edb478" failed: rpc error: code = Unknown desc = failed to pull and unpack image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-c1edb478": failed to extract layer sha256:f126afed68c068bc47bf2c4468f4764794b6206a18a47a57789da42ca55eb1cb: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount075234526: permission denied: unknown
Jul 08 23:15:25 ip-10-12-34-56.ec2.internal kubelet[4159]: E0708 23:15:25.724896    4159 kuberuntime_manager.go:815] init container &Container{Name:init-logs,Image:registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-c1edb478,Command:[sh -c touch /logs-9754-1414355/output.log && (chmod 777 /logs-9754-1414355/output.log || exit 0)],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:scripts,ReadOnly:false,MountPath:/scripts-9754-1414355,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:logs,ReadOnly:false,MountPath:/logs-9754-1414355,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:repo,ReadOnly:false,MountPath:/builds,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:default-token-jbvrl,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod runner-byxzzoom-project-9754-concurrent-0z4zg6_default(b7593111-1d5f-4b49-8f6d-a6fbd99caf83): ErrImagePull: rpc error: code = Unknown desc = failed to pull and unpack image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-c1edb478": failed to extract layer sha256:f126afed68c068bc47bf2c4468f4764794b6206a18a47a57789da42ca55eb1cb: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount075234526: permission denied: unknown
Jul 08 23:15:25 ip-10-12-34-56.ec2.internal kubelet[4159]: E0708 23:15:25.724968    4159 pod_workers.go:191] Error syncing pod b7593111-1d5f-4b49-8f6d-a6fbd99caf83 ("runner-byxzzoom-project-9754-concurrent-0z4zg6_default(b7593111-1d5f-4b49-8f6d-a6fbd99caf83)"), skipping: failed to "StartContainer" for "init-logs" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-c1edb478\": failed to extract layer sha256:f126afed68c068bc47bf2c4468f4764794b6206a18a47a57789da42ca55eb1cb: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount075234526: permission denied: unknown"
Jul 08 23:15:26 ip-10-12-34-56.ec2.internal kubelet[4159]: E0708 23:15:26.011749    4159 pod_workers.go:191] Error syncing pod b7593111-1d5f-4b49-8f6d-a6fbd99caf83 ("runner-byxzzoom-project-9754-concurrent-0z4zg6_default(b7593111-1d5f-4b49-8f6d-a6fbd99caf83)"), skipping: failed to "StartContainer" for "init-logs" with ImagePullBackOff: "Back-off pulling image \"registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-c1edb478\""
Jul 08 23:15:26 ip-10-12-34-56.ec2.internal containerd[3531]: time="2021-07-08T23:15:26.139206529Z" level=info msg="StopPodSandbox for \"f9e509c805a42f3407d4d01af96df4b40cf24308d61c1d1b2bdc2b1fde070ad6\""

Sample Helm Chart Config

# ref: https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/master/values.yaml
gitlabUrl: https://git.myorg.net
runnerRegistrationToken: abc12345
unregisterRunners: true

# The maximum number of pods allowed at a single time
concurrent: 10

# Allow Helm to create a Service Account for Gitlab Runner
rbac:
  create: true

runners:
  name: "test-runner"
  tags: "test-runner"
  # privileged: true
  config: |
    log_level = "debug"
    [[runners]]
      [runners.kubernetes]
        namespace = "{{ .Release.Namespace }}"
        image = "alpine"
        # privileged = true
        cpu_request = "2"
        memory_request = "4Gi"

Related: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28050

[bdwyertech]

It appears this is an issue with the gitlab/gitlab-runner-helper image -- when I swapped to the alpinelinux/gitlab-runner-helper the problem went away -- very strange.

[jhaynes]

Thanks for this report @bdwyertech. Happy to keep this open to track this issue but I'm inclined to close it if you're happy with the workaround. If I don't hear back from you, I'll plan on closing in a few days.

[zmrow]

I'll close this. As mentioned, @bdwyertech feel free to re-open if you're not ok with using the alpine image!

[anthr76]

The alpine image didn't work for me though a year old tag "gitlab/gitlab-runner-helper:x86_64-448c28a9" did resolve the issue. Hope to have this resolved soon.

[bcressey]

Re-opening since it's still an issue for you, @anthr76 - I'm doing some work on the SELinux policy now and will investigate before our next release.

[anthr76]

Thanks @bcressey . The linked gitlab issue has some good context as well. Sadly this seems to mostly boil down to GitLab not exposing the Selinux portion of the pod spec to users.

[bcressey]

This should be fixed as of the 1.3.0 release.