BUG: Cannot pull awx image from private container registry

[avwede]

ISSUE TYPE

• Bug Report

SUMMARY

I cannot pull the awx image from my Gitlab private container registry, yet the other image dependencies are pulling fine.

ENVIRONMENT

• AWX version: 0.17.0
• Operator version: 0.17.0
• Kubernetes version: v1.22.6+k3s1
• AWX install method: docker on AlmaLinux (Airgap install)

STEPS TO REPRODUCE

• Clone and build awx-operator
• Create private container registry on GitLab and push docker images for postgres:12
• Create secret for GitLab credentials and CRD for AWX instance
• Apply the AWX instance

EXPECTED RESULTS

Successfully running AWX Instance that is correctly pulling all images from private registry.

ACTUAL RESULTS

Failed to pull awx image from private container registry.

ADDITIONAL INFORMATION

AWX Instance Configuration YAML

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-instance-1
  namespace: awx
spec:
  admin_user: admin
  admin_email: admin@dev
  admin_password_secret: awx-instance-1-admin-password
  projects_persistence: true
  postgres_storage_class: local-path
  service_type: ClusterIP
  postgres_image: dev.awx/devops/awx-operator/postgres
  postgres_image_version: "12"
  redis_image: dev.awx/devops/awx-operator/redis
  redis_image_version: "6.2.6"
  image_pull_secret: awx-puller
  web_resource_requirements: {}
  task_resource_requirements: {}
  ee_resource_requirements: {}
  image: dev.awx/devops/awx-operator/awx
  image_version: "20.0.1"
  ee_images:
    - name: awx-ee
      image: dev.awx/devops/awx-operator/awx-ee:0.6.0
  control_plane_ee_image: dev.awx/devops/awx-operator/awx-ee:0.6.0
  init_container_image: dev.awx/devops/awx-operator/centos
  init_container_image_version: "8"
  projects_storage_class: local-path
  projects_storage_access_mode: ReadWriteOnce
---
apiVersion: v1
kind: Secret
metadata:
  name: awx-instance-1-admin-password
  namespace: awx
stringData:
  password: awxpassword

Docker secret

sudo kubectl create secret docker-registry awx-puller --docker-server=dev.awx --docker-username=dev.svc --docker-password=dev -n awx

AWX-OPERATOR LOGS

  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  41m                  default-scheduler  Successfully assigned awx/awx-instance-1-64d5fcbfb5-57wrz to dev-awx
  Normal   Pulled     41m                  kubelet            Container image "dev.awx/devops/awx-operator/centos:8" already present on machine
  Normal   Created    41m                  kubelet            Created container database-check
  Normal   Started    41m                  kubelet            Started container database-check
  Normal   Pulled     41m                  kubelet            Container image "dev.awx/devops/awx-operator/centos:8" already present on machine
  Normal   Created    41m                  kubelet            Created container init
  Normal   Started    41m                  kubelet            Started container init
  Normal   Pulled     41m                  kubelet            Container image "dev.awx/devops/awx-operator/redis:6.2.6" already present on machine
  Normal   Created    41m                  kubelet            Created container redis
  Normal   Started    41m                  kubelet            Started container redis
  Normal   Pulling    41m                  kubelet            Pulling image "dev.awx/devops/awx-operator/awx:20.0.1"
  Warning  Failed     41m                  kubelet            Failed to pull image "dev.awx/devops/awx-operator/awx:20.0.1": rpc error: code = Unknown desc = failed to pull and unpack image "dev.awx/devops/awx-operator/awx:20.0.1": failed to extract layer sha256:49ec979a67ca5a2f58231972544c2e40d859dd9793f3bdfd048ecd7e23141488: failed to unmount /var/lib/rancher/k3s/agent/containerd/tmpmounts/containerd-mount4075416591: failed to unmount target /var/lib/rancher/k3s/agent/containerd/tmpmounts/containerd-mount4075416591: device or resource busy: unknown
  Warning  Failed     41m                  kubelet            Error: ErrImagePull
  Normal   Pulled     41m                  kubelet            Container image "dev.awx/devops/awx-operator/awx-ee:0.6.0" already present on machine
  Normal   Created    41m                  kubelet            Created container awx-instance-1-ee
  Normal   Started    41m                  kubelet            Started container awx-instance-1-ee
  Warning  Failed     41m (x2 over 41m)    kubelet            Error: ImagePullBackOff
  Normal   BackOff    41m (x3 over 41m)    kubelet            Back-off pulling image "dev.awx/devops/awx-operator/awx:20.0.1"
  Warning  Failed     41m (x3 over 41m)    kubelet            Error: ImagePullBackOff
  Normal   BackOff    65s (x166 over 41m)  kubelet            Back-off pulling image "dev.awx/devops/awx-operator/awx:20.0.1"

[rebeccahhh]

Hi @avwede can you provide more info into the k8s secrets that authenticate to your gitlab? This may be a configuration issue.

[lals1]

Experiencing the same problem with pulling the image from private registry. Custom pod with pull_secret can successfully pull the image but it doesn't work with AWX CRD.

[rebeccahhh]

Circling back, @avwede I realize it may have looked like I was asking for K8s secrets, which I am not, but I am looking for the configuration of how you use them. 😄
@lals1 thanks for the comment, you're also using Kubernetes?

[lals1]

@rebeccahhh Yes, AWX is installed via AWX-operator on k8s. My AWX install definition looks like below. In AWX Web UI it does show EE image URL reference but its missing pull secret info. When launching some job with the EE image, it fails. kubectl event show Error: ImagePullBackOff even though I have made regcred available via k8s secrets before instantiating AWX instance.

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: my-awx
  namespace: awx
spec:
  service_type: clusterip
  image: quay.io/ansible/awx
  image_version: 20.0.1
  postgres_image: postgres
  postgres_image_version: "12"
  ee_images:
    - name: awx-execution-environment
      image: myregistry/awx-ee:latest
  image_pull_secret: regcred

[jompins]

Can confirm this for my setup here as well. I configured everything according to the specs but the default awx-ee will always use quay.io instead of my private registry only for the execution image. But this causes the deployment to stall. No secrets in use since it is all public for testing. Totally Air gaped setup.

spec:
  service_type: nodeport
  image: server.local:9090/ansible/awx
  image_version: 19.5.0
  image_pull_policy: Always
  postgres_image: server.local:9090/ansible/postgres
  postgres_image_version: v12
  redis_image: server.local:9090/ansible/redis
  redis_image_version: latest
  init_container_image: server.local:9090/ansible/centos
  init_container_image_version: stream8
  ee_images:
    - name: ngs2-awx
      image: server.local:9090/ansible/awx-ee:latest

[ssawulski]

Same issue here
This is a problem for deployments in environments where images from untrusted sources are blocked.

[jompins]

For what its worth: adding the "control_plane_ee_image: " key and pointing it to the awx-ee image in my private registry, did the trick.

[beerfranz]

Hello, another workaround is to

1. Create an Instance Group, of type "Container group" and with the imagePullSecrets in "Pod spec override"
2. Link your inventory to this Instance Group to use it

Pod spec example:

apiVersion: v1
kind: Pod
metadata:
  namespace: awx
spec:
  imagePullSecrets:
    - name: ecr-docker-login

[sunshine69]

Same issues here, why the init image can not be overridden.

[sunshine69]

For what its worth: adding the "control_plane_ee_image: " key and pointing it to the awx-ee image in my private registry, did the trick.
This does not work for me though

[rooftopcellist]

@avwede It looks like you were running in to a bug which was fixed by this PR:

• Add back image_pull_secret field for backwards compatibility #870

In summary, we added a new variable image_pull_secrets, which takes a list, not a string, which allows users to pass more than one pull secret if needed. Unfortunately, we didn't have the foresight to add a backwards compatibility shim so that users who had the old secret wouldn't be broken. That backwards compatibility was added in the PR linked above.

This fix became available in awx-operator 0.20.1:

• https://github.com/ansible/awx-operator/releases/tag/0.20.1

@sunshine69 The init container image can be overwritten:

• https://github.com/ansible/awx-operator/blob/devel/config/crd/bases/awx.ansible.com_awxs.yaml#L750-L755

I will close this issue for now. Please open a new issue if you are still experiencing an issue on the latest release. Thanks!

[sunshine69]

@rooftopcellist I found it yesterday after reading the source code, commented on other ticket for some others.

What I see is a lack of documentation, or in accurate. The information is not easily shown in the first README.md or reference. Otherwise I do not need to start reading source code (remember trace everything from the architecture of the operator sdk and then ansible operator then the ansible role ...

If we can improve documentation then it would be a lot of time savers.

Thanks