AWX operator is over writing awx config map and causing awe-web to crash

[rakesh561]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

Recently I have upgraded my awx instance from 20.0.1 to 21.10.2 and operator from 0.19.0 to 1.1.3.Every thing went fine I was using make deploy to do operator deployment and then run awx deployment to upgrade to 21.10.2 I have passed the value ipv6_disabled: true upgrade went fine after two days awx-web pod started crashing.When did a deep dig into the issue
awx-awx-configmap is getting overwritten and has entry of ipv6 ngnix config.Wat I did for work around is kubectl edit awx and manually replace ipv6_disabled: false to ipv6_disabled: true then delete awx-awx-configmap manually and restart the awx deployment after that issue is getting fixed.Just trying to understand where I'm going wrong also I see crd awxs.awx.ansible.com is not configured per namespace any reason why it's not configured at namespace level instead its one CRD for entire cluster ?

AWX Operator version

1.1.3

AWX version

21.10.2

Kubernetes platform

kubernetes

Kubernetes/Platform version

v1.19.0

Modifications

no

Steps to reproduce

N/A

Expected results

N/A

Actual results

N/A

Additional information

N/A

Operator Logs

No response

[rooftopcellist]

@rakesh561 the CRD is a cluster-scoped resource. This is a k8s thing, not something we can change. I agree, it is not ideal...

Can you share your AWX CR yaml?

I just tested this out and when ipv6_disabled: true is set, it appears to be templated correctly (no [::] in the nginx.conf). Your changes didn't take effect until you restarted your deployment because of a bug. See this issue for the full explanation:

• Helm upgrade is not detecting AWX kind updates #1239 (comment)

[rakesh561]

@rooftopcellist you mean cm ?

[rakesh561]

@rooftopcellist this is the example where my spec is getting overwritten by operator and ipv6_disabled value is removed from my config.

Please edit the object below. Lines beginning with a '#' will be ignored,

and an empty file will abort the edit. If an error occurs while saving this file will be

reopened with the relevant failures.

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
annotations:
meta.helm.sh/release-name: awx
meta.helm.sh/release-namespace: devops-appvirt
creationTimestamp: "2023-02-09T16:55:08Z"
generation: 1
labels:
app.kubernetes.io/component: awx
app.kubernetes.io/managed-by: awx-operator
app.kubernetes.io/name: awx
app.kubernetes.io/operator-version: 1.1.4
app.kubernetes.io/part-of: awx
name: awx
namespace: devops-appvirt
resourceVersion: "296104382"
selfLink: /apis/awx.ansible.com/v1beta1/namespaces/devops-appvirt/awxs/awx
spec:
admin_password_secret: awx-admin-password
admin_user: admin
create_preload_data: true
ee_images:

• image:
  name: ansible-2.10-stable
• image:
  name: ansible-2.11-stable
  ee_resource_requirements:
  requests:
  cpu: 250m
  memory: 300Mi
  garbage_collect_secrets: false
  image: quay.io/ansible/awx
  image_pull_policy: IfNotPresent
  image_version: 21.10.2
  ldap_cacert_secret: c
  loadbalancer_port: 80
  loadbalancer_protocol: http
  nodeport_port: 30080
  projects_persistence: false
  projects_storage_access_mode: ReadWriteMany
  projects_storage_size: 8Gi
  replicas: 1
  route_tls_termination_mechanism: Edge
  task_privileged: false
  task_resource_requirements:
  requests:
  cpu: 250m
  memory: 300Mi
  web_resource_requirements:
  requests:
  cpu: 250m
  memory: 200Mi
  status:

[rakesh561]

You can see from the below output ipv6_disabled: true was set to true initially and its getting overwritten
kubectl edit awx

Please edit the object below. Lines beginning with a '#' will be ignored,

and an empty file will abort the edit. If an error occurs while saving this file will be

reopened with the relevant failures.

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
annotations:
meta.helm.sh/release-name: awx
meta.helm.sh/release-namespace: devx-mongo
creationTimestamp: "2023-02-03T19:48:21Z"
generation: 2
labels:
app.kubernetes.io/component: awx
app.kubernetes.io/managed-by: awx-operator
app.kubernetes.io/name: awx
app.kubernetes.io/operator-version: 1.1.4
app.kubernetes.io/part-of: awx
name: awx
namespace: devx-mongo
resourceVersion: "31414229"
selfLink: /apis/awx.ansible.com/v1beta1/namespaces/devx-mongo/awxs/awx
spec:
admin_password_secret: awx-admin-password
admin_user: admin
auto_upgrade: true
create_preload_data: true
ee_images:

• image:
  name: ansible-2.10-stable
• image:
  name: ansible-2.11-stable
  ee_resource_requirements:
  requests:
  cpu: 250m
  memory: 100Mi
  garbage_collect_secrets: false
  image: quay.io/ansible/awx
  image_pull_policy: IfNotPresent
  image_version: 21.10.2
  ipv6_disabled: true

[rooftopcellist]

There seems to be a bit of confusion here. I will try to clarify:

• There was a bug that was causing deployments to not be properly cycled after changes to the ConfigMap. The result is that someone would make a change to the AWX spec, and the change would not end up in the resulting pod (with the configmap mounted in it) until the deployment/pod was scaled down of deleted.
• cm is the abbreviation/shorthand for ConfigMap for k8s.
• Other than the fact that the indentation is not right on your AWX CR spec, the content looks to be otherwise valid.
• CRD == CustomResourceDefinition, which defines the fields on a custom resource. This is a global resource (cluster-wide). CR == CustomResource, an instance of a CRD.

Since the bug described in the issue was fixed by this commit that merged yesterday, I think we can close this.

• 94d68bf#diff-29de64e6284e07a8a16b351b132d812618c7f4e3485cec09987c2374257b94fcR39

A release with those changes should be happening on Monday.