Helm upgrade is not detecting AWX kind updates

[danielmorillas]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

I have a custom deployment with helm and fluxcd which no updates from values.yaml were detected.

So, I manually tested the official helm chart for version 1.1.3 and 1.1.4 and same issues.

I install the chart with a specific values.yaml and all is ok, once I upgrade the chart with a new change (i.e LDAP Manager DN update) it's not completely detected.

AWX resource is modified but the configmap keeps the value of the first installation.

AWX Operator version

1.1.4

AWX version

latest

Kubernetes platform

kubernetes

Kubernetes/Platform version

1.23
kind v0.11.0 go1.16.4 linux/amd64

Modifications

no

Steps to reproduce

#After helm upgrade ...

kubectl get configmap -n dev-ops-playground -o yaml awx-awx-configmap | grep "testldap"
AUTH_LDAP_BIND_DN = "testldap"
...

kubectl get awx -n dev-ops-playground -o yaml awx | grep "testldap2"
value: '"testldap2"'

Expected results

Configmap should be updated but once execute helm upgrade command it's not detecting any update in the configmap.

Actual results

helm upgrade awx awx-operator/awx-operator -n dev-ops-playground -f ../helm-chart-awx-operator/values.yaml --debug
upgrade.go:142: [debug] preparing upgrade for awx
upgrade.go:150: [debug] performing update for awx
upgrade.go:322: [debug] creating upgraded release for awx
client.go:218: [debug] checking 12 resources for changes
client.go:501: [debug] Looks like there are no changes for ServiceAccount "awx-operator-controller-manager"
client.go:501: [debug] Looks like there are no changes for ConfigMap "awx-operator-awx-manager-config"
...

--

Source: awx-operator/templates/awx-deploy.yaml

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
name: awx
...
spec:
...

• setting: AUTH_LDAP_BIND_DN
  value: '"testldap2"'
  ...
  Release "awx" has been upgraded. Happy Helming!

Additional information

The only way that I saw how this can change is deleting the awx deployment and the new value is applied.

Same error in Openshift cluster!

Operator Logs

No response

[danielmorillas]

I guess as there are no templates related to awx configmap and awx deployment (they are in the image build creation) to be detected those changes in helm, it's updated the AWX Kind resource but no change is taking until the awx deployment is deleted.

[rooftopcellist]

I can't speak to helm.. but as long as it got changed on the AWX spec and the syntax for the extra_settings added is correct, it should also get changed in the configmap. However, it will take ~1-2 minutes to be reflected in the awx configmap. This is because once you change the AWX spec, the reconciliation loop gets triggered and runs the installer role, it takes a bit of time to get to the task that applies changes to the configmap.

---

Correction: When testing this out locally, I find that the ConfigMap is updated properly, but the /etc/tower/settings.py files is not updated without cycling the deployment pods.

To test this, I added the following change in my AWX spec:

  spec:
    extra_settings:
    - setting: MAX_PAGE_SIZE
      value: "500"

Useful command for checking if the change was reflected in the ConfigMap:

$ oc get cm awx-awx-configmap -o yaml | grep MAX_PAGE_SIZE

Useful command for checking if the change was reflected in the container:

$ oc exec -it deployments/awx -c awx-task -- cat /etc/tower/settings.py | grep MAX_PAGE_SIZE

[rooftopcellist]

It seems that the task is not getting marked as changed=true if only the value of an existing field on the ConfigMap is changed.

However, when a new key/value pair is added to a configMap, the task is marked as changed.

Note: changing the task that applies the ConfigMap to force=true does not fix it.

---

I wasn't able to reproduce with a playbook outside the reconciliation loop. Here is what I tried:

• https://gist.github.com/rooftopcellist/dba80e140e9793cf0e37c124bb8bb17d

I tried running this inside the container and on my dev machine, so that rules out different kubernetes.core versions.

Maybe there is some nuance to how the installer role is run by the operator/reconciliation loop that I am missing...

[rooftopcellist]

If I create a run.yml file in the roles/ directory like this:

- hosts: localhost
  vars:
    ansible_operator_meta:
      name: awx
      namespace: awx
    service_type: nodeport
    extra_settings:
    - setting: MAX_PAGE_SIZE
      value: "222"

  roles:
    - installer

And run the installer role directly:

ansible-playbook run.yml

I see that the task is marked as changed:

However, I don't see this when it is run as part of the reconciliation loop...

cc @TheRealHaoLiu any ideas here?

[danielmorillas]

Thanks for your feedback!

Focused on helm... Helm is upgrading based on the templates built for the chart, for instance using a helm upgrade (with debug option), we can see what helm is trying to upgrade:

root@dmorillas-dev:~/dmorillas/awx-iac-gitops# helm upgrade awx -n dev-ops-playground helm-chart-awx-operator/ -f helm-chart-awx-operator/values.yaml --debug
upgrade.go:142: [debug] preparing upgrade for awx
upgrade.go:150: [debug] performing update for awx
upgrade.go:322: [debug] creating upgraded release for awx
client.go:218: [debug] checking 12 resources for changes
client.go:501: [debug] Looks like there are no changes for ServiceAccount "awx-operator-controller-manager"
client.go:501: [debug] Looks like there are no changes for ConfigMap "awx-operator-awx-manager-config"
client.go:501: [debug] Looks like there are no changes for ClusterRole "awx-operator-metrics-reader"
client.go:501: [debug] Looks like there are no changes for ClusterRole "awx-operator-proxy-role"
client.go:501: [debug] Looks like there are no changes for ClusterRoleBinding "awx-operator-proxy-rolebinding"
client.go:510: [debug] Patch Role "awx-operator-awx-manager-role" in namespace dev-ops-playground
client.go:501: [debug] Looks like there are no changes for Role "awx-operator-leader-election-role"
client.go:501: [debug] Looks like there are no changes for RoleBinding "awx-operator-awx-manager-rolebinding"
client.go:501: [debug] Looks like there are no changes for RoleBinding "awx-operator-leader-election-rolebinding"
client.go:501: [debug] Looks like there are no changes for Service "awx-operator-controller-manager-metrics-service"
client.go:510: [debug] Patch Deployment "awx-operator-controller-manager" in namespace dev-ops-playground
client.go:510: [debug] Patch AWX "awx" in namespace dev-ops-playground
upgrade.go:157: [debug] updating status for upgraded release for awx
Release "awx" has been upgraded. Happy Helming!

Of course, it's patching changes in AWX resource but not the configmap (That should do it for the LDAP change...).
I've been based on LDAP change (that should be reflected in that configmap) because it's the change that I've been testing days ago.

Besides that, Helm is not able to upgrade any of the values.yaml configuration , for instance, I tested with image and image_version parameters.
Helm does not take any action after executing a helm upgrade, bad solution: helm upgrade (update AWX resource)+ delete awx deployment (Execute changes on awx pod)
If this works in that way this is a limitation for helm / GitOps environments.

I guess Helm should take any action once AWX resource is updated (redeploy the pod, update the specific configmap without redeploy it...)

[danielmorillas]

Does anyone have any ideas or is the same thing happening?

[rooftopcellist]

@danielmorillas this should be fixed as part of this commit which was merged yesterday as part of :

• Refine deployment rollouts #1222

This will be in the next release, which I believe is Monday.

Please open another issue linking to this one if it is not fixed by that patch.