Checksum changing without config changing

[mikebryant]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): checksum

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version: 0.21.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.10", GitCommit:"098570796b32895c38a9a1c9286425fb1ececa18", GitTreeState:"clean", BuildDate:"2018-08-02T17:19:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.10", GitCommit:"098570796b32895c38a9a1c9286425fb1ececa18", GitTreeState:"clean", BuildDate:"2018-08-02T17:11:51Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Openstack
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened: nginx.conf is being changed, and nginx reloaded, but there appears to be no actual change

What you expected to happen: For the config not to reload if there's no change

How to reproduce it (as minimally and precisely as possible):
Unsure

Anything else we need to know:

Debug, from a controller pod, at different times:

www-data@nginx-ingress-controller-5767dc895f-sgtm2:/etc/nginx$ cp nginx.conf /tmp/nginx.conf.$(date +%s)



gave:
www-data@nginx-ingress-controller-5767dc895f-sgtm2:/etc/nginx$ diff /tmp/nginx.conf.1543512863 /tmp/nginx.conf.1543513124
2c2
< # Configuration checksum: 15373257790483374758
---
> # Configuration checksum: 2651252075192297349
www-data@nginx-ingress-controller-5767dc895f-sgtm2:/etc/nginx$ diff /tmp/nginx.conf.1543512863 /tmp/nginx.conf.1543513477
2c2
< # Configuration checksum: 15373257790483374758
---
> # Configuration checksum: 3523881217680486521

[mikebryant]

Some of our Ingress objects are having their .metadata.ownerReferences and .metadata.annotations.kubectl.kubernetes.io/last-applied-configuration fields updated, by our deployment process.

I would not have expected those changes to result in an nginx reload (and the corresponding drop of tcp connections)

[aledbf]

@mikebryant please add the flag --v=2 in the ingress controller deployment

[mikebryant]

(After some conversation with @aledbf - many thanks!) - looks like this is because the ownerReferences and last-applied-configuration annotation are being used to compute the config hash, and they shouldn't be

[horkyada]

Hi, can I ask what is the status of this? I see the merged PR, but the issue remains opened. We actually suffer from ingress nginx being reloaded too often and killing connections even if there are no changes in config. Fixing this would resolve our problems, right?

[aledbf]

Closing. Please update to 0.22.0. Reopen if the issue persist after the upgrade

[stepanselyuk]

It seems I have like this issue, mentioned above.

Wed Mar 13 17:44:35 UTC 2024: Checksum changed to cc1036f4ab401780b3c983e13722736a
Diff:
2c2
< # Configuration checksum: 590285667225438696
---
> # Configuration checksum: 8675996187203755368
Wed Mar 13 17:44:39 UTC 2024: Checksum changed to 41358552eee6f9ca8773b4c0be55d51e
Diff:
2c2
< # Configuration checksum: 8675996187203755368
---
> # Configuration checksum: 17966220645503737419
Wed Mar 13 17:44:42 UTC 2024: Checksum changed to 050110d89b8a32096c7dd26a80a971d6
Diff:
2c2
< # Configuration checksum: 17966220645503737419
---
> # Configuration checksum: 4767872212545264337
Wed Mar 13 17:44:46 UTC 2024: Checksum changed to 42783606f0250d7355ac9991334ad494
Diff:
2c2
< # Configuration checksum: 4767872212545264337
---
> # Configuration checksum: 15736275022131449068
Wed Mar 13 17:44:53 UTC 2024: Checksum changed to c145c4ccae0e2e456603e99720a8e263
Diff:
2c2
< # Configuration checksum: 15736275022131449068
---
> # Configuration checksum: 18323211867010836222
Wed Mar 13 17:44:56 UTC 2024: Checksum changed to 348142b692272153096064e9189748e4
Diff:
2c2
< # Configuration checksum: 18323211867010836222
---
> # Configuration checksum: 13989427767816962574
^C

#!/bin/bash

prev_checksum=""
prev_content=""

while true; do
  current_content=$(cat nginx.conf)
  current_checksum=$(openssl md5 nginx.conf | awk '{print $2}')

  if [[ "$prev_checksum" != "" && "$prev_checksum" != "$current_checksum" ]]; then
    echo "$(date): Checksum changed to $current_checksum"
    echo "Diff:"
    diff <(echo "$prev_content") <(echo "$current_content")
  fi

  prev_checksum=$current_checksum
  prev_content="$current_content"
  
  sleep 1 # Adjust the sleep as needed
done

ingress-nginx/controller:v1.7.0