High CPU usage in 2.8.0 in application controller

[akselleirv]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When I upgrade from version v2.7.10 to v2.8.0 I have large increase CPU and memory usage as seen in the screenshot below.

Screenshots

Resource usage:

You can see the downgrade from v2.8.0 to v.2.7.10 around 11:00 which results in a large decrease of resource usage.

The application controller:

Nothing major changes in the other pods:

Argo metrics:

Version

v2.8.0

Logs

I've looked through the logs, but yet to find any hints of what the cause could be.

Here is a pprof profile of the application-controller: argocd-application-controller-1.Profile.zip

Let me know if you need any more information.

[leoluz]

Thank you for the great details in your issue (pprof helps a lot)!

Analyzing the graph below, Argo CD is mainly doing diff calculation for your apps. Your cache is probably invalidated in 2.8 if you are using ignore diffs. Can you confirm if you have many apps with ignore diffs configuration? This was a bug in previous versions (#12610).

In your case, I suspect that once the cache is rebuilt the CPU will stabilize. We are running 2.8 in several instances and didn't notice a general increase in CPU.

[akselleirv]

The left view is the CPU usage and the right view is the log lines. When the new version is rolled out then a the high CPU usage starts, and it logs these warnings pasted below.

Logs:

2023-08-25 11:22:41.186 | W0825 09:22:41.186328       6 warnings.go:70] unknown field "status.sync.comparedTo.ignoreDifferences"  
2023-08-25 11:22:41.186 | W0825 09:22:41.186305       6 warnings.go:70] unknown field "status.controllerNamespace"

The ignoreDifferences error might the related to the high CPU usage as you indicated in your comment regarding previous issues.

[akselleirv]

Found the issue! The problem was that the CRDs was not installed thus related in the log output.