azure-npm daemonset pods request 250m CPU instead of 10m. Is this configurable?

[DaveOHenry]

Duplicate of #2033 which was closed without an answer.

[ghost]

Hi DaveOHenry, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

1. If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
2. Please abide by the AKS repo Guidelines and Code of Conduct.
3. If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
4. Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
5. Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
6. If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

[ghost]

Triage required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[ngbrown]

I would also like an answer on this. The last posting on #2033 was by @paulgmiller:

Azure NPM has had some improvements recently to try and address OOMs.
That said we also have a setting on outside that can boost memory limit so
if you're still effected file a support ticket and we'll try and help out.

But the issue described is the baseline reservation of CPU in the manifest for azurn-npm, not the amount of memory the process uses.

In my cluster, azure-npm is reserving 13% of the CPU per node while really using ~0%.

Both @miwithro and @juan-lee were tagged on the closed, but unresolved issue.

[vakalapa]

Because azure-npm is a daemonset, we apply default memory and cpu limits irrespective of the cluster size. Can you give your AKS cluster fqdn, i can check if it is recommended to reduce those limits or not.

Even though NPM steady state is not using a ton of CPU, when a flood of events come in/ NPM restarts for some reason, if CPU limit is reduced and the cluster is fairly large in size then there is a high chance for NPM to be in OOM kill loop.

[ngbrown]

Lowering requested CPU should not generate a out of memory kill loop as it has nothing to do with memory. It also doesn't limit the maximum allowed CPU usage. By lowering the resources.requests.cpu, more pods will fit per node. This is especially acute on 2 vCPU nodes.

[vakalapa]

NPM pods watch pod, namespace and netpol related events and due to some inefficiencies (which we are actively working to solve) can only work on one event at a time. On start of NPM pod, this results in incoming events to be piled up and increasing memory usage. So reducing CPU limit will result in less number of events processed, in turn further aggravating memory usage. We are working on an improved design and until that we want to review a cluster's size before we can reduce the CPU limits.

I agree that this limit can be costly for smaller Node sizes, we have been exploring options on making these limits dynamic based on node size, but i am afraid we do not have a solution in sight. Until then we need to rely on adhoc requests to either increase/decrease the limits based on cluster size. Sorry for the inconvenience.

[ngbrown]

CPU time is a much more fluid resource than memory is. The CPU will get to everything eventually, even in a highly loaded machine. Most clusters are going to be way over subscribed if you add up the maximum limits. Kubernetes prevents scheduling anything additional on a node if the requests add up to more than 100%. So requesting a minimum reservation of 13% on a two vCore instance (or 25% on a single vCore instance) and not consistently using it unreasonably prevents pods from being scheduled.

Other daemon sets usually set their requests.cpu right above idle, while keeping the limits.cpu much higher.

(purple: limits, yellow: requests, blue: usage-max over 30 min)

In the azure-npm case, this issue and the one before was about just lowering the minimum reserved CPU down to above idle, and keeping the maximum limit at the current value.

[ghost]

Triage required from @Azure/aks-pm @vakalapa

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[ngbrown]

This should still be implemented in AKS.

#not-stale

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[DaveOHenry]

ping

…

________________________________ From: msftbot[bot] ***@***.***> Sent: Monday, June 27, 2022 4:01:17 AM To: Azure/AKS ***@***.***> Cc: David Heinrich ***@***.***>; Author ***@***.***> Subject: Re: [Azure/AKS] azure-npm daemonset pods request 250m CPU instead of 10m. Is this configurable? (Issue #2792) This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment. — Reply to this email directly, view it on GitHub<#2792 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AE77SLMIPV62BRL5P3DNAK3VREDO3ANCNFSM5OK7LTYA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[ngbrown]

The advice I've gotten from Microsoft employees is that our cluster's networkPolicy should be switched from azure to calico because of this and other issues aren't being taken care of.

[Kapsztajn]

This should be addressed from MS side... 250m is way to much for CPU Request as it takes basically nothing...

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[Kapsztajn]

Not stale, bump.

[westy-crey]

I'd like to join and also ping this issue. Quarter of a whole CPU to be requested is not justified for such usage. Limits naturally is okay to stay. This is a heavily contributing factor why a k8s system overutilizing itself and allocates more nodes than it is actually necessary.

[mieky]

We also encountered a case where trying to apply a NetworkPolicy to multiple namespaces at once had our azure-npm CPU usage go through the roof (related to #2823).

[jwstauber2]

Bump, currently experiencing this problem as well, needs a resolution.

[kethahel99]

+1 ,we are also experiencing this issue. It is very disruptive. 250mCores is alot

[cjdell]

Needs resolving. We have 3 instances of this thing using up 0.75 cores and we are now at the limit of resources for our cluster. There is barely any CPU activity so it obviously does not need it.

[welersonlisboa]

Needs resolving.. issue still happening for azure-npm

[tomkukral]

I'd like to get this solved ... we are running 400 aks nodes and this azure-npm costs a lot for money due to incorrect requests configuration.

[joshcoburn]

I would also like to see this resolved.. at least give users the ability to tune this.

[thilan3547]

need to resolve this, costing us too much money at the moment

[viktor-gustafsson]

Would also like to see this resolved, seems ridiculous with this high amount of requested resources for what it is doing. This is our primary cost for our AKS cluster.

[yashak]

Needs resolving.. issue is still happening for azure-npm

[restenb]

Same issue. azure-npm pods have high request:

     Limits:                                                                                                                                                                              
       cpu:     250m                                                                                                                                                                      
       memory:  1000Mi                                                                                                                                                                    
     Requests:                                                                                                                                                                            
       cpu:     250m                                                                                                                                                                      
       memory:  300Mi  

While usage over time in a running cluster is usually quite low, 1-3% of request ...

[seguler]

The team is discussing a few improvements here. We will evaluate automatic scaling in the future, but possibly can release a reduction in the short term. @vakalapa will update this thread once the plan is solidified.

[yashak]

Hi @vakalapa ! Any updates on this?