Add an option to automatically set worker_connections based on worker_rlimit_nofile

[ramnes]

What this PR does / why we need it:

This PR makes the max-worker-connections option use maxOpenFile by default rather than 16384, and adds a max-worker-open-files option for those who'd want to manually set both.

See #3557 for rationale.

Which issue this PR fixes:

It closes #3557.

Special notes for your reviewer:

First of all, thanks for passing by! Please note that I'm not a Gopher; any remark is more than welcome.

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please email the CNCF helpdesk: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[ramnes]

@k8s-ci-robot <3

[aledbf]

/lgtm
/approve

[aledbf]

@ramnes thanks!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aledbf, ramnes

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [aledbf]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[aledbf]

@ramnes I am going to rever the default value for MaxWorkerConnections. Using 0 as default increases the default memory usage ~3.5x something that is not acceptable.

Screenshot of the controller in an empty cluster without ingresses

[ramnes]

@aledbf This is really weird, I don't see any reason that it would change the memory usage when idle. Are you sure that it's related to that commit? Also, in your example, what is the value that got generated?

[aledbf]

Are you sure that it's related to that commit?

Yes. The value was 261120 (same than worker_rlimit_nofile). Once I returned the value to the previous level the memory usage returned to the "normal" value

[ramnes]

I just tried it myself and I can confirm that Nginx RAM usage on idle and worker_connections are highly correlated. I wouldn't have expected this and it's good to know, thanks for the find! Looks like Nginx is pre-allocating its events space, most likely to avoid the penalty of doing it when scaling up.