override least recently used entries when certificate_data dict is full

[ElvinEfendi]

What this PR does / why we need it:

For #4191 (comment)

I feel like we can do better than this. Here are some alternatives

1. Flush all entries right before the loop - this is not good because while we are in the middle of the loop, the existing certificates will not be available and there might be brief downtime for some domains

2. Get all keys in the dictionary and diff it with hostnames in servers and delete the ones that are not in servers. We can do this, but getting all keys locks the dictionary - if we have many keys/certificates then we might end up locking the dictionary for too long. I presume this is an issue if there are over few thousands of certificates - so maybe this is an acceptable solution?

3. Refactor the code completely and use the logic we use for backends, a.k.a store the servers as raw JSON and have Nginx workers to periodically fetch raw servers, parse it and store it locally. Then when doing TLS handshake the worker would lookup the certificate from worker memory.

I think 3. is the best option.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (master@ecce3fd). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #4232   +/-   ##
=========================================
  Coverage          ?   57.94%           
=========================================
  Files             ?       87           
  Lines             ?     6539           
  Branches          ?        0           
=========================================
  Hits              ?     3789           
  Misses            ?     2322           
  Partials          ?      428

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ecce3fd...e85ccfa. Read the comment docs.

[gork74]

We definitely need this. We create a lot of certificates while building and will hit any fixed limit of the certificate store.

[aledbf]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aledbf, ElvinEfendi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ElvinEfendi,aledbf]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment