Use timer instead of time.After to prevent memory leaks in logger

[ivan-valkov]

What this PR does / why we need it:

A kind person at gophercon pointed out that our usage of time.After here could lead to a memory leak. This can happen because the actual timer in time.After is not cleaned up until the timer fires. This can be after the work has been accepted on the work queue. In our default setup the timer is only 2s which should be mostly fine, but it is user configurable so this could end up as a memory leak (ex. you set the timer to 1h, and send 1mil requests - you end up with 1 mil timers sitting there for 1h even though the work has been processed).

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

[seldondev]

Hi @ivan-valkov. Thanks for your PR.

I'm waiting for a SeldonIO or todo member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository.

[axsaucedo]

Awesome - nice one @ivan-valkov, very much appreciated 😀

[agrski]

From skimming this article, this confirms the point raised about GC of Timers.
It also indicates the changes here are valid (as we aren't doing anything too clever with this timer).

Nice spot @ivan-valkov!

[seldondev]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: agrski

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [agrski]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[axsaucedo]

It does seem we're getting a consistent operator unit test fail, I can't seem to reproduce in master so seems will require updating (the docs test can be ignored as it seems a new broken link which we'll fix separately) - let me know if you need any pointers. Thanks again for the contribution @ivan-valkov !

/hold

[agrski]

It does seem we're getting a consistent operator unit test fail, I can't seem to reproduce in master so seems will require updating

This is strange, as no operator code is affected by this PR

[axsaucedo]

Yeah not sure why these failures are being caused, I've just fetched the branch and run the test locally but cannot reproduce the failure, so seems to be a CI issue, I will have a deeper look and sync internally so we can merge if indeed just a CI issue

[axsaucedo]

Ok it seems it was a cosmic ray - merging now, thanks again for the contribution @ivan-valkov 🚀