feat(insights): jitter zone insights upsert

[jakubdyszkiewicz]

Checklist prior to review

• Use upsert settings once again (it was used at some point and then dropped?). This way we can customize those values without releasing new version
• Add jitter for storing envoy streams
• Add jitter to upserts
• Change default values to 200ms, max 10 retries and 30% jitter.

Fix ##7921

• Link to relevant issue as well as docs and UI issues --
• This will not break child repos: it doesn't hardcode values (.e.g "kumahq" as a image registry) and it will work on Windows, system specific functions like syscall.Mkfifo have equivalent implementation on the other OS --
• Tests (Unit test, E2E tests, manual test on universal and k8s) --
• Do you need to update UPGRADE.md? --
• Does it need to be backported according to the backporting policy? (this GH action will add "backport" label based on these file globs, if you want to prevent it from adding the "backport" label use no-backport-autolabel label) --
• Do you need to explicitly set a > Changelog: entry here or add a ci/ label to run fewer/more tests?

[lahabana]

LGTM. Random question not directly related to this:

Why do we need to keep a globalInstanceId per EnvoyStream type? I thought there was a single KDS connection per Zone no?

Answering myself: I guess that's not true. Each Envoy Admin is a different stream so it's possible they'd end up on different connections and therefore LoadBalanced to different GCP instance

Correct?

[jakubdyszkiewicz]

Correct.