Track store readiness

[nightkr]

Motivation

Fixes #1226.

Solution

We add a new wait_until_ready() method to Store, which returns a Future that completes after any write has been done to the store. If the store has already been changed in the past then it completes immediately.

Further, controller::Runner is extended with a mechanism that allows us to gate all jobs until a trigger Future has completed. This is then used in controller::applier to gate reconciliation based on the store readiness introduced earlier.

[codecov]

Codecov Report

Merging #1243 (047a73f) into main (bda3b90) will increase coverage by 0.70%.
The diff coverage is 95.37%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1243      +/-   ##
==========================================
+ Coverage   71.89%   72.60%   +0.70%     
==========================================
  Files          71       72       +1     
  Lines        5729     5899     +170     
==========================================
+ Hits         4119     4283     +164     
- Misses       1610     1616       +6     

Impacted Files	Coverage Δ
kube-runtime/src/utils/mod.rs	64.15% <ø> (ø)
kube-runtime/src/controller/mod.rs	35.15% <85.71%> (+1.42%)	⬆️
kube-runtime/src/controller/runner.rs	94.36% <93.67%> (-1.09%)	⬇️
kube-runtime/src/utils/delayed_init.rs	97.40% <97.40%> (ø)
kube-runtime/src/reflector/store.rs	98.94% <100.00%> (+2.47%)	⬆️

[clux]

Initial comments. Short story is that I really like the approach (even though there's a lot more here than I expected from a one-day nerd snipe). Very good tests.

Some questions mainly around complexity and public interfaces.

[clux]

Very transparent from a user POV with debug logs kube=debug (trimmed timestamps for the 0.1s this took):

INFO kube_runtime::controller: press ctrl+c to shut down gracefully
DEBUG kube_runtime::controller: applier runner held until store is ready
DEBUG HTTP{http.method=GET http.url=https://0.0.0.0:51145/apis/nullable.se/v1/configmapgenerators? otel.name="list" otel.kind="client"}: kube_client::client::builder: requesting
DEBUG HTTP{http.method=GET http.url=https://0.0.0.0:51145/api/v1/configmaps? otel.name="list_metadata" otel.kind="client"}: kube_client::client::builder: requesting
DEBUG kube_runtime::controller: store is ready, starting runner
DEBUG HTTP{http.method=GET http.url=https://0.0.0.0:51145/apis/nullable.se/v1/configmapgenerators?&watch=true&timeoutSeconds=290&allowWatchBookmarks=true&resourceVersion=779 otel.name="watch" otel.kind="client"}: kube_client::client::builder: requesting
DEBUG HTTP{http.method=GET http.url=https://0.0.0.0:51145/api/v1/configmaps?&watch=true&timeoutSeconds=290&allowWatchBookmarks=true&resourceVersion=779 otel.name="watch_metadata" otel.kind="client"}: kube_client::client::builder: requesting