fix: remediator missing custom resource events

[karlkfi]

Prior to this change, the remediator watches were only being started
for new custom resources after the apply attempt had fully completed.
This left some time after the object was applied that the remediator
could miss events made by third-parties. Normally, this would be fine,
because the remediator would revert any change after the watch was
started. But if a DELETE event was missed, the object wouldn't be
recreated until the next apply attempt.

This change adds a CRD Controller to the remediator that watches CRDs
and executes any registered handlers when the CRD is established,
unestablished, or deleted. The remediator now registers CRD handlers
for each resource type it watches, starting watchers as soon as
possible, without waiting for the next apply attempt.

This change also adds a ClusterRole and ClusterRoleBinding specifically
for RepoSync reconcilers, to allow watching of CRDs. RootSync
reconcilers now also watch CRDs, but they already have a CR & CRD.

Fixes: b/355532135

Extracted:

• test: fix flaky controller-manager tests #1445
• chore: cleanup some logs #1446
• chore: remove ClearManagementConflictsWithKind #1447

[karlkfi]

/retest

[nan-yu]

Do we need the lock? The Register function is only invoked once and won't be re-registered.

[karlkfi]

The lock protects the controller from read/write race condition. But the way we're calling Register today it probably won't ever be called in parallel.

[karlkfi]

We don't technically need it today, it's just good defensive practice to protect variables that could be used in parallel.

Locking is cheap. Lock contention is what's expensive.

[karlkfi]

For posterity, since inline comment threads can get lost...

Unfortunately, neither the DynamicRESTMapper in controller-runtime nor the DeferredDiscoveryRESTMapper/CachedDiscoveryClient in client-go implement auto-invalidation of resources, unless aggregated discovery is enabled on the server. While aggregated discovery is beta in 1.27+ and thus enabled by default, that doesn't guarantee that it will be enabled on non-GKE clusters.

1. DynamicRESTMapper only recently added support for aggregated discovery, but it's in a newer controller-runtime than we're using today. While it does handle auto-discovery when aggregated discovery is disabled, it does not handle auto-invalidation.
2. CachedDiscoveryClient does support with aggregated discovery, but when disabled it doesn't handle auto-discovery or auto-invalidation. It just has a cache TTL.

So I've added a new ReplaceOnResetRESTMapper that can be used to replace the RESTMapper when Reset is called. Then I added some code to the watch.Manager to handle calling Reset on the mapper when a CRD is established or unestablished, if the mapper still knows about the deleted resource or doesn't know about a new resource. This handles auto-discovery and auto-invalidation of resources in the RESTMapper, but it's a relatively inefficient.

Hopefully at some point in the future we can make aggregated discovery a requirement and use a simplified DiscoveryClient to handle both auto-discovery and auto-invalidation.

[nan-yu]

/lgtm

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nan-yu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [nan-yu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment