feat: vap generation

[ritazh]

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged):
Fixes #2874

Special notes for your reviewer:

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 24.06639% with 183 lines in your changes are missing coverage. Please review.

Project coverage is 54.18%. Comparing base (6d1e4ee) to head (f5ea086).
Report is 3 commits behind head on master.

Files	Patch %	Lines
pkg/controller/constraint/constraint_controller.go	0.00%	126 Missing ⚠️
...onstrainttemplate/constrainttemplate_controller.go	49.09%	42 Missing and 14 partials ⚠️
pkg/target/target.go	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3266      +/-   ##
==========================================
- Coverage   54.28%   54.18%   -0.10%     
==========================================
  Files         134      134              
  Lines       12356    10247    -2109     
==========================================
- Hits         6707     5552    -1155     
+ Misses       5152     4190     -962     
- Partials      497      505       +8     

Flag	Coverage Δ
unittests	54.18% <24.06%> (-0.10%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[maxsmythe]

Minor comments, but very nice!

[maxsmythe]

I see.

Where is the constraint controller reading from constraintEvents? Should we just be writing to regEvents instead?

If we want to use a second channel, we'd need to plumb it through.

[ritazh]

@maxsmythe @sozercan PTAL when you get a chance

[maxsmythe]

nit: I'd make this an else if statement, otherwise we make two writes to the API server instead of one (on account of currentVAP being nil, guaranteeing reflect.DeepEqual() will be false.

[maxsmythe]

nit: I'd make this an else if statement, otherwise we make two writes to the API server instead of one (on account of currentVAP being nil, guaranteeing reflect.DeepEqual() will be false.

[maxsmythe]

I think we actually want to do this after create/delete (not update), since we want to re-reconcile constraints if the "use VAP" annotation ever changes (so constraints can re-evaluate whether their default behavior should change).

On update "use VAP" was true and will continue to be true. On create/delete "use VAP" has changed (or the CT is brand new/just deleted)

[maxsmythe]

We should add the "listObjects/replay constraints" behavior here as well (might make sense to factor that out into its own function to avoid code duplication)

[maxsmythe]

Thinking more deeply about this, I think we have a general race condition. Let's pretend:

1. constraint controller is fast
2. CT controller is slow (likely, since it may require compilation of Rego/CEL)
3. there is some constraint Foo
4. It is backed by some template FooTemplate
5. there are 3 events for FooTemplate: no-generate-vap, generate-vap, no-generate-vap
6. FooTemplate reconciler is slow enough that only two events get reconciled: no-generate-vap, no-generate-vap
7. constraint controller gets a reconcile while generate-vap is set

If this sequence of events happens, you could get a dangling VAP binding b/c when constraint controller reconciles, it's gonna run client.Get() on the informer cache and see the annotation to create a binding, but the constraint template controller sees no change (from its view no-generate-vap has always been set), so it will not trigger a reconcile.

I think I see a way out of this, but it's not super pretty:

1. create a shared cache between the constraint template and constraint controllers that stores the CT controller's opinion as to whether a constraint kind should/should not generate VAP objects
2. create a function on the CT reconciler called shouldGenerateVAP(kind string, shouldGenerate bool), when shouldGenerate changes for a particular kind, that's what updates the cache and triggers a re-reconcile of all constraints for a given template
3. call shouldGenerateVAP() where we are currently running list calls (might need to think about the timing)
4. The constraint controller reads from the cache, rather than the object
5. Because the results of reconciliation are dependent on internal state, we should run reconciliation as a singleton process (i.e. throw it on the status/audit Pod), this will prevent thrashing of the k8s resources whenever the pods' watch events are out of sync

(5) is probably a good thing to address generally, since there is also a chance of thrashing on G8r upgrade. I'd suggest that any controller that creates a downstream object (CRDs, VAP objects), the creation runs as a singleton (we can carve it out as an operation, pehaps, like audit or status updates). The controllers should still run for the purposes of caching state (e.g. loading constraint templates into CF), and they should still write out pod status objects.

This is medium-to-large body of work. I would recommend we do this before graduating to beta (or at least before allowing users to set failure policy to Fail). However, I don't think we need to block the current PR on this because:

• This is an in-development feature that is off-by-default
• We currently fail open, so a dangling binding will not cause any issues

[maxsmythe]

LGTM, but see my caveat about a race condition above.

[maxsmythe]

Thinking more deeply about this, I think we have a general race condition. Let's pretend:

1. constraint controller is fast
2. CT controller is slow (likely, since it may require compilation of Rego/CEL)
3. there is some constraint Foo
4. It is backed by some template FooTemplate
5. there are 3 events for FooTemplate: no-generate-vap, generate-vap, no-generate-vap
6. FooTemplate reconciler is slow enough that only two events get reconciled: no-generate-vap, no-generate-vap
7. constraint controller gets a reconcile while generate-vap is set

If this sequence of events happens, you could get a dangling VAP binding b/c when constraint controller reconciles, it's gonna run client.Get() on the informer cache and see the annotation to create a binding, but the constraint template controller sees no change (from its view no-generate-vap has always been set), so it will not trigger a reconcile.

I think I see a way out of this, but it's not super pretty:

1. create a shared cache between the constraint template and constraint controllers that stores the CT controller's opinion as to whether a constraint kind should/should not generate VAP objects
2. create a function on the CT reconciler called shouldGenerateVAP(kind string, shouldGenerate bool), when shouldGenerate changes for a particular kind, that's what updates the cache and triggers a re-reconcile of all constraints for a given template
3. call shouldGenerateVAP() where we are currently running list calls (might need to think about the timing)
4. The constraint controller reads from the cache, rather than the object
5. Because the results of reconciliation are dependent on internal state, we should run reconciliation as a singleton process (i.e. throw it on the status/audit Pod), this will prevent thrashing of the k8s resources whenever the pods' watch events are out of sync

(5) is probably a good thing to address generally, since there is also a chance of thrashing on G8r upgrade. I'd suggest that any controller that creates a downstream object (CRDs, VAP objects), the creation runs as a singleton (we can carve it out as an operation, pehaps, like audit or status updates). The controllers should still run for the purposes of caching state (e.g. loading constraint templates into CF), and they should still write out pod status objects.

This is medium-to-large body of work. I would recommend we do this before graduating to beta (or at least before allowing users to set failure policy to Fail). However, I don't think we need to block the current PR on this because:

• This is an in-development feature that is off-by-default
• We currently fail open, so a dangling binding will not cause any issues

[sozercan]

LGTM