Policy CRD allows everything until first policy CR

[hash-d]

Today, when applying a Policy CRD to a cluster, it takes effect immediately and prohibits all of the policy-enforceable features, until a Policy CR is created and policy is allowed to whatever it specifies.

This will cause disruption to an existing site, which may extend past the creation of the CR, given that some of the skupper items will be removed when policy does not allow them (skupper services, for example), and the admin will have to recreate them.

The suggestion is to make the Skupper Policy engine work in a similar fashion to how Kubernetes Network Policies work. There, policies are only enforced on a namespace once at least one policy applies to that namespace, otherwise all traffic is allowed.

The difference here on skupper is that the policies should be enforcing or permissive cluster-wide, instead of namespace-wide. This would simplify applying policies for the first time:

• Install Policy CRD (no policies are in effect)
• Install all-allowing Policy CR (policies are now in effect, but this CR keeps everything up)
• Install actual policies that are desired on the cluster (the actual policies are installed, but the all-allowing one supersedes them and everything is still allowed)
• Remove the all-allowing policy (now, only the desired policies are in effect)

[hash-d]

Note #759 removes the 'extension past creation of the CR part', but still one would have some disruption when adding the CRD to the cluster.

[fgiorgetti]

One more thing to add here. If a given site has been installed by a namespace admin (no permission to read policies),
policies must be enforced, regardless of whether there are policies for the respective namespace or not.

[grs]

Why?

[fgiorgetti]

Because there is no way block the site if controller can't read policies. In this case, lack of permission to read would imply it's enabled.

[grs]

Of course, makes sense. That does make the overall picture somewhat less clear for this change in my view. (I wonder if we could add a subcommand to skupper cli that would indicate whether a given site - or perhaps even scan all sites - would be affected by policy installation?

[fgiorgetti]

That is a good point. We had a discussion about it last week and we have been requested to write a "dry-run" command
for policies, which, I believe, would do exactly what you suggested.

Along with that, I believe skupper status should report that a given site is not operational as policies are enabled on
the cluster, but controller cannot read them. Same for skupper init performed by namespace admins after policy is enabled.