Tanks vs Commanders permissions

[josibake]

In the context of war-games, we don't want teams to be able to create more of their own tanks: they start with a fixed set of vulnerable nodes and if the opposing team kills the node, they cannot restart it or create new ones.

However, we do want them to be able to spin up their own commander pods, where a commander packages some sort of attack script. One solution would be tanks in one namespace and commanders in the other. But commanders are just pods, so if we give the user permissions to create pods in namespace x , they could also create tanks in namespace x! There's likely more that could be done here but it feels like barking up the wrong tree to me.

I started looking at CRDs for commanders and this feels like the better approach. Here are some links I've been reading on the subject:

• https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
• https://www.loft.sh/blog/extending-kubernetes-with-custom-resource-definitions-crds
• https://overcast.blog/kubernetes-custom-resource-definitions-crd-a-practical-guide-dbffa5fd5b2e

This feels like the right direction to go, imo. This would involve creating the custom resource definition (commander.yaml), creating permissions for creating these resources, and creating a controller. A controller is to a CRD as a scheduler is to a pod. The nice about this is the controller can apply more complex application logic based on the type of resource, e.g., might be a nice fit to model complex start up conditions when deploying a lightning node (future work).

The main benefit I see today is we can more natively use RBACs , where a use can create and delete resource Commander but can only get, view, list, etc Pods (we could also model Tank or BitcoinNode as a CRD in the future if this pattern seems good). I haven't validated RBAC control on CRDs vs pod in the same namespace, but it seems entirely possible.

Opening for discussion by my proposal would be to start with a minimal example of creating a CRD that is essentially a no-op (doesn't even need to create a container or have a controller), but validate that a users permissions can be restricted such that they cannot create pods in a namespace but can create CRDs. If that works, next step would be to write a controller for creating these resources and then package the commander as one. Ideally , the controller can validate that the a CRD is only created with the commander image (looks like this is possible).

cc @mplsgrant since you've been looking at this and @m3dwards since we briefly discussed this off-line.

[pinheadmz]

I'm not convinced we need to prevent players deploying tanks. Even if we decide that is a rule it seems easy enough for an admin to run a script that kills all unauthorized pods.

[pinheadmz]

You're proposal is definitely a cool idea I just want to make sure we have enough time to get everything done by tabconf

[josibake]

I just want to make sure we have enough time to get everything done by tabconf

That's the main reason I started this discussion. I do think players deploying their own tanks is a valid concern for Tabconf, which means we should probably have a mitigation in place even if its "okay, can you guys stop deploying tanks, please."

I spent a few hours researching this approach and feel it's something I could knock out in a few days or less with @mplsgrant 's help if we keep it scoped to a CRD for just commanders. I also think it sets a foundation for a pattern we will want to keep building on in the future. That being said, if you have concerns about this approach and how long it will take, it would be great to hear concretely what you are concerned about and then we can decide on a short term mitigation (including do nothing), if its deemed this approach will take too long.

What I don't want is for any of us to invest hours researching a solution that we agree feels conceptually wrong, but would work as a short term mitigation. This will necessarily take hours of research/testing/building for something that we will eventually throw away later or carry with us legacy tech debt. A few of the other proposed solutions feel like they fall in this category for me, namely:

• Trying to create custom networking logic to limit what can happen between namespaces
• Creating service accounts that reach across namespaces, e.g., service account for blue-team user Alice can now create pods in namespace blue-team-commanders
• Writing custom scripts just for Tabconf to get around the fact we don't have proper permissions management

[m3dwards]

I think it's a nice pattern and does give more flexibility regarding permissions etc however I too am a bit concerned that it wouldn't be strictly necessary as we could just ask participants not to spin up tanks with a "rules of the game" written on a whiteboard.

I'd like to see it but I'd like to see all the other components in place first I think with this as a cherry on the top. For example, do we need to write all the attacks ourselves to prove that everything works in practice?

[m3dwards]

tl;dr concept ack with correct priority applied vs other required elements

[josibake]

Happy to bump this in priority of actually building attacks, so long as we are fine with not hacking an interim solution (which was my main concern).

[mplsgrant]

1. I can envision some downsides to CRDs with respect to managing permissions. For example, if we wrap a Commander Pod in a CRD, then we will not be able to do this: kubectl logs pod/my-commander-crd. Instead, we'll need to create some logic that determines the underlying pod within the CRD and then get the logs from that. I imagine we would also need to do some kind of logic to get our current graphana setup aligned with the CRDs.

2. Another approach is using explicit contexts which looks like this kubectl --context=my-context-2 logs pod/my-pod. The way we use warnet today would not change: admin users of warnet who don't need permission restrictions still have their regular context and nothing changes for them. Player users have their Commander context, and they can do whatever they want in that context given some resource constraints (total number of pods, cpu, etc). Players would also have their Vulnerable Tanks context which would hold their tanks, and they would not be able to create new pods in that context.

3. Another approach is reactive: a cron job nukes pods younger than some datetime that have a certain label, and it also nukes pods that don't have the right label. So, a pod with a commander label never gets nuked. A pod with a tank label that's too young gets nuked. Any pod that doesn't have either a "commander" or "tank" label gets nuked. This might be a good approach because we want a job that monitors connectivity of pods anyway. So, this is in line with our current goals and doesn't really change much in our setup because it is reactive.

[josibake]

logging + CRDs is a good call out, I’m not sure how a CRD works with tools like grafana etc. I’m not convinced that logs just “just don’t work” but would be a good thing to validate.

Regarding contexts, I don’t think this works: if a user can create pods in a namespace they can create whatever the want, it doesn’t matter if we say “this namespace is for commands and the other one is for tanks.”

I’m also pretty strongly opposed to the cron job approach and would rather we just say “we’ll ask people not to do this, until we can build a proper solution.” So far, that seems to be the consensus, i.e., tackle this later once we have other more critical tasks finished, which is fine with me.

[mplsgrant]

Using the contexts approach, it seems like we can use a NetworkPolicy to firewall the Alice Commanders namespace from the Alice Vulnerable Tanks namespace. That way, Alice can spawn whatever she wants in the Commander namespace, but she won't be able to hook her Vulnerable Tanks up to her Commander nodes. That means she can't, for example, insulate them from the rest of the network.

[josibake]

Firewalling namespaces seems like a dead end to me. First, Alice can create whatever she wants in the commanders namespaces, and from the other team's perspective they can't tell what's in Alice's tank namespace vs what's in her commanders namespace. This requires building more tooling on the admin side to monitor the namespaces. Second, if firewalling the namespaces, Alice now can't talk to the vulnerable nodes from the commander nodes, which limits strategies Alice can use for attack/defense and makes the games more complicated imo.

This also seems very specific to the tabconf war-games use case and doesn't seem generally useful for warnet. CRDs (assuming there are no gotcha's) provide a way to natively model "this pod is different from this pod," which allows us to use all of the native tooling for applying permissions and managing the lifecycle of the resource.

[mplsgrant]

Attempted to spike an admission webhook here: https://github.com/mplsgrant/warnet/tree/2024-09-05-admission-webhook/resources/admission-webhook

Was interesting, but it reached its timebox.