nomad namespace apply -region=otherregion mynamespace does not propagate to federated region

[benvanstaveren]

Nomad version

1.6.2

Issue

Running nomad namespace apply -region=otherregion mynamespace results in "mynamespace" showing up in the region of the server the command was run against, instead of the otherregion specified in the command.

Reproduction steps

Federate 2 clusters together, create a namespace in the "other" region.

Expected Result

Namespace to show up in the region specified using the -region flag

Actual Result

Namespace shows up on the region of the nomad server the command was run against

[tgross]

Hi @benvanstaveren! Applying a namespace to a single region when regions are federated doesn't work the way you might expect. When two regions are federated, the namespace apply command should be forwarded to the authoritative region, and the namespace will be first written there and then replicated to the other regions. So if you're passing the -region flag, all that should be doing is sending the command to the -region region first, and then forwarding to the authoritative region anyways (so there's no point in doing it, as it just creates more hops).

If that's not what you're seeing, it'd be good to know more about the cluster topology involved here.

[benvanstaveren]

Hey @tgross, that's a little confusing... I don't see any replication taking place. On the authoritative region for instance:

$ nomad namespace list
Name          Description
developers    Developer namespace
database      Database Servers
default       Default shared namespace
ops           Ops namespace

but then if you do this:

$ nomad namespace list -region=other-region
Name     Description
default  Default shared namespace

Which would seem to suggest the other-region has only the default namespace. I tested it with a job, and indeed the other-region doesn't seem to have anything other than default.

Topology wise, we have the authoritative region, with 3 other clusters joined to that. TLS is enabled, but ACL is not enabled.

[tgross]

but ACL is not enabled

Oh, well that's the problem then! I think there's probably a documentation gap around this. Namespaces are access control objects, and only access control objects like ACLs, auth methods, etc. are replicated between regions, and that only happens if ACLs are enabled. (See leader.go#L414-L434.)

So when you federated your regions, all you're doing is allowing RPC forwarding between them (which can be useful, no doubt) but there's no "authoritative region" because ACLs are disabled. So when you're writing an ACL-related object like a namespace, it's trying to forward it to an authoritative region (ref namespace_endpoint.go#L33 but that value is effectively empty and no forwarding is happening.

I just took a quick look through a bunch of documentation and I don't see this relationship between federation and namespaces explicitly called out anywhere in the federation tutorial, the nomad server join docs, the authoritative_region config docs, the namespace spec docs, or even the ACL policy docs. So that's one of those big docs gaps that must have been hard for us as Nomad engineers to see because it's "obvious" because we're too close to it. Sorry to hear you stumbled onto this gap, but we'll get that fixed.

We should probably also have a validation step in the Namespace.Apply RPC (and maybe similar RPCs) so that if the region has been set explicitly and doesn't match the current region, and the authoritative region is empty, that we return an error that explains you can't forward these RPCs without ACLs enabled.

I've marked this issue for roadmapping.

[benvanstaveren]

So that's one of those big docs gaps that must have been hard for us as Nomad engineers to see because it's "obvious" because we're too close to it. Sorry to hear you stumbled onto this gap, but we'll get that fixed.

Hah! I know all about that, I fall into the same trap with my own projects way too often 😅 Anyway, at least if it's in the docs that'll be nice. I guess I can cheat the system by just creating the namespace on the cluster directly. Cheers!

[tgross]

Docs PR is at #20196. I've pulled a discussion about the RPC's behavior out to #20197.