Add ability to exclude client nodes without explicit constraint

[mikenomitch]

Proposal

Datacenters are currently the only way to have nodes be ineligible to receive a job by default. Other constraints are all opt-in by the jobspec writer.

If a Nomad operator wants other members of their team to not deploy to certain nodes, they have to either use Sentinel (which is difficult and pay gated) or build a layer of validation/templating between their teammates and Nomad.

Ideally, Nomad job submitters could submit jobs and have them deployed to a constrained set of datacenter without knowledge of these constraints. See Proposal Details and Alternatives for details.

Use-cases

Nomad users who do not have permission to deploy to clients on certain datacenters can deploy a job without specifying this constraint themselves.

Additional benefit/use-case: Nomad operators do not have to add validation step themselves to check for datacenter eligibility.

Additional benefit/use-case: Users who deploy to different data centers can share jobspecs without replacing datacenters value.

Proposal Details

To accomplish the goals above, I propose extending ACL Policies with two fields in the namespace stanza:

• allowed_datacenters: list(string)
• default_datacenters: list(string)

When submitting a job, while validating the jobspec, Nomad should reject any requests that have a datacenter outside the allowed_datacenters list.

If allowed_datacenters is not specified in a policy, it falls back to no validation.

If default_datacenters is specified in a policy, a job submitter can leave out the “datacenters” value in their jobspec and the value will be autopopulated based on their ACL policy.

If default_datacenters is not specified, datacenter must be specified on the jobspec.

Additionally, allowed_datacenters and default_datacenters default values should be configurable in the nomad server config. If these values are given at the server level, the default namespace inherits them, and any policies that do not explicitly override these values do as well.

allowed_datacenters would provide a backwards compatible way to make sets of clients ineligible to certain Nomad users

default_datacenters would simplify jobspec writing and sharing. Groups within the same organization could share jobspecs without having the overwrite or parameterize the datacenter value. Additionally, the community could more easily share job specs.

Alternatives

An alternative approach is to keep datacenters the same, but add a new attribute on client nodes, like “node_pool” and then add implicit constraints onto that. This may become confusing when added on top of datacenters, but could allow for more complex hierarchies.

[pznamensky]

See also: #2299

[shishir-a412ed]

@mikenomitch
To address this
Instead of allowed_datacenters (whitelist), can we have deny_datacenters (blacklist)?

Basically, every ACL policy by default has access to all the nodes in the pool (*), unless specified otherwise (deny_datacenters). On upgrade, all the existing policies will have no effect thereby maintaining backward compatibility. Then the operator can define deny_datacenters for the policies it wants to restrict access to those DCs.

With the current solution (allowed_datacenters) the problem is, let's say I have 5 DCs [dc1, dc2, dc3, dc4, dc5] and I have 100 ACL policies in the system (and associated tokens with these policies).

I want only one of the ACL policy to be able to deploy to dc5. I have to add all the DCs except dc5 {allowed_datacenters = dc1, dc2, dc3, dc4} to all the other 99 policies thereby making dc5 not allowed for those 99 policies. and have dc5 as allowed_datacenters for my policy. When more datacenter e.g. dc6 and dc7 gets added, I have to update these policies! This is not very scalable.

With the blacklist (deny_datacenters), I just need to add dc5 to the rest of the 99 policies and nothing to my policy.

[ygersie]

I'm not a fan of (ab)using the datacenter for this purpose. We've always used the datacenter as a failure boundary within a region, not as a constraint to ACL on. Ideally we would be able to restrict which namespaces can land on a node(pool). That way ACLs are bound to the namespace which is already the case for most job related operations in Nomad.

[tgross]

@ygersie you're in luck, as we've got the upcoming node pools feature in 1.6.0, which will allow restricting which namespaces can land on a node pool as part of Nomad Enterprise.See #2299 (comment) and follow-along with the theme/node-pools label for more info.

[lgfa29]

Node pools have been implemented and are set to be released in Nomad 1.6.0.

You will be able to define node pools similarly to how namespaces are created:

$ cat dev-pool.hcl
node_pool "dev" {
  description = "Node pool for development workloads."

  meta {
    environment = "development"
  }
}

$ nomad node pool apply dev-pool.hcl
Successfully applied node pool "dev"!

You can then register nodes and register jobs to this pool using the `node_pool configuration:

client {
  enabled = true
  # ...
  node_pool = "dev"
  # ..
}

job "..." {
  # ...
  node_pool = "dev"
  # ...
}

When this job is evaluated by the Nomad scheduler, only the nodes in the dev node pool will be used, meaning that jobs are restricted to the nodes in the pool it's assigned to.

By default Nomad creates two node pools, called default and all. These node pools cannot be deleted nor modified. The default node pool is used by all nodes and jobs that don't specify a node_pool, and the all node pool can be used by jobs to be placed in nodes in all node pools, which is useful for system and sysbatch jobs.

In Nomad Enterprise node pools have an additional block called scheduler_config. This block allows you to adjust some of the scheduler settings per node pool. At this time the available options are scheduler algorithm and memory oversubscription.

node_pool "dev" {
  description = "Node pool for development workloads."

  meta {
    environment = "development"
  }

  scheduler_config {
    scheduler_algorithm             = "spread"
    memory_oversubscription_enabled = true
  }
}

Nomad Enterprise users are also able to associate a namespace to a node pool, and allow or deny access to them.

namespace "prod" {
  description = "Production namespace."

  node_pool_config {
    default = "prod"
    allowed = ["all"]
  }
}

Jobs in this namespace that don't specify a node pool will use prod by default. Jobs are also allowed to use the all node pool, but no others. There is also a deny configuration to restrict which node pools are not allowed be used by jobs in a given namespace.

[tgross]

Shipped!