guidance for updates on jobs with multiple volumes

[tgross]

In #8045 we encountered a common use case that was a bit underdesigned in the CSI/host volumes implementation.

Suppose you want to run a multiple-node database (ex. MySQL with replication). You need to specify a specific volume to mount to each task. Your options are:

• Run each DB node as its own job, orchestrate the deployment outside of Nomad, and lose features like constraints.
• Run each DB node as its own task group with a single job, but lose the ability to control concurrent updates via the update stanza, which means when you update the job you can have all 3 nodes stop at the same time and end up taking an outage.
• Run each DB node as a task within a single task group with count = n. But then you need some kind of consistent metadata for the tasks to be interpolated in the volume ID. You can't use NOMAD_ALLOC_INDEX because there's no guarantee that (job_version=1, alloc_index=1) is going to be stopped and have its volumes released before (job_version=2, alloc_index=1).

We need to provide some documentation guidance for operators on this, and/or come up with a better way to orchestrate tracking resources across job versions.

[tgross]

Example of using HCL2 to accomplish this is in #9449

[ayZagen]

@tgross I came here from #8045. HCL2 example doesn't play well with nomad autoscaler as autoscaler will increase count. Is there any way/workaround to apply requirement with autoscaler ?

[tgross]

Not that I can think of currently. You'd need to have a pool of volumes ready-to-go and unclaimed, because Nomad can't create volumes on its own (see #8212). And even then, you'd need to be able to interpolate the volume source, which is #7877. I've been working on a design this week for both #8212 and #7877, but I don't have a timeline on when that work would be complete.

[joliver]

I'm late to the party, but there's a slightly related scenario that I at least wanted to document: using host_volume rather than the CSI plugin, it's not obvious how to assign a particular allocation (e.g. 0, 1, 2, etc.) to a particular node. When using stateful resources like MySQL, PostgreSQL, etc. having a particular node be assigned to an allocation makes things operationally easier and reduces the potential for changes to cause problems in stateful application.

Using Nomad Pack, we are able to create multiple instances of the same job with a slightly different name so they can be independently managed to protect the state's integrity.

job "mysql-[[ .my.instance_name ]]" {
 ...
  group "default" {
    count = 1 # instances are individually controlled and thus there's only ever a single instance per job, but multiple jobs
    constraint {
      attribute = meta.tags
      operator  = "set_contains"
      value     = "mysql-[[ .my.instance_name ]]" # a constraint that ties the name of the job ("-a", "-b" or "-1", "-2" to a node)
    }
 ...
 }

When this is rendered using Nomad Pack, I can now control which job I am interacting with by specifying the instance_name variable which will only ever manage that particular job instance.

So I'm late to the party and this uses an additional tool, but it's one more potential way to deal with the complexities of stateful applications.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.