Periodic Parameterized Batch behavior

[EtienneBruines]

Nomad version

Nomad v1.9.1
BuildDate 2024-10-21T09:00:50Z
Revision d9ec23f

Operating system and Environment details

Ubuntu 22.04.5 LTS

Issue

A batch job that is both parameterized and periodic cannot be run. When invoking it as parameterized, it results in something like this:

# nomad job status sync-periodically-or-invoked/dispatch-1730190526-56bb20a5                                                                                                                                                        [8:33:17]

ID                   = sync-periodically-or-invoked/dispatch-1730190526-56bb20a5
Name                 = sync-periodically-or-invoked/dispatch-1730190526-56bb20a5
Submit Date          = 2024-10-29T08:28:46Z
Type                 = batch
Priority             = 50
Datacenters          = dc1
Namespace            = default
Node Pool            = default
Status               = running
Periodic             = true
Parameterized        = false
Next Periodic Launch = 2024-10-30T02:30:00Z (17h56m24s from now)

When trying to force-launch it via the CLI: nomad job periodic force`

Error forcing periodic job "sync-periodically-or-invoked": Unexpected response code: 500 (rpc error: force launch for job "sync-periodically-or-invoked" failed: can't force run non-tracked job "sync-periodically-or-invoked" (default))

Reproduction steps

• Create the job using the example job file (this step will succeed)
• Try to run it via periodic force
• Or try to run it via dispatch

Expected Result

The job being able to run periodically (e.g. a daily sync) as well as on-demand (e.g. as a response to a webhook).

If this use-case is not supported (which would be sad), then at least the creating of such a parameterized periodic job should be blocked.

Actual Result

Neither of use-cases work - the job is not run at all.

Job file (if appropriate)

job "sync-periodically-or-invoked" {
  datacenters = [ "dc1" ]
  type = "batch"

  periodic {
    crons = [
      "30 2 * * * *" // At 2:30 AM every day
    ]
  }

  parameterized {
    payload = "optional"
  }

  group "sync" {
    task "sync" {
      driver = "docker"

      config {
        image = "hello-world"
      }
    }
  }
}

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

Update

I do seem to be able to periodic-force launch the parameterized job, resulting in some kind of a job like this:

ID            = sync-periodically-or-invoked/dispatch-1730190526-56bb20a5/periodic-1730191323
Name          = sync-periodically-or-invoked/dispatch-1730190526-56bb20a5/periodic-1730191323
Submit Date   = 2024-10-29T08:42:03Z
Type          = batch
Priority      = 50
Datacenters   = dc1
Namespace     = default
Node Pool     = default
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost  Unknown
sync        0       0         1        0       0         0     0

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created  Modified
a78d346b  568c32eb  sync        0        run      running  24s ago  8s ago

Which does not show up in the web UI.

I can't argue against that use-case (creating a number of cronjobs, each with different dispatched payloads), but it's not quite what I'd expect, nor does the UI support that either.

[gulducat]

Heya, thanks for the report!

Just in case you haven't seen this, you may consider HCL variables, which may or may not suit your needs. It's not uncommon to run whole separate batch jobs using variables instead of parameters, with pros and cons in either direction, but as for this particular issue:

As you mention in your Update, a sorta-hidden hierarchy comes into play when you combine parameterized and periodic job blocks. To rephrase your findings, when you nomad job run your example, what's "running" is the parameterized job, waiting to be dispatched. When you nomad job dispatch that job, you get a periodic job.

So the relationship is:

run: parameterized
  ↳ dispatch: periodic
    ↳ cron/force: batch

I don't know all of the history for why this is the case, but my thinking is: a batch job that takes no parameters and can be either run on a cron or run on demand is just a periodic job that you may sometimes force. Further, since parameterized jobs can't have default values, if it were treated simultaneously as a force-periodic, then what parameters should be provided? So instead, you can provide the parameters when dispatching the periodic job. HCL variables, on the other hand, can have default values, which is sometimes useful.

At any rate, I can see how this is not obvious behavior 😋. I don't think our docs explain it well (or at all?), and you're also right that the web UI doesn't handle it very well. There is a way to get to it in the UI -- the Evaluations page should have a link to what I'll call the "grandchild" job (provided it has not been garbage collected), or you can formulate the URL manually, but that's not exactly obvious, either.

So I do think we could stand to improve those 2 things: Docs and the UI presentation.

[gulducat]

I put in #24338 for the web UI side, and I'll say this ticket can track the documentation updates.