"cores" parameter in resource stanza not behaving as expected

[bjornbyte]

Nomad version

Output from nomad version
Nomad v1.3.5 (1359c25)

Operating system and Environment details

Windows 11 WSL

Issue

When attempting to schedule a job with

resources {
  cores = 1
}

in the task stanza, plan is returning Dimension cores exhausted on 1 node even though there are no other jobs running and the client attributes in the UI shows cpu.numcores | 16

Reproduction steps

in windows WSL ubuntu prompt run
nomad agent -dev
try to submit the below simple job

Expected Result

it should get scheduled

Actual Result

hello 1 unplaced
Resources exhausted on 1 node
Dimension cores exhausted on 1 node

Job file (if appropriate)

job "example" {
  datacenters = ["dc1"]

  group "hello" {
    task "hello" {
      driver = "docker"

      config {
	image   = "redis:3.2"
        command = "/bin/bash"
	args    = ["-c", "echo 'Hello World!' && sleep infinity"]
      }

      resources {
        cores    = 1
        memory = 20
      }
    }
  }
}

Client Node details

I do notice that in the below output, although Attributes.cpu.numcores is 16, Resources.Cores is zero, so I expect that's the problem here, but not sure why that would be.

> curl http://localhost:4646/v1/node/ae67729b-44e7-9bcf-002f-0fcf7ea3351d
{
    "Attributes": {
        "cpu.frequency": "2303",
        "nomad.version": "1.3.5",
        "cpu.totalcompute": "36848",
        "unique.storage.bytesfree": "247538786304",
        "cpu.modelname": "11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz",
        "kernel.version": "5.10.16.3-microsoft-standard-WSL2",
        "kernel.arch": "x86_64",
        "nomad.service_discovery": "true",
        "cpu.numcores": "16",
        "unique.storage.bytestotal": "269490393088",
        "driver.docker": "1",
        "unique.storage.volume": "/dev/sdc",
        "driver.docker.os_type": "linux",
        "unique.hostname": "Byter",
        "os.version": "20.04",
        "unique.cgroup.version": "v1",
        "nomad.revision": "1359c2580fed080295840fb888e28f0855e42d50",
        "os.signals": "SIGSYS,SIGTERM,SIGXFSZ,SIGNULL,SIGPIPE,SIGIO,SIGIOT,SIGQUIT,SIGSEGV,SIGCONT,SIGILL,SIGTSTP,SIGTTOU,SIGTRAP,SIGTTIN,SIGABRT,SIGALRM,SIGBUS,SIGINT,SIGHUP,SIGPROF,SIGSTOP,SIGWINCH,SIGUSR1,SIGFPE,SIGKILL,SIGUSR2,SIGXCPU",
        "unique.cgroup.mountpoint": "/sys/fs/cgroup/cpuset",
        "driver.docker.version": "20.10.18",
        "driver.docker.runtimes": "io.containerd.runc.v2,io.containerd.runtime.v1.linux,runc",
        "kernel.name": "linux",
        "driver.docker.bridge_ip": "172.17.0.1",
        "nomad.advertise.address": "127.0.0.1:4646",
        "unique.network.ip-address": "127.0.0.1",
        "os.name": "ubuntu",
        "cpu.arch": "amd64",
        "memory.totalbytes": "8204963840",
        "driver.raw_exec": "1"
    },
    "CSIControllerPlugins": null,
    "CSINodePlugins": null,
    "CgroupParent": "/nomad",
    "ComputedClass": "v1:9026653487005831442",
    "CreateIndex": 7,
    "Datacenter": "dc1",
    "Drain": false,
    "DrainStrategy": null,
    "Drivers": {
        "qemu": {
            "Attributes": null,
            "Detected": false,
            "Healthy": false,
            "HealthDescription": "",
            "UpdateTime": "2022-09-23T09:49:20.989328168-07:00"
        },
        "java": {
            "Attributes": null,
            "Detected": false,
            "Healthy": false,
            "HealthDescription": "Driver must run as root",
            "UpdateTime": "2022-09-23T09:49:20.954550754-07:00"
        },
        "raw_exec": {
            "Attributes": {
                "driver.raw_exec": "true"
            },
            "Detected": true,
            "Healthy": true,
            "HealthDescription": "Healthy",
            "UpdateTime": "2022-09-23T09:49:20.95506124-07:00"
        },
        "exec": {
            "Attributes": null,
            "Detected": false,
            "Healthy": false,
            "HealthDescription": "Driver must run as root",
            "UpdateTime": "2022-09-23T09:49:20.955093879-07:00"
        },
        "docker": {
            "Attributes": {
                "driver.docker.os_type": "linux",
                "driver.docker": "true",
                "driver.docker.version": "20.10.18",
                "driver.docker.bridge_ip": "172.17.0.1",
                "driver.docker.runtimes": "io.containerd.runc.v2,io.containerd.runtime.v1.linux,runc"
            },
            "Detected": true,
            "Healthy": true,
            "HealthDescription": "Healthy",
            "UpdateTime": "2022-09-23T09:49:20.975171369-07:00"
        }
    },
    "Events": [
        {
            "Message": "Node registered",
            "Subsystem": "Cluster",
            "Details": null,
            "Timestamp": "2022-09-23T09:49:20-07:00",
            "CreateIndex": 0
        }
    ],
    "HTTPAddr": "127.0.0.1:4646",
    "HostNetworks": null,
    "HostVolumes": null,
    "ID": "ae67729b-44e7-9bcf-002f-0fcf7ea3351d",
    "LastDrain": null,
    "Links": null,
    "Meta": {
        "connect.log_level": "info",
        "connect.proxy_concurrency": "1",
        "connect.sidecar_image": "envoyproxy/envoy:v${NOMAD_envoy_version}",
        "connect.gateway_image": "envoyproxy/envoy:v${NOMAD_envoy_version}"
    },
    "ModifyIndex": 9,
    "Name": "Byter",
    "NodeClass": "",
    "NodeResources": {
        "Cpu": {
            "CpuShares": 36848,
            "TotalCpuCores": 16,
            "ReservableCpuCores": null
        },
        "Memory": {
            "MemoryMB": 7824
        },
        "Disk": {
            "DiskMB": 236071
        },
        "Devices": null,
        "NodeNetworks": [
            {
                "Mode": "host",
                "Device": "lo",
                "MacAddress": "",
                "Speed": 1000,
                "Addresses": [
                    {
                        "Family": "ipv4",
                        "Alias": "default",
                        "Address": "127.0.0.1",
                        "ReservedPorts": "",
                        "Gateway": ""
                    },
                    {
                        "Family": "ipv6",
                        "Alias": "default",
                        "Address": "::1",
                        "ReservedPorts": "",
                        "Gateway": ""
                    }
                ]
            }
        ],
        "Networks": [
            {
                "Mode": "host",
                "Device": "lo",
                "CIDR": "127.0.0.1/32",
                "IP": "127.0.0.1",
                "Hostname": "",
                "MBits": 1000,
                "DNS": null,
                "ReservedPorts": null,
                "DynamicPorts": null
            },
            {
                "Mode": "host",
                "Device": "lo",
                "CIDR": "::1/128",
                "IP": "::1",
                "Hostname": "",
                "MBits": 1000,
                "DNS": null,
                "ReservedPorts": null,
                "DynamicPorts": null
            }
        ],
        "MinDynamicPort": 20000,
        "MaxDynamicPort": 32000
    },
    "Reserved": {
        "CPU": 0,
        "Cores": 0,
        "MemoryMB": 0,
        "MemoryMaxMB": 0,
        "DiskMB": 0,
        "IOPS": 0,
        "Networks": null,
        "Devices": null
    },
    "ReservedResources": {
        "Cpu": {
            "CpuShares": 0,
            "ReservedCpuCores": null
        },
        "Memory": {
            "MemoryMB": 0
        },
        "Disk": {
            "DiskMB": 0
        },
        "Networks": {
            "ReservedHostPorts": ""
        }
    },
    "Resources": {
        "CPU": 36848,
        "Cores": 0,
        "MemoryMB": 7824,
        "MemoryMaxMB": 0,
        "DiskMB": 236071,
        "IOPS": 0,
        "Networks": [
            {
                "Mode": "host",
                "Device": "lo",
                "CIDR": "127.0.0.1/32",
                "IP": "127.0.0.1",
                "Hostname": "",
                "MBits": 1000,
                "DNS": null,
                "ReservedPorts": null,
                "DynamicPorts": null
            },
            {
                "Mode": "host",
                "Device": "lo",
                "CIDR": "::1/128",
                "IP": "::1",
                "Hostname": "",
                "MBits": 1000,
                "DNS": null,
                "ReservedPorts": null,
                "DynamicPorts": null
            }
        ],
        "Devices": null
    },
    "SchedulingEligibility": "eligible",
    "SecretID": "",
    "Status": "ready",
    "StatusDescription": "",
    "StatusUpdatedAt": 1663951762,
    "TLSEnabled": false
}

[lgfa29]

Hi @bjornbyte 👋

Nomad detected 16 cores in the machine, but none of them are set as reservable:

    "NodeResources": {
        "Cpu": {
            "CpuShares": 36848,
            "TotalCpuCores": 16,
            "ReservableCpuCores": null
        },

This may be due the issue described in microsoft/WSL#4189 where WSL doesn't mount cgroups by default, which is what Nomad uses to detect reservable cores.

Could you try the fix mentioned in this comment and restart the Nomad agent to see if it's able to detect the researvable cores?

[bjornbyte]

That appears not to have helped, unfortunately.

[shoenig]

Hi @bjornbyte, without cgroups (and specifically the cpuset controller) you won't be able to make use of resources.cores, which is based on using cgroups to isolate work across physical cores. As a workaround you can use resources.cpu to specify the amount of CPU resource for the Nomad scheduler to allocate to the task in terms of MHz.

If cgroups are enabled you should get output like

➜ mount -l | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)

And if the cpuset controller is enabled you should see the reservable cores via

➜ cat /sys/fs/cgroup/cpuset.cpus.effective 
0-23

[bjornbyte]

Thanks. could it be that WSL (or the ubuntu version I'm running there) is not putting these where nomad expects them?
I have

> mount -l |grep cgroup
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)

and

> cat /sys/fs/cgroup/cpuset/cpuset.cpus
0-15

I took a quick look at the code you linked to around detecting reservable cores, and there seems to be a conditional branch around using v1 or v2, and some of the comments etc there seem to indicate that I've got a "hybryd system" with cgroup2 mounted under /sys/fs/unified and so it should be trying to use V1? I'm not sure what that implies about how nomad will try to determine reservable cores.

This stack overflow question suggests I should be able to have WSL not use v1 but when I tried taht .wslconfig then WSL would not start. Even so, It seems like the unified/cgroups2 does not have a cpuset.cpus.effective file, so I'm not sure it would help even if I could make that work:

> ls /sys/fs/cgroup/unified/
cgroup.controllers  cgroup.max.descendants  cgroup.stat             cgroup.threads
cgroup.max.depth    cgroup.procs            cgroup.subtree_control  cpu.stat

[shoenig]

Oh interesting, so it's setup with cgroups v1, but we should be able to support that. I'm not sure why you're getting an empty cpuset. When you first startup the Nomad Client agent, do you see any of the WARN log messages from
https://github.com/hashicorp/nomad/blob/v1.3.5/client/lib/cgutil/cpuset_manager_v1.go#L32 ?

[bjornbyte]

I don't see any of those, but I do see this one, which looks suspiciously like it might be important:

    2022-09-26T13:54:26.699-0700 [WARN]  client.cpuset.v1: failed to ensure cgroup parent exists; disable cpuset management: error="mkdir /sys/fs/cgroup/cpuset/nomad: permission denied"
``

[bjornbyte]

so, yea, if I run nomad with sudo then:

    "NodeResources": {
        "Cpu": {
            "CpuShares": 36864,
            "TotalCpuCores": 16,
            "ReservableCpuCores": [
                0,
                1,
                2,
                3,
                4,
                5,
                6,
                7,
                8,
                9,
                10,
                11,
                12,
                13,
                14,
                15
            ]
        },

[bjornbyte]

after that, since the /sys/fs/cgroup/cpuset/nomad directory exists, if I start it without sudo I get a warning
2022-09-26T14:06:27.997-0700 [WARN] client.cpuset.v1: failed to ensure reserved cpuset.cpus interface exists; disable cpuset management: error="mkdir /sys/fs/cgroup/cpuset/nomad/reserved: file exists"
but it still works.

[shoenig]

Ah, that'll do it it! Just FYI running Nomad as a non-root user isn't supported yet, but it's something we're exploring in #13669

[bjornbyte]

roger that, but it is a handy mechanism to create the right filesystem nodes so that it works when run as normal.

Now what I really was after was getting cores working in my local docker compose nomad cluster with multiple client agents running in containers, but what I learned here should help with that and I will open a new issue if I encounter something else that is a nomad problem and not my particular setup problem. Thanks!

[bjornbyte]

oops, I misread. it's not supported as non-root, although that usually seems to work just fine in dev mode.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.