Meaning of the WaitingJobs and ScheduledJobs metrics?

[edmund-huber]

Neither in the README of this project, nor in the API docs https://buildkite.com/docs/apis/agent-api/metrics, do I find an explanation of what these different metrics mean. Most of the metrics are self-explanatory, but I'm still not clear about WaitingJobs and ScheduledJobs.

In my graphs, I only very occasionally see ScheduledJobs go over 0. Since I'm overprovisioned with agents, I suppose this means that ScheduledJobs is measuring how many jobs want to find a free agent but can't. Is this correct?

In my graphs, WaitingJobs is very often greater than 0. It often trends pretty closely with RunningJobs. My current best guess is that jobs are "waiting" when they have an agent to run on, but the git clone/checkout is still happening. Is that correct? What does WaitingJobs mean?

[edmund-huber]

My current best guess is that jobs are "waiting" when they have an agent to run on, but the git clone/checkout is still happening.

I don't think this can be correct, because when I have high numbers of "waiting jobs", the number of "busy agents" has not increased.