Possibly add something like pools?

[savagematt]

I can see what they're going for here:

https://pythonhosted.org/airflow/concepts.html#pools

But I don't think the server should be responsible for holding back jobs. Agents should be able to (possibly dynamically) decide how many of a particular job type should be in-flight at one time. i.e. "give me work, but only if there aren't loads of this type of job already running"

If we denormalised the running agent count into each job, the agent could just use that as another filter in addition to job type- no need for special code.

[savagematt]

There's probably some thorny code to write here around controlling agent count in the face of multiple concurrent requests for work.

Also not quite sure how this plays when we switch to push rather than polling.