WIP: Remove the lock at the scheduler level

[abronan]

fixes #1194 #786

This PR removes the Lock at the scheduler level and moves it to node.Node for better granularity. cluster now maintains a list of Nodes with resource accounting protected by a Lock which avoids races for resource reservation. Processes not lucky enough to get a resource slice on a machine that gets full in the meantime are retrying the process of scheduling unless they get the resources they need (it fails if there is no more resource available cluster-wide).

This is I think the most simple approach to remove the lock at the scheduler level for now and avoid the races. Note that this does not solve the naming issue (we should settle if we allow duplicate names and solve the ambiguity through the prefix which is the node name or if each container must absolutely have a unique name).

Downside: The blocked pull problem still persists so some resources are going to be locked up for a while if this happens because we have no way to know if this is due to the issue or just a very slow pull.

TODO items:

• Integration tests
• Take a look at the implications on strategies and constraints, and parallel create/remove/build

Signed-off-by: Alexandre Beslic [email protected]

[abronan]

CI failing on all the constraints/affinity tests, kind of makes sense.. This solves the simple parallel create issue on resource constraints but impacts the rest. Should look into it..

[abronan]

Failure is obvious, I give a list of Nodes emptied from their containers and images to SelectNodeForContainer thus it fails on constraint and affinities. Works fine otherwise. Will fix the issue with listNodes to statically account for used resources while including the actual state of existing Containers and Images.

[tnachen]

I think you should check that this is >= 0, otherwise something really wrong happened

[abronan]

Agreed 👍

[abronan]

Closing this one in favor of #1261, will maybe revisit for 0.6 but until then the solution in #1261 plays nicer with scheduler plugins while this one could've ended with some side effects.