More general scheduling constraints

[thockin]

There have been a few folks who have asked about machine constraints for scheduling. Let's use this issue as a place to gather ideas and requirements.

@timothysc

[verdverm]

I have noticed the FirstFit (default?) scheduler co-locates pods when there are open machines available. Each of these machines has a single cpu.

It would be nice to use information about available cpu and a pod's expected cpu requirements

sed 's/cpu/other_machine_stat/'

[monnand]

Currently, kubelet could get stats from cAdvisor which would be useful for scheduler. It could provide different percentiles of CPU and memory usage of a container (including the root container, i.e. the machine).

[thockin]

That's just "scheduling", as opposed to machine constraints, though very
coarsely they feel similar :)

On Mon, Jul 7, 2014 at 1:32 PM, Tony Worm [email protected] wrote:

I have noticed the FirstFit (default?) scheduler co-locates pods when
there are open machines available. Each of these machines has a single cpu.

It would be nice to use information about available cpu and a pod's
expected cpu requirements

sed 's/cpu/other_machine_stat/'

Reply to this email directly or view it on GitHub
#367 (comment)
.

[timothysc]

Labels "seems" like the ideal place to enable a rank & requirements to define constraints. However labels would need to be regularly published by minions.

e.g.
rank = memory
requirements = gpu & clusterXYZ

I have a couple of concerns here:

1. This treads into the full scale scheduling world.
2. Config syntax = ?, DSL? ...

[thockin]

Let's worry about semantics before syntax. We have a similar issue open
for label selectors in general - we can discuss syntax there.

On Mon, Jul 7, 2014 at 1:42 PM, Timothy St. Clair [email protected]
wrote:

Labels "seems" like the ideal place to enable a rank & requirements to
define constraints. For example:

rank = memory
requirements = gpu & clusterXYZ

I have a couple of concerns here:

1. This treads into the full scale scheduling world.
2. Config syntax = ?, DSL? ...

Reply to this email directly or view it on GitHub
#367 (comment)
.

[timothysc]

FWIW I often view constraints as a SQL query on a nvp store.

SELECT Resources
FROM Pool
WHERE Requirements
ORDER BY Rank
...

The hardest part are the 'fields' in an nvp store.

[bgrant0607]

Scheduling based on resources and constraints are 2 significantly different issues.

We have several issues open about resource (and QoS) awareness: #147 , #160 , #168 , #274 , #317.

Constraint syntax/semantics: We should start with the proposed label selector mechanism, #341 .

[timothysc]

I'm ok with doing the selection from a set of offers/resources from the scheduler.

Provided the offers have enough NVP information to enable discrimination.

[thockin]

I don't know about NVP - where can I read more on it?
On Jul 8, 2014 7:45 PM, "Timothy St. Clair" [email protected]
wrote:

I'm ok with doing the selection from a set of offers/resources from the
scheduler.

Provided the offers have enough NVP information to enable discrimination.

Reply to this email directly or view it on GitHub
#367 (comment)
.

[bgrant0607]

Searching for "NVP SQL" or "name value pair SQL" or "key value pair SQL" comes up with lots of hits. Common arguments against are performance and loss of control over DB schema. But I'm getting the feeling that we're barking up the wrong forest.

@timothysc What are you trying to do? Right now, k8s has essentially no intelligent scheduling. However, that's not a desirable end state. If what you want is a scheduler, we should figure out how to support scheduling plugins and/or layers on top of k8s.

[thockin]

Name Value Pairs? Now I feel dumb :)

On Tue, Jul 8, 2014 at 7:51 PM, Tim Hockin [email protected] wrote:

I don't know about NVP - where can I read more on it?
On Jul 8, 2014 7:45 PM, "Timothy St. Clair" [email protected]
wrote:

I'm ok with doing the selection from a set of offers/resources from the
scheduler.

Provided the offers have enough NVP information to enable discrimination.

Reply to this email directly or view it on GitHub
#367 (comment)
.

[bgrant0607]

Something somewhat different than label selectors is per-attribute limits for spreading. Aurora is one system that supports this model:
https://aurora.incubator.apache.org/documentation/latest/configuration-reference/#specifying-scheduling-constraints

This is more relevant to physical rather than virtual deployments. I'd consider it a distinct mechanism from constraints. @timothysc If you'd like this, we should file a separate issue. However, I'd prefer a a new failure tolerance scheduling policy object that specifies a label selector to identify the set of instances to be spread. We could debate about how to describe what kind and/or how much spreading to apply, but I'd initially just leave it entirely up to the infrastructure.

[timothysc]

I completely agree its more relevant to physical rather then virtual deployments.

I was somewhat testing the possibility of enabling the capabilities for more general purpose scheduling, on par with a mini-Condor approach but it's not a requirement.

Aurora or Marathon -esk capabilities will fill the gap.
https://github.com/mesosphere/marathon/wiki/Constraints

[bgrant0607]

Note that in order to add constraints, we'd need a way to attach labels to minions/nodes.

[timothysc]

That is what I had alluded to earlier, but it received luke warm attention. In fact, I believe Wilkes had chimed in on a different thread regarding this topic.

[brendandburns]

I think we should have labels for worker nodes, but they need to be
dynamic, and that's tough without a re-scheduler.

For now, I think we should use resources on nodes, since they are already
there, and the are known to be static.

You can add resource requests to pods, to achieve appropriate scheduling.

Brendan
On Oct 20, 2014 8:26 AM, "Timothy St. Clair" [email protected]
wrote:

That is what I had alluded to earlier, but it received luke warm
attention. In fact, I believe Wilkes had chimed in on a different thread
regarding this topic.

—
Reply to this email directly or view it on GitHub
#367 (comment)
.

[erictune]

Replication controllers reschedule pods when the machines they are on are no longer available. Seems like replication controller could do the same if the machine becomes infeasible for scheduling. A fairly simple loop can recheck predicates as a background task in the scheduler, and move pods to terminated state if they no longer fit.

Questions:

1. If a pod running pod is updated so that it's requirements no longer match the machine it is bound to, what happens?
   • pod moves to terminated state
   • refuse update
   • both, and let users be able to control the behavior (yay)

[bgrant0607]

The minion/node controller (#1366) should be responsible for killing pods with mismatched label selectors, and then, yes, replication controllers would recreate them.

Re. @erictune's question: Yes, we could support both, for instance, using a URL parameter to select the desired behavior.

[brendandburns]

Yeah, having the kubelet kill pods that don't match makes the most sense.

--brendan

On Mon, Oct 20, 2014 at 9:39 AM, bgrant0607 [email protected]
wrote:

The minion/node controller (#1366
#1366) should
be responsible for killing pods with mismatched label selectors, and then,
yes, replication controllers would recreate them.

Re. @erictune https://github.com/erictune's question: Yes, we could
support both, for instance, using a URL parameter to select the desired
behavior.

—
Reply to this email directly or view it on GitHub
#367 (comment)
.

[timothysc]

@davidopp

Constraint evals to simlpe true or false expressions, so it should really be &&, || this way you can connect them including simple predicate matching.

GROUPBY (cluster) || rack==funkytown

The more I think about it, the less I want to tread into config language space given the cattle idiom on services. Weather they are soft or hard could either be denoted kia keyword or some other semantics.

[davidopp]

So IIUC the way the thing you're proposing would work would like this?

GROUPBY expr1 || expr2 => put a virtual label X on all the machines that match expr1 or expr2, and then try to co-locate all the pods of the service on machines with label X
GROUPBY expr1 && expr2 => put a virtual label X on all the machines that match expr1 and expr2, and then try to co-locate all the pods of the service on machines with label X
SPREADBY expr1 || expr2 => put a virtual label X on all the machines that match expr1 or expr2, and then try to spread all the pods of the service across machines with label X
SPREADBY expr1 && expr2 => put a virtual label X on all the machines that match expr1 and expr2, and then try to spread all the pods of the service across machines with label X

It would be good to flesh out some use cases...

[bgrant0607]

I agree that flavors of affinity and anti-affinity are the basic 2 features that would satisfy most use cases.

With respect to As Simple As Possible, specifying just whether to group or spread seams like the simplest possible API. That needs to be associated with some set of pods, via label selector (in which object TBD). Node groups to concentrate in or spread across could be configured in the scheduler in most cases.

[jcderr]

+1

I deploy some fairly hefty celery tasks in our cluster, and definitively do not ever want more than one running on the same host at the same time. I'd rather some get left unscheduled and run a monitoring task to pick them up by scaling my cluster up.

[dchen1107]

cc/ @pwittrock

[dchen1107]

#13524

[davidopp]

This is part of #18261

[timothysc]

@davidopp I think it reasonable to close this issue, in lieu of the assorted proposal.

[davidopp]

@timothysc Let's wait until we merge the proposal.

[bgrant0607]

Affinity/anti-affinity proposals merged and implementations are underway.