Insufficient cpu error when creating complex graphs

[Maximophone]

When applying the seldon deployment below on a GCP cluster, the main pod status stays on pending (the canary works) and shows the following warning:

Warning FailedScheduling No nodes are available that match all of the predicates: Insufficient cpu (5).

The cluster has 5 nodes.

seldon deployment JSON:

{
"apiVersion": "machinelearning.seldon.io/v1alpha1",
"kind": "SeldonDeployment",
"metadata": {
"labels": {
"app": "seldon"
},
"name": "seldon-deployment-example"
},
"spec": {
"annotations": {
"project-name":"FX Market Prediction",
"deployment_version": "v1"
},
"name": "test-deployment-complex",
"oauth_key": "oauth-key",
"oauth_secret": "oauth-secret",
"predictors": [
{
"componentSpec": {
"spec": {
"containers": [
{
"image": "seldonio/mean_classifier:0.6",
"imagePullPolicy": "IfNotPresent",
"name": "mean-classifier-1",
"resources": {
"requests": {
"memory": "1Mi",
"cpu": "0.1"
}
}
},
{
"image": "seldonio/mean_classifier:0.6",
"imagePullPolicy": "IfNotPresent",
"name": "mean-classifier-2",
"resources": {
"requests": {
"memory": "1Mi",
"cpu": "0.1"
}
}
},
{
"image": "seldonio/mean_classifier:0.6",
"imagePullPolicy": "IfNotPresent",
"name": "mean-classifier-3",
"resources": {
"requests": {
"memory": "1Mi",
"cpu": "0.1"
}
}
},
{
"image": "seldonio/mock_outlier_detector:1.0",
"imagePullPolicy": "IfNotPresent",
"name": "outlier-detector",
"resources": {
"requests": {
"memory": "1Mi",
"cpu": "0.1"
}
}
},
{
"image": "seldonio/mock_transformer:1.0",
"imagePullPolicy": "IfNotPresent",
"name": "mean-transformer",
"resources": {
"requests": {
"memory": "1Mi",
"cpu": "0.1"
}
}
}
],
"terminationGracePeriodSeconds": 20
}
},
"name": "fx-market-predictor",
"replicas": 1,
"annotations": {
"predictor_version": "v1"
},
"graph": {
"name": "outlier-detector",
"type": "TRANSFORMER",
"endpoint": {
"type": "REST"
},
"children": [
{
"name": "random-abtest",
"implementation": "RANDOM_ABTEST",
"type": "UNKNOWN_TYPE",
"children": [
{
"name": "mean-transformer",
"type": "TRANSFORMER",
"endpoint": {
"type": "REST"
},
"children": [
{
"name": "mean-classifier-1",
"type": "MODEL",
"endpoint": {
"type": "REST"
}
}
]
},
{
"name": "ensemble",
"type": "UNKNOWN_TYPE",
"implementation": "AVERAGE_COMBINER",
"children": [
{
"name": "mean-classifier-2",
"type": "MODEL",
"endpoint": {
"type": "REST"
}
},
{
"name": "mean-classifier-3",
"type": "MODEL",
"endpoint": {
"type": "REST"
}
}
]
}
]
}
]
}
},
{
"componentSpec": {
"spec": {
"containers": [
{
"image": "seldonio/mean_classifier:0.6",
"imagePullPolicy": "IfNotPresent",
"name": "mean-classifier",
"resources": {
"requests": {
"memory": "1Mi",
"cpu": "0.1"
}
}
},
{
"image": "seldonio/mock_transformer:1.0",
"imagePullPolicy": "IfNotPresent",
"name": "mean-transformer",
"resources": {
"requests": {
"memory": "1Mi",
"cpu": "0.1"
}
}
}
],
"terminationGracePeriodSeconds": 20
}
},
"name": "fx-market-predictor-canary",
"replicas": 1,
"annotations": {
"predictor_version": "v1"
},
"graph": {
"name": "mean-transformer",
"type": "TRANSFORMER",
"endpoint": {
"type": "REST"
},
"children": [
{
"name": "mean-classifier",
"endpoint": {
"type": "REST"
},
"type": "MODEL"
}
]
}
}
]
}
}

[ukclivecox]

We should change the Helm scripts to specify smaller initial CPU requests for the default components.

[ukclivecox]

Close this as the initial problem is fixed by hard wiring resource request to engine.