Autoscaling cluster experiment design

[typhoonzero]

Evironment requiremnets:

• Cluster with kubernetes 1.6.x installed, best to have paddlecloud installed.
• At least 4 nodes, each node should have multiple GPUs
• Dataset prepared to recordio format and split to multiple files.

Test cases:

1. Simple:
   • start a mnist training job with 2 pservers, 2 trainers and 1 master, 2~100 pods. Trainer will scale immediently to use maximum free resources in the cluster.
   • start another deployment simulating online cluster loads. Trainer should scale down if not enouph resource left.
2. Simple with GPU:
   • same to case 1, but the mnist job requests GPU resource.
3. Many Jobs
   • start 5~10 mnist training jobs with different parallelism settings, scale 2~100 pods. Jobs should scale equally and fair.
   • each time add one load to cluster, see the down scaling of the jobs.

[putcn]

👍
can we also publish a white paper describing this experiment with the PR article?

[typhoonzero]

@putcn sure. Put this experiment in the PR article is also fine.

[Yancey1989]

Some additional ideas:

• Computing Resources
  • CPU: 120 Cores
  • GPU: 8 GPU
• Experiment metrics:
  • Computing utils, maybe we can use (requests/total) as the target.
  • Training time for each job.
• Additional Test Case
  • Inference serving with 5 pods in the cluster.
  • Start a training job with 100 trainers(2 pservers + 1 master)
  • Start another training job with 20 trainers ( 2 pservers + 1 master)
  • Compare the experiment features.

[helinwang]

Some additional ideas:

We need to test the cluster running Pods other than PaddlePaddle Pods (e.g., nginx, databases), and show that the training job is scaled down when the QPS (actually we will measure the CPU limit, but we can plot the QPS) of the nginx server increases. This use case is very good for training that only requires CPU and memory resources - share the CPU and memory with the general purpose Pods.