Add "streams/scale/" API

[sabbyanandan]

As a continuation of spring-cloud/spring-cloud-deployer#62, we will have a platform-specific implementation of the SPI, which can then be automatically backed by a public-facing RESTful endpoint exposed in SCDF.

We will have to decide whether this API would be chained under the /streams API or something more application-specific.

Depends on:

• Implement scaling for request flows spring-cloud-skipper#911
• Update SCDF docker-compose suite #3562

Acceptance:

• Include new API under /streams/scale
• The new API is included in the Java DSL and DataFlowTemplate
• Include IT tests to validate this via DockerComposeIT

[jvalkeal]

Before trying to add new rest api's we could simply try to use existing stream update and see if simply updating deployment property deployer.*.count=2 would suit our needs. This would surely result minimal changes as we wouldn't necessarily need to change anything on a scdf side.

[tzolov]

The update command changes the functionality of the deployed pipeline. E.g. by update/rollback an app in a pipeline you effectively change the functional behavior of the running pipeline. The non-functional operations such as scale or binder (re)partition on the other hand are used to improve the performance or tackle overload issues but they do NOT alter the logic of the running pipeline.

IMO it is important to keep this separation and keep the functional (e.g. update) and non-functional (e.g. scale, repartition ...) operations orthogonal to each other. We should not aim to reuse update in place of scale just because it might feel easy to hack it that way.

[sabbyanandan]

@tzolov: Once when we have this API, let's make sure we have the necessary infrastructure to include it in the Java DSL and DataflowTemplate.

[jvalkeal]

Btw, this depends on spring-cloud/spring-cloud-skipper#910 which will be on other PR.

[tzolov]

The scale method uses deploymentId, count and properties (e.g. AppScaleRequest) for every application. Two Data Flow Scale API extension are provided below for bulks (multi-app) and single app scaling:

• Streams Deployments Scale API - as the scale is only applicable for deployed streams it is natural to make it part of the existing /streams/deployments resource instead of creating a new /streams/scale.

POST /streams/deployments/scale/{streamName}
Content-Type: application/json
BODY: 
[ 
 {"appName": "", "count": "", "properties": {...optional...}},
 {"appName": "", "count": "", "properties": {...optional...}},
 ...
]

appName stands for the name/label as defined in the stream DSL, not the depmoment-Id!

• Runtime Scale API - The GET /runtime/apps/ offers runtime information about deployed apps identified by the deploymentid. Later can be used to as reference to scale the deployed app:

POST /runtime/apps/{appId}/scale/{count}
Content-Type: application/json
BODY: { ... properties ...} // optional

appId stands for the deployementId not the app name/label in the DSL