Add scale to SPI and tests

[dturanski]

Resolves #62

[sabbyanandan]

@markpollack suggests void scale(String deploymentId, int desiredCount)

[markfisher]

I wonder if we should support (instead or in-addition) a +/- N option... otherwise it's the user's responsibility to know the current state before calling this, and that seems to create a race condition

[dturanski]

@markfisher I'm not sure I follow. If the user would scale, don't they already know the current count? This follows the protocol of k8s and PCF just tell me how many instances you want.

[dturanski]

@markfisher - I agree with your suggestion now that we are revisiting this, especially for autoscaling use cases.

[markfisher]

I'm not so sure about my own suggestion after reading this: https://hackernoon.com/level-triggering-and-reconciliation-in-kubernetes-1f17fe30333d

[jvalkeal]

Requesting a new count is easiest way to get started as then you don't get race conditions. We had scaling in our good ol' yarn deployer and at least in a hadoop world you had to set a new count.

[markfisher]

Kubernetes is the same (as described in the edge vs level triggering article I posted above), and that avoids the race conditions which are exacerbated by the fact that scaling can take some time, during which multiple requests for changes may occur.

[dturanski]

True new count is idempotent

[tzolov]

@markfisher , thanks for sharing this article! Imo, we should stick with the level approach. It would be enough if the scale function is idempotent - usually available OOTB in the underlying platform such as k8s.

Avoid keeping state (e.g. changes relative to previous state) or try to be too smart. It is impossible to protect agains all concurrency or stability issues from within the scale function alone. I used to be good in tuning PID controllers and solving Laplace transformations to stabilize control systems. It's all gone now but it makes me aware that one can not avoid stability issues without considering all components in the control loop (in our case auto-scaling chain).
So keep the scale simple without integral or derivative functionality ;) If needed this can be added externally.
Having said this we still can have some configurable guards such as max-instance count or so.

[tzolov]

@dturanski , here is a snipped from my s1p demo. A web hook is configured to receive 2 types of (json) alerts, coming from a Prometheus AlertManager: (1) ThroughtputDeltaHigh - the throughput between the source and processor (e.g. fraud-detection) apps is higher than a given threshold; (2) ThroughtputDeltaZero - the throughputs of the source and the processor apps are the same.
When the delta is high the fraud-detection processor is scaled to 4 instances and reduced to 1 otherwise. I believe this is aligned with the level approach. We express the desired state/level and let the platform reach it eventually. Most of the PID control logic is done though the AlertManager configuration.
The scale API looks like scale(streamName, AppName, Count)

@RequestMapping(value = "/alert", method = RequestMethod.POST)
public void alertMessage(@RequestBody List<Map<String, Object>> alerts) {
   for (Map<String, Object> alert : alerts) {
      String alertStatus =  fn(from alert);
      String alertName =  fn(alert);
      String streamName = fn(alert) ;

      if (alertStatus.equals("firing")) {
         if (alertName.equals("ThroughtputDeltaHigh")) {
            kubeScaleService.scale(streamName, "fraud-detection", 4);
         }  else if (alertName.equals("ThroughtputDeltaZero")) {
            kubeScaleService.scale(streamName, "fraud-detection", 1);
         }
      }
   }
}

[sabbyanandan]

We will revisit in the context of Spinnaker integration in the future.