Proposal: scaling interface

[bgrant0607]

Auto-scaling has been discussed in many issues/PRs: #1007, #1178, #170, ...

This is about how an auto-scaler should be layered on the API.

In the case of auto-scaling, the currently desired number of replicas won't be statically configured by the user, and should not be among the fields diff'ed by the configuration system. #1201 addresses that problem.

It should be possible to auto-scale applications with multiple release tracks, and it should be possible to auto-scale and perform a rolling update (#1353) simultaneously. If the auto-scaler were tied to a specific replication controller, then it would at least need a label selector (or possibly reference to a service object, but I prefer direct use of label selectors) to identify the whole serving set in order to assess current capacity.

Additionally, if the approach of creating a new replication controller were used to deploy new images, the auto-scaler would need to be updated to resize the new replication controller.

And, if we introduced higher-level API objects to manage multiple underlying objects or other new controller types (e.g., JobController - #1624), the auto-scaler would need to support all those API types.

Instead of declaratively manipulating all these object types directly, the auto-scaler should target a narrower API: an imperative resize "verb" (i.e., POST to a URL). Any API object capable of being auto-scaled (e.g., replicationController) could support that verb. The verb would update the desired replica count for that object. With this approach, resizing could also be supported generically by the CLI (lots of discussion in #1325), would simplify client code more generally, and could simplify authorization logic.

Yes, verbs of all flavors are discouraged by REST purists, but this approach avoids per-object-type verb proliferation and facilitates metaprogramming, addressing their main concrete concerns. The alternative would be a generic declarative subobject containing (essentially) just the size. That seems like a lot of metadata and object-lifecycle overhead for little gain.

Details to be worked out:

• Whether everyone is ok with URL-encoded API verbs
• URL format for API verbs generally. Simple strawman proposal: collection or object URL followed by /.verb -- period prefix to avoid collisions with sub-collections/sub-objects (only unescaped punctuation across all RFCs are -, _, and .).
• Whether we support absolute and/or relative resizing -- either could work in most cases, though relative can be useful for pure reactive behavior
• Preconditions for resize (e.g., current size), which would be optional (though necessary for idempotence in the relative case)
• JSON vs. URL parameters for operation parameters. I like URL parameters, since they facilitate simple web hooks.

[smarterclayton]

We already have 3 API verbs URL-encoded - watch, proxy, and redirect. I think I'm ok with a few more.

Regarding relative vs absolute - In practice we started with relative and realized we wanted absolute for idempotence. I would argue relative should be optional and absolute should be the default (like specifying a name is required and generating it is optional).

Is resize limited to a single dimension for all verbs? Is it parameterizable in the future? Or should all objects have a dimensionality of one for resize?

[bgrant0607]

Verbs: These verbs appear early in URL paths, preceding the resource collections (kinds), namespaces, and names?

Relative vs. absolute: I agree.

Dimensionality: I was only thinking about a single dimension: horizontal scaling. What I imagine is that an autoscaler would watch a set of pods and invoke a resize operation. It wouldn't need to understand what was controlling the pods or what was implementing resize. It might be a single replication controller, a replication controller controller, replication and nominal service controller, a multi-release deployment controller, a job controller, etc. However, it would need to have an expectation that tweaking resize would change the number of monitored pods accordingly.

I consider vertical resource resizing (cpu, memory, etc.) of pods to be a separate issue (#2072), one which is more complex not only due to multiple dimensions, but also due to pod resources vs. container resources, request vs. limit, in place update vs. rescheduling, and other issues.

[derekwaynecarr]

I am ok with this approach if the resize verb is something implemented by a runtime.Object, and not that a RESTStorage resize operation is authored. I want to make sure we have a single choke point for intercepting all Update operations as part of my ability to weave in admission control. I have some other concerns on a resize verb and policy sychronization.

[derekwaynecarr]

On policy, it sounds like I need to separately manage /resize/ns/{namespace}/object/name from /ns/{namespace}/object/name. That kind of complicates the system in my opinion so we should be careful before taking on new verbs without thinking it through.

[bgrant0607]

I was thinking of something like POST .../ns/{namespace}/{kind}/{name}/op/resize

That would make it easier to delegate the handling of special verbs to each registry object, and for the set of verbs to be discoverable. This is sort of the approach used by GCE, though I don't think it puts verbs into a sub-collection.

[smarterclayton]

Structurally, it would be better to watch via GET /ns/kind/obj/name/op/watch because then watch can be properly sharded

On Dec 17, 2014, at 7:42 PM, bgrant0607 [email protected] wrote:

I was thinking of something like POST .../ns/{namespace}/{kind}/{name}/op/resize

That would make it easier to delegate the handling of special verbs to each registry object, and for the set of verbs to be discoverable. This is sort of the approach used by GCE, though I don't think it puts verbs into a sub-collection.

—
Reply to this email directly or view it on GitHub.

[brendandburns]

+1 to this proposal in general.

Caveats:
1)
We need to support a resource version on this request to prevent read -
update - write races.

This is likely to make client codegen harder (not that we're doing it
today...)
We should look at what (if anything) swagger supports in this space.

I think we should intercept these even higher in the stack, likely the
apiserver itself, and handle them in with a VerbHandler interface or some
such that parallels RESTStorage.

I don't think we should support incremental resize, as it seems inherently
racy.

Brendan
On Dec 17, 2014 5:36 PM, "Clayton Coleman" [email protected] wrote:

Structurally, it would be better to watch via GET
/ns/kind/obj/name/op/watch because then watch can be properly sharded

On Dec 17, 2014, at 7:42 PM, bgrant0607 [email protected]
wrote:

I was thinking of something like POST
.../ns/{namespace}/{kind}/{name}/op/resize

That would make it easier to delegate the handling of special verbs to
each registry object, and for the set of verbs to be discoverable. This is
sort of the approach used by GCE, though I don't think it puts verbs into a
sub-collection.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#1629 (comment)
.

[bgrant0607]

This proposal is really about an abstract interface for resizable entities.

So, to put forth a more RESTful alternative -- feel free to shoot it down:

We could create a new virtual REST object:

type Scaler struct {
    TypeMeta   `json:",inline"`
    ObjectMeta `json:"metadata,omitempty"`
    Spec ScalerSpec `json:"spec,omitempty"`
    Status ScalerStatus `json:"status,omitempty"`
}
type ScalerSpec struct {
    Count int `json:"count"`
}
type ScalerStatus struct {
    ScaledObjectRef *ObjectReference `json:"scaledObjectRef,omitempty"`
    Count int `json:"count"`
}

Any object that wanted to be scalable, such as ReplicationController, would automatically create a Scaler object. Modifications to the Scaler object would in turn modify the corresponding ReplicationController.

This would achieve the goal of making the auto-scaler not need to understand the full details of all types of controllers that wanted to be scalable.

Note that etcd is adding multi-object transactions, which should make such multi-object "macros" easier to manage.

[smarterclayton]

Or similarly, expose

/replicationcontrollerscalers/<name of rc>

type Scaler struct {
  Metadata struct {
    ResourceVersion
  } 
  Spec struct {
    Count int64
  }
  Status struct {
    Count int64
  }
}

which supports GET / PUT.

I don't think /replicationcontrollers/<name>/op/scaler is more or less special than /replicationcontrollerscalers/<name> - all that really matters is can a client easily figure out whether X has a scaler, and then do something consistent to it. That argues a bit more for <resource>/.../op/<op>, because as a client I need to know these things:

• What I want to scale (replicationcontroller)
• What is its current scale
• What do I want its current scale to be

So the mechanical transformation you want is

• Given resource name, check whether it has scaler
• Given scaler, "get" its current scale
• Given scaler, "update" the current scale

Unlike spec/status sub objects, there's no real need to mutate the metadata at the same time as the scaler. So having a totally decoupled resource representation in the API makes sense.

----- Original Message -----

This proposal is really about an abstract interface for resizable entities.

So, to put forth a more RESTful alternative -- feel free to shoot it down:

We could create a new virtual REST object:

type Scaler struct {
  TypeMeta   `json:",inline"`
  ObjectMeta `json:"metadata,omitempty"`
  Spec ScalerSpec `json:"spec,omitempty"`
  Status ScalerStatus `json:"status,omitempty"`
}
type ScalerSpec struct {
  Count int `json:"count"`
}
type ScalerStatus struct {
        ScaledObjectRef *ObjectReference `json:"scaledObjectRef,omitempty"`
  Count int `json:"count"`
}

Any object that wanted to be scalable, such as ReplicationController, would
automatically create a Scaler object. Modifications to the Scaler object
would in turn modify the corresponding ReplicationController.

This would achieve the goal of making the auto-scaler not need to understand
the full details of all types of controllers that wanted to be scalable.

Note that etcd is adding multi-object transactions, which should make such
multi-object "macros" easier to manage.

---

Reply to this email directly or view it on GitHub:
#1629 (comment)

[pweil-]

I agree with @smarterclayton that <resource>/.../op/<op> seems like the best option. Exposing a resize operation is exposing the Resizable object, not the scaler itself. The scaler is simply a boundary definition that triggers resize actions on a Resizable target.

[davidopp]

Could someone explain the resize verb concept a little more? What object is the target? From some of the examples above it seemed the target was a pod (for example), but I don't understand what it means to horizontally resize a pod (wouldn't you just create a copy of it, or delete it)? Or maybe the target is some kind of meata-object?

[davidopp]

Sorry for being dense, I just figured out that the replication controller is the object with a resize verb. That makes sense.

[smarterclayton]

#5054 contains some fixes required to allow sub resources like /resize to work, as well as an example with /binding. I made "Binding" generic to type, (it's a name, target, and an object reference to be bound to).

[bgrant0607]

Note that the current proposal is /size. See #2726 (comment)

[bgrant0607]

After further thought, I'm thinking we should call this /scale, and rename kubectl resize to kubectl scale, in order to match standard terminology (auto-scaling), to distinguish from cpu/memory resizing/auto-sizing, and to be more similar to other systems (e.g., CF).

[bgrant0607]

In progress: #12217

[stormltf]

when atuoscaling can be done? do you need our team help to do this?

[fgrzadkowski]

Several issues here:

• /scale subresource is already done. See Experimental Scale subresource #12217
• Horizontal pod autoscaling is tracked in Horizontal pod autoscaler basing on scale subresource #12087
• Vertical pod autoscaling is tracked in Vertical pod auto-sizer #10782; currently we work on Setting Initial Resources #12149 which is a first step
• Cluster autoscaling (adding/removing nodes) is tracked in Implement node autoscaler for GCE #11748 (almost done; just wrapping up last things like documentation)

I think we should close this particular issue as it seems it's redundant.