Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KEP-2819: CNI traffic shaping support #2808

Closed
wants to merge 3 commits into from
Closed
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 133 additions & 0 deletions keps/sig-network/2819-CNI-traffic-shaping-support/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# KEP-2819: CNI traffic shaping support

<!-- toc -->
- [Release Signoff Checklist](#release-signoff-checklist)
- [Summary](#summary)
- [Motivation](#motivation)
- [Goals](#goals)
- [Non-goals](#non-goals)
- [Proposal](#proposal)
- [Pod Setup](#pod-setup)
- [Pod Teardown](#pod-teardown)
- [Graduation Criteria](#graduation-criteria)
- [Implementation History](#implementation-history)
- [CNI plugin part](#cni-plugin-part)
- [Kubernetes part](#kubernetes-part)
- [Drawbacks](#drawbacks)
- [Alternatives](#alternatives)
<!-- /toc -->

## Release Signoff Checklist

Items marked with (R) are required *prior to targeting to a milestone / release*.

- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
- [ ] (R) KEP approvers have approved the KEP status as `implementable`
- [ ] (R) Design details are appropriately documented
- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
- [ ] (R) Graduation criteria is in place
- [ ] (R) Production readiness review completed
- [ ] Production readiness review approved
- [ ] "Implementation History" section is up-to-date for milestone
- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
- [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

<!--
**Note:** This checklist is iterative and should be reviewed and updated every time this enhancement is being considered for a milestone.
-->

## Summary

Make kubelet cni network plugin support basic traffic shapping capability `bandwidth`.

## Motivation

Currently the kubenet code supports applying basic traffic shaping during pod setup.
This will happen if bandwidth-related annotations have been added to the pod's metadata.
Comment on lines +45 to +46
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this still true now we've removed the docker shim?


Kubelet CNI code doesn't support it yet, though CNI has already added a [traffic sharping plugin](https://github.com/containernetworking/plugins/tree/master/plugins/meta/bandwidth).
uablrek marked this conversation as resolved.
Show resolved Hide resolved
We can replicate the behavior we have today in kubenet for kubelet CNI network plugin if we feel this is an important feature.

### Goals

* Support traffic shaping for CNI network plugin in Kubernetes.

### Non-goals

* CNI plugins to implement this sort of traffic shaping guarantee.


## Proposal

If kubelet starts up with `network-plugin = cni` and user enabled traffic shaping via the network plugin configuration,
it would then populate the runtimeConfig section of the config when calling the bandwidth plugin.
Comment on lines +62 to +63
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that Kubernetes only supports CNI network plugins now, at least on the default branch. I might be wrong.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically. Networking is now delegated to the CRI runtime.


Traffic shaping in Kubelet CNI network plugin can work with ptp and bridge network plugins.

### Pod Setup
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few comments/questions

  • Do we really have to do it via annotation? can't we do it with apis?
  • Have you considered dynamic updates? e.g., rate limit to x but later to y in response to some external condition?
  • FYI The implementation will suffer from the same problems we have with NPol, the idea is users can not make a clear determination that CNI they configured can in fact support the rate limiting.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation will suffer from the same problems we have with NPol, the idea is users can not make a clear determination that CNI they configured can in fact support the rate limiting.

A Kubernetes style API, if that were the approach taken, could in theory use some part of its .status field to at least let a CNI / related controller record whether the limit has been applied (I'm not sure why NetworkPolicy doesn't do that). Still doesn't let you check for feature availability before you create the resource though.


When we create a pod with bandwidth configuration in its metadata, for example,

```json
{
"kind": "Pod",
"metadata": {
"name": "iperf-slow",
"annotations": {
"kubernetes.io/ingress-bandwidth": "10M",
"kubernetes.io/egress-bandwidth": "10M"
Comment on lines +77 to +78
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}
}
}
```

Kubelet would firstly parse the ingress and egress bandwidth values and transform them to ingressRate and egressRate for cni bandwidth plugin.
Kubelet would then detect whether user has enabled the traffic shaping plugin by checking the following CNI config file:

```json
{
"type": "bandwidth",
"capabilities": {"trafficShaping": true}
}
```

If traffic shaping plugin is enabled, kubelet would populate the runtimeConfig section of the config when call the bandwidth plugin:

```json
{
"type": "bandwidth",
"runtimeConfig": {
"trafficShaping": {
"ingressRate": "X",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of the problem with the existing annotation is that they rely on plugin-specific definitions of bandwidth and burst.

We should probably clearly define either

  1. Burst isn't required
  2. Burst has a precise definition (e.g. Bucket size is N percent / ratio of bandwidth)

"egressRate": "Y"
}
}
}
```

### Pod Teardown

When we delete a pod, kubelet will bulid the runtime config call cni plugin DelNetworkList API, which will remove this pod's bandwidth configuration.

## Graduation Criteria

* Add traffic shaping as part of the Kubernetes e2e runs and ensure tests are not failing.

## Implementation History

### CNI plugin part

* [traffic shaping plugin](https://github.com/containernetworking/plugins/pull/96)
* [support runtime config](https://github.com/containernetworking/plugins/pull/138)

### Kubernetes part

* [add traffic shaping support](https://github.com/kubernetes/kubernetes/pull/63194)

## Drawbacks
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only drawback is a larger feature: bandwidth is not currently a scheduled resource. Do we want to consider adding node-egress-bandwidth to a pod's resources?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is made complicated by the fact that bandwidth is only accountable as a pod-level resource, rather than a container-level resource (barring serious trickery)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

brief outline of an approach

Someone who wants to can define an extended resource and assign that to a Pod. Pick something like “bits per second” because you can only have whole numbers of these.

Nodes can advertise their actual uplink as the available amount of that resource. Or, if we decide to, we could implement some cleverness around a particular extended resource where this doesn't hold (allowing overcommitment).


None

## Alternatives

None
35 changes: 35 additions & 0 deletions keps/sig-network/2819-CNI-traffic-shaping-support/kep.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
title: CNI traffic shaping support
kep-number: 2819
authors:
- "@Lion-Wei"
- "@m1093782566"
owning-sig: sig-network
participating-sigs:
status: implemented
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would that mean that this is generally available?

Suggested change
status: implemented
status: implementable

creation-date: 2018-05-31
reviewers:
- "@thockin"
- "@m1093782566"
approvers:
- "@thockin"
- "@m1093782566"

see-also:
replaces:

stage: alpha

# The most recent milestone for which work toward delivery of this KEP has been
# done. This can be the current (upcoming) milestone, if it is being actively
# worked on.
latest-milestone: "v1.9"

# The milestone at which this feature was, or is targeted to be, at each stage.
milestone:
alpha: "v1.9"
beta: "v1.23"
stable: "v1.24"

feature-gates:
disable-supported: false