-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KEP-2819: CNI traffic shaping support #2808
Changes from 2 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,133 @@ | ||
# KEP-2819: CNI traffic shaping support | ||
|
||
<!-- toc --> | ||
- [Release Signoff Checklist](#release-signoff-checklist) | ||
- [Summary](#summary) | ||
- [Motivation](#motivation) | ||
- [Goals](#goals) | ||
- [Non-goals](#non-goals) | ||
- [Proposal](#proposal) | ||
- [Pod Setup](#pod-setup) | ||
- [Pod Teardown](#pod-teardown) | ||
- [Graduation Criteria](#graduation-criteria) | ||
- [Implementation History](#implementation-history) | ||
- [CNI plugin part](#cni-plugin-part) | ||
- [Kubernetes part](#kubernetes-part) | ||
- [Drawbacks](#drawbacks) | ||
- [Alternatives](#alternatives) | ||
<!-- /toc --> | ||
|
||
## Release Signoff Checklist | ||
|
||
Items marked with (R) are required *prior to targeting to a milestone / release*. | ||
|
||
- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) | ||
- [ ] (R) KEP approvers have approved the KEP status as `implementable` | ||
- [ ] (R) Design details are appropriately documented | ||
- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input | ||
- [ ] (R) Graduation criteria is in place | ||
- [ ] (R) Production readiness review completed | ||
- [ ] Production readiness review approved | ||
- [ ] "Implementation History" section is up-to-date for milestone | ||
- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] | ||
- [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes | ||
|
||
<!-- | ||
**Note:** This checklist is iterative and should be reviewed and updated every time this enhancement is being considered for a milestone. | ||
--> | ||
|
||
## Summary | ||
|
||
Make kubelet cni network plugin support basic traffic shapping capability `bandwidth`. | ||
|
||
## Motivation | ||
|
||
Currently the kubenet code supports applying basic traffic shaping during pod setup. | ||
This will happen if bandwidth-related annotations have been added to the pod's metadata. | ||
|
||
Kubelet CNI code doesn't support it yet, though CNI has already added a [traffic sharping plugin](https://github.com/containernetworking/plugins/tree/master/plugins/meta/bandwidth). | ||
uablrek marked this conversation as resolved.
Show resolved
Hide resolved
|
||
We can replicate the behavior we have today in kubenet for kubelet CNI network plugin if we feel this is an important feature. | ||
|
||
### Goals | ||
|
||
* Support traffic shaping for CNI network plugin in Kubernetes. | ||
|
||
### Non-goals | ||
|
||
* CNI plugins to implement this sort of traffic shaping guarantee. | ||
|
||
|
||
## Proposal | ||
|
||
If kubelet starts up with `network-plugin = cni` and user enabled traffic shaping via the network plugin configuration, | ||
it would then populate the runtimeConfig section of the config when calling the bandwidth plugin. | ||
Comment on lines
+62
to
+63
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think that Kubernetes only supports CNI network plugins now, at least on the default branch. I might be wrong. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Basically. Networking is now delegated to the CRI runtime. |
||
|
||
Traffic shaping in Kubelet CNI network plugin can work with ptp and bridge network plugins. | ||
|
||
### Pod Setup | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. few comments/questions
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
A Kubernetes style API, if that were the approach taken, could in theory use some part of its |
||
|
||
When we create a pod with bandwidth configuration in its metadata, for example, | ||
|
||
```json | ||
{ | ||
"kind": "Pod", | ||
"metadata": { | ||
"name": "iperf-slow", | ||
"annotations": { | ||
"kubernetes.io/ingress-bandwidth": "10M", | ||
"kubernetes.io/egress-bandwidth": "10M" | ||
Comment on lines
+77
to
+78
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We should add these to https://kubernetes.io/docs/reference/labels-annotations-taints/ |
||
} | ||
} | ||
} | ||
``` | ||
|
||
Kubelet would firstly parse the ingress and egress bandwidth values and transform them to ingressRate and egressRate for cni bandwidth plugin. | ||
Kubelet would then detect whether user has enabled the traffic shaping plugin by checking the following CNI config file: | ||
|
||
```json | ||
{ | ||
"type": "bandwidth", | ||
"capabilities": {"trafficShaping": true} | ||
} | ||
``` | ||
|
||
If traffic shaping plugin is enabled, kubelet would populate the runtimeConfig section of the config when call the bandwidth plugin: | ||
|
||
```json | ||
{ | ||
"type": "bandwidth", | ||
"runtimeConfig": { | ||
"trafficShaping": { | ||
"ingressRate": "X", | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One of the problem with the existing annotation is that they rely on plugin-specific definitions of bandwidth and burst. We should probably clearly define either
|
||
"egressRate": "Y" | ||
} | ||
} | ||
} | ||
``` | ||
|
||
### Pod Teardown | ||
|
||
When we delete a pod, kubelet will bulid the runtime config call cni plugin DelNetworkList API, which will remove this pod's bandwidth configuration. | ||
|
||
## Graduation Criteria | ||
|
||
* Add traffic shaping as part of the Kubernetes e2e runs and ensure tests are not failing. | ||
|
||
## Implementation History | ||
|
||
### CNI plugin part | ||
|
||
* [traffic shaping plugin](https://github.com/containernetworking/plugins/pull/96) | ||
* [support runtime config](https://github.com/containernetworking/plugins/pull/138) | ||
|
||
### Kubernetes part | ||
|
||
* [add traffic shaping support](https://github.com/kubernetes/kubernetes/pull/63194) | ||
|
||
## Drawbacks | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The only drawback is a larger feature: bandwidth is not currently a scheduled resource. Do we want to consider adding node-egress-bandwidth to a pod's resources? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is made complicated by the fact that bandwidth is only accountable as a pod-level resource, rather than a container-level resource (barring serious trickery) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. brief outline of an approach Someone who wants to can define an extended resource and assign that to a Pod. Pick something like “bits per second” because you can only have whole numbers of these. Nodes can advertise their actual uplink as the available amount of that resource. Or, if we decide to, we could implement some cleverness around a particular extended resource where this doesn't hold (allowing overcommitment). |
||
|
||
None | ||
|
||
## Alternatives | ||
|
||
None |
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,35 @@ | ||||||
title: CNI traffic shaping support | ||||||
kep-number: 2819 | ||||||
authors: | ||||||
- "@Lion-Wei" | ||||||
- "@m1093782566" | ||||||
owning-sig: sig-network | ||||||
participating-sigs: | ||||||
status: implemented | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Would that mean that this is generally available?
Suggested change
|
||||||
creation-date: 2018-05-31 | ||||||
reviewers: | ||||||
- "@thockin" | ||||||
- "@m1093782566" | ||||||
approvers: | ||||||
- "@thockin" | ||||||
- "@m1093782566" | ||||||
|
||||||
see-also: | ||||||
replaces: | ||||||
|
||||||
stage: alpha | ||||||
|
||||||
# The most recent milestone for which work toward delivery of this KEP has been | ||||||
# done. This can be the current (upcoming) milestone, if it is being actively | ||||||
# worked on. | ||||||
latest-milestone: "v1.9" | ||||||
|
||||||
# The milestone at which this feature was, or is targeted to be, at each stage. | ||||||
milestone: | ||||||
alpha: "v1.9" | ||||||
beta: "v1.23" | ||||||
stable: "v1.24" | ||||||
|
||||||
feature-gates: | ||||||
disable-supported: false | ||||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this still true now we've removed the docker shim?