API Aggregation

[lavalamp]

Feature Description

• One-line feature description (can be used as a release note): User-provided apiservers can be aggregated (served along with) the rest of the Kubernetes API.
• Primary contact (assignee): @lavalamp, @deads2k, @cheftako
• Responsible SIGs: API Machinery
• Design proposal link (community repo): https://github.com/kubernetes/community/blob/master/contributors/design-proposals/aggregated-api-servers.md
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: @deads2k, @lavalamp
• Approver (likely from SIG/area to which feature belongs): @lavalamp
• Initial target stage (alpha/beta/stable) and release (x.y):
  • Alpha: v1.6
  • Beta (and on-by-default): v1.7
  • GA: v1.10

[bklau]

Is this feature going to be Beta in 1.7 release?

[lavalamp]

That is the goal

[philips]

How did this feature go directly to beta? It is a huge impacting feature.

cc @brendandburns

[smarterclayton]

Looks like it was alpha in 1.6? https://github.com/kubernetes/features/blob/master/release-1.6/release-notes-draft.md#api-server-aggregator

[smarterclayton]

I see https://github.com/kubernetes/kube-aggregator/tree/release-1.6/pkg/apis/apiregistration which was the alpha API.

[philips]

@smarterclayton that was when aggregator was a separate binary. I understand this feature is the built-in aggregator/proxy to kube-apiserver.

[smarterclayton]

So alpha / beta is typically related to stability of features and API. I guess you're saying the checked in code should be considered alpha even though the API is beta? Seems weird that the SIG wouldn't decide whether the API is stable or not (especially having evolved it), and the relative health of the feature.

[philips]

What I am trying to understand is that this feature seems net-new in 1.7 and I haven't seen features jump directly to beta in the first release they are introduced. I could be wrong two ways:

1. This isn't a net new feature in this release
2. This is a new feature but other features have jumped directly to beta in the past

I don't have a strong opinion on this but it was flagged during the community hangout as a "feature going directly to beta".

Also, I would say that having the API server proxy other APIs is a new API which should go through the usual alpha/beta/stable. But, again, I am just trying to understand the thinking and where the discussion happened.

[jbeda]

Regardless of process, I assume that this will have impact on how folks deploy and manage k8s. We haven't seen any discussion of this in SIG-cluster-lifecycle and I suspect that it hasn't shown up in SIG-cluster-ops either. Features probably shouldn't be moved to beta until there is a way for a wider set of users to get access.

[smarterclayton]

I don't see it in the 1.6 feature tracking spreadsheet, but the proposal was merged prior to 1.6 and it looks like the functionality was considered alpha by the sig and part of 1.6 (correct me if I'm wrong). I remember the discussions were about whether it should be in process or out, and the resolution prior to 1.6 was in process. Looks like this was also broadly communicated in the kubernetes-dev mailing list in 1.6 as well as in community meetings.

[smarterclayton]

Looks like this was the PR merging the proposal kubernetes/community#261, the discussion in sig-apps https://groups.google.com/forum/#!msg/kubernetes-sig-apps/0gbmMNvZWUo/fgYSHNoWCQAJ

Raises the interesting question about alpha features with beta apis, for sure.

[smarterclayton]

I thought CRD was the feature going directly to beta since it's TPR changing

[brendandburns]

Having this built into the apiserver and on-by default for the first time in 1.7 (if that is the case) is concerning to me.

I have less concerns about the API itself and many more about the deploy/operator/stability issues.

--brendan

[lavalamp]

Look at the date on the OP here; this has been plan-of-record for even longer than that. :)

The standalone was our alpha.

This has been in head for a while now and the tests seem fine, so I'm not super worried.

[idvoretskyi]

@lavalamp please, provide us with the design proposal link and docs PR link (and update the features tracking spreadsheet with it).
cc @kubernetes/sig-api-machinery-feature-requests

[philips]

The plan of record sounds like the api-aggregator binary IMHO.

@lavalamp Where are the docs on non-standalone mode?

[lavalamp]

@philips Weren't you one of the people arguing it ought to be built in? e.g. look at your comment here: https://docs.google.com/document/d/1lU1SnVtEec2iIfYx5U3L5N0za2YhfEOik0uPen276Ks/edit#heading=h.6ae9qgk4mhne

[philips]

@lavalamp I 100% think this is the right architectural thing.

But, turning it on by default on the very first release to have the functionality doesn't seem in-line with the way we have approached these things on other API server features. e.g. --experimental-bootstrap-token-auth

[philips]

Chatted with @smarterclayton and @lavalamp on this. Essentially I want to ensure that when components start to behave differently in the system, particularly on the network, that there is solid documentation on how to enable/disable those behaviors.

To set a good precedent the docs I think we need here are:

• The RBAC rules required to turn this off. Idea: As a feature flag default RBAC roles default aggr off for all roles in 1.7.
• A doc, which @lavalamp said should already exist, on the network behavior once turned on

I think having the standalone kube-aggr being the alpha is fine. Probably would be good if on this repo when something moves from alpha to beta a comment is left so there is a trail to follow

WDYT @brendandburns? @jbeda?

[lavalamp]

Or you could turn the entire APIService api off with --runtime-config (if that is plumbed correctly-- @deads2k?).

It sounds like you've got a specific requirement to know about all network traffic changes, I think staying on top of such things is going to be an uphill battle. I would not have guessed that that was the sticking point! It may be worth working on a central location for discussing architectural network traffic changes, maybe sig architecture. But I don't think we've said anywhere that we need to pay special attention to communicating such changes in particular.

Control plane components already sometimes made calls to nodes. If a user api is registered, the master will now proxy just that api's traffic to the service, which is probably located in a pod on some node. See: https://docs.google.com/document/d/1KNT4iS_Y2miLARrfSPumBIiFo_h7eb5B2pVOZJ0ZmjQ/edit

[brendandburns]

Assuming this delegation is totally off *until* I create the first custom API that delegates to that apiserver and the default is to shop with no built in aggregated apis then this seems ok to me. If the default is to turn it all the way on by default I'm not comfortable with that...

…

--brendan

________________________________ From: Brandon Philips <[email protected]> Sent: Wednesday, June 21, 2017 3:45:10 PM To: kubernetes/features Cc: Brendan Burns; Mention Subject: Re: [kubernetes/features] API Aggregation (#263) Chatted with @smarterclayton<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsmarterclayton&data=02%7C01%7Cbburns%40microsoft.com%7C0d5a014d50dd4f35544d08d4b8f72eb0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636336819154573879&sdata=D6b316DkBVQTvWbqqwYrVlLyVjdJSTRV%2F7AN%2FtzEfmo%3D&reserved=0> and @lavalamp<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flavalamp&data=02%7C01%7Cbburns%40microsoft.com%7C0d5a014d50dd4f35544d08d4b8f72eb0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636336819154573879&sdata=4sGZ2i%2B5RghmccGTJA%2B6fRjQN7p7S0pKx9i4d5kegTA%3D&reserved=0> on this. Essentially I want to ensure that when components start to behave differently in the system, particularly on the network, that there is solid documentation on how to enable/disable those behaviors. To set a good precedent the docs I think we need here are: * The RBAC rules required to turn this off. Idea: As a feature flag default RBAC roles<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkubernetes.io%2Fdocs%2Fadmin%2Fauthorization%2Frbac%2F%23default-roles-and-role-bindings&data=02%7C01%7Cbburns%40microsoft.com%7C0d5a014d50dd4f35544d08d4b8f72eb0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636336819154573879&sdata=bffo%2FQT6rqX1ukcoo1lrDAB%2Fyc4xjqZ96KOTEX4iYbY%3D&reserved=0> default aggr off for all roles in 1.7. * A doc, which @lavalamp<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flavalamp&data=02%7C01%7Cbburns%40microsoft.com%7C0d5a014d50dd4f35544d08d4b8f72eb0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636336819154573879&sdata=4sGZ2i%2B5RghmccGTJA%2B6fRjQN7p7S0pKx9i4d5kegTA%3D&reserved=0> said should already exist, on the network behavior once turned on I think having the standalone kube-aggr being the alpha is fine. Probably would be good if on this repo when something moves from alpha to beta a comment is left so there is a trail to follow WDYT @brendandburns<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbrendandburns&data=02%7C01%7Cbburns%40microsoft.com%7C0d5a014d50dd4f35544d08d4b8f72eb0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636336819154573879&sdata=BXbuyImHcNHJw7A0cjUyFLdKlJY75O23D%2F8KcI%2FYevU%3D&reserved=0>? @jbeda<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjbeda&data=02%7C01%7Cbburns%40microsoft.com%7C0d5a014d50dd4f35544d08d4b8f72eb0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636336819154573879&sdata=d4Zz%2Fju53RihQiLWIyVoAGAoTVckcuORBSpjmzeZSRs%3D&reserved=0>? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Ffeatures%2Fissues%2F263%23issuecomment-310226124&data=02%7C01%7Cbburns%40microsoft.com%7C0d5a014d50dd4f35544d08d4b8f72eb0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636336819154573879&sdata=d2Wjsr0J51YpPsnPX2eupkVQ3HhXWt%2BnUn83nOY5tAo%3D&reserved=0>, or mute the thread<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFfDgvvcItozIRM7S_iB2Qq7X888ORWwks5sGZz2gaJpZM4NGt0m&data=02%7C01%7Cbburns%40microsoft.com%7C0d5a014d50dd4f35544d08d4b8f72eb0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636336819154573879&sdata=rClHdkLnBNBajcflVlJBSEaY1cKZwZ8m%2FqNu92keWCw%3D&reserved=0>.

[philips]

@lavalamp I can't tell from that doc what exists today vs future proposals. What are the next steps to getting this doc mergeable into the docs. Can I help?

Some of the comments from deads makes it seem like some of it doesn't exist. Also, a number of the comments are unaddressed.

[jbeda]

I'll trust @philips and @brendandburns to make the call. I think they are capturing my concerns.

[lavalamp]

I think this issue can be closed at this point. Only one issue turned up at launch, and it affected few people. I'll consider that a success. :)

[luxas]

Going to GA in v1.8?

[lavalamp]

I think we would have needed to graduate the API for that, so probably not at this point. Definitely for 1.9.

…

On Fri, Sep 1, 2017 at 5:03 PM, Lucas Käldström ***@***.***> wrote: Going to GA in v1.8? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#263 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAngloG1k3RdEcDpXSqOp3tTi97GO4biks5seJtJgaJpZM4NGt0m> .

[luxas]

Reopened, added the right labels and retargeted for v1.9
Also edited the first comment to reflect the v1.9 -> GA target

[smarterclayton]

For 1.9, server side table printing to alpha with client side support so that aggregators can expose CLI output
For 1.10, server side table printing to beta and enabled in kubectl by default.

[deads2k]

Reopened, added the right labels and retargeted for v1.9
Also edited the first comment to reflect the v1.9 -> GA target

Aggregated API didn't move to GA in 1.9, but I'm planning to update it for 1.10.

[idvoretskyi]

@deads2k updated milestone to 1.10

[Bradamant3]

@deads2k 1.10 feature tracking spreadsheet indicates docs need updating. I don't see anything obvious in the original docs PR for 1.7, but could you please check? If not needed, please indicate on spreadsheet. Any questions, lmk.

[deads2k]

@deads2k 1.10 feature tracking spreadsheet indicates docs need updating. I don't see anything obvious in the original docs PR for 1.7, but could you please check? If not needed, please indicate on spreadsheet. Any questions, lmk.

Updated the spreadsheet. No new docs, it was a promotion without updates.

[justaugustus]

@lavalamp @luxas @deads2k
Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

• Description
• Milestone
• Assignee(s)
• Labels:
  • stage/{alpha,beta,stable}
  • sig/*
  • kind/feature

cc @idvoretskyi

[deads2k]

@lavalamp @luxas @deads2k
Any plans for this in 1.11?

No plans in 1.11. It remains stable.

[lavalamp]

Isn't this GA in 1.10? I think the next step here would be a v2, but we have no plans for that and it'd deserve its own issue.

[justaugustus]

Thanks for the update, @deads2k + @lavalamp!