Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API Aggregator #4173

Merged
merged 8 commits into from
Jun 27, 2017
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions _data/concepts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ toc:
section:
- docs/concepts/architecture/nodes.md
- docs/concepts/architecture/master-node-communication.md
- docs/concepts/architecture/apiserver-aggregation.md
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might make more sense to group this with/below the pre-existing Kubernetes API section.

You could either do this by bringing this section up to the Overview, or bringing the Kubernetes API section down underneath Kubernetes Architecture.

Feel free to disagree--I'm lacking context about dependencies or the history of the existing informational hierarchy, this is just my two cents. :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree this should probably go along with the api section, or we could make a new api extension section and include this, TPR, CRD, & admission controllers / initializers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'm going to create a new subsection under Concepts called Extending the Kubernetes API and put this in there. We can then start adding other docs into that. I did originally try to put it in the Overview subsection next to Kubernetes API, but this doc isn't really an overview so it didn't quite fit.


- title: Containers
section:
Expand Down
1 change: 1 addition & 0 deletions _data/tasks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ toc:
- docs/tasks/access-kubernetes-api/http-proxy-access-api.md
- docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions.md
- docs/tasks/access-kubernetes-api/extend-api-third-party-resource.md
- docs/tasks/access-kubernetes-api/setup-api-aggregator.md

- title: TLS
section:
Expand Down
32 changes: 32 additions & 0 deletions docs/concepts/architecture/apiserver-aggregation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
title: Extending the Kubernetes API with Aggregator
assignees:
- chenopis
---

{% capture overview %}

The *Aggregator* is a system to allow the Kubernetes apiserver to be extended with additional APIs, which are not part of the core Kubernetes APIs.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it best practice to refer to the "apiserver" or "API Server"? I've seen it both ways---it's more of a nit, but consistency would be nice.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the sentence reads a little easier as: "The Aggregator is a system that allows the Kubernetes apiserver to be extended with additional APIs, beyond what is offered by the core Kubernetes APIs."?

Let me know if this comment is too fine-grained.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to refer to this as the "aggregation layer" since it is built in and not a separate component.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The aggregation layer allows Kubernetes to be extended with additional APIs which are not included in the kubernetes core APIs"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that makes a lot of sense.


{% endcapture %}

{% capture body %}

## Overview

The Aggregator allows both third party extension via something like [service-catalog](https://github.com/kubernetes-incubator/service-catalog/blob/master/README.md) and the creation of extensions to allow seamless addition of the resources a customer needs. The latter can be easily bootstrapped using [apiserver-builder](https://github.com/kubernetes-incubator/apiserver-builder/blob/master/README.md).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not entirely clear to me what the difference between these two are, sorry! I'll read up on it later and see if I have suggestions, but do you mind clarifying?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Third party extensions are resource which are not built by either the cluster owner or the kubernetes team. Service Catalog is an example of this and is in fact an example which will make it easy for cluster owners to easily add other third party extensions to their cluster. This sort of extension is already built but does need to be deployed properly to the kubernetes cluster. If the cluster owner wants to create their own extension they first have to build something which adheres to the extension api-server requirements and then they have to work out how to deploy it correctly. So the story difference here is largely about the use case, though the technical details are very similar.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"aggregation layer"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The aggregation layer allows installing additional Kubernetes-style APIs in your cluster. These can either be pre-built, existing 3rd party solutions (e.g., service catalog), or user-created APIs (apiserver-builder can get you started)."

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes this is much more clear!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed


In 1.7, the Aggregator runs in a process with the kube-apiserver; however, until an extension resource is registered, the Aggregator will do nothing. Once a new resource is registered, Aggregator will proxy the request to the extension-apiserver to be served.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "to be served" might be redundant

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, is extension-apiserver an official term? I can infer what you're referring to, but it definitely made me pause. Would be nice to italicize it and mention it in the first paragraph.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly I think we're still trying to come up with the official term. However I've heard several complaints about user-apiserver as that seems to exclude the third party use case. As this is meant as an extension mechanism we seem to be moving toward consensus on that term.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My suggested wording in the doc was:

"In 1.7 the aggregation layer runs in-process with the kube-apiserver. Until an extension resource is registered, the aggregation layer will do nothing. Users can add a APIService object to register their API, which “claims” the URL path in the Kubernetes API. At this point, the aggregation layer will proxy anything sent to that api path (e.g. /apis/myextension.mycompany.io/v1/…) to the registered APIService. Ordinarily, the APIService will be implemented by an extension-apiserver in a pod running in the cluster. This extension-apiserver will normally need to be paired with one or more controllers if active management of the added resources is needed. As a result, the apiserver-builder will actually provide a skeleton for both. For example, when the service-catalog is installed, it provides both the extension-apiserver and controller for the services it provides."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

swapped in


**Note:** The extension-apiserver will normally need to be paired with a controller to actually provide any functionality. As a result, things like the apiserver-builder will actually provide a skeleton for both. The service-catalog should provide both the extension-apiserver and controller for the services it provides.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ a bit wordy----am I correctly interpreting that both the apiserver-builder and the service-catalog provide the extension-apiserver and the controller? (A little confused if "should provide" means that service-catalog does it by default)

Copy link
Member

@cheftako cheftako Jun 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The apiserver-build will provide both a skeleton extension-apiserver and a skeleton controller. The service catalog can install extensions for you. When it does this it should provide/deploy both the extension apiserver and any other resources it needs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reworded this paragraph to clarify.


{% endcapture %}

{% capture whatsnext %}

* To get the aggregator working in your environment, see [Setup the API aggregator](/docs/tasks/access-kubernetes-api/setup-api-aggregator/).
* Learn how to [Extend the Kubernetes API Using Third Party Resources](/docs/tasks/access-kubernetes-api/extend-api-third-party-resource/).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like TPRs are being deprecated in 1.7? Swap out with Extend the Kubernetes API Using Custom Resource Definitions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this might be the Google doc that you are already aware of, but it seems like TPR/CRDs are distinct from API Aggregation? It would be great to make this more clear, since this is a high-level conceptual doc.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aggregator is completely distinct from TPR/CRD. However the Aggregator has no opinion on how the the extension resource data is persisted (or if it needs persisted data). Practically the extension resources will need to persist their data. The sample-apiserver and apiserver-builder both assume you will run your own etcd pod which they can use. TPR/CRD are also good answers to this need and allow kubernetes to solve the management portion of persisting the data.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't call these third party resources. Extension APIs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the link has been changed


{% endcapture %}

{% include templates/concept.md %}
68 changes: 68 additions & 0 deletions docs/tasks/access-kubernetes-api/setup-api-aggregator.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
title: Setup the API Aggregator
assignees:
- chenopis
---

{% capture overview %}

Setting up the API Aggregator will allow the Kubernetes apiserver to be extended with additional APIs, which are not part of the core Kubernetes APIs.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "allows" instead of "will allow" (there are a couple unnecessary future tenses scattered around...might want to just Ctrl-f "will")

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would call this doc "turn on the aggregation layer" or "configure the aggregation layer" or "configure API Aggregation"


{% endcapture %}

{% capture prerequisites %}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are not prereqs, these are what this doc is going to tell you how to do.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Are there any other prereqs then that they'll need before they can get started?


* You need to have a Kubernetes cluster running.
* Make sure there are Aggregator certificates (both CA and proxy) setup on the Kubernetes apiserver.
* The proxy cert must have been signed by the relevant CA.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clarify what "relevant" means?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kubernetes (and the kube-apiserver) have multiple CAs. I want to make sure the proxy is signed by the aggregator CA and not say the master CA.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like this should go together with the previous bullet point then?


{% endcapture %}

{% capture steps %}

## Enable apiserver flags

The system needs to be enabled via the following apiserver flags; however, they may have already been taken care of by your provider.

--requestheader-client-ca-file=<path to aggregator CA cert>
--requestheader-allowed-names=aggregator
--requestheader-extra-headers-prefix=X-Remote-Extra-
--requestheader-group-headers=X-Remote-Group
--requestheader-username-headers=X-Remote-User
--proxy-client-cert-file=<path to aggregator proxy cert>
--proxy-client-key-file=<path to aggregator proxy key>

If you are not running kube-proxy on your master, as the [Kubernetes Architectural Roadmap](https://docs.google.com/a/google.com/document/d/1XkjVm4bOeiVkj-Xt1LgoGiqWsBfNozJ51dyI-ljzt1o/edit?usp=sharing) recommends, then you need to make sure that the system is enabled with the following apiserver flag. Again, this may have already been taken care of by your provider.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found this a tad confusing. Just to be clear the roadmap recommends not running kube-proxy on the master.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reworded it


--enable-aggregator-routing=true

## Get an extension api-server working with Aggregator
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO this and everything below should go in a separate doc, "how to setup an extension API server".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'm splitting this into two docs.


The generated skeleton, which comes with [apiserver-builder](https://github.com/Kubernetes-incubator/apiserver-builder/blob/master/README.md), should setup the skeleton server it creates to do all of these steps. You can also look at the sample-apiserver under Kubernetes for ways the following can be done.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

provide link to sample-apiserver? Also consider rewording the first sentence.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1. Make sure the APIService api is enabled (check --runtime-config); it is on by default, so it should be on unless it’s been deliberately turned off in your cluster.
1. You may need to make an RBAC rule allowing you to add APIService objects, or get your cluster administrator to make one. (Since API extensions affect the entire cluster, it is not recommended to do testing/development/debug of an API extension in a live cluster.)
1. Create the Kubernetes namespace you want to run your extension api-service in.
1. Create/get a CA cert to be used to sign the server cert the extension api-server uses for HTTPS.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do these easily map to code? sample commands might make it easier to follow.

1. Create a server cert/key for the api-server to use for HTTPS. This cert should be signed by the above CA. It should also have a CN of the Kube DNS name. This will generally be derived from the Kubernetes service and be of the form <service name>.<service name namespace>.svc
1. Create a Kubernetes secret with the server cert/key in your namespace.
1. Create a Kubernetes deployment for the extension api-server and make sure you are loading the secret as a volume. It will also need to contain a reference to a working image of your extension api-server. The deployment should also be in your namespace.
1. Make sure that your extension-apiserver loads those certs from that volume and that they are used in the HTTPS handshake.
1. Create a Kubernetes service account in your namespace.
1. Create a Kubernetes cluster role for the operations you want to allow on your resources.
1. Create a Kubernetes cluster role binding from the default service account in your namespace to the cluster role you just created.
1. Create a Kubernetes apiservice. The CA cert above should be base 64 encoded, stripped of new lines and used as the spec.caBundle in the apiservce. This should not be namespaced.
1. Use kubectl to get your resource. It will probably return "No resources found." Which means that everything worked but you currently have no objects of that resource type created yet.

{% endcapture %}

{% capture whatsnext %}

* For more information, see the [Aggregator Networking Requirements](https://docs.google.com/document/d/1KNT4iS_Y2miLARrfSPumBIiFo_h7eb5B2pVOZJ0ZmjQ/edit) doc.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have access to this Google doc---might want to check the permissions on it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to take this out until it's publicly accessible.

* For a high level overview, see [Extending the Kubernetes API with Aggregator](/docs/concepts/architecture/apiserver-aggregation/).
* Learn how to [Extend the Kubernetes API Using Third Party Resources](/docs/tasks/access-Kubernetes-api/extend-api-third-party-resource/).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see the earlier comment about CRDs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it


{% endcapture %}

{% include templates/task.md %}