Custom Resource Documentation #4071
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
|
@enisoc Is this ready for review now? |
|
@chenopis Sorry, not yet. I'm planning to push a ready-for-review commit by the end of today. |
|
@enisoc No worries, just wasn't sure. Ping me when it's ready. Thx |
enisoc
force-pushed
the
crd-1.7
branch
2 times, most recently
from
0926785 to
de1ab47
Compare
|
nice doc. lgtm. |
|
@abiogenesis-now Can you take a look at this one as well? Thx |
| It can take up to 10 seconds for the TPR controller to notice that you deleted the TPR definition | ||
| and initiate the migration. TPR data will remain accessible during this time. | ||
|
|
||
| Once the migration completes, the resource will begin serving through the CRD. |
There was a problem hiding this comment.
[RW] Once the migration completes, the resource begins serving through the CRD.
Eliminates future tense
|
|
||
| After verifying the CRD data, restart any clients you stopped before the migration, such as | ||
| custom controllers. | ||
| These controllers will now access CRD data when they make requests on the same API endpoints |
There was a problem hiding this comment.
delete "will"
Eliminates future tense
|
|
||
| For custom resources that need specialized implementations, you can write and deploy a standalone | ||
| API server, then make the resource available to clients of the main API server through aggregation. | ||
| The main API server will delegate requests to you for the custom resources that you handle. |
There was a problem hiding this comment.
[RW] The main API server delegates requests to you...
avoids future tense
| 1. **Install the CustomResourceDefinition** | ||
|
|
||
| While the source TPR is still active, install the matching CRD with `kubectl create`. | ||
| Existing TPR data will remain accessible because TPRs take precedence over CRDs when both try |
There was a problem hiding this comment.
[RW] Existing TPR data remains accessible...
avoids future tense
|
|
||
| 1. **Stop all clients that use the TPR** | ||
|
|
||
| The API server will attempt to prevent mutation of TPR data for the resource while it |
There was a problem hiding this comment.
[RW] The API server attempts to prevent TPR data for the resource from mutation while it...
avoids future tense and nominalization (easier to read)
| Stopping clients, such as TPR-based custom controllers, helps to avoid inconsistencies in | ||
| the copied data. | ||
|
|
||
| In addition, clients that watch TPR data will not receive any more events once the migration |
There was a problem hiding this comment.
[RW] In addition, clients that watch TPR data do not receive any more events once the migration begins.
avoids future tense
|
|
||
| Normally, when you delete a TPR definition, the API server tries to clean up any objects stored | ||
| in that resource. | ||
| Since you created a matching CRD, the server will copy objects to the CRD instead of deleting |
There was a problem hiding this comment.
[RW] After you create a matching CRD, CRD, the server copies objects to the CRD instead of deleting...
There was a problem hiding this comment.
I think "After" makes it sound like creating the CRD triggers something to happen, which it doesn't. Deleting the TPR is the trigger, and at that point it checks whether the CRD exists. I've reworded to hopefully make this more clear.
|
|
||
| 1. **Verify the new CRD data** | ||
|
|
||
| It can take up to 10 seconds for the TPR controller to notice that you deleted the TPR definition |
There was a problem hiding this comment.
[RW] It can take up to 10 seconds for the TPR controller to notice that you delete the TPR definition and to initiate the migration. The data remains accessible during this time.
Once the migration completes, the resource begins serving through the CRD.
avoids future tense and adds parallel construction
|
|
||
| This endpoint URL can then be used to create and manage custom objects. | ||
| The `kind` of these objects will be `CronTab` from the spec of the | ||
| `CustomResourceDefinition` object we created above. | ||
| CustomResourceDefinition object we created above. |
There was a problem hiding this comment.
replace "we" with "you"
| custom object of kind `CronTab`. The kind `CronTab` comes from the spec of the | ||
| `CustomResourceDefinition` object we created above. | ||
| CustomResourceDefinition object we created above. |
There was a problem hiding this comment.
replace "we" with "you"
|
|
||
| This frees you from writing your own API server to handle the custom resource, | ||
| but the generic nature of the implementation means you have less flexibility than with an | ||
| [aggregated API server](#aggregated-api-servers). |
There was a problem hiding this comment.
There was a problem hiding this comment.
#namingishard
would also like to be part of this meeting! @chenopis dunno who organizes this kind of stuff, but let me know if you need help.
There was a problem hiding this comment.
@lavalamp I'm game. @abiogenesis-now I'll loop you in when we setup a time.
_data/concepts.yml
Outdated
| @@ -89,4 +89,6 @@ toc: | |||
| - docs/concepts/policy/security-context.md | |||
| - docs/concepts/policy/pod-security-policy.md | |||
|
|
|||
|
|
|||
| - title: Extensions | |||
There was a problem hiding this comment.
In terms of organizing this content, might want to sync up with @chenopis regarding #4173 (comment) (Docs PR for API Aggregation)? e.g. it doesn't matter which PR implements it, but make sure you guys are aware in the case of merge conflicts.
There was a problem hiding this comment.
I'm creating a subsection in Concepts called Extending the Kubernetes API, so we should group this in that as well.
- title: Extending the Kubernetes API
section:
- docs/concepts/api-extension/apiserver-aggregation.md
|
|
||
| Custom resources can appear and disappear in a running cluster through dynamic registration, | ||
| and cluster admins can update custom resources independently of the cluster itself. | ||
| Once it's installed, users can create and access objects in a custom resource with |
There was a problem hiding this comment.
[RW for clarity] Once a custom resource is installed, users can create and access its objects with kubectl, just as they do for built-in resources like pods.
| certain kind. | ||
| For example, the built-in *pods* resource contains a collection of Pod objects. | ||
|
|
||
| A *custom resource* is an extension of the Kubernetes API that is not necessarily available on every |
There was a problem hiding this comment.
[RW] Second sentence: In other words, it represents a customization of a particular Kubernetes installation.
| independently of the cluster's own lifecycle. | ||
| Custom controllers can work with any kind of resource, but they fit especially well with custom | ||
| resources. | ||
| For example, the [Operator](https://coreos.com/blog/introducing-operators.html) pattern is a |
There was a problem hiding this comment.
[RW] Custom controllers can work with any kind of resource, but they are especially effective when combined with custom resources. The Operator pattern is one example of such a combination. It allows developers to encode domain knowledge for specific applications into an extension of the Kubernetes API.
|
|
||
| This frees you from writing your own API server to handle the custom resource, | ||
| but the generic nature of the implementation means you have less flexibility than with an | ||
| [aggregated API server](#aggregated-api-servers). |
There was a problem hiding this comment.
| Some or all of these steps may be unnecessary if the custom controller handles the migration for | ||
| you. | ||
| * Be familiar with the concept of [custom resources](/docs/concepts/extensions/custom-resources/), | ||
| which were known as *third-party resources* until Kubernetes 1.7. |
There was a problem hiding this comment.
oh boy, that's a bit confusing >< (just a comment about "third-party-resource", not asking for an edit here)
There was a problem hiding this comment.
Is the confusion because we called the general concept "third-party resources" and the API object "ThirdPartyResource", and we are renaming both at the same time? We tried to make this better going forward by calling the API object CustomResourceDefinition so we could use the term "custom resource" to refer to the general concept, which is a thing you can achieve either with CRD or some other way. (see #45277)
There was a problem hiding this comment.
Would it be helpful to add an additional sentence to hammer home the point? Or is that overkill:
- Be familiar with the concept of custom resources, which were known as third-party resources until Kubernetes 1.7.
- Be familiar with CustomResourceDefinitions, which are the simplest way to make custom resources API-accessible.
| * Be familiar with the concept of [custom resources](/docs/concepts/extensions/custom-resources/), | ||
| which were known as *third-party resources* until Kubernetes 1.7. | ||
| * Be familiar with [how to use CustomResourceDefinitions](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/). | ||
| * **Before performing a migration on real data, conduct a dry run of these steps in a test cluster.** |
There was a problem hiding this comment.
instead phrase this as "conduct a dry run by going through these steps in a test cluster"? It might be just me but for a second I thought "dry run" was a special command
| 1. **Stop all clients that use the TPR** | ||
|
|
||
| The API server will attempt to prevent mutation of TPR data for the resource while it | ||
| copies objects to the CR, but it can't guarantee consistency in all cases, such as with |
| 1. **Restart clients** | ||
|
|
||
| After verifying the CRD data, restart any clients you stopped before the migration, such as | ||
| custom controllers. |
There was a problem hiding this comment.
maybe not 100% necessary, but add "and watchers" ?
| Check that all your objects were correctly copied: | ||
|
|
||
| ```shell | ||
| kubectl get crontabs --all-namespaces -o yaml |
There was a problem hiding this comment.
Out of scope for this PR, but @chenopis, what do you think of eventually implementing custom Markdown like DigitalOcean has (https://www.digitalocean.com/community/tutorials/digitalocean-s-writing-guidelines#variables)? That way it's clearer that things like crontab are meant to be swapped out and are not part of the real command.
See:
There was a problem hiding this comment.
@abiogenesis-now Yeah, that looks more clear. I'm not satisfied w/ our current flavor of markdown. Can you open an Issue and assign it to me? Maybe we can tackle this after moving the production site over to Netlify.
|
@enisoc FYI, all feedback must be addressed and LGTMs given by EOD Tue, June 27th so that this can be merged for the 1.7 release on June 28th. |
enisoc
force-pushed
the
crd-1.7
branch
2 times, most recently
from
6937b1f to
1fe0e88
Compare
| kubectl get ct -o yaml | ||
| ``` | ||
|
|
||
| You should see that it contains the custom `cronSpec` and `image` fields |
There was a problem hiding this comment.
Yes, if you use kubectl 1.7+. Are you suggesting to show that output instead? It looks like this:
Name: my-new-cron-object
Namespace: default
Labels: <none>
Annotations: <none>
API Version: stable.example.com/v1
Kind: CronTab
Metadata:
Cluster Name:
Creation Timestamp: 2017-06-26T20:23:26Z
Generation: 0
Resource Version: 427
Self Link: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object
UID: 4f18b46a-5aad-11e7-9f5e-a0481c9805b1
Spec:
Cron Spec: * * * * /5
Image: old-image
Events: <none>
| CustomResources (objects created in the schema defined by CustomResourceDefintions) | ||
| support finalizers. If you add a `metadata.finalizers` stanza like | ||
|
|
||
| Custom objects support finalizers. If you add a `metadata.finalizers` stanza like: |
There was a problem hiding this comment.
The rest of this section is essentially a definition of what a finalizer is. I can't think of a good way to define it in one line, and I haven't found a good doc to link to. The one you found in the namespaces proposal appears to be a bit out of date, and refers only to the usage with namespaces, not the general concept of finalizers which can be on any object.
| @@ -0,0 +1,163 @@ | |||
| --- | |||
| title: Migrate a ThirdPartyResource to CustomResourceDefinition | |||
There was a problem hiding this comment.
I guess that's grammatically correcterer, but the second "a" feels weird to me... I think in my head I'm expanding my version of the title as, "Migrate a [single] ThirdPartyResource [object] to [use the] CustomResourceDefinition [API]."
If I'm the only one who thinks it feels weird, I'm fine going with your suggestion.
| Some or all of these steps may be unnecessary if the custom controller handles the migration for | ||
| you. | ||
| * Be familiar with the concept of [custom resources](/docs/concepts/extensions/custom-resources/), | ||
| which were known as *third-party resources* until Kubernetes 1.7. |
There was a problem hiding this comment.
Is the confusion because we called the general concept "third-party resources" and the API object "ThirdPartyResource", and we are renaming both at the same time? We tried to make this better going forward by calling the API object CustomResourceDefinition so we could use the term "custom resource" to refer to the general concept, which is a thing you can achieve either with CRD or some other way. (see #45277)
|
|
||
| Normally, when you delete a TPR definition, the API server tries to clean up any objects stored | ||
| in that resource. | ||
| Since you created a matching CRD, the server will copy objects to the CRD instead of deleting |
There was a problem hiding this comment.
I think "After" makes it sound like creating the CRD triggers something to happen, which it doesn't. Deleting the TPR is the trigger, and at that point it checks whether the CRD exists. I've reworded to hopefully make this more clear.
|
@cody-clark @abiogenesis-now PTAL. Thx |
|
responded with some rewording suggestions, but these are optional (e.g. use them if you think they're useful). otherwise LGTM |
|
Added some more edits for the parts about finalizers and prereqs for migration. |
|
@enisoc Sorry, due to some other recent PR merges, there are now conflicts w/ the |
|
@chenopis lgtm |
|
Conflicts resolved. |
|
Review status: 0 of 10 files reviewed at latest revision, 30 unresolved discussions. Comments from Reviewable |
Docs for CustomResourceDefinition and the concept of custom resources in general.
Preview links:
This change is