CustomResourceDefinitions

[deads2k]

Enhancement Description

• One-line enhancement description (can be used as a release note): Custom Resource Definitions allow defining kubernetes-style APIs as add-ons
• Kubernetes Enhancement Proposal:
  • https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20180415-crds-to-ga.md
• See also:
  • Moving ThirdPartyResources to beta
  • Versioning
  • Validation
  • Subresources
• Primary contact (assignee): @sttts, @jpbetz
• Responsible SIGs: api-machinery
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): Already released
  • Beta release target (x.y): v1.7
  • Stable release target (x.y): v1.16

Scope of work planned for v1.15

• Define allowed OpenAPI subset (sig-api-machinery KEP: Vanilla OpenAPI subset: structural schema #1002, Publish CRD OpenAPI Schema #692)
• Define and perform scale testing for CRD (sig-api-machinery: Add scale targets to CRDs to GA KEP #1015)
• Bring CRD conversion webhooks to beta (sig-api-machinery: port CRD conversion webhook design doc to KEP, add v1beta1 criteria #1004, Webhook Conversion for Custom Resources #598)

Scope of work planned for v1.11

• Implement first steps of CRD versioning. This has another features issue: Custom Resource Definition Versioning #544. Design proposal: CRD versioning design proposal community#2026.
• Add pruning and defaulting for custom resources.
• Support maps in the CRD validation schema. CustomResourceDefinitions should support maps in OpenAPI schema kubernetes#59485
• Implement quota support for custom resources (this can be considered as a bug fix too) quota support for custom resources kubernetes#53777

Scope of work planned for v1.10

• Finish CRD sub-resources as alpha Proposal: SubResources for CustomResources community#913
• Support categories for CRDs.

Scope of work planned for v1.9

• CRD validation to beta Requirements for ThirdPartyResource to graduate to beta kubernetes#22768 Getting CRD Validation to Beta kubernetes#53829

Scope of work planned for v1.8

• Remove deprecated ThirdPartyResource API.
• Add validation and defaulting for CustomResourceDefinition.
• Add subresources for CustomResourceDefinition.
  • Support Spec/Status split (/status subresource) on custom resources.
  • Support incrementing object Generation on custom resource data mutation (requires Spec/Status split).
• Support OwnerReference-based garbage collection with CRD.

Scope of work planned for v1.7

• Move TPR to a new API group (tentatively called apiextensions) to support deprecation of the extensions group
  • Ideally, implement the new TPR in a separate API server, to be integrated into kube-apiserver via API Aggregation.
• For now, only allow 1 version at a time per TPR. In the absence of conversion (which is out of scope for this release), this is necessary to remain consistent with the expectations of other components.
  • Support for multiple versions could be added (with or without conversion) in a later release.
• Fix name conflicts due to lossy conversion of TPR name into resource/kind.
• Allow TPRs to specify their own names for resources and kinds, rather than tying them to the TPR name.
• Allow TPRs to register short names that will be discoverable by kubectl.
• Allow TPRs to optionally be cluster-scoped rather than namespaced.
• Define and document a process to migrate from extensions/v1beta1 TPR, possibly requiring brief downtime for TPR custom controllers and operators.
  • Where possible, provide automated tools to help with migration.
• A finalizer ensures CR data is deleted if a CRD is deleted.
• Fix TPR/CRD data cleanup upon namespace deletion for the 3rd time, this time with a regression test.

Other plans not in scope for this release

• Support multiple versions at the same time for a given TPR.
  • Other components (e.g. GC, namespace finalizers) expect automatic conversion. TPR currently does not support that.
  • Note that it's possible to change the single registered version of a TPR, but it requires brief downtime for TPR custom controllers and operators.
  • The extensions/v1beta1 TPR gives the appearance of supporting multiple versions, but multiple version support was never implemented.
• Support customizing where TPR APIs appear in discovery, relative to other TPRs or other APIs.
• Support namespace-scoped CRD whose CRs are only visible in one namespace.

Plans with unclear status

Still investigating or TBD. Please comment/edit with any updates.

• Improve the display of TPRs in kubectl/dashboard.
  • There may be other feature trackers addressing this.

[deads2k]

@lavalamp I've created this to try to have a place where we can at least consolidate our thoughts and track progress on third party resources. I've tried to create a list of known shortcomings to be resolved before promotion to stable.

I don't have an owner in mind, but recognition of the problem seems like step 1.

[adohe-zz]

@deads2k I am learning third party resource recently, also wish to help with something.

[deads2k]

@deads2k I am learning third party resource recently, also wish to help with something.

I've re-ordered the list in terms of what I see as tactical priority. People are trying to use this now and these problems will burn them badly.

If you're comfortable taking the "multiple resources" item, that would be a great start. You could create a separate issue and we can talk about implementation in there.

[adohe-zz]

@deads2k I spent some time trying to reproduce the first issue:

Multiple Resources, single version, different add times - Adding resource A, waiting for it to appear, then adding resource B fails. Resource B is never added.

but with unluck. Below is my reproduce steps:

1. create a custom thirdparty resource&wait it to appear

[root@localhost kubernetes]# cat /home/tony/Desktop/debug/lbclaim.yaml
kind: ThirdPartyResource
apiVersion: extensions/v1beta1
metadata:
  name: loadbalancerclaim.k8s.io
description: "Allow user to claim a loadbalancer instance"
versions:
- name: v1
[root@localhost kubernetes]# kc create -f /home/tony/Desktop/debug/lbclaim.yaml
thirdpartyresource "loadbalancerclaim.k8s.io" created
[root@localhost kubernetes]# curl  http://localhost:8080/apis/extensions/v1beta1/thirdpartyresources/
{
  "kind": "ThirdPartyResourceList",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "selfLink": "/apis/extensions/v1beta1/thirdpartyresources/",
    "resourceVersion": "170"
  },
  "items": [
    {
      "metadata": {
        "name": "loadbalancerclaim.k8s.io",
        "selfLink": "/apis/extensions/v1beta1/thirdpartyresources/loadbalancerclaim.k8s.io",
        "uid": "dcb88b3a-9857-11e6-a19b-08002767e1f5",
        "resourceVersion": "146",
        "creationTimestamp": "2016-10-22T13:03:01Z"
      },
      "description": "Allow user to claim a loadbalancer instance",
      "versions": [
        {
          "name": "v1"
        }
      ]
    }
  ]
}

1. after a moment(more than 10s), create another custom thirdparty resource

[root@localhost kubernetes]# cat /home/tony/Desktop/debug/loadbalancer.yaml
kind: ThirdPartyResource
apiVersion: extensions/v1beta1
metadata:
  name: loadbalancer.k8s.io
description: "Allow user to curd a loadbalancer instance"
versions:
- name: v1
[root@localhost kubernetes]# kc create -f /home/tony/Desktop/debug/loadbalancer.yaml
thirdpartyresource "loadbalancer.k8s.io" created

1. verify both resources exist

[root@localhost kubernetes]# curl http://localhost:8080/apis/k8s.io/v1/
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "k8s.io/v1",
  "resources": [
    {
      "name": "loadbalancerclaims",
      "namespaced": true,
      "kind": "Loadbalancerclaim"
    },
    {
      "name": "loadbalancers",
      "namespaced": true,
      "kind": "Loadbalancer"
    }
  ]
}
[root@localhost kubernetes]# kc get loadbalancers
No resources found.
[root@localhost kubernetes]# kc get loadbalancerclaims
No resources found.

seems we already support multiple resources, single version.

[adohe-zz]

And I take a deep look at TPR related code. The thirdparty_controller will do periodically sync(every 10 seconds), it will install every new TPR, and also do some deletion job. The ThirdPartyResourceServer contains all installed TPR mappings. As we can see from SyncOneResource and InstallThirdPartyResource, even this this group exists, it will still update the group with the new API.

[adohe-zz]

Also I found that I am able to delete a TPR schema def even there are TPR instances in the system. I think this should not be allowed.

[deads2k]

@deads2k I spent some time trying to reproduce the first issue:

Try to enable this test: https://github.com/kubernetes/kubernetes/blob/master/test/integration/thirdparty/thirdparty_test.go#L137 . If it works, we're good. If it fails, something is wrong.

[adohe-zz]

@deads2k Hi David, please take a look at the message I sent on Slack. Besides, I add a fix to the failed integration test, the third party resource controller will remove the corresponding routes handler when a TPR get deleted, this will help with the integration test, but I am not sure whether this will bring in any other problems.

[brendandburns]

For problem #1, it was fixed here:

kubernetes/kubernetes#28414

[adohe-zz]

@brendandburns actually not, you can run the comment out integration test, and it will fail.

[adohe-zz]

@brendandburns More correctly, we did support multiple resources, single version, but the deletion logical has some problem.

[brendandburns]

@adohe did you file an issue? I can take a look.

[adohe-zz]

@brendandburns you can see here:

https://github.com/kubernetes/kubernetes/blob/master/test/integration/thirdparty/thirdparty_test.go#L137 

enable this test, and you will see it will fail. I have tried to fix this on my local, and I will open a PR later today.

[adohe-zz]

@brendandburns I am afraid I don't file an issue.

[nikhiljindal]

Also ref kubernetes/kubernetes#32306 (TPR should be deleted when namespace is deleted)

[grodrigues3]

@deads2k can you update the checklist ?

[deads2k]

@deads2k can you update the checklist ?

All issues still outstanding. This is actually a feature to track the problems in the (already) beta thirdparyresources implementation from 1.3. We needed a place to keep track of our problems, but had to devote energy to other efforts in 1.5.

[adohe-zz]

@deads2k I am already working on Multiple Resources, single version and Multiple versions, I think a lot of code need to be update.

[idvoretskyi]

@deads2k does still feature still target 1.5?

[kacole2]

Hello @nikhita @deads2k , I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. A KEP will need to be merged for 1.15 inclusion as well. Thanks!

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

[liggitt]

this will remain in beta stage. work on validation, conversion, and OpenAPI publishing is happening in 1.15

[liggitt]

updated description with links to relevant KEPs for 1.15

[daminisatya]

Hey, @liggitt @deads2k @jpbetz @sttts I'm the v1.15 docs release shadow.

Does this enhancement (or the work planned for v1.15) require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.15) due by Thursday, May 30th. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions! 😄

[daminisatya]

@deads2k @jpbetz @sttts @liggitt

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.15) due by Thursday, May 30th. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions! 😄

[sttts]

Docs PR for 1.15: kubernetes/website#14583

@deads2k can you update the issue description?

[liggitt]

/milestone v1.16
/stage stable

[simplytunde]

Hey, @liggitt @jpbetz @sttts I'm the v1.16 docs release lead.

Does this enhancement (or the work planned for v1.16) require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.16) due by Friday,August 23rd. Let me know if you have any questions!

[sttts]

@simonswine the placeholder PR kubernetes/website#15982

[kacole2]

@liggitt @jpbetz @sttts Thursday is code freeze. Are there any outstanding k/k PRs that will prevent this from graduating to Stable? Everything in the original post for planned 1.15* work looks to be merged.

[jpbetz]

I believe the outstanding PRs are just the feature gate version bump (kubernetes/kubernetes#81965) and two outstanding bug fixes that should go in this week: kubernetes/kubernetes#81436, kubernetes/kubernetes#78707

[liggitt]

docs ready for review in kubernetes/website#15982

[liggitt]

Released as stable in v1.16.0

Post-GA work tracked in https://github.com/orgs/kubernetes/projects/28

/close

[k8s-ci-robot]

@liggitt: Closing this issue.

In response to this:

Released as stable in v1.16.0

Post-GA work tracked in https://github.com/orgs/kubernetes/projects/28

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.