Enabling CRDs themselves for federation does not work

[vasanth89]

What happened:
I tried doing a kubefedctl enable CustomResourceDefinition to enable federation of CustomResourceDefinitions (which are also resources similar to pods/deployments/etc). This caused a stack overflow error. Including part of the log from my setup -

I0318 00:03:40.800471    2738 enable.go:202] Found type "customresourcedefinitions.apiextensions.k8s.io/v1"
runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow
runtime stack:
runtime.throw(0x140967f, 0xe)
    /usr/local/go/src/runtime/panic.go:617 +0x72
runtime.newstack()
    /usr/local/go/src/runtime/stack.go:1041 +0x6f0
runtime.morestack()
    /usr/local/go/src/runtime/asm_amd64.s:429 +0x8f
goroutine 1 [running]:
sigs.k8s.io/kubefed/pkg/kubefedctl/enable.(*jsonSchemaVistor).VisitKind(0xc013dc2ec0, 0xc00122e150)
    /go/src/sigs.k8s.io/kubefed/pkg/kubefedctl/enable/schema.go:167 +0x38b fp=0xc0240004a0 sp=0xc024000498 pc=0x1136b6b
sigs.k8s.io/kubefed/vendor/k8s.io/kube-openapi/pkg/util/proto.(*Kind).Accept(0xc00122e150, 0x160cda0, 0xc013dc2ec0)

What you expected to happen:
kubefedctl enable CustomResourceDefinition should succeed and we should see a new CRD called FederatedCustomResourceDefinition in the API Server along with a FederatedTypeConfig entry to map FederatedCustomResourceDefinition to CustomResourceDefinition (the target type)

How to reproduce it (as minimally and precisely as possible):
After deploying KubeFed using Helm, attempt to run kubefedctl enable CustomResourceDefinition

Anything else we need to know?:
The KubeFed User Guide talks about the ability to enable federation of CRDs and mentions the following note -

NOTE: Federation of a CRD requires that the CRD be installed on all member clusters. If the CRD is not installed on a member cluster, propagation to that cluster will fail.

I think this talks about enabling a specific CRD for federation - e.g. for a CRD called X, we can run kubefedctl enable X and that will yield FederatedX

I attempted to use KubeFed to address the requirement of installing CRDs on member clusters before propagating custom resources of a specific CRD

Environment:

• Kubernetes version (use kubectl version)

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

• KubeFed version
  0.1.0-rc6

• Scope of installation (namespaced or cluster)
  Namespaced

/kind bug

[hectorj2f]

@vasanth89 you're right, this is failing. I workaround for now until I have more information would be to create yourself the Federated equivalent. We are doing that using the CustomResourceDefinition and creating one for FederatedCustomResourceDefinition so we can federate CRDs.

[hectorj2f]

@vasanth89 I found the bug, I am pushing a PR

[vasanth89]

@hectorj2f Thanks for getting this done! We might need to update the documentation in kubefed user-guide (see the description of this issue). We can add a note to say that kubefed can be used to federate the CRD to the member clusters before federating resources of that CRD type.

[hectorj2f]

@vasanth89 sure, I'd do it. Could you also test #1207 changes to validate this statement on your own environment ?

[vasanth89]

I am not sure how to go about doing that since I have always used the canary image of KubeFed. Any suggestions? I guess I might have to follow the Development Guide and build my own version of KubeFed with your changes

[vasanth89]

Or if you have the image hosted somewhere with a specific tag, I can just point my deployment to pull that image

[hectorj2f]

@vasanth89 it is just the kubefectl binary generated by make build which I used to validate these changes. I hope I clarified the changes here #1207, can you review it before I ping one of the owners :) ?

[vasanth89]

Oh shucks. Sorry I didnt realize that its the kubefedctl binary. I will try it and get back to you by today.

[vasanth89]

• I pulled your changes and did a make build
• I was able to do kubefedctl enable customresourcedefinitions which resulted in a FederatedCustomResourceDefinition CRD. I noticed the following logs from KubeFed

I0403 22:28:02.437869       1 controller.go:100] Starting sync controller for "FederatedCustomResourceDefinition"
I0403 22:28:02.437978       1 controller.go:330] Started sync controller for "FederatedCustomResourceDefinition"
I0403 22:28:02.492126       1 controller.go:362] refreshing sync controller for "customresourcedefinitions.apiextensions.k8s.io"
I0403 22:28:02.492154       1 controller.go:354] Stopping controller for "customresourcedefinitions.apiextensions.k8s.io"
I0403 22:28:02.495458       1 federated_informer.go:212] Cluster kube-federation-system/mem-cluster-2 not added; it is not ready.
I0403 22:28:02.827793       1 federated_informer.go:207] Cluster kube-federation-system/mem-cluster-1 is ready
I0403 22:28:03.030466       1 controller.go:100] Starting sync controller for "FederatedCustomResourceDefinition"
I0403 22:28:03.030511       1 controller.go:330] Started sync controller for "FederatedCustomResourceDefinition"
E0403 22:28:03.127835       1 controller.go:264] Could not update status fields of the CRD: "kube-federation-system/customresourcedefinitions.apiextensions.k8s.io": Operation cannot be fulfilled on federatedtypeconfigs.core.kubefed.io "customresourcedefinitions.apiextensions.k8s.io": the object has been modified; please apply your changes to the latest version and try again

• I tried to federate the dnsendpoints.multiclusterdns.kubefed.io CRD but I did not see it propagate to the member cluster. Here are some more logs -

kubectl describe federatedcustomresourcedefinitions.types.kubefed.io

Name:         dnsendpoints.multiclusterdns.kubefed.io
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  types.kubefed.io/v1beta1
Kind:         FederatedCustomResourceDefinition
Metadata:
  Creation Timestamp:  2020-04-03T23:18:44Z
  Finalizers:
    kubefed.io/sync-controller
  Generation:        1
  Resource Version:  228748
  Self Link:         /apis/types.kubefed.io/v1beta1/federatedcustomresourcedefinitions/dnsendpoints.multiclusterdns.kubefed.io
  UID:               673ce7dd-b6b0-4b13-ba22-c756f2213895
Spec:
  Placement:
    Cluster Selector:
      Match Labels:
  Template:
    Spec:
      Conversion:
        Strategy:  None
      Group:       multiclusterdns.kubefed.io
      Names:
        Kind:                   DNSEndpoint
        List Kind:              DNSEndpointList
        Plural:                 dnsendpoints
        Singular:               dnsendpoint
      Preserve Unknown Fields:  true
      Scope:                    Namespaced
      Versions:
        Name:  v1alpha1
        Schema:
          openAPIV3Schema:
            Description:  DNSEndpoint is the CRD wrapper for Endpoint which is designed to act as a source of truth for external-dns.
            Properties:
              API Version:
                Description:  APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
                Type:         string
              Kind:
                Description:  Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
                Type:         string
              Metadata:
                Type:  object
              Spec:
                Description:  DNSEndpointSpec defines the desired state of DNSEndpoint
                Properties:
                  Endpoints:
                    Items:
                      Description:  Endpoint is a high-level association between a service and an IP.
                      Properties:
                        Dns Name:
                          Description:  The FQDN of the DNS record.
                          Type:         string
                        Labels:
                          Additional Properties:
                            Type:       string
                          Description:  Labels stores labels defined for the Endpoint.
                          Type:         object
                        Record TTL:
                          Description:  TTL for the record in seconds.
                          Format:       int64
                          Type:         integer
                        Record Type:
                          Description:  RecordType type of record, e.g. CNAME, A, SRV, TXT etc.
                          Type:         string
                        Targets:
                          Description:  The targets that the DNS record points to.
                          Items:
                            Type:  string
                          Type:    array
                      Type:        object
                    Type:          array
                Type:              object
              Status:
                Description:  DNSEndpointStatus defines the observed state of DNSEndpoint
                Properties:
                  Observed Generation:
                    Description:  ObservedGeneration is the generation as observed by the controller consuming the DNSEndpoint.
                    Format:       int64
                    Type:         integer
                Type:             object
            Type:                 object
        Served:                   true
        Storage:                  true
        Subresources:
          Status:
Status:
  Clusters:
    Name:    mem-cluster-1
    Status:  CreationFailed
    Name:    mem-cluster-2
    Status:  ClusterNotReady
  Conditions:
    Last Transition Time:  2020-04-03T23:18:45Z
    Last Update Time:      2020-04-03T23:18:45Z
    Reason:                CheckClusters
    Status:                False
    Type:                  Propagation
  Observed Generation:     1
Events:
  Type     Reason                 Age                    From                                          Message
  ----     ------                 ----                   ----                                          -------
  Warning  ClusterNotReady        2m17s (x3 over 2m18s)  federatedcustomresourcedefinition-controller  Cluster not ready
  Normal   CreateInCluster        2m17s (x3 over 2m17s)  federatedcustomresourcedefinition-controller  Creating CustomResourceDefinition "dnsendpoints.multiclusterdns.kubefed.io" in cluster "mem-cluster-1"
  Warning  CreateInClusterFailed  2m16s (x3 over 2m17s)  federatedcustomresourcedefinition-controller  Failed to create CustomResourceDefinition "dnsendpoints.multiclusterdns.kubefed.io" in cluster "mem-cluster-1": CustomResourceDefinition.apiextensions.k8s.io "dnsendpoints.multiclusterdns.kubefed.io" is invalid: spec.preserveUnknownFields: Invalid value: true: cannot set to true, set x-preserve-unknown-fields to true in spec.versions[*].schema instead

P.S: Ignore the errors related to mem-cluster-2

[hectorj2f]

federated crd needs to have preserveUnknownFields: true while normal CRD has it false. I can see my federated crd set to true while the crds are false. I don't see any error due to that. I'll check again.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.