-
Notifications
You must be signed in to change notification settings - Fork 407
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
2 changed files
with
259 additions
and
0 deletions.
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
259 changes: 259 additions & 0 deletions
259
docs/proposals/20210301-enhancement_of_YurtHub_caching_ability.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,259 @@ | ||
| --- | ||
| title: Proposal to enhance the caching ability of YurtHub | ||
| authors: | ||
| - "@qclc" | ||
| - "@GsssC" | ||
| reviewers: | ||
| - "@rambohe-ch" | ||
| creation-date: 2021-03-01 | ||
| last-updated: 2021-03-01 | ||
| status: provisional | ||
| --- | ||
|
|
||
| # Proposal to enhance the caching ability of YurtHub | ||
|
|
||
| ## Table of Contents | ||
|
|
||
| - [Title](#title) | ||
| - [Table of Contents](#table-of-contents) | ||
| - [Summary](#summary) | ||
| - [Motivation](#motivation) | ||
| - [Goals](#goals) | ||
| - [Description of Yurthub's existing cache ability](#Description of Yurthub's existing cache ability) | ||
| - [serializerManager](#serializerManager) | ||
| - [Problem](#Problem) | ||
| - [cacheManager](#cacheManager) | ||
| - [Problem](#Problem) | ||
| - [Proposal](#proposal) | ||
| - [Delete resourceToKindMap and resourceToListKindMap](#Delete resourceToKindMap and resourceToListKindMap) | ||
| - [Design detail](#Design detail) | ||
| - [Decode the unregistered Custom Resources into Unstructured structure](#Decode the unregistered Custom Resources into Unstructured structure) | ||
| - [Design detail](#Design detail) | ||
| - [Add unstructuredNegotiatedSerializer](#Add unstructuredNegotiatedSerializer) | ||
| - [Determine the resource type based on request.URL and select the appropriate serializer](#Determine the resource type based on request.URL and select the appropriate serializer) | ||
| - [Implementation History](#implementation-history) | ||
|
|
||
| ## Summary | ||
|
|
||
| YurtHub can currently only cache resources defined in [resourceToKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L46) and [resourceToListKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L62). This proposal mainly tries to put forward and realize a set of scheme, so that YurtHub is no longer limited by resource types, and Custom Resources can be cached. | ||
|
|
||
| ## Motivation | ||
|
|
||
| when network between cloud and edge disconnected, if any pod(eg: calico) on the edge node that used some resources(like crd) not in the above map want to run continuously, that is to say, the pod can not restarted successfully because resources(like crd) are not cached by yurt-hub. | ||
|
|
||
| ### Goals | ||
|
|
||
| - Delete [resourceToKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L46) and [resourceToListKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L62) so that we can remove the restriction that yurthub can only cache certain resources; | ||
| - Decode Custom Resources that are not registered in the scheme into `Unstructured` structures; | ||
| - The Custom Resource is distinguished from the core resource based on `request.URL`. The core resource uses the existing serializer and the Custom Resource uses the `unstructuredNegotiatedSerializer`. | ||
|
|
||
| ## Description of Yurthub's existing cache ability | ||
|
|
||
| This section briefly describes how Yurthub receives and caches response with resources from Kube-apiserver, mainly involving the CacheManager and SerializerManager modules. Yurthub caches resources, which roughly requires two steps: | ||
|
|
||
| **Step 1:** Decode the byte stream contained in `Response.Body` into `Runtime.Object` type in memory, so that YurtHub can operate the Object; | ||
|
|
||
| **Step 2:** Encode the Runtime.Object type into the corresponding byte stream and save it to the hard disk. | ||
|
|
||
| The limitations of the existing caching ability in YurtHub are mainly in the first step: One is the limitation of [resourceToKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L46) and [resourceToListKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L62); The second is the lack of a serializer to decode and encode Custom Resources. | ||
|
|
||
| ### serializerManager | ||
|
|
||
| SerializerManager is used to manage the serializers that decode and encode K8S resources. When SerializerManager is initialized, the structure of the built-in resources in the Client-Go library is registered in the scheme, such as the Pod structure example: | ||
|
|
||
| ```go | ||
| type Pod struct { | ||
| metav1.TypeMeta `json:",inline"` | ||
| metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"` | ||
| Spec PodSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"` | ||
| Status PodStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"` | ||
| } | ||
| ``` | ||
|
|
||
| Currently Yurthub can only encode and decode resources under [these paths](https://github.com/kubernetes/client-go/blob/master/kubernetes/scheme/register.go): | ||
|
|
||
| ```go | ||
| admissionregistrationv1 "k8s.io/api/admissionregistration/v1" | ||
| admissionregistrationv1beta1 "k8s.io/api/admissionregistration/v1beta1" | ||
| appsv1 "k8s.io/api/apps/v1" | ||
| appsv1beta1 "k8s.io/api/apps/v1beta1" | ||
| appsv1beta2 "k8s.io/api/apps/v1beta2" | ||
| auditregistrationv1alpha1 "k8s.io/api/auditregistration/v1alpha1" | ||
| authenticationv1 "k8s.io/api/authentication/v1" | ||
| authenticationv1beta1 "k8s.io/api/authentication/v1beta1" | ||
| authorizationv1 "k8s.io/api/authorization/v1" | ||
| authorizationv1beta1 "k8s.io/api/authorization/v1beta1" | ||
| autoscalingv1 "k8s.io/api/autoscaling/v1" | ||
| autoscalingv2beta1 "k8s.io/api/autoscaling/v2beta1" | ||
| autoscalingv2beta2 "k8s.io/api/autoscaling/v2beta2" | ||
| batchv1 "k8s.io/api/batch/v1" | ||
| batchv1beta1 "k8s.io/api/batch/v1beta1" | ||
| batchv2alpha1 "k8s.io/api/batch/v2alpha1" | ||
| certificatesv1beta1 "k8s.io/api/certificates/v1beta1" | ||
| coordinationv1 "k8s.io/api/coordination/v1" | ||
| coordinationv1beta1 "k8s.io/api/coordination/v1beta1" | ||
| corev1 "k8s.io/api/core/v1" | ||
| discoveryv1alpha1 "k8s.io/api/discovery/v1alpha1" | ||
| eventsv1beta1 "k8s.io/api/events/v1beta1" | ||
| extensionsv1beta1 "k8s.io/api/extensions/v1beta1" | ||
| networkingv1 "k8s.io/api/networking/v1" | ||
| networkingv1beta1 "k8s.io/api/networking/v1beta1" | ||
| nodev1alpha1 "k8s.io/api/node/v1alpha1" | ||
| nodev1beta1 "k8s.io/api/node/v1beta1" | ||
| policyv1beta1 "k8s.io/api/policy/v1beta1" | ||
| rbacv1 "k8s.io/api/rbac/v1" | ||
| rbacv1alpha1 "k8s.io/api/rbac/v1alpha1" | ||
| rbacv1beta1 "k8s.io/api/rbac/v1beta1" | ||
| schedulingv1 "k8s.io/api/scheduling/v1" | ||
| schedulingv1alpha1 "k8s.io/api/scheduling/v1alpha1" | ||
| schedulingv1beta1 "k8s.io/api/scheduling/v1beta1" | ||
| settingsv1alpha1 "k8s.io/api/settings/v1alpha1" | ||
| storagev1 "k8s.io/api/storage/v1" | ||
| storagev1alpha1 "k8s.io/api/storage/v1alpha1" | ||
| storagev1beta1 "k8s.io/api/storage/v1beta1" | ||
| v1 "k8s.io/apimachinery/pkg/apis/meta/v1" | ||
| runtime "k8s.io/apimachinery/pkg/runtime" | ||
| schema "k8s.io/apimachinery/pkg/runtime/schema" | ||
| serializer "k8s.io/apimachinery/pkg/runtime/serializer" | ||
| utilruntime "k8s.io/apimachinery/pkg/util/runtime" | ||
| ``` | ||
|
|
||
| #### Problem | ||
|
|
||
| Because scheme currently only saves the above fixed resources' structures, when YurtHub receives Custom Resources returned from Kube-apiserver, it cannot find the corresponding GO type structure to decode Custom Resources. | ||
|
|
||
| ### cacheManager | ||
|
|
||
| CacheManager provides methods for managing resources cached on edge nodes, which are divided into three main functions: | ||
|
|
||
| - Determining whether the response message needs to be cached | ||
| - The resource byte stream from Kube-apiserver is decoded into `runtime.Object` and stored in memory. When the YurtHub operation is completed, the object is converted to a byte stream and stored on the hard disk | ||
|
|
||
| - When cloud-edge communication is unhealthy, edge nodes use `QueryCache(...)` to find resources cached locally. | ||
|
|
||
| ```go | ||
| // CacheManager is an adaptor to cache runtime object data into backend storage | ||
| type CacheManager interface { | ||
| CacheResponse(ctx context.Context, prc io.ReadCloser, stopCh <-chan struct{}) error | ||
| QueryCache(req *http.Request) (runtime.Object, error) | ||
| UpdateCacheAgents(agents []string) error | ||
| ListCacheAgents() []string | ||
| CanCacheFor(req *http.Request) bool | ||
| } | ||
| ``` | ||
|
|
||
| #### Problem | ||
|
|
||
| Since the serializer currently used by YurtHub will lose GVK information when decoding resource objects. So we need to [resourceToKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L46) and [resourceToListKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L62) to obtain the corresponding kind information according to resource name. The two maps have the following three main functions: | ||
|
|
||
| - `CanCacheFor(req *http.Request)` function uses a resourceToKindMap to determine if the response message needs to be cached; | ||
|
|
||
| - `SaveOneObject(...)`, `SaveListObject(...)` and `SaveWatchObject(...)` use resourceToKindMap to obtain the kind information corresponding to the resource name, which is used to cache the response message; | ||
|
|
||
| Because the resolved object does not contain GVK information. But when caching objects, GVK information needs to be set. The above two maps pre-store some resource-to-kind mappings, which can be used to cache resources and retrieve the corresponding kind based on the resource name in the request.URL . | ||
|
|
||
| - The `queryListObject(...)` function uses the resourceToListKindMap member to query the List object | ||
|
|
||
| Due to the above limitations, Yurthub can only cache resources defined in [resourceToKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L46) and [resourceToListKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L62). | ||
|
|
||
| ## Proposal | ||
|
|
||
| ### Delete [resourceToKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L46) and [resourceToListKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L62) | ||
|
|
||
| Remove the restriction that only certain resources can be cached by removing [resourceToKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L46) and [resourceToListKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L62). Removing the above restriction requires solving the problem which is preserving the original GVK information while decoding the corresponding resources; | ||
|
|
||
| #### Why the WithoutConversionCodecFactory loses GVK information | ||
|
|
||
| The reason why object misses the field of apiverison and kind is YurtHub using [serializer.WithoutConversionCodecFactory](https://github.com/alibaba/openyurt/blob/7095962687e2666a27d14e436b96ef893b228beb/pkg/yurthub/kubernetes/serializer/serializer.go#L48) as NegotiatedSerializer. Because the WithoutConversionCodecFactory is a CodecFactory that will explicitly ignore requests to perform conversion, the GVK information will be ignored when this WithoutConversionCodecFactory use the decoder (which is this [Decoder](https://github.com/kubernetes/kubernetes/blob/994b5c6cc20bda188ac9bc33217f2dbc7ecf45bb/staging/src/k8s.io/apimachinery/pkg/runtime/helper.go#L256), the following is the old decoder's code). | ||
|
|
||
| ```go | ||
| // Decode does not do conversion. It removes the gvk during deserialization. | ||
| func (d WithoutVersionDecoder) Decode(data []byte, defaults *schema.GroupVersionKind, into Object) (Object, *schema.GroupVersionKind, error) { | ||
| obj, gvk, err := d.Decoder.Decode(data, defaults, into) | ||
| if obj != nil { | ||
| kind := obj.GetObjectKind() | ||
| // clearing the gvk is just a convention of a codec | ||
| kind.SetGroupVersionKind(schema.GroupVersionKind{}) | ||
| } | ||
| return obj, gvk, err | ||
| } | ||
| ``` | ||
|
|
||
| #### Design detail | ||
|
|
||
| Define a new `WithVersionCodecFactory` replace the [serializer.WithoutConversionCodecFactory](https://github.com/alibaba/openyurt/blob/7095962687e2666a27d14e436b96ef893b228beb/pkg/yurthub/kubernetes/serializer/serializer.go#L48) as a negotiatedSerializer, which can retain GVK information while decoding resources. Here is part of the WithVersionCodecFactory code | ||
|
|
||
| ```go | ||
| // WithVersionCodecFactory is a CodecFactory that will explicitly ignore requests to perform conversion. | ||
| // It keeps the gvk during deserialization. | ||
| // This wrapper is used while code migrates away from using conversion (such as external clients) | ||
| type WithVersionCodecFactory struct { | ||
| serializer.CodecFactory | ||
| } | ||
|
|
||
| // DecoderToVersion returns an decoder that does not do conversion, and keeps the gvk information | ||
| func (f WithVersionCodecFactory) DecoderToVersion(serializer runtime.Decoder, _ runtime.GroupVersioner) runtime.Decoder { | ||
| //return versioning.NewDefaultingCodecForScheme(s.scheme, nil, decoder, nil, gv) | ||
| return WithVersionDecoder{ | ||
| Decoder: serializer, | ||
| } | ||
| } | ||
|
|
||
| // WithVersionDecoder keeps the group version kind of a deserialized object. | ||
| type WithVersionDecoder struct { | ||
| runtime.Decoder | ||
| } | ||
|
|
||
| // Decode does not do conversion. It keeps the gvk during deserialization. | ||
| func (d WithVersionDecoder) Decode(data []byte, defaults *schema.GroupVersionKind, into runtime.Object) (runtime.Object, *schema.GroupVersionKind, error) { | ||
| obj, gvk, err := d.Decoder.Decode(data, defaults, into) | ||
| return obj, gvk, err | ||
| } | ||
| ``` | ||
|
|
||
| ### Decode the unregistered Custom Resources into `Unstructured` structure | ||
|
|
||
| Unstructured allows objects that do not have Golang structs registered to be manipulated generically. This can be used to deal with the API objects from a plug-in. | ||
|
|
||
| ```go | ||
| //Unstructured objects still have functioning TypeMeta features-- kind, version, etc. | ||
| type Unstructured struct { | ||
| // Object is a JSON compatible map with string, float, int, bool, []interface{}, or | ||
| // map[string]interface{} | ||
| // children. | ||
| Object map[string]interface{} | ||
| } | ||
| ``` | ||
|
|
||
| #### Design detail | ||
|
|
||
| ##### Add unstructuredNegotiatedSerializer | ||
| On the basis of the existing NegotiatedSerializer increase a new unstructuredNegotiatedSerializer, used to decode the structure of the unregistered Custom Resources. | ||
|
|
||
| ```go | ||
| type unstructuredNegotiatedSerializer struct { | ||
| scheme *runtime.Scheme | ||
| typer runtime.ObjectTyper | ||
| creator runtime.ObjectCreater | ||
| } | ||
|
|
||
| // SerializerManager is responsible for managing *rest.Serializers | ||
| type SerializerManager struct { | ||
| // NegotiatedSerializer is used for obtaining encoders and decoders for multiple | ||
| // supported media types. | ||
| NegotiatedSerializer runtime.NegotiatedSerializer | ||
| //TO-ADD: use to decode the CR | ||
| unstructuredNegotiatedSerializer runtime.NegotiatedSerializer | ||
| } | ||
| ``` | ||
| ##### Determine the resource type based on request.URL and select the appropriate serializer | ||
|
|
||
| When cache resources, determine resource's type according to the `request.URL`, if the resource type has been registered, use the original serializer, otherwise use the serializer returned by `unstructuredNegotiatedSerializer` . | ||
|
|
||
|  | ||
|
|
||
| ## Implementation History | ||
|
|
||
| - [ ] 03/01/2021: Proposed idea. | ||
| - [ ] 03/01/2021: Commit the PR about delete [resourceToKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L46) 和 [resourceToListKindMap](https://github.com/openyurtio/openyurt/blob/4d7463a40801c29d09c4f7d10ba46b73cb019915/pkg/yurthub/cachemanager/cache_manager.go#L62) (https://github.com/openyurtio/openyurt/pull/225) | ||
|
|