fix: resourceregistry is incompatible in the upgrade scenario

[zhzhuang-zju]

What type of PR is this?

/kind cleanup

What this PR does / why we need it:
Referring to #4144, there is a problem in the upgrade scenario of resource resourceregistry. We need to solve it, otherwise the relevant data will be lost after the upgrade.

Which issue(s) this PR fixes:
Refer to #4141

Special notes for your reviewer:
none
Does this PR introduce a user-facing change?:

`karmada-search`: Fixed the ResourceRegistry is incompatible in the upgrade scenario.

[codecov-commenter]

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (fc006ce) 52.93% compared to head (23187f6) 52.93%.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4171   +/-   ##
=======================================
  Coverage   52.93%   52.93%           
=======================================
  Files         239      239           
  Lines       23522    23522           
=======================================
  Hits        12452    12452           
  Misses      10395    10395           
  Partials      675      675           

Flag	Coverage Δ
unittests	52.93% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
pkg/registry/search/storage/resourceregistry.go	0.00% <0.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zhzhuang-zju]

the root cause is:

karmada/vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go

Lines 1408 to 1420 in fc006ce

	opts, err := options.RESTOptions.GetRESTOptions(e.DefaultQualifiedResource)
	if err != nil {
	return err
	}
	// ResourcePrefix must come from the underlying factory
	prefix := opts.ResourcePrefix
	if !strings.HasPrefix(prefix, "/") {
	prefix = "/" + prefix
	}
	if prefix == "/" {
	return fmt.Errorf("store for %s has an invalid prefix %q", e.DefaultQualifiedResource.String(), opts.ResourcePrefix)
	}

Field DefaultQualifiedResource is used to construct storage paths to resource directories. So, changing it will cause the storage path to change, and eventually cause upgrate problem. I did some experiments, and the results were consistent with the code analysis.

1. use version release-1.7.0 to build the karmada environment.
2. Create ResourceRegistry with command kubectl --kubeconfig $HOME/.kube/karmada.config --context karmada-apiserver create -f deployment-search.yaml

### deployment-search.yaml
apiVersion: search.karmada.io/v1alpha1
kind: ResourceRegistry
metadata:
  name: deployment-search
spec:
  targetCluster:
    clusterNames:
      - member1
      - member2
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment

3. check the resource path in etcd.

sh-5.1#                                                                                                                                                                                sh-5.1# etcdctl --cert="/etc/karmada/pki/etcd-client.crt" --key="/etc/karmada/pki/etcd-client.key" --cacert="/etc/karmada/pki/etcd-ca.crt" get /registry --prefix --keys-only
/registry/search.karmada.io/resourceRegistries/deployment-search

4. Upgrade the component karmada-search with the latest image.
5. do the step 2 and 3 again.

sh-5.1#                                                                                                                                                                                sh-5.1# etcdctl --cert="/etc/karmada/pki/etcd-client.crt" --key="/etc/karmada/pki/etcd-client.key" --cacert="/etc/karmada/pki/etcd-ca.crt" get /registry --prefix --keys-only
/registry/search.karmada.io/resourceRegistries/deployment-search
/registry/search.karmada.io/resourceregistries/deployment-search

so, field DefaultQualifiedResource should not change.
Besides, run the command kubectl --kubeconfig $HOME/.kube/karmada.config --context karmada-apiserver api-resources| grep resourceregistries before and after the upgrade. The command output is as follows:

➜  karmada git:(1.7) ✗ ka api-resources|grep resourceregistries       
resourceregistries                                      search.karmada.io/v1alpha1             false        ResourceRegistry

The plural name of the resource has always been resourceregistries and is not affected by filed DefaultQualifiedResource, so I start looking for plural name of the resource.

karmada/vendor/k8s.io/client-go/restmapper/discovery.go

Lines 107 to 112 in 4e7084c

	plural := gv.WithResource(resource.Name)
	singular := gv.WithResource(resource.SingularName)
	// this is for legacy resources and servers which don't list singular forms. For those we must still guess.
	if len(resource.SingularName) == 0 {
	_, singular = meta.UnsafeGuessKindToResource(gv.WithKind(resource.Kind))
	}

Tracing the code, I found the source of the value of pluralName. The key of the map is its pluralName. That's why the plural name of the resource has always been resourceregistries.

karmada/pkg/search/apiserver.go

Lines 78 to 79 in 4e7084c

	v1alpha1search := map[string]rest.Storage{}
	v1alpha1search["resourceregistries"] = resourceRegistryStorage.ResourceRegistry

As you can see from the following code, pluralNames are forced to be lowercase only.

karmada/vendor/k8s.io/apiserver/pkg/server/genericapiserver.go

Lines 904 to 911 in 4e7084c

	func (s *GenericAPIServer) getAPIGroupVersion(apiGroupInfo *APIGroupInfo, groupVersion schema.GroupVersion, apiPrefix string) (*genericapi.APIGroupVersion, error) {
	storage := make(map[string]rest.Storage)
	for k, v := range apiGroupInfo.VersionedResourcesStorageMap[groupVersion.Version] {
	if strings.ToLower(k) != k {
	return nil, fmt.Errorf("resource names must be lowercase only, not %q", k)
	}
	storage[k] = v
	}

[zhzhuang-zju]

Here is my test report.

1. use version release-1.5.0 to build the karmada environment.
2. Create ResourceRegistry with command kubectl --kubeconfig $HOME/.kube/karmada.config --context karmada-apiserver create -f deployment-search.yaml

### deployment-search.yaml
apiVersion: search.karmada.io/v1alpha1
kind: ResourceRegistry
metadata:
  name: deployment-search
spec:
  targetCluster:
    clusterNames:
      - member1
      - member2
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment

3.Executing kubectl get resourceregisty and kubectl get resourceregistries.

➜  karmada git:(1.5) ✗ kubectl --kubeconfig $HOME/.kube/karmada.config --context karmada-apiserver get ResourceRegistry  
NAME                CREATED AT
deployment-search   2023-10-25T07:04:14Z
➜  karmada git:(1.5) ✗ kubectl --kubeconfig $HOME/.kube/karmada.config --context karmada-apiserver get resourceregistries
NAME                CREATED AT
deployment-search   2023-10-25T07:04:14Z

4.Upgrade the component karmada-search with the latest image.
5.Executing kubectl get resourceregisty and kubectl get resourceregistries.

➜  karmada git:(1.7) ✗ kubectl --kubeconfig $HOME/.kube/karmada.config --context karmada-apiserver get ResourceRegistry  
NAME                CREATED AT
deployment-search   2023-10-25T07:04:14Z
➜  karmada git:(1.7) ✗ kubectl --kubeconfig $HOME/.kube/karmada.config --context karmada-apiserver get resourceregistries
NAME                CREATED AT
deployment-search   2023-10-25T07:04:14Z

[XiShanYongYe-Chang]

Ask @RainbowMango to help rerun the CI.
/cc @RainbowMango

[RainbowMango]

/approve

[karmada-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RainbowMango

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/registry/OWNERS [RainbowMango]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[RainbowMango]

/lgtm

[XiShanYongYe-Chang]

/lgtm

[RainbowMango]

/hold

[RainbowMango]

/hold cancel