bugfix: filter repeated resources and optimize memory usage.

[qicz]

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #4395 and optimize for #4181

before:

after

[arkodg]

@qicz thanks for picking this one up, its worth investigating why this happening before working around it by using a set

[zirain]

we need some test to guratee this

[qicz]

@qicz thanks for picking this one up, its worth investigating why this happening before working around it by using a set

for my analysis, the endpointslice with more repeated value for the service

gateway/internal/provider/kubernetes/controller.go

Lines 434 to 455 in aab669d

	if endpointSliceLabelKey != "" {
	endpointSliceList := new(discoveryv1.EndpointSliceList)
	opts := []client.ListOption{
	client.MatchingLabels(map[string]string{
	endpointSliceLabelKey: string(backendRef.Name),
	}),
	client.InNamespace(string(*backendRef.Namespace)),
	}
	if err := r.client.List(ctx, endpointSliceList, opts...); err != nil {
	r.log.Error(err, "failed to get EndpointSlices", "namespace", string(*backendRef.Namespace),
	backendRefKind, string(backendRef.Name))
	} else {
	for _, endpointSlice := range endpointSliceList.Items {
	endpointSlice := endpointSlice //nolint:copyloopvar
	r.log.Info("added EndpointSlice to resource tree", "namespace", endpointSlice.Namespace,
	"name", endpointSlice.Name)
	gwcResource.EndpointSlices = append(gwcResource.EndpointSlices, &endpointSlice)
	}
	}
	}
	}
	}

logging:

[codecov]

Codecov Report

Attention: Patch coverage is 19.65318% with 139 lines in your changes missing coverage. Please review.

Project coverage is 65.88%. Comparing base (04fc944) to head (79d7928).
Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/provider/kubernetes/controller.go	8.41%	93 Missing and 5 partials ⚠️
internal/provider/kubernetes/routes.go	0.00%	41 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4454      +/-   ##
==========================================
- Coverage   65.97%   65.88%   -0.10%     
==========================================
  Files         203      204       +1     
  Lines       31154    31234      +80     
==========================================
+ Hits        20554    20578      +24     
- Misses       9415     9475      +60     
+ Partials     1185     1181       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[arkodg]

ah thanks for pointing to the controller/provider code, it looks like since the service was referenced twice, we added the corresponding endpoint slice twice. The fix in this PR is not going to work since it has visibility into a single EndpointSlice

We can either fix this in the provider or in the gateway api layer

[qicz]

ah thanks for pointing to the controller/provider code, it looks like since the service was referenced twice, we added the corresponding endpoint slice twice. The fix in this PR is not going to work since it has visibility into a single EndpointSlice

We can either fix this in the provider or in the gateway api layer

I 'll fix this in gateway API layer

[shawnh2]

thanks for the fix @qicz, now the mem usage decrease dramatically !

[qicz]

thanks for the fix @qicz, now the mem usage decrease dramatically !

Yep, the duplicate resources have already been filtered, which will optimize the memory for many big objects used, such as ConfigMap, Secret, EndpointSlice, etc.
cc @arkodg

xref #3980 #3698 #4181 #4263

[arkodg]

looks like we are already doing something similar in the provider layer using this structure to ensure deduplication, can we implement this there instead ?

[qicz]

looks like we are already doing something similar in the provider layer using this structure to ensure deduplication, can we implement this there instead ?

They are similar,but cache used in controller and translate, IMO should in gateway layer resource pkg. resourcemap just for controller @arkodg

[arkodg]

looks like we are already doing something similar in the provider layer using this structure to ensure deduplication, can we implement this there instead ?

They are similar,but cache used in controller and translate, IMO should in gateway layer resource pkg. resourcemap just for controller @arkodg

since we already have an existing way to do this, prefer the provider approach to reduce code complexity

[qicz]

looks like we are already doing something similar in the provider layer using this structure to ensure deduplication, can we implement this there instead ?

They are similar,but cache used in controller and translate, IMO should in gateway layer resource pkg. resourcemap just for controller @arkodg

since we already have an existing way to do this, prefer the provider approach to reduce code complexity

you mean reuse resourcemap struct? or reuse resourcemap set properties? @arkodg

[arkodg]

looks like we are already doing something similar in the provider layer using this structure to ensure deduplication, can we implement this there instead ?

They are similar,but cache used in controller and translate, IMO should in gateway layer resource pkg. resourcemap just for controller @arkodg

since we already have an existing way to do this, prefer the provider approach to reduce code complexity

you mean reuse resourcemap struct? or reuse resourcemap set properties? @arkodg

create an addition field within resourceMappings for service to ensure we add one unique item into the list

[arkodg]

@qicz my suggestion is to solve the problem in the provider package similar to what we are doing for other resources

[qicz]

@qicz my suggestion is to solve the problem in the provider package similar to what we are doing for other resources

done, ptal @arkodg

[arkodg]

LGTM thanks !
great to see the massive drop in mem usage