NETOBSERV-578: decrease informers' memory consumption #317
Conversation
mariomac
added
the
ok-to-test
|
New image: ["quay.io/netobserv/flowlogs-pipeline:7314a97"]. It will expire after two weeks. |
github-actions
bot
removed
the
ok-to-test
|
I did some tests on higher load with
|
|
@mariomac I haven't reviewed the code in deep, but remember the discussion we had about potential blocking calls and fact the cache seemed pre-loaded? Does this PR change anything in that regard / should we try to parallelize transformer processing? |
|
@jotak after working deeply with the informers' code, I can confirm that there aren't blocking calls (neither before nor now). The initial consideration about parallelizing workers was to share the big amount of memory that the informers' spent so we expected to minimize the overall memory usage by scaling vertically and downscaling horizontally. There are some reasons that changed my mind with respect to parallelizing (I'll annotate them also in the related JIRA to justify it):
Given the internal complexity of FLP, I'd say that the effort of reworking some internals to allow this parallelization won't worth the limited improvements we could achieve. |
| if ownerReference.Kind == "ReplicaSet" { | ||
| item, ok, err := k.replicaSetInformer.GetIndexer().GetByKey(info.Namespace + "/" + ownerReference.Name) | ||
| if err != nil { | ||
| panic(err) |
There was a problem hiding this comment.
never noticed this panic .. that was abrupt!
This PR adds a Transformer for each Kubernetes informer that converts the watched Pods, Services, and Nodes to
Infoinstances, that are stored by the informers cache and contain only the data that is required for the decoration.This also alleviates the load on the Garbage Collector: before this patch, a new
Infoinstance was created for each flow decoration. Now the same objects are reused onGetInfoinvocations for the same IP.In Flowlogs-Pipelines with mid-to-low loads, no improvements have been observed in terms of CPU and slight improvements in terms of memory. In the image below, the middle part corresponds to the new version of FLP.