Huge RAM consumption after updating to 0.15.0 #247
Comments
|
I'm seeing the same across three different clusters. Memory usage steadily increases and eventually hits the default 1GB limit after about 30 minutes. I updated from |
|
That's exactly what we're seeing. We use the vertical pod autoscaler to ensure that requirements change automatically so that's how we got to the 20 Gb. |
|
I am on Flux 0.19.1 with IAC 0.15.0 and I have 3x image update automations, I don't seem to be able to repro this: If you have a consistent repro of this issue, besides passing along as much detailed information about your ImageUpdate and related configurations as you can, it would most likely help to collect a profile and include it in the report: https://fluxcd.io/docs/gitops-toolkit/debugging/#collecting-a-profile When I increased the reconciliation frequency as high as it would go, and sent a huge number of I'll leave it running this way overnight and see if anything changes, but for now I'm going to need some more information from anyone affected by this. It looks like it's definitely a real issue, as we have multiple reporters and several folks on this Slack thread which links here as well: https://cloud-native.slack.com/archives/CLAJ40HV3/p1635358682258400 |
|
Could you give |
|
@kingdonb I just sent a heap dump to your email. I'm far from being an expert in Go and this application, but checking the profiling information it seems that all this memory is being used by go-git.
@hiddeco Thanks for the tip. Your PR mentions that that change depends on another change in the source-controller. Would I need to also update the latter? |
|
@gmaiztegi if the only misbehaving controller is this one, you shouldn't have to. The reason it "depends" on it is because of shared code, not because it matters during runtime :-). |
|
@hiddeco with your image the memory stays steady at around 470MiB. |
|
@gmaiztegi I would advise you to use this image for now then, and we'll ensure to get the changes into an official release fast. Thanks a lot for reporting this 🌻 |
This commit makes use of the refactored `git` package, which has been reworked to increase stability and test coverage, and ensures implementation details do not leak out into the "main wrapper". This indirectly seems to resolve a memory leak that happenedd with the previous wiring, thereby fixing #247. The code changes for this controller itself are minimal, mostly ensuring the auth and checkout configurations are created in the "new way". Signed-off-by: Hidde Beydals <[email protected]>
|
Patch from the PR is now available as |
This commit makes use of the refactored `git` package, which has been reworked to increase stability and test coverage, and ensures implementation details do not leak out into the "main wrapper". This indirectly seems to resolve a memory leak that happenedd with the previous wiring, thereby fixing fluxcd#247. The code changes for this controller itself are minimal, mostly ensuring the auth and checkout configurations are created in the "new way". Signed-off-by: Hidde Beydals <[email protected]>
After updating to 0.15.0 the
image-automation-controllerwent from using around 700MB to 20GB. Nothing else relevant was updated in our environment, so we don't know what might be causing this. Reverting to 0.14.1 fixes the issue.We have around 80 image repositories and policies, in case it is relevant.
Is this happening to someone else?
The text was updated successfully, but these errors were encountered: