High memory usage on elastic agent

[mistic]

A friend of mine using the elastic agent reported to me an unreasonably high memory when using the elastic agent.

The following is the feedback that came up as a result from a little investigation they ran:

The registration strategy used in beats means we perform package initialisation for all modules at startup, and this means it walks into the vsphere packages at startup - that 40mb one uses a serialization/deserialization preparation pattern that means it executes tons of code over all of it's types to generate plan tables in package init, which also means it has to bring all that stuff into the heap on startup - this almost certainly contributes very significantly to startup performance too. Ousting that module cuts 10% of the rss requirement for printing --help from the program

In addition the "add fields" processor depends on the scripts processor which depends on a WIP javascript runtime written in go, that round-trips through json a couple of extra times.

I believe this is something that is worth it at least to get checked so I'm opening this issue here.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[pierrehilbert]

@alexsapran putting this in your radar.

[alexsapran]

Thanks @pierrehilbert 👍🏼 .

Reading about this issue, I was wondering if, @mistic could share the configuration (or part of the configuration) of what your friend tried as well as what version of elastic agent he was using, what OS, and what methodology he used to test this.

[cmacknz]

The way Go initializes dependencies is the major factor here and is well described in grafana/alloy#529, where it also affects the startup time for services with many dependencies.

CC @andrewkroh, since we were just talking about the memory usage of the agent and this seems to be one contributing factor.

For @alexsapran what we would need from our performance testing is to capture the memory usage for each test, and ideally capture heap profiles in addition to CPU profiles.

[mistic]

Thanks for paying attention to this team! @alexsapran do you still find the info you mentioned previously as needed?

[raggi]

See raggi@f069ddd for an example, this cuts the rss substantially