memguardian - backpressure

[Mzack9999]

Adding memguardian component. It's goal is to allow detection of potential OOM kill conditions, and offer the caller the chance to activate mitigation actions to slow down execution.
It can be used both as a standalone struct, or activated and controlled via the following environment variables:

MEMGUARDIAN: Enable/disable. Set to 1 to enable
MEMGUARDIAN_MAX_RAM_RATIO: Maximum ram ratio from 1 to 100
MEMGUARDIAN_MAX_RAM: Maximum amount of RAM (in size units ex: 10gb)
MEMGUARDIAN_INTERVAL: detection interval (with unit ex: 30s)

In case environment variables are used, the component can be accessed through memguardian.DefaultMemGuardian.
The warning state (activated when the used RAM exceedes the defined thresholds) offers two approaches to initiate the back-pressure mechanism:

• Passive by checking the bool MemGuardianInstance.Warning
• Active via callback invoked at each duration when the warning state is reached

The caller should insert in hot-paths reduction factors, or for global settings, either use the callback mechanism or check periodically the state and activate/disable mitigation actions.

Closes #361

[stan-threatmate]

Are you sure that the virtual memory is the right metric? Virtual memory can be larger than the physical available memory. How about considering the real memory or physical memory?

[Mzack9999]

I thought about retrieving directly via OS system call (eg. sysinfo but the gopsutil seems to do already a decent job about obtaining it with standard system tools/files (for example on linux by parsing /proc/meminfo which according to linux kernel docs represent the avilable effective RAM). Do you think this is correct, or there is any better way on the metrics to monitor (for example swap is actually not considered)?

[stan-threatmate]

This will probably differ per OS:
On MacOS Virtual means something else:
https://apple.stackexchange.com/a/107

Additionally misleading info from gopls:
shirou/gopsutil#119

Windows is probably the same

[Mzack9999]

Thanks for the links, I'll have a closer look and see if it makes sense to implement more specific os calls

[stan-threatmate]

I'd also leave this comment here: would it be better if instead of a memguardian there is a simple memory allocation rate limiter? For example if a buffer can be 10MB we can say at most 50 allocations per second regardless of the concurrency settings. This way if templates don't allocate memory they can be very concurrent while memory heavy templates will be less so. The rate limit can be implemented per template or make it global.

[Mzack9999]

During tests imposing a rate limiting on buffer allocations was leading anyway to OOM kill, just later in time as the rate of threads was anyway piling up.
The backpressure mechanism turned out to be the most effective against the cumulative effect. This is part of a major planned feature involving the implementation of an execution planner (see projectdiscovery/nuclei#4808), where memguardian would act on the planner in order to autotune the speed according to resource availability/constraints.
Anyway that's a great suggestion, I think that we can impose a further rate limit on allocations upon memguardian trigger, so we ensure that memory heavy template are both thread and allocation limited.

[stan-threatmate]

I can see how per-template rate limiter will still lead to OOM kill if there are enough concurrent templates. A global rate limiter that is tuned based on the amount of available memory at startup might be a good approach. You're basically doing this by controlling the concurrency with memguardian.

Another suggestion is to have a tag for bruteforce templates that we know can take a ton of RAM. Or even a high-mem tag. This way we can exclude all templates that negatively impact the memory. We could also keep memory allocation stats per template execution so we can log it and quickly identify bad templates.