Split CPE file to reduce sync memory usage (9.0) #901
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This reduces the maximum amount of memory used by the CPE update during the SCAP sync, by parsing 40Mb chunks of the CPE dictionary at a time, instead of parsing the whole 162Mb dictionary.
My gvmd sync process is now maxing at about 3.5% of 16GB where it was 9% before.
gvmd creates the chunks itself, in a temp directory, using system to run a shell command. Not the best, but g_spawn_sync has trouble here because we're already handling child signals (because we've forked a process to run the sync). Also the XML wrapper replacements done by the shell command would be tedious to do in C.
This also add details to the proctitle during sync to make it easier to see where memory is spiking.
Checklist: