Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split CPE file to reduce sync memory usage (9.0) #901

Merged
merged 9 commits into from
Dec 12, 2019

Conversation

mattmundell
Copy link
Contributor

@mattmundell mattmundell commented Dec 11, 2019

This reduces the maximum amount of memory used by the CPE update during the SCAP sync, by parsing 40Mb chunks of the CPE dictionary at a time, instead of parsing the whole 162Mb dictionary.

My gvmd sync process is now maxing at about 3.5% of 16GB where it was 9% before.

gvmd creates the chunks itself, in a temp directory, using system to run a shell command. Not the best, but g_spawn_sync has trouble here because we're already handling child signals (because we've forked a process to run the sync). Also the XML wrapper replacements done by the shell command would be tedious to do in C.

This also add details to the proctitle during sync to make it easier to see where memory is spiking.

Checklist:

src/manage_sql_secinfo.c Show resolved Hide resolved
src/manage_sql_secinfo.c Outdated Show resolved Hide resolved
src/manage_sql_secinfo.c Show resolved Hide resolved
@timopollmeier timopollmeier merged commit 7df53af into greenbone:gvmd-9.0 Dec 12, 2019
@mattmundell mattmundell deleted the split-cpe-file-9.0 branch December 16, 2019 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants