-
Notifications
You must be signed in to change notification settings - Fork 201
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Avoid memory exhaustion during data migration #1630
Conversation
- Avoid loading all records at once in memory Signed-off-by: Keshav Priyadarshi <[email protected]>
@keshav-space thanks for this, can we use our own paginate here? |
@TG1999 We cannot use vulnerablecode/vulnerabilities/models.py Lines 874 to 972 in 289f4b8
Also, afaik custom querysets are not directly available in data migrations. In this case it would be better to use the built-in queryset |
Merging this now! |
There are nearly 15 million package vulnerability relationships in VCIO, and loading them all at once during data migration consumes all the memory.