Huge memory usage when importing large quantities of products

[piotrekkaminski]

imported from: magento/magento2#10914

When importing huge amounts of products Magento 2 is loading contents of importexport_importdata to memory and then iterate it in memory which results in huge memory usage

Preconditions

1. Magento 2.1.7
2. CSV with 1 500 000 products

Steps to reproduce

1. Validate 1 500 000 products
2. Run import

Expected result

1. Memory usage for iterating trough import data should be constant

Actual result

1. Whole table contents is loaded to memory

[piotrekkaminski]

Pull request: magento/magento2#10918

[piotrekkaminski]

_From @magento-engcom-team: @diwipl, thank you for your report.
We've created internal ticket(s) MAGETWO-84592 to track progress on the issue. _

[nemesis-back]

@dmanners @piotrekkaminski assign to me please

[dmanners]

Hi @nemesis-back thank you for reaching out to us. I have sent you and invitation to the github team. Once you have accepted this invite you will have permissions to assign a task to yourself. If you need any further help feel free to reach out to me.

[dmanners]

Hi @nemesis-back did you get anything working on this? Or have any debugging information that might be useful for people to continue working with?