Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[#5453] [YSQL] Fix OOM on File Sourced YB Relations in COPY FROM Query
Summary: This change resets memory context regularly on all YB relations that are copied from files. Instead of resetting memory context at batch size using `rows_per_transaction` query option, memory context will be reset per row when certain conditions hold. See below for details: With this change: 1. Memory context resets per row instead of per batch size when memory is resettable. - eg. `COPY tab FROM 'absoluteFilePath'` will reset memory at every row when conditions are met. 2. Memory context reset per row occurs on YB relations (which excludes temporary tables) and when rows are read from file, not stdin. - if `cstate->filename != NULL`, file is consumed either directly or passed in using program option - if `cstate->filename == NULL`, query is consumed from stdin. Note, large files read through stdin will run into OOM since old context path must be used to hold all rows in memory before inserting to table. That issue is tracked in [[ #5603 | here ]]. Test Plan: Rebuilt and reran Java tests. Manually verified performance on large queries in ysqlsh. Reviewers: alex, mihnea, jason Reviewed By: jason Subscribers: yql Differential Revision: https://phabricator.dev.yugabyte.com/D9313
- Loading branch information