Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sqlccl: make IMPORT CSV tolerant of data errors #25751

Open
knz opened this issue May 21, 2018 · 1 comment
Open

sqlccl: make IMPORT CSV tolerant of data errors #25751

knz opened this issue May 21, 2018 · 1 comment
Labels
A-import Issues related to IMPORT syntax C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) E-quick-win Likely to be a quick win for someone experienced. S-3-ux-surprise Issue leaves users wondering whether CRDB is behaving properly. Likely to hurt reputation/adoption. T-sql-queries SQL Queries Team

Comments

@knz
Copy link
Contributor

knz commented May 21, 2018 •
edited by exalate-issue-sync bot
Loading

Reported in #25480

If any line in the CSV file contains erroneous (unparsable/incorrect data), the entire IMPORT job fails. This is poor UX, because the amount of work needed to correct the data might be negligible compared to the time needed to complete the IMPORT job.

Each of the following alternatives would be better UX, and the combination of them even better:

  • make IMPORT save erroreous CSV lines in a separate file (possibly also with location information) for manual inspection, and skip over them
  • make IMPORT suspend on data error with a status message in the job table that indicates where the error was found, to let the user fix the error and then resume the IMPORT job
  • make IMPORT accept SQL options to specify what to do in case of data errors, including skipping over erroneous lines

Jira issue: CRDB-5695
@knz knz added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) S-3-ux-surprise Issue leaves users wondering whether CRDB is behaving properly. Likely to hurt reputation/adoption. A-disaster-recovery labels May 21, 2018
@knz knz mentioned this issue May 21, 2018
Closed
@msbutler
Copy link
Collaborator

msbutler commented Aug 30, 2022 •
edited
Loading

triaging to sql-exp as new owners of this.

Ideally ,the log_ignored_statements option should no longer be undocumented and experimental for IMPORT INTO.

@msbutler msbutler added the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label Aug 30, 2022
@ZhouXing19 ZhouXing19 added the A-import Issues related to IMPORT syntax label Oct 20, 2022
@rafiss rafiss added the E-quick-win Likely to be a quick win for someone experienced. label Nov 17, 2022
@exalate-issue-sync exalate-issue-sync bot added E-quick-win Likely to be a quick win for someone experienced. and removed E-quick-win Likely to be a quick win for someone experienced. labels Nov 17, 2022
@rafiss rafiss removed the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label Apr 4, 2023
@blathers-crl blathers-crl bot added the T-sql-queries SQL Queries Team label Apr 4, 2023
@mgartner mgartner moved this to New Backlog in SQL Queries Jul 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-import Issues related to IMPORT syntax C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) E-quick-win Likely to be a quick win for someone experienced. S-3-ux-surprise Issue leaves users wondering whether CRDB is behaving properly. Likely to hurt reputation/adoption. T-sql-queries SQL Queries Team
Projects
SQL Queries
Status: Backlog
Milestone
No milestone
Development

No branches or pull requests
5 participants
@kenliu @knz @rafiss @msbutler @ZhouXing19