Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql: add a roachtest to benchmark COPY FROM STDIN #81725

Closed
cucaroach opened this issue May 24, 2022 · 0 comments
Closed

sql: add a roachtest to benchmark COPY FROM STDIN #81725

cucaroach opened this issue May 24, 2022 · 0 comments
Assignees
@cucaroach
Labels
A-migrations Migrating to CRDB from another database vendor C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-sql-queries SQL Queries Team

Comments

@cucaroach
Copy link
Contributor

cucaroach commented May 24, 2022 •
edited by cockroach-jira-scripts
Loading

We need a benchmark representative of the dataloading path used by DMS services. AWS DMS is a client side streamed COPY of a CSV. The benchmark should use a reasonably complex schema (ie tpcc lineitem) and should have a quick variation (ie 10k rows) for simple single node spot testing and a larger multinode test with a dataset that won't fit in memory and can give us an indication of how load speed varies over time. Because of limited native support for COPY the roachtest will install postgresql and use psql to perform the COPY.

Jira issue: CRDB-16060
@cucaroach cucaroach added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-sql-queries SQL Queries Team A-migrations Migrating to CRDB from another database vendor labels May 24, 2022
@cucaroach cucaroach self-assigned this May 24, 2022
cucaroach added a commit to cucaroach/cockroach that referenced this issue Jun 2, 2022
@cucaroach
sql: COPY benchmark roachtest
bf7f53e 
Simple roachtest benchmark performing a single node copy of a 750k line
CSV (TPCH lineitem table).  On my gceworker CRDB does it in 5773 rows/s
and PG does it in 36945 rows/s.

Fixes: cockroachdb#81725

Release note: none
cucaroach added a commit to cucaroach/cockroach that referenced this issue Jun 7, 2022
@cucaroach
sql: COPY benchmark roachtest
f4cad41 
Simple roachtest benchmark performing a single node copy of a 750k line
CSV (TPCH lineitem table).  On my gceworker CRDB does it in 5773 rows/s
and PG does it in 36945 rows/s.

Fixes: cockroachdb#81725

Release note: none
@mgartner mgartner moved this to Done in SQL Queries Jul 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cucaroach cucaroach

Labels
A-migrations Migrating to CRDB from another database vendor C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-sql-queries SQL Queries Team
Projects
SQL Queries
Archived in project
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cucaroach