Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve the performance of TextSource by reducing how many byte[]s are copied (fixes #23193) #23196

Merged
merged 3 commits into from
Sep 15, 2022
Merged

Improve the performance of TextSource by reducing how many byte[]s are copied (fixes #23193) #23196

merged 3 commits into from
Sep 15, 2022

Commits on Sep 12, 2022

  1. Improve the performance of TextSource by reducing how many byte[]s ar… 

    …e copied (fixes apache#23193)

This makes TextSource take about 2.3x less CPU resources during decoding.

Before this change:
```
TextSourceBenchmark.benchmarkTextSource        thrpt    5  0.248 ± 0.029  ops/s
```

After this change:
```
TextSourceBenchmark.benchmarkHadoopLineReader  thrpt    5  0.465 ± 0.064  ops/s
TextSourceBenchmark.benchmarkTextSource        thrpt    5  0.575 ± 0.059  ops/s
```
    @lukecwik
    lukecwik committed Sep 12, 2022
    Configuration menu
    Copy the full SHA
    8617f4a View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2022

  1. Write file in pieces instead of pre-allocating entire buffer

    @lukecwik
    lukecwik committed Sep 13, 2022
    Configuration menu
    Copy the full SHA
    8f921ea View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2022

  1. Address PR comments

    @lukecwik
    lukecwik committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    f1a60ba View commit details
    Browse the repository at this point in the history