Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ch): use writev when possible #413

Merged
merged 11 commits into from
Sep 25, 2024
Merged

Conversation

tdakkota
Copy link
Contributor

@tdakkota tdakkota commented Aug 2, 2024

Reduce memory footprint by avoiding unnecessary data copying during inserts.

goos: linux
goarch: amd64
pkg: github.com/ClickHouse/ch-go
cpu: AMD Ryzen 9 5950X 16-Core Processor            
                        │   old.txt    │               new.txt               │
                        │    sec/op    │   sec/op     vs base                │
Insert/Rows10000-32       396.3µ ±  6%   421.2µ ± 1%   +6.27% (p=0.000 n=15)
Insert/Rows100000-32      521.8µ ±  6%   687.5µ ± 1%  +31.76% (p=0.000 n=15)
Insert/Rows1000000-32     3.562m ± 12%   3.485m ± 1%        ~ (p=0.305 n=15)
Insert/Rows10000000-32    32.29m ±  3%   31.77m ± 2%   -1.61% (p=0.023 n=15)
Insert/Rows100000000-32   421.7m ±  6%   230.5m ± 9%  -45.35% (p=0.000 n=15)
geomean                   6.314m         5.939m        -5.93%

                        │    old.txt    │               new.txt                │
                        │      B/s      │     B/s       vs base                │
Insert/Rows10000-32       192.5Mi ±  7%   181.1Mi ± 1%   -5.90% (p=0.000 n=15)
Insert/Rows100000-32      1.428Gi ±  6%   1.084Gi ± 1%  -24.10% (p=0.000 n=15)
Insert/Rows1000000-32     2.092Gi ± 14%   2.138Gi ± 1%        ~ (p=0.305 n=15)
Insert/Rows10000000-32    2.307Gi ±  2%   2.345Gi ± 2%   +1.64% (p=0.023 n=15)
Insert/Rows100000000-32   1.767Gi ±  6%   3.233Gi ± 8%  +82.97% (p=0.000 n=15)
geomean                   1.180Gi         1.255Gi        +6.31%

                        │      old.txt      │                new.txt                │
                        │       B/op        │     B/op      vs base                 │
Insert/Rows10000-32           10.31Ki ±  0%   10.45Ki ± 0%    +1.42% (p=0.000 n=15)
Insert/Rows100000-32          10.64Ki ±  0%   10.66Ki ± 0%    +0.19% (p=0.000 n=15)
Insert/Rows1000000-32         32.03Ki ±  9%   10.94Ki ± 0%   -65.85% (p=0.000 n=15)
Insert/Rows10000000-32      2309.09Ki ±  8%   11.59Ki ± 0%   -99.50% (p=0.000 n=15)
Insert/Rows100000000-32   260431.62Ki ± 25%   12.43Ki ± 1%  -100.00% (p=0.000 n=15)
geomean                       291.7Ki         11.19Ki        -96.16%

                        │  old.txt   │              new.txt              │
                        │ allocs/op  │ allocs/op   vs base               │
Insert/Rows10000-32       325.0 ± 0%   328.0 ± 0%  +0.92% (p=0.000 n=15)
Insert/Rows100000-32      325.0 ± 0%   328.0 ± 0%  +0.92% (p=0.000 n=15)
Insert/Rows1000000-32     326.0 ± 0%   329.0 ± 0%  +0.92% (p=0.000 n=15)
Insert/Rows10000000-32    326.0 ± 0%   329.0 ± 0%  +0.92% (p=0.000 n=15)
Insert/Rows100000000-32   331.0 ± 0%   333.0 ± 0%  +0.60% (p=0.000 n=15)
geomean                   326.6        329.4       +0.86%

@tdakkota tdakkota force-pushed the perf/use-writev branch 2 times, most recently from 333a54b to 949f3cb Compare September 25, 2024 02:07
`proto.Writer` API still reduces total memory consumption and prevents unnecessary data copying
@tdakkota
Copy link
Contributor Author

At c420f0b

goos: linux
goarch: amd64
pkg: github.com/ClickHouse/ch-go
cpu: AMD Ryzen 9 5950X 16-Core Processor            
                        │   old.txt    │               new.txt                │
                        │    sec/op    │    sec/op     vs base                │
Insert/Rows10000-32       454.3µ ±  1%    471.0µ ± 2%   +3.66% (p=0.000 n=15)
Insert/Rows100000-32      851.8µ ±  3%   1003.2µ ± 2%  +17.77% (p=0.000 n=15)
Insert/Rows1000000-32     7.611m ±  1%    6.609m ± 1%  -13.16% (p=0.000 n=15)
Insert/Rows10000000-32    62.03m ± 17%    58.12m ± 3%   -6.30% (p=0.000 n=15)
Insert/Rows100000000-32   758.8m ± 13%    515.4m ± 5%  -32.08% (p=0.000 n=15)
geomean                   10.68m          9.867m        -7.57%

                        │    old.txt     │                new.txt                │
                        │      B/s       │      B/s       vs base                │
Insert/Rows10000-32        167.9Mi ±  1%    162.0Mi ± 2%   -3.53% (p=0.000 n=15)
Insert/Rows100000-32       895.7Mi ±  4%    760.5Mi ± 2%  -15.09% (p=0.000 n=15)
Insert/Rows1000000-32     1002.4Mi ±  1%   1154.4Mi ± 1%  +15.16% (p=0.000 n=15)
Insert/Rows10000000-32     1.201Gi ± 14%    1.282Gi ± 3%   +6.72% (p=0.000 n=15)
Insert/Rows100000000-32   1005.4Mi ± 11%   1480.4Mi ± 5%  +47.24% (p=0.000 n=15)
geomean                    714.7Mi          773.2Mi        +8.19%

                        │      old.txt      │                new.txt                │
                        │       B/op        │     B/op      vs base                 │
Insert/Rows10000-32           10.38Ki ±  0%   10.89Ki ± 0%    +4.95% (p=0.000 n=15)
Insert/Rows100000-32          10.93Ki ±  0%   10.92Ki ± 0%    -0.16% (p=0.002 n=15)
Insert/Rows1000000-32         60.91Ki ±  2%   11.33Ki ± 0%   -81.41% (p=0.000 n=15)
Insert/Rows10000000-32      4123.41Ki ± 36%   11.92Ki ± 0%   -99.71% (p=0.000 n=15)
Insert/Rows100000000-32   390642.05Ki ±  0%   13.67Ki ± 6%  -100.00% (p=0.000 n=15)
geomean                       406.8Ki         11.70Ki        -97.12%

                        │  old.txt   │              new.txt              │
                        │ allocs/op  │ allocs/op   vs base               │
Insert/Rows10000-32       325.0 ± 0%   329.0 ± 0%  +1.23% (p=0.000 n=15)
Insert/Rows100000-32      325.0 ± 0%   329.0 ± 0%  +1.23% (p=0.000 n=15)
Insert/Rows1000000-32     326.0 ± 0%   330.0 ± 0%  +1.23% (p=0.000 n=15)
Insert/Rows10000000-32    326.0 ± 0%   330.0 ± 0%  +1.23% (p=0.000 n=15)
Insert/Rows100000000-32   333.0 ± 1%   334.0 ± 0%       ~ (p=0.685 n=15)
geomean                   327.0        330.4       +1.04%

@tdakkota tdakkota marked this pull request as ready for review September 25, 2024 02:45
@ernado ernado merged commit e4a2d07 into ClickHouse:main Sep 25, 2024
25 checks passed
@tdakkota tdakkota deleted the perf/use-writev branch September 25, 2024 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants