Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use @node-rs/crc32 v1.6.0 #3

Merged
merged 4 commits into from
Jan 12, 2023
Merged

Use @node-rs/crc32 v1.6.0 #3

merged 4 commits into from
Jan 12, 2023

Conversation

twoeths
Copy link

@twoeths twoeths commented Jan 11, 2023

Motivation
- sse4_crc32 has comparable performance to @node-rs/crc32 (or better in some cases) but without memory issue (see napi-rs/node-rs#655)
- This is a backup solution in case it takes time to resolve that issue

  • Update: v1.6.0 has released which resolved the memory issue so migrated to it

Description
- Switch to sse4_crc32

  • Update to @node-rs/crc32 v1.6.0
  • Add memory test
  • Move benchmark to test folder

Benchmark result

  • sse4_crc32 wins for smaller data but lose from 1MB so in general, they are comparable
crc32 implementations
    ✔ @node-rs/crc32c 500 bytes                                            1293661 ops/s    773.0000 ns/op        -    9391541 runs   10.1 s
    ✔ sse4_crc32c_hw 500 bytes                                             1972387 ops/s    507.0000 ns/op        -   12195986 runs   10.1 s
    ✔ sse4_crc32c_sw 500 bytes                                             1342282 ops/s    745.0000 ns/op        -    9686755 runs   10.3 s
    ✔ js_crc32c 500 bytes                                                 103605.5 ops/s    9.652000 us/op        -     980747 runs   10.0 s
    ✔ js_crc32 500 bytes                                                  99810.36 ops/s    10.01900 us/op        -     955473 runs   10.0 s
    ✔ @node-rs/crc32c 1000 bytes                                           1001001 ops/s    999.0000 ns/op        -    7665479 runs   10.1 s
    ✔ sse4_crc32c_hw 1000 bytes                                            1811594 ops/s    552.0000 ns/op        -   11863414 runs   10.1 s
    ✔ sse4_crc32c_sw 1000 bytes                                           938967.1 ops/s    1.065000 us/op        -    7392259 runs   10.1 s
    ✔ js_crc32c 1000 bytes                                                52356.02 ops/s    19.10000 us/op        -     516993 runs   10.1 s
    ✔ js_crc32 1000 bytes                                                 53084.19 ops/s    18.83800 us/op        -     518707 runs   10.0 s
    ✔ @node-rs/crc32c 10000 bytes                                         308451.6 ops/s    3.242000 us/op        -    2804143 runs   10.1 s
    ✔ sse4_crc32c_hw 10000 bytes                                          625390.9 ops/s    1.599000 us/op        -    5305390 runs   10.1 s
    ✔ sse4_crc32c_sw 10000 bytes                                          164554.9 ops/s    6.077000 us/op        -    1582521 runs   10.1 s
    ✔ js_crc32c 10000 bytes                                               5650.771 ops/s    176.9670 us/op        -      55812 runs   10.1 s
    ✔ js_crc32 10000 bytes                                                5675.852 ops/s    176.1850 us/op        -      56072 runs   10.1 s
    ✔ @node-rs/crc32c 100000 bytes                                        45285.75 ops/s    22.08200 us/op        -     443297 runs   10.0 s
    ✔ sse4_crc32c_hw 100000 bytes                                         82230.08 ops/s    12.16100 us/op        -     808063 runs   10.1 s
    ✔ sse4_crc32c_sw 100000 bytes                                         17901.58 ops/s    55.86100 us/op        -     178632 runs   10.1 s
    ✔ js_crc32c 100000 bytes                                              553.7822 ops/s    1.805764 ms/op        -       5280 runs   10.0 s
    ✔ js_crc32 100000 bytes                                               554.3047 ops/s    1.804062 ms/op        -       5287 runs   10.0 s
    ✔ @node-rs/crc32c 1000000 bytes                                       4501.017 ops/s    222.1720 us/op        -      46191 runs   10.5 s
    ✔ sse4_crc32c_hw 1000000 bytes                                        3060.135 ops/s    326.7830 us/op        -      29439 runs   10.1 s
    ✔ sse4_crc32c_sw 1000000 bytes                                        1115.471 ops/s    896.4820 us/op        -      10546 runs   10.0 s
    ✔ js_crc32c 1000000 bytes                                             47.05664 ops/s    21.25099 ms/op        -        450 runs   10.1 s
    ✔ js_crc32 1000000 bytes                                              31.19031 ops/s    32.06124 ms/op        -        297 runs   10.0 s

@twoeths twoeths changed the title Switch to sse4_crc32 Use @node-rs/crc32 v1.6.0 Jan 12, 2023
@twoeths twoeths marked this pull request as ready for review January 12, 2023 02:14
@twoeths twoeths requested a review from a team as a code owner January 12, 2023 02:14
@Brooooooklyn
Copy link

@tuyennhv Could you rerun the benchmark since I also improved performance in the latest @node-rs/crc32 release

@twoeths twoeths merged commit e0dad99 into master Jan 12, 2023
@twoeths
Copy link
Author

twoeths commented Jan 12, 2023

@Brooooooklyn I didn't see a clear difference, are there big improvements between v1.5.1 and v1.6.0?

@twoeths twoeths deleted the tuyen/sse4_crc32 branch January 12, 2023 02:47
@Brooooooklyn
Copy link

It's about 10% improvement on my computer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants