Releases · kwai/blaze

New features

supports spark3.0/3.1/3.2/3.3/3.4/3.5.
supports integrating with Apache Celeborn.
supports native ORC input format.
supports bloom filter join introduced in spark 3.5.
supports forceShuffledHashJoin for running tpch/tpcds benchmarks.
new supported native expression/functions: year, month, day, md5.

Bug fixes

add missing UDTF.terminate() invokes.
fix NPE while executing some native spark physical plans.

Performance

use custom implemented hash table for faster joining, supporting SIMD, bulk searching, memory prefetching, etc.
improve shuffle write performance.
reuse FSDataInputStream for same input file.

What's Changed

spark351 + supports bloom filter join by @richox in #532
Support Spark 3.2.0 by @XorSum in #538
Bump sonic-rs from 0.3.8 to 0.3.9 by @dependabot in #533
Bump tonic-build from 0.12.0 to 0.12.1 by @dependabot in #525
Bump prost from 0.13.0 to 0.13.1 by @dependabot in #521
supports UDTF.terminate by @richox in #536
Bump bytes from 1.6.1 to 1.7.0 by @dependabot in #539
Bump tokio from 1.38.0 to 1.39.2 by @dependabot in #534
Bump tempfile from 3.10.1 to 3.12.0 by @dependabot in #542
Bump bytes from 1.7.0 to 1.7.1 by @dependabot in #540
supports forceShuffledHashJoin. by @richox in #545
init RSS framework by @richox in #551
Bump radsort from 0.1.0 to 0.1.1 by @dependabot in #550
Bump postcard from 1.0.8 to 1.0.10 by @dependabot in #552
Bump tonic-build from 0.12.1 to 0.12.2 by @dependabot in #554
Bump sonic-rs from 0.3.9 to 0.3.12 by @dependabot in #555
improve shuffle performance by @richox in #559
Bump prost from 0.13.1 to 0.13.2 by @dependabot in #556
Bump async-trait from 0.1.81 to 0.1.82 by @dependabot in #557
Bump tokio from 1.39.2 to 1.40.0 by @dependabot in #558
delete walkaround_sliced_boolean_array_issue by @DDDominik in #560
fix CI by @richox in #562
optimize hash joins by @richox in #563
support year/month/day functions by @richox in #565
Support Spark 3.1.3 by @harveyyue in #561
Bump sonic-rs from 0.3.12 to 0.3.13 by @dependabot in #566
WIP: improve shuffle write performance by @richox in #564
remove variables from artifacts in pom.xml by @richox in #569
add --allow-no-vcs to cargo fix command by @richox in #570
use memory prefetch in hash map building by @richox in #571
support native scan orc format by @harveyyue in #544
add spark prefix to tpcds.gen.defaultParallel to take affect by @TJX2014 in #572
support md5 command by @KnightChess in #581
Bump bytes from 1.7.1 to 1.7.2 by @dependabot in #578
update doc to add protobuf installation guides by @richox in #583
Bump prost from 0.13.2 to 0.13.3 by @dependabot in #586
make RUSTFLAGS configurable by @XorSum in #585
[BLAZE-587] Replace spark pattern with spark- for maven profile and shim name by @SteNicholas in #588
use hadoop fs positioned-reading by @richox in #582
[BLAZE-573] Support Spark 3.4 version by @SteNicholas in #589
docs/fix: fix error message by @caicancai in #591
reuse FSDataInputStream for same input file by @richox in #592
make RUSTFLAGS effective before running cargo commands by @harveyyue in #594
[BLAZE-287] Integrate Blaze with Celeborn by @RexXiong in #596
fix spark3.1 compile error by @richox in #598
fix timer in AggExec by @richox in #597
update docs (celeborn related) by @richox in #599
Change the output qualifier from var to def will avoid invoking NPE by @harveyyue in #608
OrcExec should read the specified file range data by @harveyyue in #605
Bump tempfile from 3.12.0 to 3.13.0 by @dependabot in #604
Bump once_cell from 1.20.0 to 1.20.1 by @dependabot in #603
Bump tonic-build from 0.12.2 to 0.12.3 by @dependabot in #595
Bump async-trait from 0.1.82 to 0.1.83 by @dependabot in #590
Bump futures-util from 0.3.30 to 0.3.31 by @dependabot in #609
Bump futures from 0.3.30 to 0.3.31 by @dependabot in #610
Bump once_cell from 1.20.1 to 1.20.2 by @dependabot in #611