Skip to content
Celeborn CI

[WIP][CELEBORN-1319] Optimize skew partition logic for Reduce Mode to avoid sorting shuffle files #10244

Sign in to view logs

[WIP][CELEBORN-1319] Optimize skew partition logic for Reduce Mode to avoid sorting shuffle files

[WIP][CELEBORN-1319] Optimize skew partition logic for Reduce Mode to avoid sorting shuffle files #10244

Run time

Learn about OS pricing on GitHub Actions
Job Run time
service (8)
8m 12s
flink (8, 1.14)
7m 33s
service (11)
8m 34s
mr (8)
6m 54s
spark2 (8, 2.4)
20m 6s
service (17)
8m 34s
mr (11)
7m 7s
flink (8, 1.15)
7m 36s
flink (8, 1.17)
7m 45s
spark3 (8, 3.0, org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO)
22m 50s
spark3 (8, 3.1, org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO)
22m 40s
spark3 (8, 3.2, org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO)
22m 49s
flink (8, 1.18)
7m 39s
spark3 (8, 3.3, org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO)
34m 5s
flink (8, 1.19)
7m 36s
spark3 (8, 3.4, org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO)
36m 41s
flink (11, 1.14)
7m 53s
spark3 (8, 3.5, org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO)
32m 36s
flink (11, 1.15)
8m 14s
spark3 (8, 3.5, org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO)
32m 40s
flink (11, 1.17)
7m 44s
spark3 (11, 3.0, org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO)
22m 55s
flink (11, 1.18)
7m 37s
spark3 (11, 3.1, org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO)
22m 30s
flink (11, 1.19)
7m 40s
spark3 (11, 3.2, org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO)
21m 29s
spark3 (11, 3.3, org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO)
24m 5s
spark3 (11, 3.4, org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO)
24m 11s
spark3 (11, 3.5, org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO)
24m 3s
spark3 (11, 3.5, org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO)
23m 46s
spark3 (17, 3.3, org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO)
25m 44s
spark3 (17, 3.4, org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO)
23m 14s
spark3 (17, 3.5, org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO)
25m 38s
spark3 (17, 3.5, org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO)
25m 50s
10h 4m 30s