Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize exchange operator & Support mpp version #6596

Merged
merged 104 commits into from
Jan 30, 2023
Merged
Changes from 1 commit
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
3ba1ffb
optimize
solotzg Nov 11, 2022
2541792
support haswell(intel 2013) arch
solotzg Nov 15, 2022
7585264
fix some modules std::chrono::system_clock->std::chrono::steady_clock
solotzg Nov 15, 2022
518b51b
test compress ch block chunck
solotzg Nov 15, 2022
d6495b1
test
solotzg Nov 22, 2022
b1dd890
use TIFLASH_COMPILER_ARCH_HASWELL_FLAG
solotzg Dec 7, 2022
ae30e4a
1
solotzg Dec 2, 2022
54921da
2
solotzg Dec 8, 2022
adac181
3
solotzg Dec 12, 2022
7fd467e
4
solotzg Dec 12, 2022
2b811fb
5
solotzg Dec 12, 2022
e1448ca
6
solotzg Dec 12, 2022
8ce25ec
7
solotzg Dec 12, 2022
f671ba3
8
solotzg Dec 14, 2022
b803984
9
solotzg Dec 15, 2022
bca36f1
10
solotzg Dec 15, 2022
42ae446
11
solotzg Dec 16, 2022
2b56ec2
12
solotzg Dec 16, 2022
f249764
13
solotzg Dec 16, 2022
8c5cd37
14
solotzg Dec 20, 2022
7010790
15
solotzg Dec 21, 2022
c87ee5a
16
solotzg Dec 22, 2022
16dd20d
17
solotzg Dec 22, 2022
e2f17be
18
solotzg Dec 23, 2022
686105d
19
solotzg Dec 27, 2022
94ddcf2
20
solotzg Dec 27, 2022
b43aeb0
21
solotzg Dec 27, 2022
d6578df
22
solotzg Dec 29, 2022
3b15dab
23
solotzg Dec 30, 2022
148235b
24
solotzg Dec 30, 2022
d1d7e82
25
solotzg Jan 2, 2023
9ce108b
26
solotzg Jan 2, 2023
409d020
27
solotzg Jan 3, 2023
2fb5379
28
solotzg Jan 3, 2023
77b36c5
29
solotzg Jan 3, 2023
436bb80
30
solotzg Jan 3, 2023
a8f1b89
31
solotzg Jan 4, 2023
09cb7d3
32
solotzg Jan 5, 2023
334654e
33
solotzg Jan 5, 2023
1efc507
34
solotzg Jan 5, 2023
357843f
35
solotzg Jan 5, 2023
950503c
36
solotzg Jan 5, 2023
78cb77c
37
solotzg Jan 5, 2023
2c4558c
38
solotzg Jan 5, 2023
0478c3f
39
solotzg Jan 5, 2023
f3ad485
40
solotzg Jan 6, 2023
12e2692
41
solotzg Jan 6, 2023
45fdeb3
42
solotzg Jan 6, 2023
f0b805e
43
solotzg Jan 6, 2023
94d5646
44
solotzg Jan 6, 2023
d80d4f1
Merge remote-tracking branch 'pingcap/master' into test-exchange-comp…
solotzg Jan 6, 2023
e70fcb5
45
solotzg Jan 6, 2023
1cf3ccd
46
solotzg Jan 6, 2023
7e2aaad
47
solotzg Jan 6, 2023
1719462
48
solotzg Jan 6, 2023
4423038
Merge remote-tracking branch 'pingcap/master' into test-exchange-comp…
solotzg Jan 6, 2023
c0be2b4
49
solotzg Jan 6, 2023
c960803
50
solotzg Jan 10, 2023
74111b8
51
solotzg Jan 10, 2023
441ad56
52
solotzg Jan 11, 2023
218d272
53
solotzg Jan 11, 2023
63b19e9
54
solotzg Jan 11, 2023
545540c
55
solotzg Jan 11, 2023
5f697d2
56
solotzg Jan 11, 2023
f42a66b
Merge remote-tracking branch 'pingcap/master' into test-exchange-comp…
solotzg Jan 11, 2023
0e6fb4b
57
solotzg Jan 11, 2023
2be58a4
58
solotzg Jan 12, 2023
1a95d51
59
solotzg Jan 12, 2023
5970ea6
60
solotzg Jan 12, 2023
68beb5f
61
solotzg Jan 12, 2023
e2b201f
Merge remote-tracking branch 'pingcap/master' into test-exchange-comp…
solotzg Jan 12, 2023
dbdb0ce
update tipb
solotzg Jan 12, 2023
0dc605f
fix gtests
solotzg Jan 12, 2023
7c3a70d
62
solotzg Jan 12, 2023
de28cc9
62 asan
solotzg Jan 13, 2023
23fe9b7
Merge branch 'test-exchange-compress' of github.com:solotzg/tiflash i…
solotzg Jan 13, 2023
1a8cc8f
support data compression in FineGrainedShuffle
solotzg Jan 13, 2023
efcb1be
63
solotzg Jan 13, 2023
a159775
64
solotzg Jan 15, 2023
77278d8
65
solotzg Jan 16, 2023
18e3ccd
66
solotzg Jan 16, 2023
95f9d9a
67
solotzg Jan 16, 2023
35457d5
68: statistic
solotzg Jan 16, 2023
e145ab7
combine FineGrainedShuffleWriter & HashPartitionWriterV1
solotzg Jan 17, 2023
22f242f
69
solotzg Jan 17, 2023
2869f1d
Merge remote-tracking branch 'pingcap/master' into test-exchange-comp…
solotzg Jan 17, 2023
e3724f1
rebase to pingcap/master
solotzg Jan 17, 2023
8848ede
remove useless code
solotzg Jan 18, 2023
89dd195
70
solotzg Jan 18, 2023
d371444
71
solotzg Jan 18, 2023
19ff254
72
solotzg Jan 18, 2023
cd91a7c
73
solotzg Jan 19, 2023
b7922d2
74
solotzg Jan 19, 2023
19facbf
75
solotzg Jan 19, 2023
2fa197b
76
solotzg Jan 19, 2023
94a8cbc
Merge remote-tracking branch 'pingcap/master' into test-exchange-comp…
solotzg Jan 28, 2023
6060432
77
solotzg Jan 23, 2023
57032d1
78
solotzg Jan 28, 2023
3c8155f
79
solotzg Jan 29, 2023
68921ee
80
solotzg Jan 30, 2023
e5376a5
Merge remote-tracking branch 'pingcap/master' into test-exchange-comp…
solotzg Jan 30, 2023
f290782
Merge branch 'master' into test-exchange-compress
solotzg Jan 30, 2023
4a1b2c4
Merge branch 'master' into test-exchange-compress
solotzg Jan 30, 2023
66df613
Merge branch 'master' into test-exchange-compress
ti-chi-bot Jan 30, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
1
Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>
solotzg committed Dec 12, 2022
commit ae30e4a5fc7a37e1b54dbd9d79093bc2ae6a5811
4 changes: 3 additions & 1 deletion dbms/src/Flash/Mpp/HashPartitionWriter.cpp
Original file line number Diff line number Diff line change
@@ -18,6 +18,7 @@
#include <Flash/Mpp/HashPartitionWriter.h>
#include <Flash/Mpp/MPPTunnelSet.h>

#include <cassert>
#include <cstddef>

#include "Common/Exception.h"
@@ -159,8 +160,9 @@ void HashPartitionWriter<ExchangeWriterPtr>::partitionAndEncodeThenWriteBlocks()
if (dest_block_rows > 0)
{
auto * codec_stream = chunk_codec_stream.get();
if (compress_chunk_codec_stream && !writer->getTunnels()[part_id]->isLocal())
if (tracked_packets[part_id]->getPacket().compress() != mpp::CompressMethod::NONE)
{
assert(compress_chunk_codec_stream);
// no need compress
codec_stream = compress_chunk_codec_stream.get();
}