Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Update](inverted index) add config for inverted index query cache shards #32666

Merged
merged 1 commit into from
Mar 26, 2024

Conversation

airborne12
Copy link
Member

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@airborne12
Copy link
Member Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 37866 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 522b2d2deeaa886975695f6185dbf122e642627f, data reload: false

------ Round 1 ----------------------------------
q1	17625	4295	4187	4187
q2	2128	168	161	161
q3	10564	1211	1218	1211
q4	10218	803	818	803
q5	7451	2947	2962	2947
q6	201	121	124	121
q7	1048	599	572	572
q8	9345	2052	1996	1996
q9	7155	6470	6435	6435
q10	8414	3398	3475	3398
q11	430	225	226	225
q12	399	202	199	199
q13	17810	2869	2825	2825
q14	232	210	206	206
q15	506	461	455	455
q16	492	375	377	375
q17	969	527	636	527
q18	7246	6452	6543	6452
q19	2425	1453	1409	1409
q20	535	264	246	246
q21	3640	2946	2823	2823
q22	348	293	303	293
Total cold run time: 109181 ms
Total hot run time: 37866 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4134	4096	4118	4096
q2	339	229	231	229
q3	2993	2835	2854	2835
q4	1840	1550	1536	1536
q5	5227	5302	5272	5272
q6	194	116	118	116
q7	2243	1890	1830	1830
q8	3158	3291	3293	3291
q9	8582	8533	8539	8533
q10	3685	3710	3668	3668
q11	547	441	443	441
q12	720	570	579	570
q13	16914	2893	2896	2893
q14	274	253	255	253
q15	500	467	447	447
q16	472	423	428	423
q17	1744	1503	1481	1481
q18	7492	7124	7174	7124
q19	1603	1485	1580	1485
q20	1932	1733	1722	1722
q21	4810	4685	4653	4653
q22	528	464	438	438
Total cold run time: 69931 ms
Total hot run time: 53336 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.28% (8733/24756)
Line Coverage: 27.08% (71522/264142)
Region Coverage: 26.32% (37107/140995)
Branch Coverage: 23.21% (18975/81742)
Coverage Report: http://coverage.selectdb-in.cc/coverage/522b2d2deeaa886975695f6185dbf122e642627f_522b2d2deeaa886975695f6185dbf122e642627f/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 181428 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 522b2d2deeaa886975695f6185dbf122e642627f, data reload: false

query1	946	370	354	354
query2	7471	2089	1987	1987
query3	6708	218	215	215
query4	31346	20659	20553	20553
query5	4338	520	484	484
query6	282	175	172	172
query7	4628	299	291	291
query8	228	162	172	162
query9	9193	2264	2247	2247
query10	552	244	251	244
query11	14616	14102	14103	14102
query12	142	85	86	85
query13	1626	436	418	418
query14	11737	10722	11014	10722
query15	255	202	197	197
query16	8267	266	267	266
query17	1964	585	550	550
query18	2112	323	294	294
query19	344	167	162	162
query20	91	89	89	89
query21	202	133	126	126
query22	4731	4419	4478	4419
query23	32050	31057	31100	31057
query24	10685	2831	2817	2817
query25	615	392	389	389
query26	1181	159	159	159
query27	2524	344	377	344
query28	6897	1901	1893	1893
query29	889	675	629	629
query30	304	155	149	149
query31	943	755	736	736
query32	97	58	55	55
query33	778	285	262	262
query34	1016	463	485	463
query35	828	632	612	612
query36	997	872	871	871
query37	127	77	75	75
query38	3522	3408	3445	3408
query39	1374	1391	1355	1355
query40	213	109	111	109
query41	48	47	45	45
query42	103	95	98	95
query43	489	464	449	449
query44	1075	743	735	735
query45	262	246	256	246
query46	1088	693	694	693
query47	1672	1597	1607	1597
query48	451	361	350	350
query49	1092	401	322	322
query50	763	367	368	367
query51	6685	6630	6565	6565
query52	106	95	90	90
query53	339	272	276	272
query54	304	248	240	240
query55	92	76	83	76
query56	257	219	237	219
query57	1085	1000	999	999
query58	234	198	200	198
query59	2793	2589	2585	2585
query60	264	244	240	240
query61	97	93	91	91
query62	589	389	377	377
query63	300	278	270	270
query64	5548	3825	3843	3825
query65	3067	3013	3021	3013
query66	870	352	348	348
query67	14821	14379	14348	14348
query68	5427	518	522	518
query69	561	377	380	377
query70	1245	1128	1118	1118
query71	425	283	283	283
query72	6395	2855	2651	2651
query73	721	328	316	316
query74	6941	6556	6555	6555
query75	3417	2870	2747	2747
query76	3578	902	875	875
query77	626	280	271	271
query78	10542	9718	9694	9694
query79	9102	525	530	525
query80	2123	410	399	399
query81	565	207	217	207
query82	1350	199	207	199
query83	333	153	155	153
query84	301	83	87	83
query85	1701	316	305	305
query86	493	289	292	289
query87	3798	3523	3542	3523
query88	5285	2378	2358	2358
query89	555	365	369	365
query90	1987	176	174	174
query91	171	137	136	136
query92	62	48	47	47
query93	6959	510	496	496
query94	1247	184	175	175
query95	432	324	333	324
query96	620	277	276	276
query97	3075	2867	2888	2867
query98	243	219	205	205
query99	1128	740	757	740
Total cold run time: 302721 ms
Total hot run time: 181428 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 522b2d2deeaa886975695f6185dbf122e642627f with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.2 seconds inserted 10000000 Rows, about 450K ops/s

Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 22, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon eldenmoon merged commit 8a4180d into apache:master Mar 26, 2024
27 of 31 checks passed
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants