Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](inverted index)Add exception check when write bkd index #39248

Merged
merged 1 commit into from
Aug 13, 2024
Merged

[fix](inverted index)Add exception check when write bkd index #39248

merged 1 commit into from
Aug 13, 2024

Conversation

qidaye
Copy link
Contributor

@qidaye qidaye commented Aug 12, 2024

Proposed changes

We are not catching the exception when add values in bkd_writer, if error throws, BE will run into segment fault.
So we add the exception check here to avoid coredump.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added the doing label Aug 12, 2024
@qidaye
Copy link
Contributor Author

qidaye commented Aug 12, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

airborne12
airborne12 approved these changes Aug 12, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 12, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40120 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 298d60279ba2b29262c028a885630d890f597316, data reload: false

------ Round 1 ----------------------------------
q1	17629	4781	4351	4351
q2	2020	182	171	171
q3	10493	1235	1127	1127
q4	10144	761	827	761
q5	7507	2580	2553	2553
q6	224	142	147	142
q7	974	604	602	602
q8	9220	1954	1907	1907
q9	8634	6577	6542	6542
q10	7050	2239	2168	2168
q11	474	249	243	243
q12	407	227	226	226
q13	18914	3021	3024	3021
q14	290	236	229	229
q15	530	471	493	471
q16	512	397	384	384
q17	991	732	726	726
q18	8291	7608	7416	7416
q19	4696	1047	1074	1047
q20	695	330	346	330
q21	5567	4781	4692	4692
q22	1143	1013	1011	1011
Total cold run time: 116405 ms
Total hot run time: 40120 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4452	4265	4278	4265
q2	374	261	272	261
q3	2861	2640	2617	2617
q4	1918	1628	1640	1628
q5	5281	5321	5273	5273
q6	222	134	133	133
q7	2071	1674	1730	1674
q8	3148	3414	3292	3292
q9	8445	8404	8342	8342
q10	3392	3192	3163	3163
q11	597	497	487	487
q12	785	593	593	593
q13	16648	2986	3005	2986
q14	295	260	282	260
q15	511	483	466	466
q16	476	417	432	417
q17	1786	1509	1485	1485
q18	7673	7541	7321	7321
q19	1665	1573	1567	1567
q20	1988	1793	1770	1770
q21	5180	5224	5107	5107
q22	1108	1039	1019	1019
Total cold run time: 70876 ms
Total hot run time: 54126 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 201972 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 298d60279ba2b29262c028a885630d890f597316, data reload: false

query1	906	371	359	359
query2	6463	1916	1897	1897
query3	6658	224	220	220
query4	30472	23266	23245	23245
query5	4188	490	479	479
query6	276	161	170	161
query7	4581	301	289	289
query8	250	212	201	201
query9	8580	2483	2475	2475
query10	509	479	441	441
query11	17616	14858	15068	14858
query12	148	99	98	98
query13	1641	388	370	370
query14	10460	7649	7090	7090
query15	258	220	237	220
query16	7845	474	532	474
query17	1764	581	562	562
query18	2035	301	289	289
query19	189	145	145	145
query20	109	105	111	105
query21	212	99	99	99
query22	4314	4144	3903	3903
query23	34064	33147	33444	33147
query24	12100	2645	2620	2620
query25	675	395	386	386
query26	1821	160	159	159
query27	2982	285	282	282
query28	7409	2069	2052	2052
query29	1130	428	430	428
query30	306	149	148	148
query31	954	738	754	738
query32	100	55	59	55
query33	751	313	294	294
query34	923	462	467	462
query35	978	852	850	850
query36	1059	921	941	921
query37	288	85	82	82
query38	4309	4126	4096	4096
query39	1441	1525	1390	1390
query40	281	117	117	117
query41	48	46	47	46
query42	120	97	101	97
query43	514	459	471	459
query44	1197	735	753	735
query45	240	208	224	208
query46	1084	696	743	696
query47	1901	1816	1791	1791
query48	390	307	307	307
query49	1214	426	429	426
query50	804	422	422	422
query51	6888	6723	6678	6678
query52	106	91	96	91
query53	261	193	184	184
query54	977	468	456	456
query55	81	77	77	77
query56	275	250	255	250
query57	1177	1106	1081	1081
query58	249	232	232	232
query59	2964	2746	2719	2719
query60	293	277	272	272
query61	117	215	96	96
query62	818	658	680	658
query63	227	191	178	178
query64	10499	2266	1761	1761
query65	3234	3155	3188	3155
query66	1370	337	331	331
query67	15827	14758	14943	14758
query68	7008	547	553	547
query69	476	440	405	405
query70	1111	1125	1155	1125
query71	574	279	278	278
query72	20185	16897	16934	16897
query73	825	338	339	338
query74	9363	8676	8828	8676
query75	5311	2697	2681	2681
query76	5134	1015	1024	1015
query77	778	302	302	302
query78	12910	9685	11286	9685
query79	9605	537	524	524
query80	1590	499	489	489
query81	589	227	222	222
query82	280	134	132	132
query83	345	147	140	140
query84	271	80	80	80
query85	768	321	270	270
query86	441	311	286	286
query87	4828	4528	4550	4528
query88	4877	2520	2514	2514
query89	399	290	291	290
query90	2202	198	199	198
query91	125	93	97	93
query92	62	51	49	49
query93	1162	534	530	530
query94	1204	292	288	288
query95	357	268	263	263
query96	590	275	276	275
query97	3276	3025	3083	3025
query98	215	208	201	201
query99	1528	1255	1284	1255
Total cold run time: 332309 ms
Total hot run time: 201972 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 298d60279ba2b29262c028a885630d890f597316, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.66	0.10	0.09
query5	0.52	0.51	0.51
query6	1.14	0.74	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.54	0.49	0.49
query10	0.55	0.52	0.54
query11	0.16	0.11	0.11
query12	0.14	0.12	0.13
query13	0.59	0.60	0.58
query14	0.76	0.79	0.78
query15	0.87	0.82	0.82
query16	0.37	0.37	0.37
query17	1.06	1.03	1.02
query18	0.23	0.22	0.21
query19	1.91	1.74	1.74
query20	0.01	0.02	0.01
query21	15.39	0.76	0.67
query22	4.08	6.99	1.83
query23	18.27	1.30	1.28
query24	2.10	0.23	0.21
query25	0.15	0.08	0.08
query26	0.31	0.21	0.21
query27	0.46	0.22	0.23
query28	13.32	1.01	1.01
query29	12.58	3.36	3.30
query30	0.24	0.05	0.05
query31	2.88	0.40	0.39
query32	3.31	0.48	0.46
query33	2.94	2.86	2.88
query34	17.20	4.41	4.40
query35	4.41	4.43	4.44
query36	0.65	0.48	0.46
query37	0.20	0.15	0.16
query38	0.16	0.16	0.16
query39	0.05	0.04	0.04
query40	0.15	0.12	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.06
query43	0.05	0.04	0.05
Total cold run time: 110 s
Total hot run time: 30.76 s

xiaokang
xiaokang approved these changes Aug 13, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang xiaokang merged commit dac6bab into apache:master Aug 13, 2024
31 of 35 checks passed
@qidaye qidaye deleted the fix_bkd_add_value_core branch August 13, 2024 02:45
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Aug 13, 2024
@qidaye
[fix](inverted index)Add exception check when write bkd index (apache…
c2aaa62 
…#39248)

We are not catching the exception when add values in `bkd_writer`, 
if error throws, BE will run into segment fault.
So we add the exception check here to avoid coredump.
@qidaye qidaye mentioned this pull request Aug 13, 2024
Merged
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Aug 13, 2024
@qidaye
[fix](inverted index)Add exception check when write bkd index (apache…
89a4ea7 
…#39248)

We are not catching the exception when add values in `bkd_writer`, 
if error throws, BE will run into segment fault.
So we add the exception check here to avoid coredump.
@qidaye qidaye mentioned this pull request Aug 13, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 13, 2024
@qidaye
[fix](inverted index)Add exception check when write bkd index (#39248) (
000ea20 
#39277)

bp #39248
xiaokang pushed a commit that referenced this pull request Aug 14, 2024
@qidaye
[fix](inverted index)Add exception check when write bkd index (#39248) (
9e9bdfb 
#39279)
@xiaokang xiaokang added p0_c and removed doing labels Aug 14, 2024
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
@qidaye @wyxxxcat
[fix](inverted index)Add exception check when write bkd index (apache…
477db9b 
…#39248)

We are not catching the exception when add values in `bkd_writer`, 
if error throws, BE will run into segment fault.
So we add the exception check here to avoid coredump.
dataroaring pushed a commit that referenced this pull request Aug 17, 2024
@qidaye @dataroaring
[fix](inverted index)Add exception check when write bkd index (#39248)
36fc3d5 
We are not catching the exception when add values in `bkd_writer`, 
if error throws, BE will run into segment fault.
So we add the exception check here to avoid coredump.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@airborne12 airborne12 airborne12 approved these changes

@xiaokang xiaokang xiaokang approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.2-merged p0_c reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@qidaye @doris-robot @xiaokang @airborne12 @yiguolei @dataroaring