Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Refactor](SchemaCache) remove redundant Schema cache #40091

Merged
merged 1 commit into from
Sep 2, 2024
Merged

[Refactor](SchemaCache) remove redundant Schema cache #40091

merged 1 commit into from
Sep 2, 2024

Conversation

eldenmoon
Copy link
Member

We already introduced TabletSchema cache and it is enough at present, and Schema cache may introduce inconsistency bettween Schema and TabletSchema

Proposed changes

Issue Number: close #xxx

@eldenmoon
[Refactor](SchemaCache) remove redundant Schema cache
40935f9 
We already introduced TabletSchema cache and it is enough at present, and Schema cache may introduce inconsistency bettween Schema and TabletSchema
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon
Copy link
Member Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 37931 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 40935f9a3eaaea74b46f299120ecc9ebe97bd396, data reload: false

------ Round 1 ----------------------------------
q1	17629	4481	4281	4281
q2	2026	180	178	178
q3	11716	981	980	980
q4	10519	718	807	718
q5	7775	2865	2790	2790
q6	225	143	140	140
q7	962	617	609	609
q8	9346	2045	2075	2045
q9	7240	6504	6579	6504
q10	7004	2258	2169	2169
q11	467	257	264	257
q12	404	231	238	231
q13	17777	3126	3021	3021
q14	272	239	243	239
q15	533	483	484	483
q16	597	511	508	508
q17	985	622	708	622
q18	7216	6880	6798	6798
q19	1393	1080	1053	1053
q20	710	340	337	337
q21	4023	3043	2966	2966
q22	1112	1002	1032	1002
Total cold run time: 109931 ms
Total hot run time: 37931 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4382	4310	4288	4288
q2	391	279	280	279
q3	2918	2699	2675	2675
q4	1969	1612	1684	1612
q5	5630	5697	5680	5680
q6	224	140	146	140
q7	2268	1815	1887	1815
q8	3292	3393	3431	3393
q9	8864	8891	8818	8818
q10	3559	3402	3378	3378
q11	608	521	513	513
q12	814	683	702	683
q13	15540	3227	3277	3227
q14	318	296	309	296
q15	520	520	489	489
q16	640	580	573	573
q17	1841	1546	1539	1539
q18	8149	7984	7885	7885
q19	1694	1527	1457	1457
q20	2160	1893	1875	1875
q21	5736	5446	5474	5446
q22	1100	1051	1057	1051
Total cold run time: 72617 ms
Total hot run time: 57112 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192347 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 40935f9a3eaaea74b46f299120ecc9ebe97bd396, data reload: false

query1	1264	886	874	874
query2	6292	1939	1931	1931
query3	10618	4062	3954	3954
query4	59145	24494	23319	23319
query5	5403	510	507	507
query6	406	157	169	157
query7	5747	295	300	295
query8	284	207	204	204
query9	9163	2496	2477	2477
query10	476	270	253	253
query11	18593	14913	15270	14913
query12	155	101	103	101
query13	1594	398	374	374
query14	11082	7397	7373	7373
query15	265	176	198	176
query16	7546	491	492	491
query17	1101	560	557	557
query18	1896	294	293	293
query19	307	144	144	144
query20	118	109	109	109
query21	210	117	104	104
query22	4758	4683	4471	4471
query23	34098	33273	33554	33273
query24	5914	2850	2807	2807
query25	525	386	376	376
query26	694	156	152	152
query27	1805	292	278	278
query28	4030	2132	2121	2121
query29	665	403	433	403
query30	237	152	157	152
query31	963	744	771	744
query32	84	50	51	50
query33	425	284	275	275
query34	864	482	484	482
query35	832	725	742	725
query36	1032	936	924	924
query37	148	91	86	86
query38	3959	3912	3949	3912
query39	1436	1377	1388	1377
query40	199	112	111	111
query41	46	46	44	44
query42	117	95	94	94
query43	517	480	465	465
query44	1095	744	767	744
query45	190	162	161	161
query46	1105	766	774	766
query47	1842	1768	1787	1768
query48	395	295	295	295
query49	739	412	433	412
query50	812	418	448	418
query51	7106	7004	7173	7004
query52	97	85	85	85
query53	257	179	183	179
query54	567	454	457	454
query55	78	77	78	77
query56	269	251	265	251
query57	1190	1056	1081	1056
query58	220	225	224	224
query59	2888	2733	2799	2733
query60	289	289	275	275
query61	108	101	100	100
query62	770	649	661	649
query63	216	179	186	179
query64	2870	678	661	661
query65	3194	3132	3150	3132
query66	704	348	336	336
query67	15350	15011	15332	15011
query68	2952	582	586	582
query69	398	282	280	280
query70	1176	1103	1141	1103
query71	342	275	275	275
query72	5364	3721	3978	3721
query73	754	336	337	336
query74	9218	8796	8813	8796
query75	3340	2676	2745	2676
query76	1496	1028	989	989
query77	531	325	313	313
query78	9637	9054	9166	9054
query79	1042	542	542	542
query80	690	520	515	515
query81	465	239	240	239
query82	252	145	142	142
query83	169	153	146	146
query84	261	73	79	73
query85	687	294	289	289
query86	318	292	297	292
query87	4363	4296	4321	4296
query88	3493	2316	2332	2316
query89	378	286	288	286
query90	1962	191	189	189
query91	122	100	163	100
query92	62	50	51	50
query93	1063	534	540	534
query94	710	279	305	279
query95	323	262	256	256
query96	593	268	273	268
query97	3166	3068	3085	3068
query98	225	201	212	201
query99	1468	1257	1243	1243
Total cold run time: 304384 ms
Total hot run time: 192347 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.2 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 40935f9a3eaaea74b46f299120ecc9ebe97bd396, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.09	0.08
query5	0.49	0.49	0.49
query6	1.14	0.73	0.73
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.56	0.49	0.48
query10	0.52	0.54	0.53
query11	0.17	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.60	0.60
query14	2.02	2.11	2.08
query15	0.86	0.83	0.82
query16	0.36	0.39	0.37
query17	1.05	1.00	1.00
query18	0.21	0.21	0.20
query19	1.89	1.86	1.86
query20	0.01	0.00	0.01
query21	15.40	0.67	0.66
query22	4.04	6.94	1.98
query23	18.26	1.39	1.28
query24	2.10	0.23	0.22
query25	0.17	0.08	0.08
query26	0.27	0.19	0.17
query27	0.08	0.08	0.08
query28	13.30	1.02	0.99
query29	12.60	3.31	3.34
query30	0.25	0.05	0.05
query31	2.87	0.40	0.40
query32	3.28	0.48	0.47
query33	3.01	3.01	3.02
query34	17.05	4.39	4.39
query35	4.48	4.48	4.40
query36	0.66	0.48	0.46
query37	0.18	0.16	0.15
query38	0.16	0.15	0.16
query39	0.04	0.04	0.04
query40	0.16	0.13	0.12
query41	0.09	0.05	0.06
query42	0.06	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 110.71 s
Total hot run time: 32.2 s

xiaokang
xiaokang approved these changes Sep 1, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 1, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 1, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 1, 2024

PR approved by anyone and no changes requested.

@eldenmoon eldenmoon merged commit 28d777e into apache:master Sep 2, 2024
28 of 31 checks passed
@eldenmoon eldenmoon deleted the rm-scc branch September 2, 2024 04:18
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Sep 2, 2024
@eldenmoon
[Refactor](SchemaCache) remove redundant Schema cache (apache#40091)
05242e3 
We already introduced TabletSchema cache and it is enough at present,
and Schema cache may introduce inconsistency bettween Schema and
TabletSchema
@eldenmoon eldenmoon mentioned this pull request Sep 2, 2024
Merged
eldenmoon added a commit that referenced this pull request Sep 2, 2024
@eldenmoon
[pick](SchemaCache) remove redundant Schema cache (#40257)
4a91b2d 
#40091
dataroaring pushed a commit that referenced this pull request Sep 3, 2024
@eldenmoon @dataroaring
[Refactor](SchemaCache) remove redundant Schema cache (#40091)
d0927b8 
We already introduced TabletSchema cache and it is enough at present,
and Schema cache may introduce inconsistency bettween Schema and
TabletSchema
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Sep 4, 2024
@eldenmoon
[Refactor](SchemaCache) remove redundant Schema cache (apache#40091)
4b97550 
We already introduced TabletSchema cache and it is enough at present,
and Schema cache may introduce inconsistency bettween Schema and
TabletSchema
@eldenmoon eldenmoon mentioned this pull request Sep 4, 2024
Merged
eldenmoon added a commit that referenced this pull request Sep 5, 2024
@eldenmoon
[Refactor](SchemaCache) remove redundant Schema cache (#40350)
ef967c2 
#40091
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qidaye qidaye qidaye approved these changes

@xiaokang xiaokang xiaokang approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@eldenmoon @doris-robot @xiaokang @qidaye @wm1581066 @dataroaring