[Enhancement] merge full/sample statistics collect #52693

Seaven · 2024-11-07T07:48:44Z

Why I'm doing:

want to merge sample/full statistics together, save in column_statistics table
want to improve full statistics collect performance
want to improve some metric in sample statsitcs

What I'm doing:

This is 1st PR
Fulll/Sample statistics collect process

split count(1)/min/max from full statistics query, use meta query to collect it.
collect ndv/count null by full statistics query only

Serious issues:
For high cardinality(NDV > 0.1%), sample statistics will get severely distorted NDV:

the unpartition table: the HLL-NDV max value should be the min(row_count * 0.1%, 20w)
the partition table: the HLL-NDV max value should be the partition_nums * min(row_count * 0.1%, 20w)

we will handle the question later, maybe not use hll or use other algorithm

modify code:

refactor some code, for merge sample/full statistic code later
add HyperStatisticsJob, to refactor the sample/full statistics job process
update column statistics query process，only query column_statistics table

the HyperStatisticsJob process, same as FullStatisticsJob

collect mertic by query, and save batch in FE (refactor to HyperQueryJob, FullQueryJob, and later will add SampleQueryJob)
insert batch value to column_statistics

RoadMap

next step:

support Analyze stmt work on partition
update Sample Statistics NDV algorithm
remove SampleStatisticsJob/FullStatisticsJob code

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

Signed-off-by: Seaven <[email protected]>

murphyatwork · 2024-11-18T06:54:05Z

fe/fe-core/src/main/java/com/starrocks/common/Config.java

@@ -2098,6 +2098,9 @@ public class Config extends ConfigBase {
            "we would use sample statistics instead of full statistics")
    public static double statistic_sample_collect_ratio_threshold_of_first_load = 0.1;

+    @ConfField(mutable = true)
+    public static boolean statistic_use_meta_statistics = true;


can you put some comment on it ?

it's temp config on main, I will remove it in next PR

murphyatwork · 2024-11-18T06:56:43Z

fe/fe-core/src/main/java/com/starrocks/statistic/HyperStatisticsCollectJob.java

+import java.util.List;
+import java.util.Map;
+
+public class HyperStatisticsCollectJob extends StatisticsCollectJob {


Hyper ?
I guess it's actually a regular sample-like collection job

I think is hyper, because Full/Sample always use it

murphyatwork · 2024-11-18T06:58:36Z

fe/fe-core/src/main/java/com/starrocks/statistic/base/ColumnClassifier.java

+
+import java.util.List;
+
+public class ColumnClassifier {


refactor from com.starrocks.statistic.sample.ColumnSampleManager, will save only one in next pr. For classifiy different column, different column type need different collect way

murphyatwork · 2024-11-18T06:59:14Z

fe/fe-core/src/main/java/com/starrocks/statistic/base/ColumnStats.java

+
+public abstract class ColumnStats {
+
+    protected final String columnName;


consider add columnId ? as SR has already supported rename column

refactor from com.starrocks.statistic.sample.ColumnStats, will save only one in next pr. I think add columnId a complex work, don't update it in this PR

murphyatwork · 2024-11-18T07:00:43Z

fe/fe-core/src/main/java/com/starrocks/statistic/base/SubFieldColumnStats.java

+import java.util.List;
+import java.util.stream.Collectors;
+
+public class SubFieldColumnStats extends PrimitiveTypeColumnStats {


how to use it ? how to store them in memory if there're thousands of fields in a struct ?

refactor from com.starrocks.statistic.sample.SubFieldColumnStats, will save only one in next pr. it's a tool-class for generate statistics SQL, don't store it in memory.

Signed-off-by: Seaven <[email protected]>

sonarcloud · 2024-11-18T13:01:37Z

Quality Gate failed

Failed conditions
6.0% Duplication on New Code (required ≤ 3%)
B Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

github-actions · 2024-11-18T13:32:25Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-11-18T13:32:45Z

[FE Incremental Coverage Report]

✅ pass : 547 / 648 (84.41%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/statistic/base/ComplexTypeColumnStats.java	2	9	22.22%	[28, 33, 38, 43, 48, 53, 58]
🔵	com/starrocks/statistic/sample/SampleInfo.java	7	17	41.18%	[44, 45, 46, 47, 48, 49, 50, 51, 52, 53]
🔵	com/starrocks/qe/SessionVariable.java	2	4	50.00%	[2583, 2584]
🔵	com/starrocks/statistic/StatisticExecutor.java	9	14	64.29%	[145, 146, 167, 168, 169]
🔵	com/starrocks/statistic/hyper/MetaQueryJob.java	52	74	70.27%	[66, 67, 68, 72, 73, 74, 109, 110, 111, 113, 114, 115, 117, 119, 120, 121, 122, 123, 124, 125, 126, 140]
🔵	com/starrocks/statistic/HyperStatisticsCollectJob.java	57	77	74.03%	[66, 95, 96, 97, 110, 111, 112, 113, 115, 116, 117, 119, 148, 149, 150, 151, 152, 154, 159, 161]
🔵	com/starrocks/statistic/base/TabletSampler.java	14	18	77.78%	[44, 45, 46, 70]
🔵	com/starrocks/statistic/base/PartitionSampler.java	54	63	85.71%	[76, 106, 107, 108, 109, 110, 111, 113, 115]
🔵	com/starrocks/statistic/StatisticsCollectJobFactory.java	13	15	86.67%	[120, 135]
🔵	com/starrocks/statistic/hyper/HyperQueryJob.java	124	135	91.85%	[82, 83, 84, 86, 87, 88, 89, 94, 211, 219, 221]
🔵	com/starrocks/statistic/hyper/HyperStatisticSQLs.java	49	53	92.45%	[35, 164, 169, 170]
🔵	com/starrocks/statistic/hyper/ConstQueryJob.java	31	33	93.94%	[45, 59]
🔵	com/starrocks/statistic/hyper/SampleQueryJob.java	15	16	93.75%	[48]
🔵	com/starrocks/statistic/base/SubFieldColumnStats.java	16	17	94.12%	[66]
🔵	com/starrocks/statistic/base/ColumnClassifier.java	44	45	97.78%	[83]
🔵	com/starrocks/statistic/base/PrimitiveTypeColumnStats.java	25	25	100.00%	[]
🔵	com/starrocks/common/Config.java	1	1	100.00%	[]
🔵	com/starrocks/qe/StmtExecutor.java	1	1	100.00%	[]
🔵	com/starrocks/statistic/base/ColumnStats.java	10	10	100.00%	[]
🔵	com/starrocks/statistic/hyper/FullQueryJob.java	21	21	100.00%	[]

github-actions · 2024-11-18T13:33:18Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

Seaven requested a review from a team as a code owner November 7, 2024 07:48

mergify bot assigned Seaven Nov 7, 2024

Seaven force-pushed the stat branch from 6338d87 to daf0a3a Compare November 7, 2024 09:34

github-actions bot added behavior_changed and removed behavior_changed labels Nov 7, 2024

Seaven force-pushed the stat branch from de3c991 to 1f75bcf Compare November 11, 2024 12:01

Seaven changed the title ~~[Enhancement] support meta statistics~~ [Enhancement] merge full/sample statistics collect Nov 13, 2024

Seaven force-pushed the stat branch from 8da4e89 to c9963a4 Compare November 13, 2024 10:30

Seaven added 14 commits November 13, 2024 20:45

[Enhancement] support meta statistics

0adb886

Signed-off-by: Seaven <[email protected]>

update

37876ff

Signed-off-by: Seaven <[email protected]>

--amend

c58270a

Signed-off-by: Seaven <[email protected]>

iiiii

4a31bc6

Signed-off-by: Seaven <[email protected]>

updateddd

1a5af26

Signed-off-by: Seaven <[email protected]>

fix

7495de4

Signed-off-by: Seaven <[email protected]>

fix ut

288b21e

Signed-off-by: Seaven <[email protected]>

update sample statistic

0b78445

Signed-off-by: Seaven <[email protected]>

fix

110d37c

Signed-off-by: Seaven <[email protected]>

fixxxx

4ee8a70

Signed-off-by: Seaven <[email protected]>

fix ut

cd1cd7d

Signed-off-by: Seaven <[email protected]>

fix

8e3e2e6

Signed-off-by: Seaven <[email protected]>

update name

e6cd7ac

Signed-off-by: Seaven <[email protected]>

uppp

243f61d

Signed-off-by: Seaven <[email protected]>

Seaven force-pushed the stat branch from c9963a4 to 243f61d Compare November 13, 2024 12:46

fixxxxx

bd1380a

Signed-off-by: Seaven <[email protected]>

Seaven requested a review from a team as a code owner November 14, 2024 08:58

add ut

910be34

Signed-off-by: Seaven <[email protected]>

Seaven force-pushed the stat branch from b2bc329 to 910be34 Compare November 14, 2024 08:58

Seaven added 3 commits November 14, 2024 18:41

fix ut

3e9a7fd

Signed-off-by: Seaven <[email protected]>

fixcomment

49d3afe

Signed-off-by: Seaven <[email protected]>

fixxxx

e44d417

Signed-off-by: Seaven <[email protected]>

murphyatwork reviewed Nov 18, 2024

View reviewed changes

fix comment

b11928e

Signed-off-by: Seaven <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] merge full/sample statistics collect #52693

[Enhancement] merge full/sample statistics collect #52693

Seaven commented Nov 7, 2024 •

edited

Loading

murphyatwork Nov 18, 2024

Seaven Nov 18, 2024

murphyatwork Nov 18, 2024

Seaven Nov 18, 2024

murphyatwork Nov 18, 2024

Seaven Nov 18, 2024 •

edited

Loading

murphyatwork Nov 18, 2024

Seaven Nov 18, 2024 •

edited

Loading

murphyatwork Nov 18, 2024

Seaven Nov 18, 2024

sonarcloud bot commented Nov 18, 2024

github-actions bot commented Nov 18, 2024

github-actions bot commented Nov 18, 2024

github-actions bot commented Nov 18, 2024


		public abstract class ColumnStats {

		protected final String columnName;

[Enhancement] merge full/sample statistics collect #52693

Are you sure you want to change the base?

[Enhancement] merge full/sample statistics collect #52693

Conversation

Seaven commented Nov 7, 2024 • edited Loading

Why I'm doing:

What I'm doing:

RoadMap

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Seaven Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Seaven Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Nov 18, 2024

Quality Gate failed

github-actions bot commented Nov 18, 2024

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Nov 18, 2024

[FE Incremental Coverage Report]

file detail

github-actions bot commented Nov 18, 2024

[BE Incremental Coverage Report]

Seaven commented Nov 7, 2024 •

edited

Loading

Seaven Nov 18, 2024 •

edited

Loading

Seaven Nov 18, 2024 •

edited

Loading