[feature](profilev2) Preliminary support for profilev2. #24881

Mryange · 2023-09-25T12:20:54Z

Proposed changes

You can set the level of counters on the backend using ADD_COUNTER_WITH_LEVEL/ADD_TIMER_WITH_LEVEL. The profile can then merge counters with level 1.
set profile_level = 1;
such as
sql
select count(*) from customer join item on c_customer_sk = i_item_sk

profile

Simple  profile  
  
  PLAN  FRAGMENT  0
    OUTPUT  EXPRS:
        count(*)
    PARTITION:  UNPARTITIONED

    VRESULT  SINK
          MYSQL_PROTOCAL


    7:VAGGREGATE  (merge  finalize)
    |    output:  count(partial_count(*))[#44]
    |    group  by:  
    |    cardinality=1
    |    TotalTime:  avg  725.608us,  max  725.608us,  min  725.608us
    |    RowsReturned:  1
    |    
    6:VEXCHANGE
          offset:  0
          TotalTime:  avg  52.411us,  max  52.411us,  min  52.411us
          RowsReturned:  8

PLAN  FRAGMENT  1

    PARTITION:  HASH_PARTITIONED:  c_customer_sk

    STREAM  DATA  SINK
        EXCHANGE  ID:  06
        UNPARTITIONED

        TotalTime:  avg  106.263us,  max  118.38us,  min  81.403us
        BlocksSent:  8

    5:VAGGREGATE  (update  serialize)
    |    output:  partial_count(*)[#43]
    |    group  by:  
    |    cardinality=1
    |    TotalTime:  avg  679.296us,  max  739.395us,  min  554.904us
    |    BuildTime:  avg  33.198us,  max  48.387us,  min  28.880us
    |    ExecTime:  avg  27.633us,  max  40.278us,  min  24.537us
    |    RowsReturned:  8
    |    
    4:VHASH  JOIN
    |    join  op:  INNER  JOIN(PARTITIONED)[]
    |    equal  join  conjunct:  c_customer_sk  =  i_item_sk
    |    runtime  filters:  RF000[bloom]  <-  i_item_sk(18000/16384/1048576)
    |    cardinality=17,740
    |    vec  output  tuple  id:  3
    |    vIntermediate  tuple  ids:  2  
    |    hash  output  slot  ids:  22  
    |    RowsReturned:  18.0K  (18000)
    |    ProbeRows:  18.0K  (18000)
    |    ProbeTime:  avg  862.308us,  max  1.576ms,  min  666.28us
    |    BuildRows:  18.0K  (18000)
    |    BuildTime:  avg  3.8ms,  max  3.860ms,  min  2.317ms
    |    
    |----1:VEXCHANGE
    |              offset:  0
    |              TotalTime:  avg  48.822us,  max  67.459us,  min  30.380us
    |              RowsReturned:  18.0K  (18000)
    |        
    3:VEXCHANGE
          offset:  0
          TotalTime:  avg  33.162us,  max  39.480us,  min  28.854us
          RowsReturned:  18.0K  (18000)

PLAN  FRAGMENT  2

    PARTITION:  HASH_PARTITIONED:  c_customer_id

    STREAM  DATA  SINK
        EXCHANGE  ID:  03
        HASH_PARTITIONED:  c_customer_sk

        TotalTime:  avg  753.954us,  max  1.210ms,  min  499.470us
        BlocksSent:  64

    2:VOlapScanNode
          TABLE:  default_cluster:tpcds.customer(customer),  PREAGGREGATION:  ON
          runtime  filters:  RF000[bloom]  ->  c_customer_sk
          partitions=1/1,  tablets=12/12,  tabletList=1550745,1550747,1550749  ...
          cardinality=100000,  avgRowSize=0.0,  numNodes=1
          pushAggOp=NONE
          TotalTime:  avg  18.417us,  max  41.319us,  min  10.189us
          RowsReturned:  18.0K  (18000)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

Mryange · 2023-09-25T12:21:13Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2023-09-25T12:28:45Z

be/src/util/runtime_profile.cpp

@@ -416,7 +420,7 @@ RuntimeProfile::Counter* RuntimeProfile::add_counter(const std::string& name, TU

    DCHECK(parent_counter_name == ROOT_COUNTER ||
           _counter_map.find(parent_counter_name) != _counter_map.end());
-    Counter* counter = _pool->add(new Counter(type, 0));
+    Counter* counter = _pool->add(new Counter(type, 0, level));


warning: variable 'counter' is not initialized [cppcoreguidelines-init-variables]

Suggested change

Counter* counter = _pool->add(new Counter(type, 0, level));

Counter* counter = nullptr = _pool->add(new Counter(type, 0, level));

github-actions · 2023-09-25T12:28:45Z

be/src/util/runtime_profile.h

+        _metadata = md;
+    }
+
+    bool is_set_metadata() const { return _is_set_metadata; }


warning: function 'is_set_metadata' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change

bool is_set_metadata() const { return _is_set_metadata; }

[[nodiscard]] bool is_set_metadata() const { return _is_set_metadata; }

doris-robot · 2023-09-25T13:21:32Z

TeamCity be ut coverage result:
Function Coverage: 36.33% (8115/22339)
Line Coverage: 28.45% (64914/228168)
Region Coverage: 27.34% (33660/123137)
Branch Coverage: 23.99% (17183/71624)
Coverage Report: http://coverage.selectdb-in.cc/coverage/68ac0dc47efd796038ae36ef3ed27a4e5804ad24_68ac0dc47efd796038ae36ef3ed27a4e5804ad24/report/index.html

doris-robot · 2023-09-25T13:25:22Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.87 seconds
stream load tsv: 558 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162099372 Bytes

yiguolei · 2023-09-26T06:16:33Z

gensrc/thrift/RuntimeProfile.thrift

@@ -25,6 +25,7 @@ struct TCounter {
  1: required string name
  2: required Metrics.TUnit type
  3: required i64 value 
+  4: required i64 level


should always be optional, never use required.

yiguolei · 2023-09-26T08:04:37Z

be/src/util/runtime_profile.h

    Counter* add_counter(const std::string& name, TUnit::type type) {
        return add_counter(name, type, "");
    }

+    Counter* add_counter_with_levle(const std::string& name, TUnit::type type, int64_t level) {


Mryange · 2023-09-26T08:33:19Z

run buildall

doris-robot · 2023-09-26T09:31:21Z

TeamCity be ut coverage result:
Function Coverage: 36.44% (8160/22391)
Line Coverage: 28.57% (65378/228874)
Region Coverage: 27.42% (33869/123518)
Branch Coverage: 24.06% (17279/71830)
Coverage Report: http://coverage.selectdb-in.cc/coverage/42b2aaf602c5b0a88de50cfeba38a6b5cb035db8_42b2aaf602c5b0a88de50cfeba38a6b5cb035db8/report/index.html

doris-robot · 2023-09-26T09:34:58Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.95 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162276507 Bytes

yiguolei · 2023-09-26T11:15:13Z

fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java

@@ -1217,7 +1217,6 @@ public String getNodeExplainString(String prefix, TExplainLevel detailLevel) {
        if (isPointQuery()) {
            output.append(prefix).append("SHORT-CIRCUIT");
        }
-
        return output.toString();


not modify useless file

yiguolei · 2023-09-26T11:18:48Z

fe/fe-core/src/main/java/org/apache/doris/common/util/RuntimeProfile.java

@@ -534,13 +508,18 @@ public String toString() {
    }

    public String getSimpleString() {


rename to getLevel1Profile

yiguolei · 2023-09-26T11:20:37Z

fe/fe-core/src/main/java/org/apache/doris/common/util/RuntimeProfile.java

        if (node.timestamp != -1 && node.timestamp < timestamp) {
            return;
        }
+        if (node.isSetMetadata()) {
+            this.nodeid = (int) node.getMetadata();


I think we should not depend on it.
Add a specific field in profile object named plan_node_id.
metadata is too normal.

Mryange · 2023-09-26T12:01:20Z

run buildall

doris-robot · 2023-09-26T12:57:58Z

TeamCity be ut coverage result:
Function Coverage: 36.34% (8142/22406)
Line Coverage: 28.46% (65149/228887)
Region Coverage: 27.42% (33783/123200)
Branch Coverage: 24.08% (17246/71624)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d15776560fa5a24ee9ebb3cb95458e7affb1e350_d15776560fa5a24ee9ebb3cb95458e7affb1e350/report/index.html

doris-robot · 2023-09-26T13:08:29Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.79 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17161993743 Bytes

Mryange · 2023-09-26T14:51:27Z

run buildall

doris-robot · 2023-09-26T15:38:09Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.3 seconds
stream load tsv: 559 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162175541 Bytes

doris-robot · 2023-09-26T15:55:55Z

TeamCity be ut coverage result:
Function Coverage: 36.34% (8146/22416)
Line Coverage: 28.45% (65176/229109)
Region Coverage: 27.40% (33790/123301)
Branch Coverage: 24.07% (17250/71666)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d15776560fa5a24ee9ebb3cb95458e7affb1e350_d15776560fa5a24ee9ebb3cb95458e7affb1e350/report/index.html

yiguolei · 2023-09-28T02:22:35Z

be/src/util/runtime_profile.cpp

@@ -278,6 +278,9 @@ RuntimeProfile* RuntimeProfile::create_child(const std::string& name, bool inden
    std::lock_guard<std::mutex> l(_children_lock);
    DCHECK(_child_map.find(name) == _child_map.end());
    RuntimeProfile* child = _pool->add(new RuntimeProfile(name));
+    if (this->is_set_metadata()) {


这样做可能是错的，因为如果meta data 里是node id，那么child 会有自己的node id

yiguolei · 2023-10-04T09:15:06Z

run buildall

yiguolei · 2023-10-06T08:58:53Z

run buildall

doris-robot · 2023-10-06T10:06:18Z

TeamCity be ut coverage result:
Function Coverage: 36.30% (8150/22450)
Line Coverage: 28.43% (65172/229256)
Region Coverage: 27.36% (33749/123339)
Branch Coverage: 24.04% (17221/71632)
Coverage Report: http://coverage.selectdb-in.cc/coverage/00a7361b68150fa4c7582a8b6f170db019d0ff17_00a7361b68150fa4c7582a8b6f170db019d0ff17/report/index.html

doris-robot · 2023-10-06T10:10:39Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.37 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162476842 Bytes

yiguolei

LGTM

github-actions · 2023-10-07T02:10:17Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-10-07T02:10:19Z

PR approved by anyone and no changes requested.

Gabriel39

LGTM

You can set the level of counters on the backend using ADD_COUNTER_WITH_LEVEL/ADD_TIMER_WITH_LEVEL. The profile can then merge counters with level 1. set profile_level = 1; such as sql select count(*) from customer join item on c_customer_sk = i_item_sk profile Simple profile PLAN FRAGMENT 0 OUTPUT EXPRS: count(*) PARTITION: UNPARTITIONED VRESULT SINK MYSQL_PROTOCAL 7:VAGGREGATE (merge finalize) | output: count(partial_count(*))[apache#44] | group by: | cardinality=1 | TotalTime: avg 725.608us, max 725.608us, min 725.608us | RowsReturned: 1 | 6:VEXCHANGE offset: 0 TotalTime: avg 52.411us, max 52.411us, min 52.411us RowsReturned: 8 PLAN FRAGMENT 1 PARTITION: HASH_PARTITIONED: c_customer_sk STREAM DATA SINK EXCHANGE ID: 06 UNPARTITIONED TotalTime: avg 106.263us, max 118.38us, min 81.403us BlocksSent: 8 5:VAGGREGATE (update serialize) | output: partial_count(*)[apache#43] | group by: | cardinality=1 | TotalTime: avg 679.296us, max 739.395us, min 554.904us | BuildTime: avg 33.198us, max 48.387us, min 28.880us | ExecTime: avg 27.633us, max 40.278us, min 24.537us | RowsReturned: 8 | 4:VHASH JOIN | join op: INNER JOIN(PARTITIONED)[] | equal join conjunct: c_customer_sk = i_item_sk | runtime filters: RF000[bloom] <- i_item_sk(18000/16384/1048576) | cardinality=17,740 | vec output tuple id: 3 | vIntermediate tuple ids: 2 | hash output slot ids: 22 | RowsReturned: 18.0K (18000) | ProbeRows: 18.0K (18000) | ProbeTime: avg 862.308us, max 1.576ms, min 666.28us | BuildRows: 18.0K (18000) | BuildTime: avg 3.8ms, max 3.860ms, min 2.317ms | |----1:VEXCHANGE | offset: 0 | TotalTime: avg 48.822us, max 67.459us, min 30.380us | RowsReturned: 18.0K (18000) | 3:VEXCHANGE offset: 0 TotalTime: avg 33.162us, max 39.480us, min 28.854us RowsReturned: 18.0K (18000) PLAN FRAGMENT 2 PARTITION: HASH_PARTITIONED: c_customer_id STREAM DATA SINK EXCHANGE ID: 03 HASH_PARTITIONED: c_customer_sk TotalTime: avg 753.954us, max 1.210ms, min 499.470us BlocksSent: 64 2:VOlapScanNode TABLE: default_cluster:tpcds.customer(customer), PREAGGREGATION: ON runtime filters: RF000[bloom] -> c_customer_sk partitions=1/1, tablets=12/12, tabletList=1550745,1550747,1550749 ... cardinality=100000, avgRowSize=0.0, numNodes=1 pushAggOp=NONE TotalTime: avg 18.417us, max 41.319us, min 10.189us RowsReturned: 18.0K (18000) --------- Co-authored-by: yiguolei <[email protected]>

Mryange added 3 commits September 25, 2023 18:03

update

f7a3f2e

rebase

88d69b1

format

68ac0dc

github-actions bot reviewed Sep 25, 2023

View reviewed changes

yiguolei added the profilev2 label Sep 26, 2023

yiguolei reviewed Sep 26, 2023

View reviewed changes

use profile level

42b2aaf

yiguolei reviewed Sep 26, 2023

View reviewed changes

format

d157765

yiguolei reviewed Sep 28, 2023

View reviewed changes

Merge branch 'master' into dev-57

fe4bfc1

Merge branch 'master' into dev-57

00a7361

yiguolei approved these changes Oct 7, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 7, 2023

github-actions bot added the reviewed label Oct 7, 2023

Gabriel39 approved these changes Oct 7, 2023

View reviewed changes

yiguolei merged commit 0631ed6 into apache:master Oct 7, 2023
15 of 17 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature](profilev2) Preliminary support for profilev2. #24881

[feature](profilev2) Preliminary support for profilev2. #24881

Mryange commented Sep 25, 2023 •

edited

Loading

Mryange commented Sep 25, 2023

github-actions bot left a comment

github-actions bot Sep 25, 2023

github-actions bot Sep 25, 2023

doris-robot commented Sep 25, 2023

doris-robot commented Sep 25, 2023

yiguolei Sep 26, 2023

yiguolei Sep 26, 2023

Mryange commented Sep 26, 2023

doris-robot commented Sep 26, 2023

doris-robot commented Sep 26, 2023

yiguolei Sep 26, 2023

yiguolei Sep 26, 2023

yiguolei Sep 26, 2023

Mryange commented Sep 26, 2023

doris-robot commented Sep 26, 2023

doris-robot commented Sep 26, 2023

Mryange commented Sep 26, 2023

doris-robot commented Sep 26, 2023

doris-robot commented Sep 26, 2023

yiguolei Sep 28, 2023

yiguolei commented Oct 4, 2023

yiguolei commented Oct 6, 2023

doris-robot commented Oct 6, 2023

doris-robot commented Oct 6, 2023

yiguolei left a comment

github-actions bot commented Oct 7, 2023

github-actions bot commented Oct 7, 2023

Gabriel39 left a comment

	Counter* counter = _pool->add(new Counter(type, 0, level));
	Counter* counter = nullptr = _pool->add(new Counter(type, 0, level));

	bool is_set_metadata() const { return _is_set_metadata; }
	[[nodiscard]] bool is_set_metadata() const { return _is_set_metadata; }

		@@ -534,13 +508,18 @@ public String toString() {
		}

		public String getSimpleString() {

[feature](profilev2) Preliminary support for profilev2. #24881

[feature](profilev2) Preliminary support for profilev2. #24881

Conversation

Mryange commented Sep 25, 2023 • edited Loading

Proposed changes

Further comments

Mryange commented Sep 25, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Sep 25, 2023

Choose a reason for hiding this comment

github-actions bot Sep 25, 2023

Choose a reason for hiding this comment

doris-robot commented Sep 25, 2023

doris-robot commented Sep 25, 2023

yiguolei Sep 26, 2023

Choose a reason for hiding this comment

yiguolei Sep 26, 2023

Choose a reason for hiding this comment

Mryange commented Sep 26, 2023

doris-robot commented Sep 26, 2023

doris-robot commented Sep 26, 2023

yiguolei Sep 26, 2023

Choose a reason for hiding this comment

yiguolei Sep 26, 2023

Choose a reason for hiding this comment

yiguolei Sep 26, 2023

Choose a reason for hiding this comment

Mryange commented Sep 26, 2023

doris-robot commented Sep 26, 2023

doris-robot commented Sep 26, 2023

Mryange commented Sep 26, 2023

doris-robot commented Sep 26, 2023

doris-robot commented Sep 26, 2023

yiguolei Sep 28, 2023

Choose a reason for hiding this comment

yiguolei commented Oct 4, 2023

yiguolei commented Oct 6, 2023

doris-robot commented Oct 6, 2023

doris-robot commented Oct 6, 2023

yiguolei left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 7, 2023

github-actions bot commented Oct 7, 2023

Gabriel39 left a comment

Choose a reason for hiding this comment

Mryange commented Sep 25, 2023 •

edited

Loading