[Opt](TabletSchema) reuse TabletColumn info to reduce mem #42448
Conversation
eldenmoon
commented
Oct 24, 2024
•
eldenmoon
commented
- When there are a large number of identical TabletColumns in the cluster, which usually occurs when VARIANT type columns are modified and added, each Rowset has an individual TabletSchema. Excessive TabletSchemas can lead to significant memory overhead. Reusing memory for identical TabletColumns would greatly reduce this memory consumption.
- Serialized TabletSchema as LRU cache key could also increase memusage when large sets of schemas are in LRU cache, so inorder to reduce the memory footprint we just record the key signature caculated by generating an UUID by hash algorithm， and lookup the key signature in LRU cache， and check the key in case of hash collision
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
eldenmoon
force-pushed
the
be_column_mem
branch
from
9b0eb93
to
02bec4e
Compare
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| @@ -942,7 +943,8 @@ void TabletSchema::clear_columns() { | |||
| _cols.clear(); | |||
| } | |||
|
|
|||
| void TabletSchema::init_from_pb(const TabletSchemaPB& schema, bool ignore_extracted_columns) { | |||
| void TabletSchema::init_from_pb(const TabletSchemaPB& schema, bool ignore_extracted_columns, | |||
There was a problem hiding this comment.
warning: function 'init_from_pb' exceeds recommended size/complexity thresholds [readability-function-size]
void TabletSchema::init_from_pb(const TabletSchemaPB& schema, bool ignore_extracted_columns,
^Additional context
be/src/olap/tablet_schema.cpp:945: 85 lines including whitespace and comments (threshold 80)
void TabletSchema::init_from_pb(const TabletSchemaPB& schema, bool ignore_extracted_columns,
^
eldenmoon
force-pushed
the
be_column_mem
branch
3 times, most recently
from
101870d
to
a98b6c0
Compare
|
run buildall |
eldenmoon
changed the title
eldenmoon
changed the title
eldenmoon
force-pushed
the
be_column_mem
branch
2 times, most recently
from
a6ae48d
to
8e31fa5
Compare
|
run buildall |
eldenmoon
force-pushed
the
be_column_mem
branch
from
8e31fa5
to
85d7e58
Compare
|
run buildall |
1 similar comment
|
run buildall |
|
TeamCity be ut coverage result: |
|
run buildall |
|
TeamCity be ut coverage result: |
eldenmoon
force-pushed
the
be_column_mem
branch
2 times, most recently
from
d0f8a69
to
0761331
Compare
|
run buildall |
|
TeamCity be ut coverage result: |
|
run buildall |
1. When there are a large number of identical TabletColumns in the cluster, which usually occurs when VARIANT type columns are modified and added, each Rowset has an individual TabletSchema. Excessive TabletSchemas can lead to significant memory overhead. Reusing memory for identical TabletColumns would greatly reduce this memory consumption. 2. Serialized TabletSchema as LRU cache key could also increase memusage when large sets of schemas are in LRU cache, so inorder to reduce the memory footprint we just record the key signature caculated by generating an UUID by hash algorithm， and lookup the key signature in LRU cache， and check the key in case of hash collision
| } | ||
|
|
||
| void TabletSchema::init_from_pb(const TabletSchemaPB& schema, bool ignore_extracted_columns) { | ||
| void TabletSchema::init_from_pb(const TabletSchemaPB& schema, bool ignore_extracted_columns, |
There was a problem hiding this comment.
warning: function 'init_from_pb' exceeds recommended size/complexity thresholds [readability-function-size]
void TabletSchema::init_from_pb(const TabletSchemaPB& schema, bool ignore_extracted_columns,
^Additional context
be/src/olap/tablet_schema.cpp:956: 88 lines including whitespace and comments (threshold 80)
void TabletSchema::init_from_pb(const TabletSchemaPB& schema, bool ignore_extracted_columns,
^
eldenmoon
force-pushed
the
be_column_mem
branch
from
9f10eaf
to
49747e9
Compare
|
run buildall |
|
TeamCity be ut coverage result: |
|
run buildall |
| } | ||
|
|
||
| void TabletSchema::init_from_pb(const TabletSchemaPB& schema, bool ignore_extracted_columns) { | ||
| void TabletSchema::init_from_pb(const TabletSchemaPB& schema, bool ignore_extracted_columns, |
There was a problem hiding this comment.
warning: function 'init_from_pb' exceeds recommended size/complexity thresholds [readability-function-size]
void TabletSchema::init_from_pb(const TabletSchemaPB& schema, bool ignore_extracted_columns,
^Additional context
be/src/olap/tablet_schema.cpp:963: 88 lines including whitespace and comments (threshold 80)
void TabletSchema::init_from_pb(const TabletSchemaPB& schema, bool ignore_extracted_columns,
^
|
TeamCity be ut coverage result: |
| FieldType::OLAP_FIELD_TYPE_ARRAY, | ||
| FieldType::OLAP_FIELD_TYPE_FLOAT, | ||
| }; | ||
| if (column.is_extracted_column() && (invalid_types.contains(column.type()))) { |
There was a problem hiding this comment.
why check is_extracted_column?
| static std::set<FieldType> invalid_types = { | ||
| FieldType::OLAP_FIELD_TYPE_DOUBLE, | ||
| FieldType::OLAP_FIELD_TYPE_JSONB, | ||
| FieldType::OLAP_FIELD_TYPE_ARRAY, |
There was a problem hiding this comment.
array is supported by inverted index
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
1. When there are a large number of identical TabletColumns in the cluster, which usually occurs when VARIANT type columns are modified and added, each Rowset has an individual TabletSchema. Excessive TabletSchemas can lead to significant memory overhead. Reusing memory for identical TabletColumns would greatly reduce this memory consumption. 2. Serialized TabletSchema as LRU cache key could also increase memusage when large sets of schemas are in LRU cache, so inorder to reduce the memory footprint we just record the key signature caculated by generating an UUID by hash algorithm， and lookup the key signature in LRU cache， and check the key in case of hash collision
1. When there are a large number of identical TabletColumns in the cluster, which usually occurs when VARIANT type columns are modified and added, each Rowset has an individual TabletSchema. Excessive TabletSchemas can lead to significant memory overhead. Reusing memory for identical TabletColumns would greatly reduce this memory consumption. 2. Serialized TabletSchema as LRU cache key could also increase memusage when large sets of schemas are in LRU cache, so inorder to reduce the memory footprint we just record the key signature caculated by generating an UUID by hash algorithm， and lookup the key signature in LRU cache， and check the key in case of hash collision
1. When there are a large number of identical TabletColumns in the cluster, which usually occurs when VARIANT type columns are modified and added, each Rowset has an individual TabletSchema. Excessive TabletSchemas can lead to significant memory overhead. Reusing memory for identical TabletColumns would greatly reduce this memory consumption. 2. Serialized TabletSchema as LRU cache key could also increase memusage when large sets of schemas are in LRU cache, so inorder to reduce the memory footprint we just record the key signature caculated by generating an UUID by hash algorithm， and lookup the key signature in LRU cache， and check the key in case of hash collision
1. When there are a large number of identical TabletColumns in the cluster, which usually occurs when VARIANT type columns are modified and added, each Rowset has an individual TabletSchema. Excessive TabletSchemas can lead to significant memory overhead. Reusing memory for identical TabletColumns would greatly reduce this memory consumption. 2. Serialized TabletSchema as LRU cache key could also increase memusage when large sets of schemas are in LRU cache, so inorder to reduce the memory footprint we just record the key signature caculated by generating an UUID by hash algorithm， and lookup the key signature in LRU cache， and check the key in case of hash collision
1. When there are a large number of identical TabletColumns in the cluster, which usually occurs when VARIANT type columns are modified and added, each Rowset has an individual TabletSchema. Excessive TabletSchemas can lead to significant memory overhead. Reusing memory for identical TabletColumns would greatly reduce this memory consumption. 2. Serialized TabletSchema as LRU cache key could also increase memusage when large sets of schemas are in LRU cache, so inorder to reduce the memory footprint we just record the key signature caculated by generating an UUID by hash algorithm， and lookup the key signature in LRU cache， and check the key in case of hash collision
eldenmoon
added
dev/3.0.3-merged
and removed
cherry-pick-conflict-in-3.0
dev/3.0.x
labels
