[Feature] Lightweight schema change of add/drop column #10136
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
area/load
|
What a fancy improvement! And after reading your issue, I'm curious will it increase the pressure on metadata storage because we persist all schema of all rowsets? |
schema is really small, and one schema per rowset is really controllable. |
Lchangliang
force-pushed
the
feature_sc_optimize
branch
9 times, most recently
from
6b0ee82
to
0e62965
Compare
github-actions
bot
added
the
area/spark-load
Lchangliang
force-pushed
the
feature_sc_optimize
branch
2 times, most recently
from
923cd3b
to
ea7d504
Compare
|
There is already a schema info for every segment. I think you could use it. |
yiguolei
reviewed
Jul 1, 2022
yiguolei
reviewed
Jul 1, 2022
Lchangliang
force-pushed
the
feature_sc_optimize
branch
2 times, most recently
from
5c1a22b
to
ccbf8f9
Compare
yiguolei
reviewed
Jul 2, 2022
morningman
added
kind/feature
morningman
reviewed
Jul 2, 2022
fe/fe-core/src/main/java/org/apache/doris/persist/OperationType.java
Outdated
Show resolved
Hide resolved
Lchangliang
force-pushed
the
feature_sc_optimize
branch
3 times, most recently
from
f1c42c3
to
3055438
Compare
Lchangliang
force-pushed
the
feature_sc_optimize
branch
9 times, most recently
from
ac6ac9b
to
47c163e
Compare
* [feature](schema-change) support fast schema change. coauthor: yixiutt * [schema change] Using columns desc from fe to read data. coauthor: Lchangliang * [feature](schema change) schema change optimize for add/drop columns. 1.add uniqueId field for class column. 2.schema change for add/drop columns directly update schema meta Co-authored-by: yixiutt <[email protected]> Co-authored-by: SWJTU-ZhangLei <[email protected]> [Feature](schema change) fix write and add regression test (apache#69) Co-authored-by: yixiutt <[email protected]> [schema change] be ssupport that delete use newest schema add delete regression test fix regression case (apache#107) tmp [feature](schema change) light schema change exclude rollup and agg/uniq/dup key type. [feature](schema change) fe olapTable maxUniqueId write in disk. [feature](schema change) add rpc iface for sc add column. [feature](schema change) add columnsDesc to TPushReq for ligtht sc. resolve the deadlock when schema change (apache#124) fix columns from fe don't has bitmap_index flag (apache#134) add update/delete case construct MATERIALIZED schema from origin schema when insert fix not vectorized compaction coredump use segment cache choose newest schema by schema version when compaction (apache#182) [bugfix](schema change) fix ligth schema change problem. [feature](schema change) light schema change add alter job. (#1) fix be ut [bug] (schema change) unique drop key column should not light schema change [feature](schema change) add schema change regression-test. fix regression test [bugfix](schema change) fix multi alter clauses for light schema change. (#2) [bugfix](schema change) fix multi clauses calculate column unique id (#3) modify PushTask process (apache#217) [Bugfix](schema change) fix jobId replay cause bdbje exception. [bug](schema change) fix max col unique id repeatitive. (apache#232) [optimize](schema change) modify pendingMaxColUniqueId generate rule. fix compaction error * fix be ut * fix snapshot load core fix unique_id error (apache#278) [refact](fe) remove redundant code for light schema change. (#4) [refact](fe) remove redundant code for light schema change. (#4) format fe core format be core fix be ut modify fe meta version fix rebase error flush schema into rowset_meta in old table [refactor](schema change) refact fe light schema change. (#5) delete the change of schemahash and support get max version schema
Lchangliang
force-pushed
the
feature_sc_optimize
branch
from
47c163e
to
672f617
Compare
dataroaring
approved these changes
Jul 12, 2022
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
eldenmoon
pushed a commit
to eldenmoon/incubator-doris
that referenced
this pull request
Aug 1, 2022
* [Schema Change] support fast add/drop column (apache#49) * [feature](schema-change) support fast schema change. coauthor: yixiutt * [schema change] Using columns desc from fe to read data. coauthor: Lchangliang * [feature](schema change) schema change optimize for add/drop columns. 1.add uniqueId field for class column. 2.schema change for add/drop columns directly update schema meta Co-authored-by: yixiutt <[email protected]> Co-authored-by: SWJTU-ZhangLei <[email protected]> [Feature](schema change) fix write and add regression test (apache#69) Co-authored-by: yixiutt <[email protected]> [schema change] be ssupport that delete use newest schema add delete regression test fix regression case (apache#107) tmp [feature](schema change) light schema change exclude rollup and agg/uniq/dup key type. [feature](schema change) fe olapTable maxUniqueId write in disk. [feature](schema change) add rpc iface for sc add column. [feature](schema change) add columnsDesc to TPushReq for ligtht sc. resolve the deadlock when schema change (apache#124) fix columns from fe don't has bitmap_index flag (apache#134) add update/delete case construct MATERIALIZED schema from origin schema when insert fix not vectorized compaction coredump use segment cache choose newest schema by schema version when compaction (apache#182) [bugfix](schema change) fix ligth schema change problem. [feature](schema change) light schema change add alter job. (apache#1) fix be ut [bug] (schema change) unique drop key column should not light schema change [feature](schema change) add schema change regression-test. fix regression test [bugfix](schema change) fix multi alter clauses for light schema change. (apache#2) [bugfix](schema change) fix multi clauses calculate column unique id (apache#3) modify PushTask process (apache#217) [Bugfix](schema change) fix jobId replay cause bdbje exception. [bug](schema change) fix max col unique id repeatitive. (apache#232) [optimize](schema change) modify pendingMaxColUniqueId generate rule. fix compaction error * fix be ut * fix snapshot load core fix unique_id error (apache#278) [refact](fe) remove redundant code for light schema change. (apache#4) [refact](fe) remove redundant code for light schema change. (apache#4) format fe core format be core fix be ut modify fe meta version fix rebase error flush schema into rowset_meta in old table [refactor](schema change) refact fe light schema change. (apache#5) delete the change of schemahash and support get max version schema * modify for review * fix be ut * fix schema change test
This was referenced Aug 3, 2022
13 tasks
13 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Issue Number: close #10135
Problem Summary:
Optimize ideas for writing in associated issue
Checklist(Required)
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...