Fix header for union stream in distributed #2226

KochetovNicolai · 2018-04-16T14:26:21Z

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

alexey-milovidov · 2018-04-18T21:15:47Z

dbms/src/DataStreams/FilterBlockInputStream.h

@@ -21,8 +21,8 @@ class FilterBlockInputStream : public IProfilingBlockInputStream

 public:
    /// filter_column_ - the number of the column with filter conditions.
-    FilterBlockInputStream(const BlockInputStreamPtr & input, const ExpressionActionsPtr & expression_, ssize_t filter_column_);
-    FilterBlockInputStream(const BlockInputStreamPtr & input, const ExpressionActionsPtr & expression_, const String & filter_column_name_);
+    FilterBlockInputStream(const BlockInputStreamPtr & input, const ExpressionActionsPtr & expression_, ssize_t filter_column_, bool remove_filter = false);


Missing comment.

alexey-milovidov · 2018-04-18T21:18:19Z

dbms/src/DataStreams/FilterBlockInputStream.cpp

-    : expression(expression_), filter_column(filter_column_)
+FilterBlockInputStream::FilterBlockInputStream(const BlockInputStreamPtr & input, const ExpressionActionsPtr & expression_,
+                                               ssize_t filter_column_, bool remove_filter)
+    : remove_filter(remove_filter), expression(expression_), filter_column(filter_column_)


When first ctor is used, header is not set.

I've removed this ctor.

alexey-milovidov · 2018-04-18T21:24:48Z

dbms/src/DataStreams/PrewhereFilterBlockInputStream.h

+namespace DB
+{
+
+class PrewhereFilterBlockInputStream : public FilterBlockInputStream


Total absence of comments.

alexey-milovidov · 2018-04-18T21:25:51Z

dbms/src/DataStreams/PrewhereFilterBlockInputStream.h

+                                     prewhere_info->prewhere_column_name, prewhere_info->remove_prewhere_column)
+            , prewhere_info(prewhere_info)
+    {
+    }


What about header?

alexey-milovidov · 2018-04-18T21:27:24Z

dbms/src/DataStreams/PrewhereFilterBlockInputStream.h

+protected:
+    Block readImpl() override
+    {
+        remove_filter = prewhere_info->remove_prewhere_column;


Does it have to be evaluated for each block?

alexey-milovidov · 2018-04-18T21:28:32Z

dbms/src/Interpreters/ExpressionActions.cpp

@@ -1016,10 +1016,25 @@ void ExpressionActionsChain::finalize()
    for (int i = static_cast<int>(steps.size()) - 1; i >= 0; --i)
    {
        Names required_output = steps[i].required_output;
+        std::unordered_map<String, size_t> required_output_indexes;


This is still Ok.

alexey-milovidov · 2018-04-19T02:09:46Z

dbms/src/Interpreters/ExpressionAnalyzer.h

@@ -103,7 +103,8 @@ class ExpressionAnalyzer : private boost::noncopyable
    /// Before aggregation:
    bool appendArrayJoin(ExpressionActionsChain & chain, bool only_types);
    bool appendJoin(ExpressionActionsChain & chain, bool only_types);
-    bool appendWhere(ExpressionActionsChain & chain, bool only_types);
+    bool appendPrewhere(ExpressionActionsChain & chain, bool only_types, std::shared_ptr<bool> & remove_filter);


Cannot understand without comments.
Why we need std::shared_ptr<bool>. Looks weird.

alexey-milovidov · 2018-04-19T02:11:14Z

dbms/src/Interpreters/ExpressionActions.h

@@ -230,7 +230,9 @@ struct ExpressionActionsChain
    struct Step
    {
        ExpressionActionsPtr actions;
+        NameSet additional_input;


What is this? Don't understand.

alexey-milovidov · 2018-04-19T02:12:48Z

dbms/src/Interpreters/ExpressionActions.h

        Names required_output;
+        std::vector<std::shared_ptr<bool>> can_remove_required_output; /// Has the same size with required_output, is filled in finalize()


If the output is required, why it's possible that we "can remove" it? No explanation.
Non obvious.

alexey-milovidov · 2018-04-19T02:18:08Z

dbms/src/Interpreters/InterpreterSelectQuery.cpp

@@ -676,12 +696,25 @@ QueryProcessingStage::Enum InterpreterSelectQuery::executeFetchColumns(Pipeline
                optimize_prewhere(*merge_tree);
        }

+        if (!dry_run && query_analyzer->appendPrewhere(chain, false, remove_prewhere_filter))
+        {
+            query_info.prewhere_info = prewhere_info = std::make_shared<PrewhereInfo>(


This style is discouraged.

alexey-milovidov · 2018-04-19T02:22:45Z

dbms/src/Storages/MergeTree/MergeTreeBaseBlockInputStream.cpp

+            block.erase(prewhere_info->prewhere_column_name);
+
+        if (!block)
+            block.insert({nullptr, std::make_shared<DataTypeNothing>(), "_nothing"});


Is it Ok to have a block without column?

alexey-milovidov · 2018-04-19T02:23:45Z

dbms/src/Storages/MergeTree/MergeTreeBlockInputStream.cpp

@@ -99,15 +111,14 @@ try
    }
    is_first_task = false;

-    Names pre_column_names, column_names = ordered_names;
-    bool remove_prewhere_column = false;
+    Names pre_column_names, column_names = required_columns;


alexey-milovidov · 2018-04-19T02:25:58Z

dbms/src/Storages/MergeTree/MergeTreeBlockInputStream.h

@@ -48,9 +48,10 @@ class MergeTreeBlockInputStream : public MergeTreeBaseBlockInputStream
    bool getNewTask() override;

 private:
-    Block header;
+    mutable Block header;


This is a red flag.
May lead to race conditions in getHeader method.

alexey-milovidov · 2018-04-19T02:26:47Z

dbms/src/Storages/MergeTree/MergeTreeBlockInputStream.h

@@ -71,6 +72,8 @@ class MergeTreeBlockInputStream : public MergeTreeBaseBlockInputStream
    bool is_first_task = true;

    Logger * log = &Logger::get("MergeTreeBlockInputStream");
+
+    const Names & getOrderedNames();


Missing comment.

alexey-milovidov · 2018-04-19T02:29:37Z

dbms/src/Storages/MergeTree/MergeTreeReader.cpp

@@ -497,7 +497,7 @@ void MergeTreeReader::fillMissingColumns(Block & res, bool & should_reorder, boo
    }
 }

-void MergeTreeReader::reorderColumns(Block & res, const Names & ordered_names)
+void MergeTreeReader::reorderColumns(Block & res, const Names & ordered_names, const String * filter_name)


std::optional?

alexey-milovidov · 2018-04-19T02:30:06Z

dbms/src/Storages/MergeTree/MergeTreeReader.h

@@ -43,7 +43,7 @@ class MergeTreeReader : private boost::noncopyable
    /// If at least one column was added, reorders all columns in the block according to ordered_names.
    void fillMissingColumns(Block & res, bool & should_reorder, bool & should_evaluate_missing_defaults);
    /// Sort columns to ensure consistent order among all blocks.
-    void reorderColumns(Block & res, const Names & ordered_names);
+    void reorderColumns(Block & res, const Names & ordered_names, const String * filter_name);


The comment above wasn't updated.

ludv1x · 2018-05-22T12:01:22Z

dbms/src/Storages/IStorage.h

@@ -174,7 +177,7 @@ class IStorage : public std::enable_shared_from_this<IStorage>, private boost::n
        const Names & /*column_names*/,
        const SelectQueryInfo & /*query_info*/,
        const Context & /*context*/,
-        QueryProcessingStage::Enum & /*processed_stage*/,
+        QueryProcessingStage::Enum /*processed_stage*/,


It seems that this parameter is unnecessary now?

ludv1x · 2018-05-22T12:10:16Z

dbms/src/Storages/StorageMerge.cpp

 BlockInputStreams StorageMerge::read(
    const Names & column_names,
    const SelectQueryInfo & query_info,
    const Context & context,
-    QueryProcessingStage::Enum & processed_stage,
+    QueryProcessingStage::Enum processed_stage,
    const size_t max_block_size,
    const unsigned num_streams)
 {


Why don't we check processed_stage here?
What will happen if new tables with a different stage will be added between calls getQueryProcessingStage() and read()?

ludv1x · 2018-05-22T12:41:28Z

dbms/src/Interpreters/ExpressionAnalyzer.cpp

+        ///    Example: select A prewhere B > 0. B can be removed at prewhere step.
+        /// 2. Store side columns which were calculated during prewhere actions execution if they are used.
+        ///    Example: select F(A) prewhere F(A) > 0. F(A) can be saved from prewhere step.
+        /// 3. Check if we can remove filer column at prewhere step. If we can, action will store single REMOVE_COLUMN.


filer -> filter

chenxing-xc · 2018-06-25T06:02:14Z

is there any progress on this fix? we hit the same problem multiple times and has to disable optimize_move_to_prewhere to workaround the problem.

ztlpn · 2018-06-28T16:26:17Z

dbms/src/Interpreters/ExpressionActions.h

+        /// True if column from required_output is needed only for current step and not used in next actions
+        /// (and can be removed from block). Example: filter column for where actions.
+        /// If not empty, has the same size with required_output; is filled in finalize().
+        std::vector<bool *> not_need_in_future_steps;


Looks quite dangerous and also hurts code readability - when we modify a flag in this vector, where is it in the code? You have to switch to another file to determine.

Alternative - have a collection (can be a vector<bool> or even a NameSet that will tell us which columns are not needed by future steps). Then the calling code can set any needed bool value itself.

ztlpn · 2018-06-29T10:15:52Z

dbms/src/DataStreams/FilterBlockInputStream.cpp

+    if (block && remove_filter)
+        block.erase(static_cast<size_t>(filter_column));
+
+    return block;


Better to add std::move here otherwise a copy will be performed (the compiler won't do copy elision for a function parameter).

ztlpn · 2018-06-29T11:09:03Z

dbms/src/Interpreters/ExpressionActions.cpp

        if (i + 1 < static_cast<int>(steps.size()))
        {
+            const NameSet & additional_input = steps[i + 1].additional_input;


Is it ok that we are considering only the next step when setting the not_need_in_future_steps variable (the name points to all future steps instead of just the next one)?

Also it's better to stick to a single name (can_remove_required_output or not_need_in_future_steps)

ztlpn · 2018-06-29T11:15:18Z

...eries/0_stateless/00614_shard_same_header_for_local_and_remote_node_in_distributed_query.sql

+create table test.tab (date Date,  time DateTime, data String) ENGINE = MergeTree(date, (time, data), 8192);
+insert into test.tab values ('2018-01-21','2018-01-21 15:12:13','test');
+select time FROM remote('127.0.0.{1,2}', test.tab)  WHERE date = '2018-01-21' limit 2;
+


Better to drop the test table at the end

ztlpn · 2018-06-29T12:33:18Z

Seems that #2083 is still not fixed in this PR.

fancyqlx · 2018-08-08T14:22:40Z

dbms/src/Interpreters/InterpreterSelectQuery.cpp

+        else if (const StorageReplicatedMergeTree * merge_tree = dynamic_cast<const StorageReplicatedMergeTree *>(storage.get()))
+            optimize_prewhere(*merge_tree);
+    }
+
    AnalysisResult expressions;

    if (dry_run)


This dry_run condition does not cover all cases. There exists some cases (i.e. case 00229 in /dbms/tests/queries/stateless/) that makes Expression::execute function throw an exception.
In this case, apply_functions in prewhere actions will not be executed if InterpreterSelectQuery::getHeader() is invoked. Then the PROJECT will not find corresponding columns, which leads to an exception.
I think it can be solved by swapping prewhere_expression with where_expression if dry_run is true.

KochetovNicolai added 6 commits April 11, 2018 22:08

remove unused columns from prewhere actions [#CLICKHOUSE-3694] #2156

c9abeed

added PrewhereInfo [#CLICKHOUSE-3694] #2156

42f9a4d

added test

d9ae4be

fix perf

325f16f

Merge branch 'master' into fix-header-for-union-stream-in-distributed-2

41229b2

fix test [#CLICKHOUSE-3694] #2156

081f0fe

KochetovNicolai force-pushed the fix-header-for-union-stream-in-distributed-2 branch from aaccf3a to 081f0fe Compare April 16, 2018 19:49

fix build

66905a1

alexey-milovidov reviewed Apr 18, 2018

View reviewed changes

alexey-milovidov added a commit that referenced this pull request Apr 18, 2018

Removed obsolete code #2226

cce4c88

alexey-milovidov reviewed Apr 18, 2018

View reviewed changes

alexey-milovidov reviewed Apr 19, 2018

View reviewed changes

KochetovNicolai added 4 commits April 19, 2018 17:47

added IStorage::getQueryProcessingStage

a3308bf

merged with master

dd10c20

fix build

f5fe383

merged with master

a6375d6

KochetovNicolai force-pushed the fix-header-for-union-stream-in-distributed-2 branch from 164fb72 to a6375d6 Compare April 20, 2018 14:15

KochetovNicolai force-pushed the fix-header-for-union-stream-in-distributed-2 branch from ece7662 to 8061046 Compare April 23, 2018 19:39

added comments and fixed tests build

7a62bb9

KochetovNicolai force-pushed the fix-header-for-union-stream-in-distributed-2 branch from 8061046 to 7a62bb9 Compare April 23, 2018 19:41

zhang2014 mentioned this pull request May 11, 2018

clickhouse query result have additional colume named _table #2338

Closed

zhang2014 mentioned this pull request May 21, 2018

Fix "DB::Exception: Block structure mismatch in UNION stream: different number of columns" #2393

Closed

ludv1x reviewed May 22, 2018

View reviewed changes

KochetovNicolai added 3 commits June 25, 2018 15:08

Merged with master.

ba4cf6f

Fixed build.

4f7c284

Added processing stage check for StorageMerge. [#CLICKHOUSE-3694] #2156

60f0ed9

KochetovNicolai force-pushed the fix-header-for-union-stream-in-distributed-2 branch from 83b2541 to 60f0ed9 Compare June 25, 2018 12:48

Spelling.

bd7c72d

ztlpn reviewed Jun 28, 2018

View reviewed changes

ztlpn reviewed Jun 29, 2018

View reviewed changes

Review fixes. [#CLICKHOUSE-3694] #2156

6e5e573

KochetovNicolai force-pushed the fix-header-for-union-stream-in-distributed-2 branch from 92a923a to 6e5e573 Compare June 29, 2018 16:19

alexey-milovidov added 3 commits August 5, 2018 10:05

Merging #2226

2016374

Merging #2226

2877140

Merging #2226

2b2869e

fancyqlx reviewed Aug 8, 2018

View reviewed changes

KochetovNicolai added 2 commits August 20, 2018 15:25

Merged with master.

391b4b5

Fix tests.

b64bb8a

alexey-milovidov merged commit 28bc6db into master Aug 20, 2018

alexey-milovidov mentioned this pull request Aug 20, 2018

alter table + prewhere: Not found column ... in block #2827

Closed

Felixoid deleted the fix-header-for-union-stream-in-distributed-2 branch April 10, 2022 12:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix header for union stream in distributed #2226

Fix header for union stream in distributed #2226

KochetovNicolai commented Apr 16, 2018

alexey-milovidov Apr 18, 2018

alexey-milovidov Apr 18, 2018

alexey-milovidov Apr 19, 2018

alexey-milovidov Apr 18, 2018

alexey-milovidov Apr 18, 2018

alexey-milovidov Apr 18, 2018

alexey-milovidov Apr 18, 2018

alexey-milovidov Apr 19, 2018

alexey-milovidov Apr 19, 2018

alexey-milovidov Apr 19, 2018

alexey-milovidov Apr 19, 2018

alexey-milovidov Apr 19, 2018

alexey-milovidov Apr 19, 2018

alexey-milovidov Apr 19, 2018

alexey-milovidov Apr 19, 2018 •

edited

Loading

alexey-milovidov Apr 19, 2018

alexey-milovidov Apr 19, 2018

alexey-milovidov Apr 19, 2018

ludv1x May 22, 2018

ludv1x May 22, 2018

ludv1x May 22, 2018

chenxing-xc commented Jun 25, 2018

ztlpn Jun 28, 2018 •

edited

Loading

ztlpn Jun 29, 2018 •

edited

Loading

ztlpn Jun 29, 2018 •

edited

Loading

ztlpn Jun 29, 2018

ztlpn commented Jun 29, 2018 •

edited

Loading

fancyqlx Aug 8, 2018

		Names required_output;
		std::vector<std::shared_ptr<bool>> can_remove_required_output; /// Has the same size with required_output, is filled in finalize()

Fix header for union stream in distributed #2226

Fix header for union stream in distributed #2226

Conversation

KochetovNicolai commented Apr 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexey-milovidov Apr 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chenxing-xc commented Jun 25, 2018

ztlpn Jun 28, 2018 • edited Loading

Choose a reason for hiding this comment

ztlpn Jun 29, 2018 • edited Loading

Choose a reason for hiding this comment

ztlpn Jun 29, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ztlpn commented Jun 29, 2018 • edited Loading

Choose a reason for hiding this comment

alexey-milovidov Apr 19, 2018 •

edited

Loading

ztlpn Jun 28, 2018 •

edited

Loading

ztlpn Jun 29, 2018 •

edited

Loading

ztlpn Jun 29, 2018 •

edited

Loading

ztlpn commented Jun 29, 2018 •

edited

Loading