From 65aa809f16370f50b50f40c6acdad4fee5b31487 Mon Sep 17 00:00:00 2001 From: timothy-e Date: Wed, 11 Sep 2024 09:48:25 -0400 Subject: [PATCH] [BACKPORT pg15-cherrypicks][#23521] YSQL: Cost YB Bitmap Table Scan remote filters Summary: Original commit: 4ea354b1b3e55e81fcd7b29fe023b037dd11b673 / D36484 Merge conflicts: * createplan.c: * `create_bitmap_subplan` declaration: * D36484 removed some of the parameters that were added by YB * Upstream realigned declarations * resolution: remove the parameters but follow new indentation practice * `create_indexscan_plan` declaration: * D36484 moved it to `planmain.h` * Upstream realigned declarations * resolution: move it to `planmain.h` * `create_bitmap_subplan`: * D36484 removed references to `indexpushdownquals` * pg15's cd1df468c56a3ea7de0eecee1b0fc98c566cc86a moved one of the references lower in the file * resolution: remove from the new location * `costsize.c` * `get_bitmap_index_quals` * D36484 copied the approach of `create_bitmap_subplan` to get index quals. * `ipath->indexquals` was removed in pg15. * Resolution: Rewrote the BitmapIndexScan case, referencing pg15's create_bitmap_subplan * planmain.h: * declarations * D36484 moved `create_indexscan_plan` from `createplan.h` and added `yb_get_bitmap_index_quals` * Upstream changed indentation of neighbouring declarations * because these new declarations were not spaced out from their neighbours, the changes caused conflicts. Include the new declarations. * yb_pg_select.out * D36484 mistakenly removed logic to account for quals implied by partial indexes, resulting in redundant storage filters being added. This is tracked by #23859. In this current diff, the BitmapIndexScan case of get_bitmap_index_quals needed to be rewritten for resolving costsize.c conflicts, and the rewritten code includes handling of partial indexes. The redundant storage filters are only introduced on master, not in the pg15 branch. * yb-pg15's 2750e71ceeebf9abdca83007a03b6ff7aef8a3ff removed yb ordering * Solution: use the pg15 test file, because the redundant filters are not added by this code. ----- Prior to this diff, Bitmap Scan CBO did not handle remote filter costs. Evaluating a remote filter was given no cost, so it was effectively "free". Therefore AND conditions were seen as better to use remote filters for, rather than Bitmap And, because Bitmap And nodes add cost but remote filters added no cost. To fix this, I 1. determine which quals will be evaluated by the index (as an index cond or a remote filter) in `yb_get_bitmap_index_quals`. This means that the previous change to `create_bitmap_subplan` that collected pushdown expressions can be reverted. The new function has a simpler interface, and allows for better comparing the given qual against the quals evaluated by the plan tree - now some redundant filters can be skipped. (more on this below) 2. determine which quals remain to be addressed by the Bitmap Table Scan node. 3. Account for the cost of evaluating those filters in the same manner that yb_cost_seqscan does, using `cost_qual_eval`. This allows for Bitmap And plans to be a reasonable alternative to remote filter plans. **Moving the discourage modifier:** I also changed the DISCOURAGE_MODIFIER to affect the startup and runtime costs of Bitmap Table Scan instead of the index scans. Now we no longer incentivise Bitmap Scan plans with more work done in the Bitmap Table Scan node. **Removing redundant tests: ** Removed some unstable tests from the yb_bitmap_scan_flags regress tests. They failed with the addition of this change, but actually don't provide anything meaningful. They answer the question: "what will the planner choose when both options are disabled?" which depends on exact details of how each plan is costed. That does not belong in a regress test. **Removing redundant remote filters**: the planner identifies which nodes are resposible for evaluating the given conditions. The Bitmap Table scan node should evaluate anything that's not already implied by the conditions on the Bitmap Index Scans. However, with the addition of a storage index filters on Bitmap Index Scans, this logic was slightly faulty. Two lists of clauses were created, one for index conditions and one for index remote filters. Consider the conditons: `a < 10 OR (b < 10 AND remote_fllter(b))`. This might create a plan like: ``` Bitmap Table Scan Storage Filter: xxx -> Bitmap Or -> Bitmap Index Scan Index Cond: a < 10 -> Bitmap Index Scan Index Cond: b < 10 Storage Index Filter: remote_flter(b)) ``` The list of index conditions was: `OR(a < 10, b < 10)`. The list of index filters was: `remote_filter(b)`. Neither of these indiviudally implied the original condition, so the planner decided it was necessary to apply the remote filters. With this change, a single unified list of index conditions and storage index filters is returned. In this case, the list is `OR(a < 10, AND(b < 10, remote_filter(b))`. Since the original condition is implied by (actually equivalent to) this expression, the planner can now easily determine that the original conditions are already met, so there is no need to add a storage filter. Jira: DB-12438 Test Plan: ``` ./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressYbBitmapScans' ./yb_build.sh --java-test 'org.yb.pgsql.TestPgCostModelSeekNextEstimation' ``` TAQO report: {F279017} Bitmap scan is often the most optimal plan for these queries, but less frequently the default plan with this change because of the movement of the discourage factor and the addition of the remote filter cost. While these are regressions, the feature is not yet in EA and further CBO changes are incoming, including [[ https://github.com/yugabyte/yugabyte-db/issues/23565 | #23565 ]] to remove the discourage factor entirely. This is a helpful step in the right direction. Reviewers: jason, tfoucher, mtakahara Reviewed By: jason, mtakahara Subscribers: mtakahara, yql, dsherstobitov Differential Revision: https://phorge.dev.yugabyte.com/D37658 --- .../org/yb/pgsql/ExplainAnalyzeUtils.java | 3 +- .../TestPgCostModelSeekNextEstimation.java | 71 ++++++ .../src/backend/optimizer/path/costsize.c | 207 +++++++++++++++--- .../src/backend/optimizer/path/indxpath.c | 3 + .../src/backend/optimizer/plan/createplan.c | 65 ++---- src/postgres/src/include/optimizer/planmain.h | 5 + .../test/regress/expected/yb_bitmap_scans.out | 167 ++++++++++++-- .../regress/expected/yb_bitmap_scans_colo.out | 167 ++++++++++++-- .../expected/yb_bitmap_scans_flags.out | 44 ---- .../src/test/regress/sql/yb_bitmap_scans.sql | 37 ++++ .../test/regress/sql/yb_bitmap_scans_colo.sql | 37 ++++ .../regress/sql/yb_bitmap_scans_flags.sql | 10 - 12 files changed, 652 insertions(+), 164 deletions(-) diff --git a/java/yb-pgsql/src/test/java/org/yb/pgsql/ExplainAnalyzeUtils.java b/java/yb-pgsql/src/test/java/org/yb/pgsql/ExplainAnalyzeUtils.java index e32d062e2770..7877a91fb463 100644 --- a/java/yb-pgsql/src/test/java/org/yb/pgsql/ExplainAnalyzeUtils.java +++ b/java/yb-pgsql/src/test/java/org/yb/pgsql/ExplainAnalyzeUtils.java @@ -2,10 +2,10 @@ import static org.junit.Assert.assertTrue; +import java.math.BigDecimal; import java.sql.ResultSet; import java.sql.Statement; import java.util.List; -import java.math.BigDecimal; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -26,6 +26,7 @@ public class ExplainAnalyzeUtils { public static final String NODE_AGGREGATE = "Aggregate"; public static final String NODE_BITMAP_INDEX_SCAN = "Bitmap Index Scan"; public static final String NODE_BITMAP_OR = "BitmapOr"; + public static final String NODE_BITMAP_AND = "BitmapAnd"; public static final String NODE_FUNCTION_SCAN = "Function Scan"; public static final String NODE_GATHER = "Gather"; public static final String NODE_GATHER_MERGE = "Gather Merge"; diff --git a/java/yb-pgsql/src/test/java/org/yb/pgsql/TestPgCostModelSeekNextEstimation.java b/java/yb-pgsql/src/test/java/org/yb/pgsql/TestPgCostModelSeekNextEstimation.java index e1704764e6b0..122ede2690b5 100644 --- a/java/yb-pgsql/src/test/java/org/yb/pgsql/TestPgCostModelSeekNextEstimation.java +++ b/java/yb-pgsql/src/test/java/org/yb/pgsql/TestPgCostModelSeekNextEstimation.java @@ -1,6 +1,7 @@ package org.yb.pgsql; import static org.junit.Assume.assumeTrue; +import static org.yb.pgsql.ExplainAnalyzeUtils.NODE_BITMAP_AND; import static org.yb.pgsql.ExplainAnalyzeUtils.NODE_BITMAP_INDEX_SCAN; import static org.yb.pgsql.ExplainAnalyzeUtils.NODE_BITMAP_OR; import static org.yb.pgsql.ExplainAnalyzeUtils.NODE_INDEX_SCAN; @@ -40,6 +41,9 @@ public class TestPgCostModelSeekNextEstimation extends BasePgSQLTest { private static final String T3_INDEX_NAME = T3_NAME + "_pkey"; private static final String T4_NAME = "t4"; private static final String T4_INDEX_NAME = T4_NAME + "_pkey"; + private static final String T_NO_PKEY_NAME = "t_no_pkey"; + private static final String T_K1_INDEX_NAME = "t_k1_idx"; + private static final String T_K2_INDEX_NAME = "t_k2_idx"; private static final double SEEK_FAULT_TOLERANCE_OFFSET = 1; private static final double SEEK_FAULT_TOLERANCE_RATE = 0.2; private static final double SEEK_LOWER_BOUND_FACTOR = 1 - SEEK_FAULT_TOLERANCE_RATE; @@ -609,6 +613,73 @@ public void testSeekNextEstimationBitmapScanWithOr() throws Exception { } } + // There's a middle ground for queries with AND. If few rows are to be + // selected, then it's cheap to apply remote filters. If a lot of rows are to + // be selected, then it's pointless to gather a large portion of the table for + // each Bitmap Index Scan under a BitmapAnd. + @Test + public void testSeekNextEstimationBitmapScanWithAnd() throws Exception { + try (Statement stmt = this.connection2.createStatement()) { + stmt.execute(String.format("CREATE TABLE %s (k1 INT, k2 INT)", T_NO_PKEY_NAME)); + stmt.execute(String.format("CREATE INDEX %s ON %s (k1 ASC)", + T_K1_INDEX_NAME, T_NO_PKEY_NAME)); + stmt.execute(String.format("CREATE INDEX %s ON %s (k2 ASC)", + T_K2_INDEX_NAME, T_NO_PKEY_NAME)); + stmt.execute(String.format("INSERT INTO %s SELECT s1, s2 FROM generate_series(1, 100) s1, " + + "generate_series(1, 100) s2", T_NO_PKEY_NAME)); + stmt.execute(String.format("ANALYZE %s", T_NO_PKEY_NAME)); + + final String query = "/*+ BitmapScan(t) */ SELECT * FROM %s AS t WHERE %s"; + testSeekAndNextEstimationBitmapScanHelper(stmt, + String.format(query, T_NO_PKEY_NAME, "k1 <= 1 AND k2 <= 1"), + T_NO_PKEY_NAME, 100, 300, 10, + makePlanBuilder().nodeType(NODE_BITMAP_INDEX_SCAN).build()); + + testSeekAndNextEstimationBitmapScanHelper(stmt, + String.format(query, T_NO_PKEY_NAME, "k1 <= 2 AND k2 <= 2"), + T_NO_PKEY_NAME, 200, 600, 10, + makePlanBuilder().nodeType(NODE_BITMAP_INDEX_SCAN).build()); + + testSeekAndNextEstimationBitmapScanHelper(stmt, + String.format(query, T_NO_PKEY_NAME, "k1 <= 5 AND k2 <= 5"), + T_NO_PKEY_NAME, 25, 75, 10, + makePlanBuilder().nodeType(NODE_BITMAP_AND).build()); + + testSeekAndNextEstimationBitmapScanHelper(stmt, + String.format(query, T_NO_PKEY_NAME, "k1 <= 10 AND k2 <= 10"), + T_NO_PKEY_NAME, 100, 300, 10, + makePlanBuilder().nodeType(NODE_BITMAP_AND).build()); + + testSeekAndNextEstimationBitmapScanHelper(stmt, + String.format(query, T_NO_PKEY_NAME, "k1 <= 20 AND k2 <= 20"), + T_NO_PKEY_NAME, 400, 1200, 10, + makePlanBuilder().nodeType(NODE_BITMAP_AND).build()); + + testSeekAndNextEstimationBitmapScanHelper(stmt, + String.format(query, T_NO_PKEY_NAME, "k1 <= 40 AND k2 <= 40"), + T_NO_PKEY_NAME, 4000, 14000, 10, + makePlanBuilder().nodeType(NODE_BITMAP_INDEX_SCAN).build()); + + testSeekAndNextEstimationBitmapScanHelper(stmt, + String.format(query, T_NO_PKEY_NAME, "k1 <= 80 AND k2 <= 80"), + T_NO_PKEY_NAME, 8000, 24000, 10, + makePlanBuilder().nodeType(NODE_BITMAP_INDEX_SCAN).build()); + + // If the two sets of ybctids are not similar sizes, it doesn't make sense + // to collect both sets. Instead, we collect the smaller and filter out + // the larger. + testSeekAndNextEstimationBitmapScanHelper(stmt, + String.format(query, T_NO_PKEY_NAME, "k1 <= 20 AND k2 <= 40"), + T_NO_PKEY_NAME, 2000, 6000, 10, + makePlanBuilder().nodeType(NODE_BITMAP_INDEX_SCAN).indexName(T_K1_INDEX_NAME).build()); + + testSeekAndNextEstimationBitmapScanHelper(stmt, + String.format(query, T_NO_PKEY_NAME, "k1 <= 20 AND k2 <= 10"), + T_NO_PKEY_NAME, 1000, 3000, 10, + makePlanBuilder().nodeType(NODE_BITMAP_INDEX_SCAN).indexName(T_K2_INDEX_NAME).build()); + } + } + /* These scans exceed work_mem, so their YB Bitmap Table Scan just performs a * sequential scan. The seeks and nexts of the Table Scan should be equivalent * to the seeks and nexts of the Sequential Scan. diff --git a/src/postgres/src/backend/optimizer/path/costsize.c b/src/postgres/src/backend/optimizer/path/costsize.c index 8bc8454fd857..5b1a5f061bcb 100644 --- a/src/postgres/src/backend/optimizer/path/costsize.c +++ b/src/postgres/src/backend/optimizer/path/costsize.c @@ -661,11 +661,7 @@ cost_index(IndexPath *path, PlannerInfo *root, double loop_count, * We don't bother to save indexStartupCost or indexCorrelation, because a * bitmap scan doesn't care about either. */ - if (IsYugaByteEnabled() && baserel->is_yb_relation) - // TODO(#20573): YB Bitmap cost - path->indextotalcost = indexTotalCost * YB_BITMAP_DISCOURAGE_MODIFIER; - else - path->indextotalcost = indexTotalCost; + path->indextotalcost = indexTotalCost; path->indexselectivity = indexSelectivity; /* all costs for touching index itself included here */ @@ -1150,7 +1146,9 @@ cost_bitmap_heap_scan(Path *path, PlannerInfo *root, RelOptInfo *baserel, run_cost += path->pathtarget->cost.per_tuple * path->rows; path->startup_cost = startup_cost; - path->total_cost = startup_cost + run_cost; + path->total_cost = (IsYugaByteEnabled() && baserel->is_yb_relation) + ? (startup_cost + run_cost) * YB_BITMAP_DISCOURAGE_MODIFIER + : startup_cost + run_cost; } /* @@ -7945,9 +7943,8 @@ yb_cost_index(IndexPath *path, PlannerInfo *root, double loop_count, (index_ybctid_num_nexts * per_next_cost); path->indextotalcost = - (yb_local_latency_cost + startup_cost + run_cost + - index_ybctid_transfer_cost + index_ybctid_paging_seek_next_costs) * - YB_BITMAP_DISCOURAGE_MODIFIER; + yb_local_latency_cost + startup_cost + run_cost + + index_ybctid_transfer_cost + index_ybctid_paging_seek_next_costs; path->indexselectivity = index_selectivity; path->ybctid_width = index_ybctid_width; @@ -8142,6 +8139,113 @@ yb_cost_index(IndexPath *path, PlannerInfo *root, double loop_count, yb_parallel_cost((Path *) path); } + +/* + * yb_get_bitmap_index_quals + * Return the list of quals used by the Bitmap Index Scans. This includes the + * index conditions themselves, and any additional remote filters that are + * applied by each Bitmap Index Scan. + * + * 'bitmapqual' is a tree of IndexPaths, BitmapAndPaths, and BitmapOrPaths + * 'scan_clauses' is a list of all conditions given to the bitmap path + */ +List * +yb_get_bitmap_index_quals(PlannerInfo *root, Path *bitmapqual, + List *scan_clauses) +{ + if (IsA(bitmapqual, BitmapAndPath)) + { + BitmapAndPath *apath = (BitmapAndPath *) bitmapqual; + List *subindexquals = NIL; + ListCell *l; + + foreach(l, apath->bitmapquals) + { + List *subindexqual = yb_get_bitmap_index_quals( + root, (Path *) lfirst(l), scan_clauses); + + subindexquals = list_concat_unique(subindexquals, subindexqual); + } + + return subindexquals; + } + else if (IsA(bitmapqual, BitmapOrPath)) + { + BitmapOrPath *opath = (BitmapOrPath *) bitmapqual; + List *subindexquals = NIL; + ListCell *l; + + foreach(l, opath->bitmapquals) + { + List *subindexqual = yb_get_bitmap_index_quals( + root, (Path *) lfirst(l), scan_clauses); + + subindexquals = lappend(subindexquals, + make_ands_explicit(subindexqual)); + } + + if (list_length(subindexquals) <= 1) + return subindexquals; + return list_make1(make_orclause(subindexquals)); + } + else if (IsA(bitmapqual, IndexPath)) + { + IndexPath *ipath = (IndexPath *) bitmapqual; + IndexScan *iscan; + List *subquals; + List *subindexquals; + ListCell *l; + + /* Use the regular indexscan plan build machinery... */ + iscan = castNode(IndexScan, + create_indexscan_plan(root, ipath, + NIL, scan_clauses, + false /* indexonly */, + true /* bitmapindex */)); + + /* Extract original index clauses, actual index quals, relevant ECs */ + subquals = NIL; + subindexquals = NIL; + foreach(l, ipath->indexclauses) + { + IndexClause *iclause = (IndexClause *) lfirst(l); + RestrictInfo *rinfo = iclause->rinfo; + + Assert(!rinfo->pseudoconstant); + subquals = lappend(subquals, rinfo->clause); + subindexquals = list_concat(subindexquals, + get_actual_clauses(iclause->indexquals)); + } + /* We can add any index predicate conditions, too */ + foreach(l, ipath->indexinfo->indpred) + { + Expr *pred = (Expr *) lfirst(l); + + /* + * We know that the index predicate must have been implied by the + * query condition as a whole, but it may or may not be implied by + * the conditions that got pushed into the bitmapqual. Avoid + * generating redundant conditions. + */ + if (!predicate_implied_by(list_make1(pred), subquals, false)) + subindexquals = lappend(subindexquals, pred); + } + + if (iscan->yb_idx_pushdown.quals) + subindexquals = lappend(subindexquals, + make_ands_explicit(iscan->yb_idx_pushdown.quals)); + + pfree(iscan); + + return subindexquals; + } + else + { + elog(ERROR, "unrecognized node type: %d", nodeTag(bitmapqual)); + } +} + + /* * yb_cost_bitmap_table_scan * Determines and returns the cost of scanning a relation using a YB bitmap @@ -8178,11 +8282,15 @@ yb_cost_bitmap_table_scan(Path *path, PlannerInfo *root, RelOptInfo *baserel, Cost per_seek_cost = 0.0; Cost per_next_cost = 0.0; List *local_clauses = NIL; - double remote_filtered_rows; + List *non_index_clauses = NIL; int num_nexts; int num_seeks; int docdb_result_width; double tuples_fetched; + double tuples_scanned; + QualCost qual_cost; + List *indexquals; + ListCell *l; /* Should only be applied to Yugabyte base relations */ Assert(IsA(baserel, RelOptInfo)); @@ -8225,14 +8333,55 @@ yb_cost_bitmap_table_scan(Path *path, PlannerInfo *root, RelOptInfo *baserel, return; } - /* TODO: account for local and remote quals */ - tuples_fetched = - clamp_row_est(baserel->tuples * - clauselist_selectivity(root, baserel->baserestrictinfo, - baserel->relid, JOIN_INNER, NULL)); + indexquals = yb_get_bitmap_index_quals(root, bitmapqual, + baserel->baserestrictinfo); + + /* Determine what clauses remain to be checked by this node */ + foreach(l, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, l); + Node *clause = (Node *) rinfo->clause; + + if (rinfo->pseudoconstant) + continue; /* we may drop pseudoconstants here */ + if (list_member(indexquals, clause)) + continue; /* simple duplicate */ + if (!contain_mutable_functions(clause) && + predicate_implied_by(list_make1(clause), indexquals, false)) + continue; /* provably implied by indexquals */ + non_index_clauses = lappend(non_index_clauses, rinfo); + } + + /* Determine remote and local quals */ + List *local_quals = NIL; + List *rel_remote_quals = NIL; + List *rel_colrefs = NIL; + + extract_pushdown_clauses(non_index_clauses, NULL /* index_info */, + false /* bitmapindex */, &local_quals, + &rel_remote_quals, &rel_colrefs, + NULL /* idx_remote_quals */, + NULL /* idx_colrefs */); + + tuples_scanned = clamp_row_est(baserel->tuples * + clauselist_selectivity(root, indexquals, baserel->relid, + JOIN_INNER, NULL)); + + cost_qual_eval(&qual_cost, rel_remote_quals, root); + startup_cost += qual_cost.startup; + run_cost += + (qual_cost.per_tuple + list_length(rel_remote_quals) * + yb_docdb_remote_filter_overhead_cycles * + cpu_operator_cost) * + tuples_scanned; + + tuples_fetched = clamp_row_est(baserel->tuples * + clauselist_selectivity(root, + list_union(indexquals, rel_remote_quals), + baserel->relid, JOIN_INNER, NULL)); /* Block fetch cost from disk */ - num_blocks = ceil(tuples_fetched * tuple_width / YB_DEFAULT_DOCDB_BLOCK_SIZE); + num_blocks = ceil(tuples_scanned * tuple_width / YB_DEFAULT_DOCDB_BLOCK_SIZE); run_cost += yb_random_block_cost * num_blocks; /* DocDB costs for merging key-value pairs to form tuples */ @@ -8240,9 +8389,9 @@ yb_cost_bitmap_table_scan(Path *path, PlannerInfo *root, RelOptInfo *baserel, yb_docdb_merge_cpu_cycles * cpu_operator_cost; /* Seek to first key cost */ - if (tuples_fetched > 0) + if (tuples_scanned > 0) { - per_seek_cost = yb_get_lsm_seek_cost(tuples_fetched, + per_seek_cost = yb_get_lsm_seek_cost(tuples_scanned, num_key_value_pairs_per_tuple, num_sst_files) + per_merge_cost; @@ -8252,9 +8401,6 @@ yb_cost_bitmap_table_scan(Path *path, PlannerInfo *root, RelOptInfo *baserel, per_next_cost = (yb_docdb_next_cpu_cycles * cpu_operator_cost) + per_merge_cost; - /* TODO: account for table pushdown quals */ - remote_filtered_rows = tuples_fetched; - /* tlist eval costs are paid per output row, not per tuple scanned */ startup_cost += path->pathtarget->cost.startup; run_cost += path->pathtarget->cost.per_tuple * path->rows; @@ -8268,8 +8414,8 @@ yb_cost_bitmap_table_scan(Path *path, PlannerInfo *root, RelOptInfo *baserel, baserel, reloid); path->yb_plan_info.estimated_docdb_result_width = docdb_result_width; - num_seeks = tuples_fetched; - num_nexts = (max_nexts_to_avoid_seek + 1) * tuples_fetched; + num_seeks = tuples_scanned; + num_nexts = (max_nexts_to_avoid_seek + 1) * tuples_scanned; path->yb_plan_info.estimated_num_nexts = num_nexts; path->yb_plan_info.estimated_num_seeks = num_seeks; @@ -8279,12 +8425,21 @@ yb_cost_bitmap_table_scan(Path *path, PlannerInfo *root, RelOptInfo *baserel, /* Network latency cost is added to startup cost */ startup_cost += yb_local_latency_cost; - run_cost += yb_compute_result_transfer_cost(remote_filtered_rows, + run_cost += yb_compute_result_transfer_cost(tuples_fetched, docdb_result_width); + /* Local filter costs */ + cost_qual_eval(&qual_cost, local_clauses, root); + startup_cost += qual_cost.startup; + run_cost += qual_cost.per_tuple * tuples_fetched; + + path->rows = + clamp_row_est(baserel->tuples * + clauselist_selectivity(root, baserel->baserestrictinfo, + baserel->relid, JOIN_INNER, NULL)); path->rows = tuples_fetched; - path->startup_cost = startup_cost; - path->total_cost = startup_cost + run_cost; + path->startup_cost = startup_cost * YB_BITMAP_DISCOURAGE_MODIFIER; + path->total_cost = (startup_cost + run_cost) * YB_BITMAP_DISCOURAGE_MODIFIER; path->yb_plan_info.estimated_num_nexts = num_nexts; path->yb_plan_info.estimated_num_seeks = num_seeks; yb_parallel_cost(path); diff --git a/src/postgres/src/backend/optimizer/path/indxpath.c b/src/postgres/src/backend/optimizer/path/indxpath.c index 192a82270485..f95658aeea29 100644 --- a/src/postgres/src/backend/optimizer/path/indxpath.c +++ b/src/postgres/src/backend/optimizer/path/indxpath.c @@ -2150,6 +2150,9 @@ yb_bitmap_scan_cost_est(PlannerInfo *root, RelOptInfo *rel, Path *ipath) bpath.path.pathkeys = NIL; bpath.bitmapqual = ipath; + /* required for yb_parallel_cost */ + bpath.path.parallel_aware = false; + /* * Check the cost of temporary path without considering parallelism. * Parallel bitmap heap path will be considered at later stage. diff --git a/src/postgres/src/backend/optimizer/plan/createplan.c b/src/postgres/src/backend/optimizer/plan/createplan.c index da749c75f387..2080696b5126 100644 --- a/src/postgres/src/backend/optimizer/plan/createplan.c +++ b/src/postgres/src/backend/optimizer/plan/createplan.c @@ -141,9 +141,6 @@ static SeqScan *create_seqscan_plan(PlannerInfo *root, Path *best_path, List *tlist, List *scan_clauses); static SampleScan *create_samplescan_plan(PlannerInfo *root, Path *best_path, List *tlist, List *scan_clauses); -static Scan *create_indexscan_plan(PlannerInfo *root, IndexPath *best_path, - List *tlist, List *scan_clauses, bool indexonly, - bool bitmapindex); static BitmapHeapScan *create_bitmap_scan_plan(PlannerInfo *root, BitmapHeapPath *best_path, List *tlist, List *scan_clauses); @@ -151,8 +148,8 @@ static YbBitmapTableScan *create_yb_bitmap_scan_plan(PlannerInfo *root, YbBitmapTablePath *best_path, List *tlist, List *scan_clauses); static Plan *create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual, - List **qual, List **indexqual, List **indexpushdownqual, - List **indexECs, List *tlist, List **scan_clauses); + List **qual, List **indexqual, List **indexECs, + List *tlist, List **scan_clauses); static void bitmap_subplan_mark_shared(Plan *plan); static TidScan *create_tidscan_plan(PlannerInfo *root, TidPath *best_path, List *tlist, List *scan_clauses); @@ -4204,7 +4201,7 @@ YbBuildIndexqualForRecheck(List *indexquals, IndexOptInfo* indexinfo) * us which to build --- we don't look at best_path->path.pathtype, because * create_bitmap_subplan needs to be able to override the prior decision. */ -static Scan * +Scan * create_indexscan_plan(PlannerInfo *root, IndexPath *best_path, List *tlist, @@ -4501,7 +4498,6 @@ create_bitmap_scan_plan(PlannerInfo *root, /* Process the bitmapqual tree into a Plan tree and qual lists */ bitmapqualplan = create_bitmap_subplan(root, best_path->bitmapqual, &bitmapqualorig, &indexquals, - NULL /* indexpushdownquals */, &indexECs, tlist, &scan_clauses); if (best_path->path.parallel_aware) @@ -4605,7 +4601,7 @@ create_yb_bitmap_scan_plan(PlannerInfo *root, Plan *bitmapqualplan; List *indexqual; List *indexquals; - List *indexpushdownquals; + List *allindexquals; List *indexECs; List *qpqual; ListCell *l; @@ -4618,8 +4614,10 @@ create_yb_bitmap_scan_plan(PlannerInfo *root, /* Process the bitmapqual tree into a Plan tree and qual lists */ bitmapqualplan = create_bitmap_subplan(root, best_path->bitmapqual, &indexqual, &indexquals, - &indexpushdownquals, &indexECs, - tlist, &scan_clauses); + &indexECs, tlist, &scan_clauses); + + allindexquals = yb_get_bitmap_index_quals(root, best_path->bitmapqual, + scan_clauses); /* * The qpqual list must contain all restrictions not automatically handled @@ -4655,15 +4653,12 @@ create_yb_bitmap_scan_plan(PlannerInfo *root, if (rinfo->pseudoconstant) continue; /* we may drop pseudoconstants here */ - if (list_member(indexquals, clause)) - continue; /* simple duplicate */ - if (list_member(indexpushdownquals, clause)) + if (list_member(allindexquals, clause)) continue; /* simple duplicate */ if (rinfo->parent_ec && list_member_ptr(indexECs, rinfo->parent_ec)) continue; /* derived from same EquivalenceClass */ if (!contain_mutable_functions(clause) && - (predicate_implied_by(list_make1(clause), indexquals, false) || - predicate_implied_by(list_make1(clause), indexpushdownquals, false))) + predicate_implied_by(list_make1(clause), allindexquals, false)) continue; /* provably implied by indexquals or * indexpushdownquals */ qpqual = lappend(qpqual, rinfo); @@ -4788,8 +4783,8 @@ create_yb_bitmap_scan_plan(PlannerInfo *root, */ static Plan * create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual, - List **qual, List **indexqual, List **indexpushdownquals, - List **indexECs, List *tlist, List **scan_clauses) + List **qual, List **indexqual, List **indexECs, + List *tlist, List **scan_clauses) { Plan *plan; @@ -4799,7 +4794,6 @@ create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual, List *subplans = NIL; List *subquals = NIL; List *subindexquals = NIL; - List *subindexpushdownquals = NIL; List *subindexECs = NIL; ListCell *l; @@ -4815,18 +4809,14 @@ create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual, Plan *subplan; List *subqual; List *subindexqual; - List *subindexpushdownqual; List *subindexEC; subplan = create_bitmap_subplan(root, (Path *) lfirst(l), &subqual, &subindexqual, - &subindexpushdownqual, &subindexEC, - tlist, scan_clauses); + &subindexEC, tlist, scan_clauses); subplans = lappend(subplans, subplan); subquals = list_concat_unique(subquals, subqual); subindexquals = list_concat_unique(subindexquals, subindexqual); - subindexpushdownquals = list_concat_unique(subindexpushdownquals, - subindexpushdownqual); /* Duplicates in indexECs aren't worth getting rid of */ subindexECs = list_concat(subindexECs, subindexEC); } @@ -4840,8 +4830,6 @@ create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual, plan->parallel_safe = apath->path.parallel_safe; *qual = subquals; *indexqual = subindexquals; - if (indexpushdownquals) - *indexpushdownquals = subindexpushdownquals; *indexECs = subindexECs; } else if (IsA(bitmapqual, BitmapOrPath)) @@ -4850,7 +4838,6 @@ create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual, List *subplans = NIL; List *subquals = NIL; List *subindexquals = NIL; - List *subindexpushdownquals = NIL; bool const_true_subqual = false; bool const_true_subindexqual = false; ListCell *l; @@ -4869,13 +4856,11 @@ create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual, Plan *subplan; List *subqual; List *subindexqual; - List *subindexpushdownqual; List *subindexEC; subplan = create_bitmap_subplan(root, (Path *) lfirst(l), &subqual, &subindexqual, - &subindexpushdownqual, &subindexEC, - tlist, scan_clauses); + &subindexEC, tlist, scan_clauses); subplans = lappend(subplans, subplan); if (subqual == NIL) const_true_subqual = true; @@ -4885,12 +4870,8 @@ create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual, if (subindexqual == NIL) const_true_subindexqual = true; else if (!const_true_subindexqual) - { subindexquals = lappend(subindexquals, make_ands_explicit(subindexqual)); - subindexpushdownquals = lappend(subindexpushdownquals, - make_ands_explicit(subindexpushdownqual)); - } } /* @@ -4925,24 +4906,11 @@ create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual, else *qual = list_make1(make_orclause(subquals)); if (const_true_subindexqual) - { *indexqual = NIL; - if (indexpushdownquals) - *indexpushdownquals = NIL; - } else if (list_length(subindexquals) <= 1) - { *indexqual = subindexquals; - if (indexpushdownquals) - *indexpushdownquals = subindexpushdownquals; - } else - { *indexqual = list_make1(make_orclause(subindexquals)); - if (indexpushdownquals) - *indexpushdownquals = list_make1( - make_orclause(subindexpushdownquals)); - } *indexECs = NIL; } else if (IsA(bitmapqual, IndexPath)) @@ -5025,11 +4993,6 @@ create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual, *qual = subquals; *indexqual = subindexquals; *indexECs = subindexECs; - - if (indexpushdownquals) - *indexpushdownquals = ipath->indexinfo->rel->is_yb_relation - ? ((YbBitmapIndexScan *) plan)->yb_idx_pushdown.quals - : NIL; } else { diff --git a/src/postgres/src/include/optimizer/planmain.h b/src/postgres/src/include/optimizer/planmain.h index 7fb9493882d8..d3585a5658f9 100644 --- a/src/postgres/src/include/optimizer/planmain.h +++ b/src/postgres/src/include/optimizer/planmain.h @@ -43,6 +43,11 @@ extern ForeignScan *make_foreignscan(List *qptlist, List *qpqual, Index scanrelid, List *fdw_exprs, List *fdw_private, List *fdw_scan_tlist, List *fdw_recheck_quals, Plan *outer_plan); +extern Scan *create_indexscan_plan(PlannerInfo *root, IndexPath *best_path, + List *tlist, List *scan_clauses, + bool indexonly, bool bitmapindex); +extern List *yb_get_bitmap_index_quals(PlannerInfo *root, Path *bitmapqual, + List *scan_clauses); extern Plan *change_plan_targetlist(Plan *subplan, List *tlist, bool tlist_parallel_safe); extern Plan *materialize_finished_plan(Plan *subplan); diff --git a/src/postgres/src/test/regress/expected/yb_bitmap_scans.out b/src/postgres/src/test/regress/expected/yb_bitmap_scans.out index 8561ecc3db4b..ebd0cd33b88e 100644 --- a/src/postgres/src/test/regress/expected/yb_bitmap_scans.out +++ b/src/postgres/src/test/regress/expected/yb_bitmap_scans.out @@ -273,7 +273,6 @@ SELECT COUNT(*) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); -------------------------------------------------------------------------------- Finalize Aggregate (actual rows=1 loops=1) -> YB Bitmap Table Scan on test_skip (actual rows=3 loops=1) - Storage Filter: ((a = 1) OR ((b < 10) AND ((b % 2) = 0))) Storage Table Read Requests: 1 Storage Table Rows Scanned: 5 Partial Aggregate: true @@ -287,7 +286,7 @@ SELECT COUNT(*) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); Storage Index Filter: ((b % 2) = 0) Storage Index Read Requests: 1 Storage Index Rows Scanned: 9 -(16 rows) +(15 rows) /*+ BitmapScan(test_skip) */ SELECT COUNT(*) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); @@ -303,7 +302,6 @@ SELECT SUM(a) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); -------------------------------------------------------------------------------- Finalize Aggregate (actual rows=1 loops=1) -> YB Bitmap Table Scan on test_skip (actual rows=3 loops=1) - Storage Filter: ((a = 1) OR ((b < 10) AND ((b % 2) = 0))) Storage Table Read Requests: 1 Storage Table Rows Scanned: 5 Partial Aggregate: true @@ -317,7 +315,7 @@ SELECT SUM(a) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); Storage Index Filter: ((b % 2) = 0) Storage Index Read Requests: 1 Storage Index Rows Scanned: 9 -(16 rows) +(15 rows) /*+ BitmapScan(test_skip) */ SELECT SUM(a) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); @@ -332,7 +330,6 @@ SELECT MAX(a) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); -------------------------------------------------------------------------------- Finalize Aggregate (actual rows=1 loops=1) -> YB Bitmap Table Scan on test_skip (actual rows=3 loops=1) - Storage Filter: ((a = 1) OR ((b < 10) AND ((b % 2) = 0))) Storage Table Read Requests: 1 Storage Table Rows Scanned: 5 Partial Aggregate: true @@ -346,7 +343,7 @@ SELECT MAX(a) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); Storage Index Filter: ((b % 2) = 0) Storage Index Read Requests: 1 Storage Index Rows Scanned: 9 -(16 rows) +(15 rows) /*+ BitmapScan(test_skip) */ SELECT MAX(a) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); @@ -361,7 +358,6 @@ SELECT 1 FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); QUERY PLAN -------------------------------------------------------------------------- YB Bitmap Table Scan on test_skip (actual rows=5 loops=1) - Storage Filter: ((a = 1) OR ((b < 10) AND ((b % 2) = 0))) Storage Table Read Requests: 1 Storage Table Rows Scanned: 5 -> BitmapOr (actual rows=5 loops=1) @@ -374,7 +370,7 @@ SELECT 1 FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); Storage Index Filter: ((b % 2) = 0) Storage Index Read Requests: 1 Storage Index Rows Scanned: 9 -(14 rows) +(13 rows) /*+ BitmapScan(test_skip) */ SELECT 1 FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); @@ -392,7 +388,6 @@ SELECT random() FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); QUERY PLAN -------------------------------------------------------------------------- YB Bitmap Table Scan on test_skip (actual rows=5 loops=1) - Storage Filter: ((a = 1) OR ((b < 10) AND ((b % 2) = 0))) Storage Table Read Requests: 1 Storage Table Rows Scanned: 5 -> BitmapOr (actual rows=5 loops=1) @@ -405,7 +400,7 @@ SELECT random() FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); Storage Index Filter: ((b % 2) = 0) Storage Index Read Requests: 1 Storage Index Rows Scanned: 9 -(14 rows) +(13 rows) -- -- test primary key queries @@ -665,13 +660,12 @@ SELECT * FROM multi WHERE a < 2 OR (b > 1797 AND (c BETWEEN 2709 AND 2712 OR c = /*+ BitmapScan(multi) */ EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF) SELECT * FROM multi WHERE (a < 3 AND a % 2 = 0) OR (b IN (10, 270, 1800) AND (c < 20 OR c > 2500)) ORDER BY a; - QUERY PLAN --------------------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +----------------------------------------------------------------------------------------------- Sort (actual rows=3 loops=1) Sort Key: a Sort Method: quicksort -> YB Bitmap Table Scan on multi (actual rows=3 loops=1) - Storage Filter: (((a < 3) AND ((a % 2) = 0)) OR ((b = ANY ('{10,270,1800}'::integer[])) AND ((c < 20) OR (c > 2500)))) -> BitmapOr (actual rows=3 loops=1) -> Bitmap Index Scan on multi_pkey (actual rows=1 loops=1) Index Cond: (a < 3) @@ -681,7 +675,7 @@ SELECT * FROM multi WHERE (a < 3 AND a % 2 = 0) OR (b IN (10, 270, 1800) AND (c Index Cond: ((b = ANY ('{10,270,1800}'::integer[])) AND (c < 20)) -> Bitmap Index Scan on multi_b_c_idx (actual rows=1 loops=1) Index Cond: ((b = ANY ('{10,270,1800}'::integer[])) AND (c > 2500)) -(14 rows) +(13 rows) /*+ BitmapScan(multi) */ SELECT * FROM multi WHERE (a < 3 AND a % 2 = 0) OR (b IN (10, 270, 1800) AND (c < 20 OR c > 2500)) ORDER BY a; @@ -1066,7 +1060,6 @@ SELECT * FROM multi WHERE (a < 5 AND a % 2 = 0) OR (c <= 10 AND a % 3 = 0) ORDER Sort Key: a Sort Method: quicksort -> YB Bitmap Table Scan on multi (actual rows=3 loops=1) - Storage Filter: (((a < 5) AND ((a % 2) = 0)) OR ((c <= 10) AND ((a % 3) = 0))) Storage Table Read Requests: 1 Storage Table Rows Scanned: 3 -> BitmapOr (actual rows=3 loops=1) @@ -1080,7 +1073,7 @@ SELECT * FROM multi WHERE (a < 5 AND a % 2 = 0) OR (c <= 10 AND a % 3 = 0) ORDER Storage Index Filter: (((a % 3) = 0) AND (((a < 5) AND ((a % 2) = 0)) OR ((c <= 10) AND ((a % 3) = 0)))) Storage Index Read Requests: 1 Storage Index Rows Scanned: 3 -(18 rows) +(17 rows) /*+ BitmapScan(multi) */ SELECT * FROM multi WHERE (a < 5 AND a % 2 = 0) OR (c <= 10 AND a % 3 = 0) ORDER BY a; @@ -1902,5 +1895,147 @@ EXPLAIN (COSTS OFF) /*+ BitmapScan(t2) */ SELECT * FROM t2 AS a JOIN t2 AS b ON Index Cond: (k1 = a.k2) (9 rows) +-- +-- BitmapAnd +-- These tests require CBO because the basic cost model does not properly cost +-- remote filters, so BitmapAnds were comparatively more expensive. +-- +-- Hints can tell the planner to use a Bitmap Scan, but they cannot tell the +-- planner how to design the plan. The planner chooses for itself how it should +-- apply remote filters, Bitmap Ands, Bitmap Ors. For these queries, we are +-- interested in testing the behaviour of a BitmapAnd, not testing the planner's +-- ability to choose between a bitmap scan and a sequential scan. IF the planner +-- must choose a bitmap scan, what plan does it choose? Does that plan work? +-- +SET yb_enable_base_scans_cost_model = true; +CREATE TABLE test_and (a INT, b INT, c INT); +CREATE INDEX ON test_and (a ASC); +CREATE INDEX ON test_and (b ASC); +CREATE INDEX ON test_and (c ASC); +INSERT INTO test_and SELECT i, j, k FROM generate_series(1, 20) i, generate_series(1, 20) j, generate_series(1, 20) k; +ANALYZE test_and; +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 6 AND b < 6; + QUERY PLAN +---------------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=500 loops=1) + -> BitmapAnd (actual rows=500 loops=1) + -> Bitmap Index Scan on test_and_b_idx (actual rows=2000 loops=1) + Index Cond: (b < 6) + -> Bitmap Index Scan on test_and_a_idx (actual rows=2000 loops=1) + Index Cond: (a < 6) +(6 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 6 AND c < 6; + QUERY PLAN +---------------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=500 loops=1) + -> BitmapAnd (actual rows=500 loops=1) + -> Bitmap Index Scan on test_and_c_idx (actual rows=2000 loops=1) + Index Cond: (c < 6) + -> Bitmap Index Scan on test_and_a_idx (actual rows=2000 loops=1) + Index Cond: (a < 6) +(6 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE b < 6 AND c < 6; + QUERY PLAN +---------------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=500 loops=1) + -> BitmapAnd (actual rows=500 loops=1) + -> Bitmap Index Scan on test_and_c_idx (actual rows=2000 loops=1) + Index Cond: (c < 6) + -> Bitmap Index Scan on test_and_b_idx (actual rows=2000 loops=1) + Index Cond: (b < 6) +(6 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 6 AND b < 6 AND c < 7; + QUERY PLAN +---------------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=150 loops=1) + Storage Filter: (c < 7) + -> BitmapAnd (actual rows=500 loops=1) + -> Bitmap Index Scan on test_and_b_idx (actual rows=2000 loops=1) + Index Cond: (b < 6) + -> Bitmap Index Scan on test_and_a_idx (actual rows=2000 loops=1) + Index Cond: (a < 6) +(7 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 10 AND b < 10; + QUERY PLAN +---------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=1620 loops=1) + Storage Filter: (a < 10) + -> Bitmap Index Scan on test_and_b_idx (actual rows=3600 loops=1) + Index Cond: (b < 10) +(4 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 10 AND c < 10; + QUERY PLAN +---------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=1620 loops=1) + Storage Filter: (a < 10) + -> Bitmap Index Scan on test_and_c_idx (actual rows=3600 loops=1) + Index Cond: (c < 10) +(4 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE b < 10 AND c < 10; + QUERY PLAN +---------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=1620 loops=1) + Storage Filter: (b < 10) + -> Bitmap Index Scan on test_and_c_idx (actual rows=3600 loops=1) + Index Cond: (c < 10) +(4 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 10 AND b < 10 AND c < 10; + QUERY PLAN +---------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=729 loops=1) + Storage Filter: ((a < 10) AND (b < 10)) + -> Bitmap Index Scan on test_and_c_idx (actual rows=3600 loops=1) + Index Cond: (c < 10) +(4 rows) + +-- complex nested queries +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 5 AND (b < 3 OR b > 16); + QUERY PLAN +---------------------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=480 loops=1) + -> BitmapAnd (actual rows=480 loops=1) + -> BitmapOr (actual rows=2400 loops=1) + -> Bitmap Index Scan on test_and_b_idx (actual rows=800 loops=1) + Index Cond: (b < 3) + -> Bitmap Index Scan on test_and_b_idx (actual rows=1600 loops=1) + Index Cond: (b > 16) + -> Bitmap Index Scan on test_and_a_idx (actual rows=1600 loops=1) + Index Cond: (a < 5) +(9 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE (b < 3 AND a < 5) OR (b > 16 AND a < 5); + QUERY PLAN +---------------------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=480 loops=1) + -> BitmapAnd (actual rows=480 loops=1) + -> BitmapOr (actual rows=2400 loops=1) + -> Bitmap Index Scan on test_and_b_idx (actual rows=800 loops=1) + Index Cond: (b < 3) + -> Bitmap Index Scan on test_and_b_idx (actual rows=1600 loops=1) + Index Cond: (b > 16) + -> Bitmap Index Scan on test_and_a_idx (actual rows=1600 loops=1) + Index Cond: (a < 5) +(9 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE (b < 3 AND a < 5) OR (b > 16 AND a < 6); + QUERY PLAN +---------------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=560 loops=1) + Storage Filter: (((b < 3) AND (a < 5)) OR ((b > 16) AND (a < 6))) + -> BitmapOr (actual rows=2400 loops=1) + -> Bitmap Index Scan on test_and_b_idx (actual rows=800 loops=1) + Index Cond: (b < 3) + -> Bitmap Index Scan on test_and_b_idx (actual rows=1600 loops=1) + Index Cond: (b > 16) +(7 rows) + +RESET yb_enable_base_scans_cost_model; RESET yb_explain_hide_non_deterministic_fields; RESET enable_bitmapscan; diff --git a/src/postgres/src/test/regress/expected/yb_bitmap_scans_colo.out b/src/postgres/src/test/regress/expected/yb_bitmap_scans_colo.out index 33e4a4071865..c3ef689d7abe 100644 --- a/src/postgres/src/test/regress/expected/yb_bitmap_scans_colo.out +++ b/src/postgres/src/test/regress/expected/yb_bitmap_scans_colo.out @@ -284,7 +284,6 @@ SELECT COUNT(*) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); -------------------------------------------------------------------------------- Finalize Aggregate (actual rows=1 loops=1) -> YB Bitmap Table Scan on test_skip (actual rows=1 loops=1) - Storage Filter: ((a = 1) OR ((b < 10) AND ((b % 2) = 0))) Storage Table Read Requests: 1 Storage Table Rows Scanned: 5 Partial Aggregate: true @@ -299,7 +298,7 @@ SELECT COUNT(*) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); Storage Table Rows Scanned: 9 Storage Index Read Requests: 1 Storage Index Rows Scanned: 9 -(17 rows) +(16 rows) /*+ BitmapScan(test_skip) */ SELECT COUNT(*) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); @@ -315,7 +314,6 @@ SELECT SUM(a) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); -------------------------------------------------------------------------------- Finalize Aggregate (actual rows=1 loops=1) -> YB Bitmap Table Scan on test_skip (actual rows=1 loops=1) - Storage Filter: ((a = 1) OR ((b < 10) AND ((b % 2) = 0))) Storage Table Read Requests: 1 Storage Table Rows Scanned: 5 Partial Aggregate: true @@ -330,7 +328,7 @@ SELECT SUM(a) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); Storage Table Rows Scanned: 9 Storage Index Read Requests: 1 Storage Index Rows Scanned: 9 -(17 rows) +(16 rows) /*+ BitmapScan(test_skip) */ SELECT SUM(a) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); @@ -345,7 +343,6 @@ SELECT MAX(a) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); -------------------------------------------------------------------------------- Finalize Aggregate (actual rows=1 loops=1) -> YB Bitmap Table Scan on test_skip (actual rows=1 loops=1) - Storage Filter: ((a = 1) OR ((b < 10) AND ((b % 2) = 0))) Storage Table Read Requests: 1 Storage Table Rows Scanned: 5 Partial Aggregate: true @@ -360,7 +357,7 @@ SELECT MAX(a) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); Storage Table Rows Scanned: 9 Storage Index Read Requests: 1 Storage Index Rows Scanned: 9 -(17 rows) +(16 rows) /*+ BitmapScan(test_skip) */ SELECT MAX(a) FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); @@ -375,7 +372,6 @@ SELECT 1 FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); QUERY PLAN -------------------------------------------------------------------------- YB Bitmap Table Scan on test_skip (actual rows=5 loops=1) - Storage Filter: ((a = 1) OR ((b < 10) AND ((b % 2) = 0))) Storage Table Read Requests: 1 Storage Table Rows Scanned: 5 -> BitmapOr (actual rows=5 loops=1) @@ -389,7 +385,7 @@ SELECT 1 FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); Storage Table Rows Scanned: 9 Storage Index Read Requests: 1 Storage Index Rows Scanned: 9 -(15 rows) +(14 rows) /*+ BitmapScan(test_skip) */ SELECT 1 FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); @@ -407,7 +403,6 @@ SELECT random() FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); QUERY PLAN -------------------------------------------------------------------------- YB Bitmap Table Scan on test_skip (actual rows=5 loops=1) - Storage Filter: ((a = 1) OR ((b < 10) AND ((b % 2) = 0))) Storage Table Read Requests: 1 Storage Table Rows Scanned: 5 -> BitmapOr (actual rows=5 loops=1) @@ -421,7 +416,7 @@ SELECT random() FROM test_skip WHERE a = 1 OR (b < 10 AND b % 2 = 0); Storage Table Rows Scanned: 9 Storage Index Read Requests: 1 Storage Index Rows Scanned: 9 -(15 rows) +(14 rows) -- -- test primary key queries @@ -688,13 +683,12 @@ SELECT * FROM multi WHERE a < 2 OR (b > 1797 AND (c BETWEEN 2709 AND 2712 OR c = /*+ BitmapScan(multi) */ EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF) SELECT * FROM multi WHERE (a < 3 AND a % 2 = 0) OR (b IN (10, 270, 1800) AND (c < 20 OR c > 2500)) ORDER BY a; - QUERY PLAN --------------------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +----------------------------------------------------------------------------------------------- Sort (actual rows=3 loops=1) Sort Key: a Sort Method: quicksort -> YB Bitmap Table Scan on multi (actual rows=3 loops=1) - Storage Filter: (((a < 3) AND ((a % 2) = 0)) OR ((b = ANY ('{10,270,1800}'::integer[])) AND ((c < 20) OR (c > 2500)))) -> BitmapOr (actual rows=3 loops=1) -> Bitmap Index Scan on multi_pkey (actual rows=1 loops=1) Index Cond: (a < 3) @@ -704,7 +698,7 @@ SELECT * FROM multi WHERE (a < 3 AND a % 2 = 0) OR (b IN (10, 270, 1800) AND (c Index Cond: ((b = ANY ('{10,270,1800}'::integer[])) AND (c < 20)) -> Bitmap Index Scan on multi_b_c_idx (actual rows=1 loops=1) Index Cond: ((b = ANY ('{10,270,1800}'::integer[])) AND (c > 2500)) -(14 rows) +(13 rows) /*+ BitmapScan(multi) */ SELECT * FROM multi WHERE (a < 3 AND a % 2 = 0) OR (b IN (10, 270, 1800) AND (c < 20 OR c > 2500)) ORDER BY a; @@ -1092,7 +1086,6 @@ SELECT * FROM multi WHERE (a < 5 AND a % 2 = 0) OR (c <= 10 AND a % 3 = 0) ORDER Sort Key: a Sort Method: quicksort -> YB Bitmap Table Scan on multi (actual rows=3 loops=1) - Storage Filter: (((a < 5) AND ((a % 2) = 0)) OR ((c <= 10) AND ((a % 3) = 0))) Storage Table Read Requests: 1 Storage Table Rows Scanned: 3 -> BitmapOr (actual rows=3 loops=1) @@ -1107,7 +1100,7 @@ SELECT * FROM multi WHERE (a < 5 AND a % 2 = 0) OR (c <= 10 AND a % 3 = 0) ORDER Storage Table Rows Scanned: 3 Storage Index Read Requests: 1 Storage Index Rows Scanned: 3 -(19 rows) +(18 rows) /*+ BitmapScan(multi) */ SELECT * FROM multi WHERE (a < 5 AND a % 2 = 0) OR (c <= 10 AND a % 3 = 0) ORDER BY a; @@ -1929,5 +1922,147 @@ SELECT * FROM test_crash_row_comp AS t WHERE (a, a) <= (10, 1) OR a IS NULL; 1 (1 row) +-- +-- BitmapAnd +-- These tests require CBO because the basic cost model does not properly cost +-- remote filters, so BitmapAnds were comparatively more expensive. +-- +-- Hints can tell the planner to use a Bitmap Scan, but they cannot tell the +-- planner how to design the plan. The planner chooses for itself how it should +-- apply remote filters, Bitmap Ands, Bitmap Ors. For these queries, we are +-- interested in testing the behaviour of a BitmapAnd, not testing the planner's +-- ability to choose between a bitmap scan and a sequential scan. IF the planner +-- must choose a bitmap scan, what plan does it choose? Does that plan work? +-- +SET yb_enable_base_scans_cost_model = true; +CREATE TABLE test_and (a INT, b INT, c INT); +CREATE INDEX ON test_and (a ASC); +CREATE INDEX ON test_and (b ASC); +CREATE INDEX ON test_and (c ASC); +INSERT INTO test_and SELECT i, j, k FROM generate_series(1, 20) i, generate_series(1, 20) j, generate_series(1, 20) k; +ANALYZE test_and; +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 6 AND b < 6; + QUERY PLAN +---------------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=500 loops=1) + -> BitmapAnd (actual rows=500 loops=1) + -> Bitmap Index Scan on test_and_b_idx (actual rows=2000 loops=1) + Index Cond: (b < 6) + -> Bitmap Index Scan on test_and_a_idx (actual rows=2000 loops=1) + Index Cond: (a < 6) +(6 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 6 AND c < 6; + QUERY PLAN +---------------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=500 loops=1) + -> BitmapAnd (actual rows=500 loops=1) + -> Bitmap Index Scan on test_and_c_idx (actual rows=2000 loops=1) + Index Cond: (c < 6) + -> Bitmap Index Scan on test_and_a_idx (actual rows=2000 loops=1) + Index Cond: (a < 6) +(6 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE b < 6 AND c < 6; + QUERY PLAN +---------------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=500 loops=1) + -> BitmapAnd (actual rows=500 loops=1) + -> Bitmap Index Scan on test_and_c_idx (actual rows=2000 loops=1) + Index Cond: (c < 6) + -> Bitmap Index Scan on test_and_b_idx (actual rows=2000 loops=1) + Index Cond: (b < 6) +(6 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 6 AND b < 6 AND c < 7; + QUERY PLAN +---------------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=150 loops=1) + Storage Filter: (c < 7) + -> BitmapAnd (actual rows=500 loops=1) + -> Bitmap Index Scan on test_and_b_idx (actual rows=2000 loops=1) + Index Cond: (b < 6) + -> Bitmap Index Scan on test_and_a_idx (actual rows=2000 loops=1) + Index Cond: (a < 6) +(7 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 10 AND b < 10; + QUERY PLAN +---------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=1620 loops=1) + Storage Filter: (a < 10) + -> Bitmap Index Scan on test_and_b_idx (actual rows=3600 loops=1) + Index Cond: (b < 10) +(4 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 10 AND c < 10; + QUERY PLAN +---------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=1620 loops=1) + Storage Filter: (a < 10) + -> Bitmap Index Scan on test_and_c_idx (actual rows=3600 loops=1) + Index Cond: (c < 10) +(4 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE b < 10 AND c < 10; + QUERY PLAN +---------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=1620 loops=1) + Storage Filter: (b < 10) + -> Bitmap Index Scan on test_and_c_idx (actual rows=3600 loops=1) + Index Cond: (c < 10) +(4 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 10 AND b < 10 AND c < 10; + QUERY PLAN +---------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=729 loops=1) + Storage Filter: ((a < 10) AND (b < 10)) + -> Bitmap Index Scan on test_and_c_idx (actual rows=3600 loops=1) + Index Cond: (c < 10) +(4 rows) + +-- complex nested queries +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 5 AND (b < 3 OR b > 16); + QUERY PLAN +---------------------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=480 loops=1) + -> BitmapAnd (actual rows=480 loops=1) + -> BitmapOr (actual rows=2400 loops=1) + -> Bitmap Index Scan on test_and_b_idx (actual rows=800 loops=1) + Index Cond: (b < 3) + -> Bitmap Index Scan on test_and_b_idx (actual rows=1600 loops=1) + Index Cond: (b > 16) + -> Bitmap Index Scan on test_and_a_idx (actual rows=1600 loops=1) + Index Cond: (a < 5) +(9 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE (b < 3 AND a < 5) OR (b > 16 AND a < 5); + QUERY PLAN +---------------------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=480 loops=1) + -> BitmapAnd (actual rows=480 loops=1) + -> BitmapOr (actual rows=2400 loops=1) + -> Bitmap Index Scan on test_and_b_idx (actual rows=800 loops=1) + Index Cond: (b < 3) + -> Bitmap Index Scan on test_and_b_idx (actual rows=1600 loops=1) + Index Cond: (b > 16) + -> Bitmap Index Scan on test_and_a_idx (actual rows=1600 loops=1) + Index Cond: (a < 5) +(9 rows) + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE (b < 3 AND a < 5) OR (b > 16 AND a < 6); + QUERY PLAN +---------------------------------------------------------------------------- + YB Bitmap Table Scan on test_and t (actual rows=560 loops=1) + Storage Filter: (((b < 3) AND (a < 5)) OR ((b > 16) AND (a < 6))) + -> BitmapOr (actual rows=2400 loops=1) + -> Bitmap Index Scan on test_and_b_idx (actual rows=800 loops=1) + Index Cond: (b < 3) + -> Bitmap Index Scan on test_and_b_idx (actual rows=1600 loops=1) + Index Cond: (b > 16) +(7 rows) + +RESET yb_enable_base_scans_cost_model; RESET yb_explain_hide_non_deterministic_fields; RESET enable_bitmapscan; diff --git a/src/postgres/src/test/regress/expected/yb_bitmap_scans_flags.out b/src/postgres/src/test/regress/expected/yb_bitmap_scans_flags.out index efa7d609d234..de755341a352 100644 --- a/src/postgres/src/test/regress/expected/yb_bitmap_scans_flags.out +++ b/src/postgres/src/test/regress/expected/yb_bitmap_scans_flags.out @@ -103,18 +103,6 @@ EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; Index Cond: (b < 5) (6 rows) -/*+ Set(enable_seqscan false) */ -EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; - QUERY PLAN ------------------------------------------------------ - YB Bitmap Table Scan on test_disable - -> BitmapOr - -> Bitmap Index Scan on test_disable_a_idx - Index Cond: (a < 5) - -> Bitmap Index Scan on test_disable_b_idx - Index Cond: (b < 5) -(6 rows) - EXPLAIN (COSTS OFF) SELECT * FROM tmp_test_disable WHERE a < 5 OR b < 5; QUERY PLAN -------------------------------- @@ -135,14 +123,6 @@ EXPLAIN (COSTS OFF) SELECT * FROM tmp_test_disable WHERE a < 5 OR b < 5; Index Cond: (b < 5) (7 rows) -/*+ Set(enable_seqscan false) */ -EXPLAIN (COSTS OFF) SELECT * FROM tmp_test_disable WHERE a < 5 OR b < 5; - QUERY PLAN --------------------------------- - Seq Scan on tmp_test_disable - Filter: ((a < 5) OR (b < 5)) -(2 rows) - SET yb_enable_bitmapscan = false; SET enable_bitmapscan = true; EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; @@ -160,14 +140,6 @@ EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; Storage Filter: ((a < 5) OR (b < 5)) (2 rows) -/*+ Set(enable_seqscan false) */ -EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; - QUERY PLAN ----------------------------------------- - Seq Scan on test_disable - Storage Filter: ((a < 5) OR (b < 5)) -(2 rows) - EXPLAIN (COSTS OFF) SELECT * FROM tmp_test_disable WHERE a < 5 OR b < 5; QUERY PLAN -------------------------------- @@ -218,14 +190,6 @@ EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; Storage Filter: ((a < 5) OR (b < 5)) (2 rows) -/*+ Set(enable_seqscan false) */ -EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; - QUERY PLAN ----------------------------------------- - Seq Scan on test_disable - Storage Filter: ((a < 5) OR (b < 5)) -(2 rows) - EXPLAIN (COSTS OFF) SELECT * FROM tmp_test_disable WHERE a < 5 OR b < 5; QUERY PLAN -------------------------------- @@ -246,13 +210,5 @@ EXPLAIN (COSTS OFF) SELECT * FROM tmp_test_disable WHERE a < 5 OR b < 5; Index Cond: (b < 5) (7 rows) -/*+ Set(enable_seqscan false) */ -EXPLAIN (COSTS OFF) SELECT * FROM tmp_test_disable WHERE a < 5 OR b < 5; - QUERY PLAN --------------------------------- - Seq Scan on tmp_test_disable - Filter: ((a < 5) OR (b < 5)) -(2 rows) - RESET enable_bitmapscan; RESET yb_enable_bitmapscan; diff --git a/src/postgres/src/test/regress/sql/yb_bitmap_scans.sql b/src/postgres/src/test/regress/sql/yb_bitmap_scans.sql index c9d16b1e5e5d..15b44be76152 100644 --- a/src/postgres/src/test/regress/sql/yb_bitmap_scans.sql +++ b/src/postgres/src/test/regress/sql/yb_bitmap_scans.sql @@ -524,5 +524,42 @@ CREATE INDEX ON t2 (k1 ASC); CREATE INDEX ON t2 (k2 ASC); EXPLAIN (COSTS OFF) /*+ BitmapScan(t2) */ SELECT * FROM t2 AS a JOIN t2 AS b ON a.k1 = b.k1 OR (a.k2 = b.k1 AND a.k1 = 1); +-- +-- BitmapAnd +-- These tests require CBO because the basic cost model does not properly cost +-- remote filters, so BitmapAnds were comparatively more expensive. +-- +-- Hints can tell the planner to use a Bitmap Scan, but they cannot tell the +-- planner how to design the plan. The planner chooses for itself how it should +-- apply remote filters, Bitmap Ands, Bitmap Ors. For these queries, we are +-- interested in testing the behaviour of a BitmapAnd, not testing the planner's +-- ability to choose between a bitmap scan and a sequential scan. IF the planner +-- must choose a bitmap scan, what plan does it choose? Does that plan work? +-- +SET yb_enable_base_scans_cost_model = true; + +CREATE TABLE test_and (a INT, b INT, c INT); +CREATE INDEX ON test_and (a ASC); +CREATE INDEX ON test_and (b ASC); +CREATE INDEX ON test_and (c ASC); +INSERT INTO test_and SELECT i, j, k FROM generate_series(1, 20) i, generate_series(1, 20) j, generate_series(1, 20) k; +ANALYZE test_and; + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 6 AND b < 6; +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 6 AND c < 6; +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE b < 6 AND c < 6; +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 6 AND b < 6 AND c < 7; + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 10 AND b < 10; +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 10 AND c < 10; +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE b < 10 AND c < 10; +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 10 AND b < 10 AND c < 10; + +-- complex nested queries +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 5 AND (b < 3 OR b > 16); +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE (b < 3 AND a < 5) OR (b > 16 AND a < 5); +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE (b < 3 AND a < 5) OR (b > 16 AND a < 6); + +RESET yb_enable_base_scans_cost_model; RESET yb_explain_hide_non_deterministic_fields; RESET enable_bitmapscan; diff --git a/src/postgres/src/test/regress/sql/yb_bitmap_scans_colo.sql b/src/postgres/src/test/regress/sql/yb_bitmap_scans_colo.sql index 7433c7c34048..9ab5608b0540 100644 --- a/src/postgres/src/test/regress/sql/yb_bitmap_scans_colo.sql +++ b/src/postgres/src/test/regress/sql/yb_bitmap_scans_colo.sql @@ -512,5 +512,42 @@ SELECT * FROM test_crash_row_comp AS t WHERE (a, a) <= (10, 1) OR a IS NULL; /*+ BitmapScan(t) */ SELECT * FROM test_crash_row_comp AS t WHERE (a, a) <= (10, 1) OR a IS NULL; +-- +-- BitmapAnd +-- These tests require CBO because the basic cost model does not properly cost +-- remote filters, so BitmapAnds were comparatively more expensive. +-- +-- Hints can tell the planner to use a Bitmap Scan, but they cannot tell the +-- planner how to design the plan. The planner chooses for itself how it should +-- apply remote filters, Bitmap Ands, Bitmap Ors. For these queries, we are +-- interested in testing the behaviour of a BitmapAnd, not testing the planner's +-- ability to choose between a bitmap scan and a sequential scan. IF the planner +-- must choose a bitmap scan, what plan does it choose? Does that plan work? +-- +SET yb_enable_base_scans_cost_model = true; + +CREATE TABLE test_and (a INT, b INT, c INT); +CREATE INDEX ON test_and (a ASC); +CREATE INDEX ON test_and (b ASC); +CREATE INDEX ON test_and (c ASC); +INSERT INTO test_and SELECT i, j, k FROM generate_series(1, 20) i, generate_series(1, 20) j, generate_series(1, 20) k; +ANALYZE test_and; + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 6 AND b < 6; +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 6 AND c < 6; +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE b < 6 AND c < 6; +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 6 AND b < 6 AND c < 7; + +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 10 AND b < 10; +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 10 AND c < 10; +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE b < 10 AND c < 10; +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 10 AND b < 10 AND c < 10; + +-- complex nested queries +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE a < 5 AND (b < 3 OR b > 16); +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE (b < 3 AND a < 5) OR (b > 16 AND a < 5); +/*+ BitmapScan(t) */ EXPLAIN (ANALYZE, SUMMARY OFF, COSTS OFF) SELECT * FROM test_and t WHERE (b < 3 AND a < 5) OR (b > 16 AND a < 6); + +RESET yb_enable_base_scans_cost_model; RESET yb_explain_hide_non_deterministic_fields; RESET enable_bitmapscan; diff --git a/src/postgres/src/test/regress/sql/yb_bitmap_scans_flags.sql b/src/postgres/src/test/regress/sql/yb_bitmap_scans_flags.sql index 6d6dfa9abf36..cd8ea29918ee 100644 --- a/src/postgres/src/test/regress/sql/yb_bitmap_scans_flags.sql +++ b/src/postgres/src/test/regress/sql/yb_bitmap_scans_flags.sql @@ -32,21 +32,15 @@ SET enable_bitmapscan = false; EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; /*+ BitmapScan(test_disable) */ EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; -/*+ Set(enable_seqscan false) */ -EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; EXPLAIN (COSTS OFF) SELECT * FROM tmp_test_disable WHERE a < 5 OR b < 5; /*+ BitmapScan(tmp_test_disable) */ EXPLAIN (COSTS OFF) SELECT * FROM tmp_test_disable WHERE a < 5 OR b < 5; -/*+ Set(enable_seqscan false) */ -EXPLAIN (COSTS OFF) SELECT * FROM tmp_test_disable WHERE a < 5 OR b < 5; SET yb_enable_bitmapscan = false; SET enable_bitmapscan = true; EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; /*+ BitmapScan(test_disable) */ EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; -/*+ Set(enable_seqscan false) */ -EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; EXPLAIN (COSTS OFF) SELECT * FROM tmp_test_disable WHERE a < 5 OR b < 5; /*+ BitmapScan(tmp_test_disable) */ EXPLAIN (COSTS OFF) SELECT * FROM tmp_test_disable WHERE a < 5 OR b < 5; @@ -58,13 +52,9 @@ SET enable_bitmapscan = false; EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; /*+ BitmapScan(test_disable) */ EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; -/*+ Set(enable_seqscan false) */ -EXPLAIN (COSTS OFF) SELECT * FROM test_disable WHERE a < 5 OR b < 5; EXPLAIN (COSTS OFF) SELECT * FROM tmp_test_disable WHERE a < 5 OR b < 5; /*+ BitmapScan(tmp_test_disable) */ EXPLAIN (COSTS OFF) SELECT * FROM tmp_test_disable WHERE a < 5 OR b < 5; -/*+ Set(enable_seqscan false) */ -EXPLAIN (COSTS OFF) SELECT * FROM tmp_test_disable WHERE a < 5 OR b < 5; RESET enable_bitmapscan; RESET yb_enable_bitmapscan;