sql: setting to disable NotVisible index feature

Prior to this commit, not visible indexes were always ignored by the optimizer
unless it is explicitly selected with index hinting or used for constraint
check. This commit adds a session variable along with a cluster setting to
control whether the optimizer can still choose to use not visible indexes for
the query plan.

Note that even if the the session variable is enabled and optimizer can choose
to use not visible indexes, not visible indexes remain as not visible in index
descriptors.

Cluster setting: `sql.defaults.optimizer_use_not_visible_indexes.enabled`
Session setting: `optimizer_use_not_visible_indexes` By default, the setting is
disabled, meaning not visible index will be properly ignored. When the setting
is enabled, optimizer treats not visible indexes as they are visible and can
choose to use them for query plan.

Assists: cockroachdb#82363

Release note (sql change): Session setting `optimizer_use_not_visible_indexes`
and cluster setting `sql.defaults.optimizer_use_not_visible_indexes.enabled` can
be used to disable not visible index feature. When they are enabled, optimizer
treats not visible indexes as they are visible and can choose to use them for
query plan. By default, they are disabled.

docs/generated/settings/settings-for-tenants.txt

@@ -171,6 +171,9 @@ This session variable default should now be configured using ALTER ROLE... SET:
 sql.defaults.optimizer_use_multicol_stats.enabled	boolean	true	"default value for optimizer_use_multicol_stats session setting; enables usage of multi-column stats in the optimizer by default
 This cluster setting is being kept to preserve backwards-compatibility.
 This session variable default should now be configured using ALTER ROLE... SET: https://www.cockroachlabs.com/docs/stable/alter-role.html"
+sql.defaults.optimizer_use_not_visible_indexes.enabled	boolean	false	"default value for optimizer_use_not_visible_indexes session setting; disable usage of not visible indexes in the optimizer by default
+This cluster setting is being kept to preserve backwards-compatibility.
+This session variable default should now be configured using ALTER ROLE... SET: https://www.cockroachlabs.com/docs/stable/alter-role.html"
 sql.defaults.override_alter_primary_region_in_super_region.enabled	boolean	false	"default value for override_alter_primary_region_in_super_region; allows for altering the primary region even if the primary region is a member of a super region
 This cluster setting is being kept to preserve backwards-compatibility.
 This session variable default should now be configured using ALTER ROLE... SET: https://www.cockroachlabs.com/docs/stable/alter-role.html"

docs/generated/settings/settings.html

@@ -138,6 +138,7 @@
 <tr><td><code>sql.defaults.on_update_rehome_row.enabled</code></td><td>boolean</td><td><code>true</code></td><td>default value for on_update_rehome_row; enables ON UPDATE rehome_row() expressions to trigger on updates<br/>This cluster setting is being kept to preserve backwards-compatibility.<br/>This session variable default should now be configured using ALTER ROLE... SET: https://www.cockroachlabs.com/docs/stable/alter-role.html</td></tr>
 <tr><td><code>sql.defaults.optimizer_use_histograms.enabled</code></td><td>boolean</td><td><code>true</code></td><td>default value for optimizer_use_histograms session setting; enables usage of histograms in the optimizer by default<br/>This cluster setting is being kept to preserve backwards-compatibility.<br/>This session variable default should now be configured using ALTER ROLE... SET: https://www.cockroachlabs.com/docs/stable/alter-role.html</td></tr>
 <tr><td><code>sql.defaults.optimizer_use_multicol_stats.enabled</code></td><td>boolean</td><td><code>true</code></td><td>default value for optimizer_use_multicol_stats session setting; enables usage of multi-column stats in the optimizer by default<br/>This cluster setting is being kept to preserve backwards-compatibility.<br/>This session variable default should now be configured using ALTER ROLE... SET: https://www.cockroachlabs.com/docs/stable/alter-role.html</td></tr>
+<tr><td><code>sql.defaults.optimizer_use_not_visible_indexes.enabled</code></td><td>boolean</td><td><code>false</code></td><td>default value for optimizer_use_not_visible_indexes session setting; disable usage of not visible indexes in the optimizer by default<br/>This cluster setting is being kept to preserve backwards-compatibility.<br/>This session variable default should now be configured using ALTER ROLE... SET: https://www.cockroachlabs.com/docs/stable/alter-role.html</td></tr>
 <tr><td><code>sql.defaults.override_alter_primary_region_in_super_region.enabled</code></td><td>boolean</td><td><code>false</code></td><td>default value for override_alter_primary_region_in_super_region; allows for altering the primary region even if the primary region is a member of a super region<br/>This cluster setting is being kept to preserve backwards-compatibility.<br/>This session variable default should now be configured using ALTER ROLE... SET: https://www.cockroachlabs.com/docs/stable/alter-role.html</td></tr>
 <tr><td><code>sql.defaults.override_multi_region_zone_config.enabled</code></td><td>boolean</td><td><code>false</code></td><td>default value for override_multi_region_zone_config; allows for overriding the zone configs of a multi-region table or database<br/>This cluster setting is being kept to preserve backwards-compatibility.<br/>This session variable default should now be configured using ALTER ROLE... SET: https://www.cockroachlabs.com/docs/stable/alter-role.html</td></tr>
 <tr><td><code>sql.defaults.prefer_lookup_joins_for_fks.enabled</code></td><td>boolean</td><td><code>false</code></td><td>default value for prefer_lookup_joins_for_fks session setting; causes foreign key operations to use lookup joins when possible<br/>This cluster setting is being kept to preserve backwards-compatibility.<br/>This session variable default should now be configured using ALTER ROLE... SET: https://www.cockroachlabs.com/docs/stable/alter-role.html</td></tr>

pkg/cli/testdata/explain-bundle/bundle/env.sql

@@ -4,6 +4,7 @@
 -- enable_zigzag_join has the default value: on
 -- optimizer_use_histograms has the default value: on
 -- optimizer_use_multicol_stats has the default value: on
+-- optimizer_use_not_visible_indexes has the default value: off
 -- locality_optimized_partitioned_index_scan has the default value: on
 -- distsql has the default value: auto
 -- vectorize has the default value: on
@@ -227,6 +228,7 @@
 --   sql.defaults.locality_optimized_partitioned_index_scan.enabled = true  (default value for locality_optimized_partitioned_index_scan session setting; enables searching for rows in the current region before searching remote regions)
 --   sql.defaults.optimizer_use_histograms.enabled = true  (default value for optimizer_use_histograms session setting; enables usage of histograms in the optimizer by default)
 --   sql.defaults.optimizer_use_multicol_stats.enabled = true  (default value for optimizer_use_multicol_stats session setting; enables usage of multi-column stats in the optimizer by default)
+--   sql.defaults.optimizer_use_not_visible_indexes.enabled = false  (default value for optimizer_use_not_visible_indexes session setting; disable usage of not visible indexes in the optimizer by default)
 --   sql.defaults.override_multi_region_zone_config.enabled = false  (default value for override_multi_region_zone_config; allows for overriding the zone configs of a multi-region table or database)
 --   sql.defaults.prefer_lookup_joins_for_fks.enabled = false  (default value for prefer_lookup_joins_for_fks session setting; causes foreign key operations to use lookup joins when possible)
 --   sql.defaults.primary_region =   (if not empty, all databases created without a PRIMARY REGION will implicitly have the given PRIMARY REGION)

pkg/settings/registry.go

@@ -177,6 +177,7 @@ var sqlDefaultSettings = map[string]struct{}{
 	"sql.defaults.on_update_rehome_row.enabled":                                 {},
 	"sql.defaults.optimizer_use_histograms.enabled":                             {},
 	"sql.defaults.optimizer_use_multicol_stats.enabled":                         {},
+	"sql.defaults.optimizer_use_not_visible_indexes.enabled":                    {},
 	"sql.defaults.override_alter_primary_region_in_super_region.enabled":        {},
 	"sql.defaults.override_multi_region_zone_config.enabled":                    {},
 	"sql.defaults.prefer_lookup_joins_for_fks.enabled":                          {},

pkg/sql/exec_util.go

@@ -389,6 +389,21 @@ var optUseMultiColStatsClusterMode = settings.RegisterBoolSetting(
 	true,
 ).WithPublic()
 
+// optUseNotVisibleIndexesClusterMode controls the cluster default for whether
+// not visible indexes can still be chosen by the optimizer for query plans. If
+// enabled, the optimizer will treat not visible indexes as they are visible.
+// Note that not visible indexes remain not visible, but the optimizer will
+// disable not visible index feature. If disabled, optimizer will ignore not
+// visible indexes unless it is explicitly selected with force index or for
+// constraint check.
+var optUseNotVisibleIndexesClusterMode = settings.RegisterBoolSetting(
+	settings.TenantWritable,
+	"sql.defaults.optimizer_use_not_visible_indexes.enabled",
+	"default value for optimizer_use_not_visible_indexes session setting; "+
+		"disable usage of not visible indexes in the optimizer by default",
+	false,
+).WithPublic()
+
 // localityOptimizedSearchMode controls the cluster default for the use of
 // locality optimized search. If enabled, the optimizer will try to plan scans
 // and lookup joins in which local nodes (i.e., nodes in the gateway region) are
@@ -3037,6 +3052,10 @@ func (m *sessionDataMutator) SetOptimizerUseMultiColStats(val bool) {
 	m.data.OptimizerUseMultiColStats = val
 }
 
+func (m *sessionDataMutator) SetOptimizerUseNotVisibleIndexes(val bool) {
+	m.data.OptimizerUseNotVisibleIndexes = val
+}
+
 func (m *sessionDataMutator) SetLocalityOptimizedSearch(val bool) {
 	m.data.LocalityOptimizedSearch = val
 }

pkg/sql/explain_bundle.go

@@ -530,6 +530,7 @@ func (c *stmtEnvCollector) PrintSessionSettings(w io.Writer) error {
 		{sessionSetting: "enable_zigzag_join", clusterSetting: zigzagJoinClusterMode, convFunc: boolToOnOff},
 		{sessionSetting: "optimizer_use_histograms", clusterSetting: optUseHistogramsClusterMode, convFunc: boolToOnOff},
 		{sessionSetting: "optimizer_use_multicol_stats", clusterSetting: optUseMultiColStatsClusterMode, convFunc: boolToOnOff},
+		{sessionSetting: "optimizer_use_not_visible_indexes", clusterSetting: optUseNotVisibleIndexesClusterMode, convFunc: boolToOnOff},
 		{sessionSetting: "locality_optimized_partitioned_index_scan", clusterSetting: localityOptimizedSearchMode, convFunc: boolToOnOff},
 		{sessionSetting: "propagate_input_ordering", clusterSetting: propagateInputOrdering, convFunc: boolToOnOff},
 		{sessionSetting: "prefer_lookup_joins_for_fks", clusterSetting: preferLookupJoinsForFKs, convFunc: boolToOnOff},

pkg/sql/logictest/testdata/logic_test/information_schema

@@ -4743,6 +4743,7 @@ opt_split_scan_limit                                  2048
 optimizer                                             on
 optimizer_use_histograms                              on
 optimizer_use_multicol_stats                          on
+optimizer_use_not_visible_indexes                     off
 override_multi_region_zone_config                     off
 parallelize_multi_key_lookup_joins_enabled            off
 password_encryption                                   scram-sha-256
@@ -4784,7 +4785,7 @@ transaction_rows_read_log                             0
 transaction_rows_written_err                          0
 transaction_rows_written_log                          0
 transaction_status                                    NoTxn
-troubleshooting_mode                          off
+troubleshooting_mode                                  off
 unconstrained_non_covering_index_scan_enabled         off
 xmloption                                             content
 

pkg/sql/logictest/testdata/logic_test/pg_catalog

@@ -4224,6 +4224,7 @@ on_update_rehome_row_enabled                          on                  NULL
 opt_split_scan_limit                                  2048                NULL      NULL        NULL        string
 optimizer_use_histograms                              on                  NULL      NULL        NULL        string
 optimizer_use_multicol_stats                          on                  NULL      NULL        NULL        string
+optimizer_use_not_visible_indexes                     off                 NULL      NULL        NULL        string
 override_multi_region_zone_config                     off                 NULL      NULL        NULL        string
 parallelize_multi_key_lookup_joins_enabled            off                 NULL      NULL        NULL        string
 password_encryption                                   scram-sha-256       NULL      NULL        NULL        string
@@ -4349,6 +4350,7 @@ on_update_rehome_row_enabled                          on                  NULL
 opt_split_scan_limit                                  2048                NULL  user     NULL      2048                2048
 optimizer_use_histograms                              on                  NULL  user     NULL      on                  on
 optimizer_use_multicol_stats                          on                  NULL  user     NULL      on                  on
+optimizer_use_not_visible_indexes                     off                 NULL  user     NULL      off                 off
 override_multi_region_zone_config                     off                 NULL  user     NULL      off                 off
 parallelize_multi_key_lookup_joins_enabled            off                 NULL  user     NULL      false               false
 password_encryption                                   scram-sha-256       NULL  user     NULL      scram-sha-256       scram-sha-256
@@ -4471,6 +4473,7 @@ opt_split_scan_limit                                  NULL    NULL     NULL
 optimizer                                             NULL    NULL     NULL     NULL        NULL
 optimizer_use_histograms                              NULL    NULL     NULL     NULL        NULL
 optimizer_use_multicol_stats                          NULL    NULL     NULL     NULL        NULL
+optimizer_use_not_visible_indexes                     NULL    NULL     NULL     NULL        NULL
 override_multi_region_zone_config                     NULL    NULL     NULL     NULL        NULL
 parallelize_multi_key_lookup_joins_enabled            NULL    NULL     NULL     NULL        NULL
 password_encryption                                   NULL    NULL     NULL     NULL        NULL

pkg/sql/logictest/testdata/logic_test/show_source

@@ -97,6 +97,7 @@ on_update_rehome_row_enabled                          on
 opt_split_scan_limit                                  2048
 optimizer_use_histograms                              on
 optimizer_use_multicol_stats                          on
+optimizer_use_not_visible_indexes                     off
 override_multi_region_zone_config                     off
 parallelize_multi_key_lookup_joins_enabled            off
 password_encryption                                   scram-sha-256
@@ -135,7 +136,7 @@ transaction_rows_read_log                             0
 transaction_rows_written_err                          0
 transaction_rows_written_log                          0
 transaction_status                                    NoTxn
-troubleshooting_mode                        off
+troubleshooting_mode                                  off
 unconstrained_non_covering_index_scan_enabled         off
 use_declarative_schema_changer                        on
 vectorize                                             on