Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql: DROP INDEX failure with a sandwiched DROP REGION results in irrecoverable state #63974

Closed
otan opened this issue Apr 21, 2021 · 6 comments · Fixed by #64117
Closed

sql: DROP INDEX failure with a sandwiched DROP REGION results in irrecoverable state #63974

otan opened this issue Apr 21, 2021 · 6 comments · Fixed by #64117
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. GA-blocker

Comments

@otan
Copy link
Contributor

otan commented Apr 21, 2021 •
edited
Loading

Consider this test, where:

  • DROP INDEX starts
  • DROP REGION intercepts and completes. All RBR partitions get repartitioned.
  • DROP INDEX fails. no re-partitioning is done.
// TestDropIndexFailureDuringAddOrDropRegion a DROP INDEX failing during
// ADD/DROP REGION, in which case the dropped index should still contain
// the new indexes.
func TestDropIndexFailureDuringAddOrDropRegion(t *testing.T) {
	defer leaktest.AfterTest(t)()
	defer log.Scope(t).Close(t)

	defer sqltestutils.SetTestJobsAdoptInterval()()

	for _, tc := range []struct {
		regionChangeSQL    string
		expectedPartitions []string
	}{
		{
			regionChangeSQL:    `ALTER DATABASE db DROP REGION "us-east2"`,
			expectedPartitions: []string{"us-east1"},
		},
		{
			regionChangeSQL:    `ALTER DATABASE db ADD REGION "us-east3"`,
			expectedPartitions: []string{"us-east1", "us-east2", "us-east3"},
		},
	} {
		t.Run(tc.regionChangeSQL, func(t *testing.T) {
			block := false
			typeChangeFinished := make(chan struct{})
			knobs := base.TestingKnobs{
				SQLSchemaChanger: &sql.SchemaChangerTestingKnobs{
					RunBeforeBackfill: func() error {
						if block {
							block = false
							<-typeChangeFinished
							return errors.New("crikey!")
						}
						return nil
					},
				},
			}

			_, sqlDB, cleanup := multiregionccltestutils.TestingCreateMultiRegionCluster(
				t, 3 /* numServers */, knobs, nil, /* baseDir */
			)
			defer cleanup()

			_, err := sqlDB.Exec(`
DROP DATABASE IF EXISTS db;
CREATE DATABASE db WITH PRIMARY REGION "us-east1" REGIONS "us-east2";
CREATE TABLE db.rbr(k INT PRIMARY KEY, v INT NOT NULL, INDEX v_idx(v)) LOCALITY REGIONAL BY ROW;
`)
			require.NoError(t, err)

			block = true
			var wg sync.WaitGroup
			go func() {
				defer wg.Done()

				_, err := sqlDB.Exec(`DROP INDEX db.rbr@v_idx`)
				if err != nil {
					if !strings.Contains(err.Error(), "crikey!") {
						t.Errorf("failed to run drop index for unknown reason: %+v", err)
					}
				} else {
					t.Errorf("expected error during DROP INDEX")
				}
			}()
			wg.Add(1)

			_, err = sqlDB.Exec(tc.regionChangeSQL)
			close(typeChangeFinished)
			require.NoError(t, err)

			// Wait for DROP INDEX to complete.
			wg.Wait()

			require.NoError(
				t,
				validateIndexAndPartitions(
					sqlDB,
					"db.rbr",
					[]string{"rbr@primary", "rbr@v_idx"},
					tc.expectedPartitions,
				),
			)
		})
	}
}

func validateIndexAndPartitions(
	sqlDB *gosql.DB, tableName string, expectedIndexes []string, expectedPartitions []string,
) error {
	rows, err := sqlDB.Query(fmt.Sprintf(
		"SELECT index_name, partition_name FROM [SHOW PARTITIONS FROM TABLE %s] ORDER BY partition_name",
		tableName,
	))
	if err != nil {
		return err
	}
	defer rows.Close()

	indexPartitions := make(map[string][]string)
	for rows.Next() {
		var indexName string
		var partitionName string
		if err := rows.Scan(&indexName, &partitionName); err != nil {
			return err
		}

		indexPartitions[indexName] = append(indexPartitions[indexName], partitionName)
	}

	for _, expectedIndex := range expectedIndexes {
		partitions, found := indexPartitions[expectedIndex]
		if !found {
			return errors.AssertionFailedf("did not find index %s", expectedIndex)
		}

		if len(partitions) != len(expectedPartitions) {
			return errors.AssertionFailedf(
				"unexpected number of partitions; expected %d, found %d",
				len(partitions),
				len(expectedPartitions),
			)
		}
		for i, expectedPartition := range expectedPartitions {
			if expectedPartition != partitions[i] {
				return errors.AssertionFailedf(
					"unexpected partitions; expected %v, found %v",
					expectedPartitions,
					partitions,
				)
			}
		}
	}
	return nil

}

In this case, we have bad state - the index does not match the regions.

Originally I tried patching this in #63968, but it's broken in the case that if we transition from RBR to Global instead of drop index, the DROP REGION would also "accidentally" partition the global indexes -- then again, we already do that today and i think that's a bug that will fall our when arul tests it.

My suggested solution would be to repartition again during CREATE INDEX/DROP INDEX when the operation is complete. This is probably similar to arul's proposed work. Can take a stab at this tomorrow. Warrants a GA-blocker.
@otan otan added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. GA-blocker labels Apr 21, 2021
@blathers-crl
Copy link

blathers-crl bot commented Apr 21, 2021

Hi @otan, please add branch-* labels to identify which branch(es) this release-blocker affects.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.

@otan
Copy link
Contributor Author

otan commented Apr 21, 2021

i've tried this alternate strategy in #63976, but hitting this on DROP REGION sandwiching:

E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2  encountered internal error:
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +while decoding partition tuple: decoding: could not find [128] in enum "public.crdb_internal_region" representation PhysicalReps: [[64]]; LogicalReps: [us-east1]
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +(1) assertion failure
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +Wraps: (2) attached stack trace
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  -- stack trace:
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql.(*optIndex).init
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:1231
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql.newOptTable
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:815
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).dataSourceForTable
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:444
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).dataSourceForDesc
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:384
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).ResolveDataSource
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:218
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/delegate.(*delegator).delegateShowPartitions
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/delegate/show_partitions.go:31
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/delegate.TryDelegate
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/delegate/delegate.go:71
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmt
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:345
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildDataSource
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:163
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildDataSource
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:59
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildFromTablesRightDeep
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:1107
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildFromTables
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:1084
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildFrom
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:1011
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelectClause
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:932
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelectStmtWithoutParens
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:880
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelect.func1
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:853
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).processWiths
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/with.go:110
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelect
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:852
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmt
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:275
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRoot
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:235
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).Build
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:209
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:521
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:194
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:940
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:821
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:659
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:122
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1527
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1529
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1450
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:484
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:626
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +Wraps: (3) while decoding partition tuple
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +Wraps: (4) decoding: could not find [128] in enum "public.crdb_internal_region" representation PhysicalReps: [[64]]; LogicalReps: [us-east1]
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | -- cause hidden behind barrier
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | decoding: could not find [128] in enum "public.crdb_internal_region" representation PhysicalReps: [[64]]; LogicalReps: [us-east1]
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | (1) attached stack trace
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   -- stack trace:
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodePartitionTuple
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/partition.go:148
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*optIndex).init
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:1226
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.newOptTable
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:815
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).dataSourceForTable
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:444
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).dataSourceForDesc
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:384
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).ResolveDataSource
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:218
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/delegate.(*delegator).delegateShowPartitions
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/delegate/show_partitions.go:31
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/delegate.TryDelegate
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/delegate/delegate.go:71
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmt
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:345
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildDataSource
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:163
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildDataSource
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:59
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildFromTablesRightDeep
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:1107
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildFromTables
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:1084
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildFrom
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:1011
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelectClause
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:932
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelectStmtWithoutParens
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:880
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelect.func1
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:853
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).processWiths
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/with.go:110
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelect
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:852
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmt
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:275
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRoot
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:235
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).Build
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:209
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:521
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:194
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:940
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:821
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:659
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:122
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1527
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1529
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1450
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | Wraps: (2) decoding
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | Wraps: (3) attached stack trace
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   -- stack trace:
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/types.(*T).EnumGetIdxOfPhysical
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/types/types.go:2403
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/sem/tree.GetEnumComponentsFromPhysicalRep
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/datum.go:3982
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodeUntaggedDatum
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/column_type_encoding.go:677
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodeTableValue
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/column_type_encoding.go:526
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodePartitionTuple
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/partition.go:146
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*optIndex).init
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:1226
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.newOptTable
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:815
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).dataSourceForTable
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:444
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).dataSourceForDesc
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:384
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).ResolveDataSource
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:218
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/delegate.(*delegator).delegateShowPartitions
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/delegate/show_partitions.go:31
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/delegate.TryDelegate
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/delegate/delegate.go:71
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmt
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:345
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildDataSource
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:163
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildDataSource
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:59
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildFromTablesRightDeep
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:1107
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildFromTables
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:1084
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildFrom
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:1011
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelectClause
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:932
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelectStmtWithoutParens
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:880
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelect.func1
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:853
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).processWiths
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/with.go:110
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelect
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:852
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmt
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:275
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRoot
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:235
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).Build
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:209
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:521
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:194
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:940
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:821
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:659
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  |   | 	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:122
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | Wraps: (4) could not find [128] in enum "public.crdb_internal_region" representation PhysicalReps: [[64]]; LogicalReps: [us-east1]
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
E210421 09:42:37.850200 5094 sql/sqltelemetry/report.go:57  [n1,client=127.0.0.1:62845,hostssl,user=root] 2 +Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.withPrefix (4) *barriers.barrierError

i'm still convinced we should block ADD/DROP REGION when a CREATE/DROP INDEX or ALTER TABLE LOCALITY is underway.

@otan
Copy link
Contributor Author

otan commented Apr 21, 2021 •
edited
Loading

i've also amended #63968 to exclude the case where this might break.
so now we can pick #63968 or #63976 to resolve this. we almost certainly need some of the latter to resolve some of the cases we've talked about for #63462. the former works, but if we get the latter working i prefer that as the solution i think.

@arulajmani
Copy link
Collaborator

so now we can pick #63968 or #63976 to resolve this. we almost certainly need some of the latter to resolve some of the cases we've talked about for #63462. the former works, but if we get the latter working i prefer that as the solution i think.

I agree here, I like the solution proposed in #63976 more as well. That'll also serve as a solution for the case I've been thinking about:

ALTER TABLE db.t SET LOCALITY REGIONAL BY ROW; <- block in the schema changer
ALTER DATABASE db ADD/DROP REGION; <- Let complete
resume the ALTER TABLE

@otan
Copy link
Contributor Author

otan commented Apr 21, 2021 •
edited
Loading

Yep, so the #63976 solution seems problematic because I'm pretty the blocked thread has no knowledge of the new enum as it used an older version of it.

@otan otan mentioned this issue Apr 22, 2021
Closed
@otan
Copy link
Contributor Author

otan commented Apr 22, 2021

#63976 works! made a silly boo boo

@otan otan mentioned this issue Apr 26, 2021
Merged
@craig craig bot closed this as completed in 2cd071a Apr 27, 2021
@craig craig bot closed this as completed in #64117 Apr 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. GA-blocker
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sql: prevent REGIONAL BY ROW and ADD/DROP REGION concurrent transitions otan-cockroach/cockroach
2 participants
@otan @arulajmani