sql: mark type strings as safe from redacting #90760

cockroach-teamcity · 2022-10-27T12:29:56Z

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3700849185/?referrer=webhooks_plugin

Panic message:

error.go:89: unexpected error from the vectorized engine: unhandled cast × -> ×
(1) assertion failure
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:89
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1038
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.InternalError
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:193
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colbuilder.NewColOperator
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colbuilder/execplan.go:1241
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupFlow.func1
| github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1183
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupFlow
| github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1122
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Setup
| github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:275
| github.com/cockroachdb/cockroach/pkg/sql/distsql.(*ServerImpl).setupFlow
| github.com/cockroachdb/cockroach/pkg/sql/distsql/server.go:411
| github.com/cockroachdb/cockroach/pkg/sql/distsql.(*ServerImpl).SetupLocalSyncFlow
| github.com/cockroachdb/cockroach/pkg/sql/distsql/server.go:584
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).setupFlows
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:373
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:555
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).planAndRunSubquery
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1313
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunSubqueries
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1213
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1497
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1163
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:699
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:144
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1891
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1895
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1817
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:763
| github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
| github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (3) unexpected error from the vectorized engine
Wraps: (4) unhandled cast × -> ×
| -- cause hidden behind barrier
| unhandled cast × -> ×
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.GetCastOperator
| | github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go:452
| | github.com/cockroachdb/cockroach/pkg/sql/colexec/colbuilder.NewColOperator
| | github.com/cockroachdb/cockroach/pkg/sql/colexec/colbuilder/execplan.go:1236
| | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupFlow.func1
| | github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1183
| | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| | github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
| | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupFlow
| | github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1122
| | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Setup
| | github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:275
| | github.com/cockroachdb/cockroach/pkg/sql/distsql.(*ServerImpl).setupFlow
| | github.com/cockroachdb/cockroach/pkg/sql/distsql/server.go:411
| | github.com/cockroachdb/cockroach/pkg/sql/distsql.(*ServerImpl).SetupLocalSyncFlow
| | github.com/cockroachdb/cockroach/pkg/sql/distsql/server.go:584
| | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).setupFlows
| | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:373
| | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:555
| | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).planAndRunSubquery
| | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1313
| | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunSubqueries
| | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1213
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1497
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1163
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:699
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:144
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1891
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1895
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1817
| | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:763
| | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
| | github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728
| | runtime.goexit
| | GOROOT/src/runtime/asm_amd64.s:1581
| Wraps: (2) unhandled cast × -> ×
| Error types: (1) *withstack.withStack (2) *errutil.leafError
Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.withPrefix (4) *barriers.barrierErr
-- report composition:
*barriers.barrierErr: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
*errutil.withPrefix: unexpected error from the vectorized engine
error.go:89: *withstack.withStack (top exception)
*assert.withAssertionFailure

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/colexecerror/error.go

Lines 88 to 90 in e438c2f

    
           	// unexpected. 
        
           	retErr = errors.NewAssertionErrorWithWrappedErrf(err, "unexpected error from the vectorized engine") 
        
           }

in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
GOROOT/src/runtime/panic.go#L1037-L1039 in runtime.gopanic

cockroach/pkg/sql/colexecerror/error.go

Lines 192 to 194 in e438c2f

    
           func InternalError(err error) { 
        
           	panic(err) 
        
           }

in pkg/sql/colexecerror.InternalError

cockroach/pkg/sql/colexec/colbuilder/execplan.go

Lines 1240 to 1242 in e438c2f

    
           if err != nil { 
        
           	colexecerror.InternalError(err) 
        
           }

in pkg/sql/colexec/colbuilder.NewColOperator

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 1182 to 1184 in e438c2f

    
           var result *colexecargs.NewColOperatorResult 
        
           result, err = colbuilder.NewColOperator(ctx, flowCtx, args) 
        
           if result != nil {

in pkg/sql/colflow.(*vectorizedFlowCreator).setupFlow.func1

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in e438c2f

    
           }() 
        
           operation() 
        
           return retErr

in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 1121 to 1123 in e438c2f

    
           ) (opChains execinfra.OpChains, batchFlowCoordinator *BatchFlowCoordinator, err error) { 
        
           	if vecErr := colexecerror.CatchVectorizedRuntimeError(func() { 
        
           		// The column factory will not change the eval context, so we can use

in pkg/sql/colflow.(*vectorizedFlowCreator).setupFlow

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 274 to 276 in e438c2f

    
           } 
        
           opChains, batchFlowCoordinator, err := f.creator.setupFlow(ctx, flowCtx, spec.Processors, f.GetLocalProcessors(), opt) 
        
           if err != nil {

in pkg/sql/colflow.(*vectorizedFlow).Setup

cockroach/pkg/sql/distsql/server.go

Lines 410 to 412 in e438c2f

    
           var err error 
        
           ctx, opChains, err = f.Setup(ctx, &req.Flow, opt) 
        
           if err != nil {

in pkg/sql/distsql.(*ServerImpl).setupFlow

cockroach/pkg/sql/distsql/server.go

Lines 583 to 585 in e438c2f

    
           ) (context.Context, flowinfra.Flow, execinfra.OpChains, error) { 
        
           	ctx, f, opChains, err := ds.setupFlow( 
        
           		ctx, tracing.SpanFromContext(ctx), parentMonitor, req, output, batchOutput, localState,

in pkg/sql/distsql.(*ServerImpl).SetupLocalSyncFlow

cockroach/pkg/sql/distsql_running.go

Lines 372 to 374 in e438c2f

    
           	} 
        
           	return dsp.distSQLSrv.SetupLocalSyncFlow(ctx, evalCtx.Mon, &setupReq, recv, batchReceiver, localState) 
        
           }

in pkg/sql.(*DistSQLPlanner).setupFlows

cockroach/pkg/sql/distsql_running.go

Lines 554 to 556 in e438c2f

    
           } 
        
           ctx, flow, opChains, err := dsp.setupFlows( 
        
           	ctx, evalCtx, leafInputState, flows, recv, localState, planCtx.collectExecStats, statementSQL,

in pkg/sql.(*DistSQLPlanner).Run

cockroach/pkg/sql/distsql_running.go

Lines 1312 to 1314 in e438c2f

    
           subqueryPlans[planIdx].started = true 
        
           dsp.Run(ctx, subqueryPlanCtx, planner.txn, subqueryPhysPlan, subqueryRecv, evalCtx, nil /* finishedSetupFn */)() 
        
           if err := subqueryRowReceiver.Err(); err != nil {

in pkg/sql.(*DistSQLPlanner).planAndRunSubquery

cockroach/pkg/sql/distsql_running.go

Lines 1212 to 1214 in e438c2f

    
           for planIdx, subqueryPlan := range subqueryPlans { 
        
           	if err := dsp.planAndRunSubquery( 
        
           		ctx,

in pkg/sql.(*DistSQLPlanner).PlanAndRunSubqueries

cockroach/pkg/sql/conn_executor_exec.go

Lines 1496 to 1498 in e438c2f

    
           defer subqueryResultMemAcc.Close(ctx) 
        
           if !ex.server.cfg.DistSQLPlanner.PlanAndRunSubqueries( 
        
           	ctx, planner, evalCtxFactory, planner.curPlan.subqueryPlans, recv, &subqueryResultMemAcc,

in pkg/sql.(*connExecutor).execWithDistSQLEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 1162 to 1164 in e438c2f

    
           ex.sessionTracing.TraceExecStart(ctx, "distributed") 
        
           stats, err = ex.execWithDistSQLEngine( 
        
           	ctx, planner, stmt.AST.StatementReturnType(), res, distribute, progAtomic,

in pkg/sql.(*connExecutor).dispatchToExecutionEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 698 to 700 in e438c2f

    
           if err := ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil { 
        
           	stmtThresholdSpan.Finish()

in pkg/sql.(*connExecutor).execStmtInOpenState

cockroach/pkg/sql/conn_executor_exec.go

Lines 143 to 145 in e438c2f

    
           } else { 
        
           	ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit) 
        
           }

in pkg/sql.(*connExecutor).execStmt

cockroach/pkg/sql/conn_executor.go

Lines 1890 to 1892 in e438c2f

    
           canAutoCommit := ex.implicitTxn() && (tcmd.LastInBatch || !implicitTxnForBatch) 
        
           ev, payload, err = ex.execStmt( 
        
           	ctx, tcmd.Statement, nil /* prepared */, nil /* pinfo */, stmtRes, canAutoCommit,

in pkg/sql.(*connExecutor).execCmd.func1

cockroach/pkg/sql/conn_executor.go

Lines 1894 to 1896 in e438c2f

    
           	return err 
        
           }() 
        
           // Note: we write to ex.statsCollector.PhaseTimes, instead of ex.phaseTimes,

in pkg/sql.(*connExecutor).execCmd

cockroach/pkg/sql/conn_executor.go

Lines 1816 to 1818 in e438c2f

    
           var err error 
        
           if err = ex.execCmd(); err != nil { 
        
           	if errors.IsAny(err, io.EOF, errDrainingComplete) {

in pkg/sql.(*connExecutor).run

cockroach/pkg/sql/conn_executor.go

Lines 762 to 764 in e438c2f

    
           	}() 
        
           	return h.ex.run(ctx, s.pool, reserved, cancel) 
        
           }

in pkg/sql.(*Server).ServeConn

cockroach/pkg/sql/pgwire/conn.go

Lines 727 to 729 in e438c2f

    
           	reservedOwned = false // We're about to pass ownership away. 
        
           	retErr = sqlServer.ServeConn(ctx, connHandler, reserved, cancelConn) 
        
           }()

in pkg/sql/pgwire.(*conn).processCommandsAsync.func1
GOROOT/src/runtime/asm_amd64.s#L1580-L1582 in runtime.goexit

pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 89
GOROOT/src/runtime/panic.go in runtime.gopanic at line 1038
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.InternalError at line 193
pkg/sql/colexec/colbuilder/execplan.go in pkg/sql/colexec/colbuilder.NewColOperator at line 1241
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlowCreator).setupFlow.func1 at line 1183
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlowCreator).setupFlow at line 1122
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Setup at line 275
pkg/sql/distsql/server.go in pkg/sql/distsql.(*ServerImpl).setupFlow at line 411
pkg/sql/distsql/server.go in pkg/sql/distsql.(*ServerImpl).SetupLocalSyncFlow at line 584
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).setupFlows at line 373
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 555
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).planAndRunSubquery at line 1313
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunSubqueries at line 1213
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 1497
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1163
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 699
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 144
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func1 at line 1891
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 1895
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1817
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 763
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 728
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1581

Tag	Value
Cockroach Release	`v22.1.9`
Cockroach SHA:	`e438c2f`
Platform	linux amd64
Distribution	CCL
Environment	v22.1.9
Command	server
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-20931

The text was updated successfully, but these errors were encountered:

yuzefovich · 2022-10-28T12:53:32Z

We can't find a cast needed for arguments of a window function inside of a subquery. The statement prefix is WITH _ AS (SELECT _, CASE WHEN substring(_, _, strpos(_, '_')) = '_' THEN '_' ELSE max(substring(_, _, strpos(_, '_'))) || '_' END AS _, substring(_, (strpos(_, '_') + _), length(_)) AS _ FROM ( [...].

yuzefovich · 2022-11-01T18:35:57Z

Maybe the action item is to mark types strings as safe from redacting.

This patch adds a `SafeSQLString` method to `types.T` that returns a `redact.SafeString`. It is used by the "unhandled cast" error message. This will make debugging sentry issues easier. Fixes cockroachdb#90760 Release note: None

This patch adds a `SQLStringForError` method to `types.T` that returns a `redact.RedactableString`. It is used by the "unhandled cast" error message to prevent safe type information from being removed from the error message. This will make debugging sentry issues easier. Fixes cockroachdb#90760 Release note: None

91892: colexec: use redact.RedactableString for types in cast error message r=DrewKimball a=DrewKimball This patch adds a `SQLStringForError` method to `types.T` that returns a `redact.RedactableString`. It is used by the "unhandled cast" error message to prevent safe type information from being removed from the error message. This will make debugging sentry issues easier. Fixes #90760 Epic: [CRDB-20535](https://cockroachlabs.atlassian.net/browse/CRDB-20535) Release note: None Co-authored-by: Drew Kimball <[email protected]>

This patch adds a `SQLStringForError` method to `types.T` that returns a `redact.RedactableString`. It is used by the "unhandled cast" error message to prevent safe type information from being removed from the error message. This will make debugging sentry issues easier. Fixes cockroachdb#90760 Release note: None

cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Oct 27, 2022

yuzefovich mentioned this issue Oct 28, 2022

sentry: error.go:89: unexpected error from the vectorized engine: unhandled cast × -> × (1) assertion failure Wraps: (2) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/col... #90611

Closed

blathers-crl bot added the T-sql-queries SQL Queries Team label Oct 28, 2022

yuzefovich assigned DrewKimball Nov 1, 2022

DrewKimball mentioned this issue Nov 15, 2022

colexec: use redact.RedactableString for types in cast error message #91892

Merged

mgartner changed the title ~~colexec: v22.1.9: unhandled cast for window function arguments~~ sql: mark type strings as safe from redacting Nov 29, 2022

craig bot closed this as completed in 1a4b094 Mar 17, 2023

DrewKimball mentioned this issue Mar 31, 2023

release-22.2: colexec: use redact.RedactableString for types in cast error message #100364

Merged

DrewKimball mentioned this issue Mar 31, 2023

release-22.1: colexec: use redact.RedactableString for types in cast error message #100367

Merged

mgartner added this to SQL Queries Jul 24, 2023

mgartner moved this to Done in SQL Queries Jul 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sql: mark type strings as safe from redacting #90760

sql: mark type strings as safe from redacting #90760

cockroach-teamcity commented Oct 27, 2022 •

edited by cockroach-jira-scripts

Loading

yuzefovich commented Oct 28, 2022

yuzefovich commented Nov 1, 2022

sql: mark type strings as safe from redacting #90760

sql: mark type strings as safe from redacting #90760

Comments

cockroach-teamcity commented Oct 27, 2022 • edited by cockroach-jira-scripts Loading

yuzefovich commented Oct 28, 2022

yuzefovich commented Nov 1, 2022

cockroach-teamcity commented Oct 27, 2022 •

edited by cockroach-jira-scripts

Loading