pgwire: v19.2.5: assertion failed: unsupported OID "anyelement"

[cockroach-teamcity]

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/1697870061/?referrer=webhooks_plugin

Panic message:

*errors.errorString
*safedetails.withSafeDetails: unsupported OID %v with format code %s (1)
encoding.go:660: *withstack.withStack (2)
*assert.withAssertionFailure
*errutil.withMessage
*safedetails.withSafeDetails: error in argument for %s (3)
conn_executor_prepare.go:304: *withstack.withStack (4)
*pgerror.withCandidateCode: 08P01 (5)
(check the extra data payloads)

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/pgwire/pgwirebase/encoding.go

Lines 659 to 661 in 4f36d0c

	default:
	return nil, errors.AssertionFailedf(
	"unsupported OID %v with format code %s", errors.Safe(id), errors.Safe(code))

in pkg/sql/pgwire/pgwirebase.DecodeOidDatum

cockroach/pkg/sql/pgwire/pgwirebase/encoding.go

Lines 805 to 807 in 4f36d0c

	buf := r.Next(int(vlen))
	elem, err := DecodeOidDatum(ctx, elemOid, code, buf)
	if err != nil {

in pkg/sql/pgwire/pgwirebase.decodeBinaryArray

cockroach/pkg/sql/pgwire/pgwirebase/encoding.go

Lines 638 to 640 in 4f36d0c

	innerOid := types.OidToType[id].ArrayContents().Oid()
	return decodeBinaryArray(ctx, innerOid, b, code)
	}

in pkg/sql/pgwire/pgwirebase.DecodeOidDatum

cockroach/pkg/sql/conn_executor_prepare.go

Lines 301 to 303 in 4f36d0c

	} else {
	d, err := pgwirebase.DecodeOidDatum(ptCtx, t, qArgFormatCodes[i], arg)
	if err != nil {

in pkg/sql.(*connExecutor).execBind

cockroach/pkg/sql/conn_executor.go

Lines 1319 to 1321 in 4f36d0c

	res = ex.clientComm.CreateBindResult(pos)
	ev, payload = ex.execBind(ctx, tcmd)
	case DeletePreparedStmt:

in pkg/sql.(*connExecutor).execCmd

cockroach/pkg/sql/conn_executor.go

Lines 1171 to 1173 in 4f36d0c

	var err error
	if err = ex.execCmd(ex.Ctx()); err != nil {
	if err == io.EOF || err == errDrainingComplete {

in pkg/sql.(*connExecutor).run

cockroach/pkg/sql/conn_executor.go

Lines 443 to 445 in 4f36d0c

	}()
	return h.ex.run(ctx, s.pool, reserved, cancel)
	}

in pkg/sql.(*Server).ServeConn

cockroach/pkg/sql/pgwire/conn.go

Lines 583 to 585 in 4f36d0c

	reservedOwned = false // We're about to pass ownership away.
	retErr = sqlServer.ServeConn(ctx, connHandler, reserved, cancelConn)
	}()

in pkg/sql/pgwire.(*conn).processCommandsAsync.func1
/usr/local/go/src/runtime/asm_amd64.s#L1336-L1338 in runtime.goexit

pkg/sql/pgwire/pgwirebase/encoding.go in pkg/sql/pgwire/pgwirebase.DecodeOidDatum at line 660
pkg/sql/pgwire/pgwirebase/encoding.go in pkg/sql/pgwire/pgwirebase.decodeBinaryArray at line 806
pkg/sql/pgwire/pgwirebase/encoding.go in pkg/sql/pgwire/pgwirebase.DecodeOidDatum at line 639
pkg/sql/conn_executor_prepare.go in pkg/sql.(*connExecutor).execBind at line 302
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 1320
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1172
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 444
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 584
/usr/local/go/src/runtime/asm_amd64.s in runtime.goexit at line 1337

Tag	Value
Cockroach Release	v19.2.5
Cockroach SHA:	4f36d0c
Platform	linux amd64
Distribution	CCL
Environment	v19.2.5
Command	server
Go Version	go1.12.12
# of CPUs	12
# of Goroutines	199

[asubiotto]

Looks like this was an OID of 2283 which is not included in lib/pq/oid/types.go and corresponds to anyelement:

asubiotto=# SELECT * FROM pg_type WHERE oid = 2283;
 oid  |  typname   | typnamespace | typowner | typlen | typbyval | typtype | typcategory | typispreferred | typisdefined | typdelim | typrelid | typelem | typarray |   typinput    |   typoutput    | typreceive | typsend | typmodin | typmodout | typanalyze | typalign | typstorage | typnotnull | typbasetype | typtypmod | typndims | typcollation | typdefaultbin | typdefault | typacl
------+------------+--------------+----------+--------+----------+---------+-------------+----------------+--------------+----------+----------+---------+----------+---------------+----------------+------------+---------+----------+-----------+------------+----------+------------+------------+-------------+-----------+----------+--------------+---------------+------------+--------
 2283 | anyelement |           11 |       10 |      4 | t        | p       | P           | f              | t            | ,        |        0 |       0 |        0 | anyelement_in | anyelement_out | -          | -       | -        | -         | -          | i        | p          | f          |           0 |        -1 |        0 |            0 |               |            |
(1 row)

[asubiotto]

@rafiss sending over to the appdev team since looks like there's a client that's sending anyelement over. I think the action items here are figure out when and why ORMs try to issue this and what CockroachDB should do about it.

[rafiss]

anyelement is in the CockroachDB pg_type table as well:

root@:26257/defaultdb> SELECT * FROM pg_type WHERE oid = 2283;
  oid  |  typname   | typnamespace | typowner | typlen | typbyval | typtype | typcategory | typispreferred | typisdefined | typdelim | typrelid | typelem | typarray |   typinput    |   typoutput    |   typreceive    |     typsend     | typmodin | typmodout | typanalyze | typalign | typstorage | typnotnull | typbasetype | typtypmod | typndims | typcollation | typdefaultbin | typdefault | typacl
-------+------------+--------------+----------+--------+----------+---------+-------------+----------------+--------------+----------+----------+---------+----------+---------------+----------------+-----------------+-----------------+----------+-----------+------------+----------+------------+------------+-------------+-----------+----------+--------------+---------------+------------+---------
  2283 | anyelement |    383994467 | NULL     |     -1 |  false   | p       | P           |     false      |     true     | ,        |        0 |       0 |     2277 | anyelement_in | anyelement_out | anyelement_recv | anyelement_send |        0 |         0 |          0 | NULL     | NULL       |   false    |           0 |        -1 |        0 |            0 | NULL          | NULL       | NULL
(1 row)

It seems like this error happens when DecodeOidDatum is called with an OID of 2283 (i.e. anyelement) and a FormatCode of FormatBinary

[rafiss]

Based on the stack trace, this looks similar to #51360

[rafiss]

Closing as unactionable.