sql: Unify the SQL type system

[andy-kimball]

This PR unifies the three different SQL type systems in CRDB:

sqlbase.ColumnType - protobuf representation of table column types
coltypes.T - in-memory representation of table column types
types.T - computation types

The type systems are quite similar, but also subtly different. This causes bugs on a regular basis throughout the codebase. Each type system supports a different subset of the PG type functionality. For example, sqlbase.ColumnType stores column widths. types.T does not store widths, but it can store a Postgres OID value, which sqlbase.ColumnType is unable to do.

In addition, even for supported functionality, the API's are often tricky and error-prone. For example, types.T values can sometimes be compared using ==, but other times Equivalent must be called, and it is up to the user to know when to use which. As another example, support for OID values require wrapping types.T instances in a special TOIDWrapper type. Our developers are expected to know when it needs to be done, but typically they either don't understand when it's needed, or else forget to do it. It also breaks == comparison semantics, leading to subtle bugs that only manifest in edge cases with unusual OID values.

Another big problem that comes up repeatedly are lossy conversions. Throughout the codebase, there are frequent conversions to/from the various type systems. Many conversions discard important type information, which often leads to incompatibilities with Postgres, as well as providing a breeding ground for bugs.

This PR unifies the three type systems into just one. The unified type is called types.T (in new sql/types package). It uses a protobuf-generated struct that is backwards-compatible with sqlbase.ColumnType so that it can be embedded in place of ColumnType in larger protobuf messages (like sqlbase.ColumnDescriptor). The API is locked down so that it's difficult to ignorantly or accidentally create invalid types. The unified type supports everything that the three previous type systems did, and is more compatible with PG as a result (with more work, even more type compatibility is possible).

Resolves #32639

[cockroach-teamcity]

This change is 

[andy-kimball]

Note to reviewers:
I know this is a real monster of a PR. However, I did it in stages, where each commit is a standalone step that builds and passes tests. This also makes it easier to rebase.

It might be easiest to review the final sql/types package first, to see where it all ends up. Then, you might review commit by commit to see each step towards the final product.

[RaduBerinde]

This is fantastic. Thank you for doing this!

[andy-kimball]

Anyone have suggestions about resolving a golint error:

pkg/sql/types/types.go:286:2: var Uuid should be UUID

I'm already using UUID to signify the name of the SemanticType. All other types follow this convention of types.INT (SemanticType) vs. types.Int (pre-constructed instance). I can't find a way to disable lint for this one line, either.

[andy-kimball]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @bobvawter, @BramGruneir, @jordanlewis, and @knz)

---

pkg/sql/pgwire/types.go, line 73 at r15 (raw file):

// mapResultOid performs any necessary mapping of Oid values to be returned to
// the client. For now, this only includes mapping T_float4 => T_float8.
func mapResultOid(o oid.Oid) oid.Oid {

@knz, I wasn't sure what to do about this issue (I believe it's a bug in libpq). I ended up mapping from float4 => float8 to work around it. But it raises larger issues about whether it's a good idea to begin returning the true OID values for selected columns. Should we map all OIDs to "canonical" OIDs (e.g. int2 => int8, varchar => string, etc) in order to match previous version behavior?

P.S. The PG client does not exhibit this bad float4 behavior when pointed at CRDB. It's a libpq thing.

[andy-kimball]

Another suggestion for reviewers: compare all changes made to test files (e.g. information_schema and pg_catalog), since that will tend to show differences in behavior as a result of this PR.

[RaduBerinde]

Anyone have suggestions about resolving a golint error:

Just add an exception here: https://github.com/cockroachdb/cockroach/blob/master/pkg/testutils/lint/lint_test.go#L1451
Something like
stream.GrepNot("pkg/sql/types/types.go.* var Uuid should be UUID")
so it's only for this file.

[andy-kimball]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @bobvawter, @BramGruneir, @jordanlewis, and @knz)

---

pkg/sql/logictest/testdata/logic_test/array, line 212 at r15 (raw file):

{1,2,3}

query T

This new behavior matches PG.

[andy-kimball]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @bobvawter, @BramGruneir, @jordanlewis, and @knz)

---

pkg/sql/parser/parse.go, line 374 at r17 (raw file):

}

// The SERIAL types are pseudo-types that are only used during parsing. After

@knz, I wanted to call this out. I'm treating SERIAL differently now by not creating a "real" type for it. Instead, it's just different instances of the INT types that I then use == reference comparison to recognize. All "memory" of these types is gone after the analysis phase is complete; they exist only for pretty-printing the parse tree.

[andy-kimball]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @bobvawter, @BramGruneir, @jordanlewis, and @knz)

---

pkg/sql/pgwire/types.go, line 73 at r15 (raw file):

Previously, andy-kimball (Andy Kimball) wrote…

@knz, I wasn't sure what to do about this issue (I believe it's a bug in libpq). I ended up mapping from float4 => float8 to work around it. But it raises larger issues about whether it's a good idea to begin returning the true OID values for selected columns. Should we map all OIDs to "canonical" OIDs (e.g. int2 => int8, varchar => string, etc) in order to match previous version behavior?

P.S. The PG client does not exhibit this bad float4 behavior when pointed at CRDB. It's a libpq thing.

As another example, not mapping to previous OIDs results in this Java acceptance test failing:

    public void testArrayWithProps() throws Exception {
        PreparedStatement stmt = conn.prepareStatement("CREATE TABLE x (a SMALLINT[])");
        stmt.execute();
        stmt = conn.prepareStatement("INSERT INTO x VALUES (ARRAY[123])");
        stmt.execute();
        stmt = conn.prepareStatement("SELECT a FROM x");
        ResultSet rs = stmt.executeQuery();
        rs.next();

        Array ar = rs.getArray(1);
        Long[] fs = (Long[]) ar.getArray();
        Assert.assertArrayEquals(new Long[]{123L}, fs);
    }

with the error: incompatible types: long cannot be converted to java.lang.Integer
since oid.T__int2 is returned rather than oid.T__int8, as before.

[andy-kimball]

bors r+

[craig]

Build succeeded

• GitHub CI (Cockroach)