sql: add proper support for hidden columns

[knz]

(Needed to implement pg_am and pg_opclass for #26504.)

Prior to this patch, CockroachDB already had full support for marking
arbitrary columns as "hidden" internally: a hidden column can be used
for queries but does not get automatically selected by a * in a
SELECT clause, and does not get automatically inserted/upserted into
when INSERT/UPSERT do not list any columns.

However, no control was given to users to determine which columns
would be hidden. The only column that could be hidden in practice was
the implicit rowid column created when CREATE does not specify a
primary key.

This patch complements the existing thorough support for hidden column
by properly exposing the "hidden" attribute through SQL:

• any column can now be marked as hidden in CREATE using the column
  attribute NOT VISIBLE.
• the attribute can now be changed with ALTER TABLE .. ALTER COLUMN .. SET [NOT] VISIBLE.

This change is further motivated by the need to annotate non-PK
columns in pg_catalog vtables, for compatibility with
PostgreSQL (this will be actually done in a later patch).

The output of SHOW CREATE now displays all columns with their
optional NOT VISIBLE attribute, instead of skipping over hidden
column as previously.

Note: the choice for the syntax "NOT VISIBLE" as opposed to "HIDDEN"
is mandated by the fact that the first word of an attribute must be a
reserved keyword, and adding reserved keywords is disruptive. This
way, the patch is properly backward-compatible.

Release note (sql change): clients can now set the visibility of a
column using the NOT VISIBLE attribute in CREATE TABLE or SET VISIBLE / SET NOT VISIBLE in ALTER TABLE .. ALTER COLUMN.

[cockroach-teamcity]

This change is 

[jordanlewis]

This is cool, but I don't understand why it's necessary to add this feature to support pg_am and so on. Is there really no way to support those tables without introducing a major new schema feature that we'll necessarily have to support forever?

---

Review status: complete! 0 of 0 LGTMs obtained

---

Comments from Reviewable

[knz]

This is cool, but I don't understand why it's necessary to add this feature to support pg_am and so on.

1. pg_am and pg_opclass (and others) are meant to have an oid column that is hidden. It must be hidden because * must not expand it.
2. our vtables do not have primary keys. Adding them would be major work.
3. if we announce a non-existing primary key for a vtable in its descriptor but without actually implementing any of its structural constraints, we would need to also pretend it does not exist in all the other virtual tables (pg_index, etc). This check would be ad-hoc and error-prone.
4. instead, this patch makes it possible to create the vtable descriptor with the proper attribute without ugly hack in the vtable generators.

Is there really no way to support those tables without introducing a [...] new schema feature

There is a way (implement proper PKs in virtual tables) but it's expensive.

[is it not better to avoid introducing] a [...] new schema feature that we'll necessarily have to support forever?

The feature was already there. It was so thoroughly there that we have the proper check if ... col.Hidden ... throughout the code already. This patch is merely revealing this fundamental design aspect of CockroachDB's SQL layer (and postgres') to the users and enables the users to utilize it.

[knz]

Two more things. - the hidden column in these vtables must be called "oid", not "rowid". Some other tables have a hidden column with a different name. So in any case we must provide a mechanism to specify the name of the hidden column(s). - the previous output of SHOW CREATE was listing columns (rowid) in constraints, family definitions and interleave specs that were not clearly defined (there was no column def for them). It makes the schema harder to understand for a human, and it makes it easy to overlook that there's an extra column with a non trivial default. This patch corrects that.

…

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.