[WIP] server,sql: make the time-to-first-query observable in telemetry

[knz]

(inspired from the discussion on #55059 and a request by @vilamurugu )

This change aims to expose “time to first query”, a product
metric that correlates with how easy it is for a user
to start using CockroachDB.

The collection of “time to first query” is made complicated
to collect because of the following:

• we cannot compute this entirely on one node using solely in-memory
  operations: we need to find the first time a query is executed across
  a fleet of nodes, and we need to remember the value when nodes get
  restarted.

• we would like to avoid a read-eval-write cycle with an access to KV
  storage, or disk, or a fan-out across the network, upon every query. The
  collection of this metric should have virtually no performance impact
  on the common case of workloads beyond the first access.

• the metric “time to first query” is different from “time of first
  query”: we want to track the amount of time between cluster
  creation and the time of first query.

• we wish to avoid taking into account “internal” SQL queries, such
  as those issued internally by CockroachDB itself and those
  issued by helper/orchestration tools.

In order to reach the stated goal, this commit proposes the following
model:

• to avoid “internal” SQL queries, the mechanism described below
  only looks at queries issued when the application_name is
  not “internal”, that is, does not start with the special
  prefix "$ ".

• nodes keep track of their “time of first query” as
  follows:

  • an in-memory only timestamp is maintained and checked upon
    every query execution. Accesses are atomic to avoid
    any performance impact.

  • separately, an asynchronous background process
    (startSynchronizeFirstQueryTimestamp) synchronizes
    the in-memory timestamp with a new cluster setting
    sql.metrics.first_query.timestamp.

    The choice is made to use a cluster setting because these
    automatically synchronize between nodes and are
    automatically embarked in telemetry already.

• separately and independently, the cluster now remembers its
  initialization time by storing a timestamp inside a new cluster
  setting sql.metrics.cluster_init.timestamp. This is set only once,
  upon cluster init.

  Again, the choice is made to use a cluster setting because these are
  automatically embarked in telemetry already.

Note that this change does not compute the time to first query,
but instead presents two different timestamp in telemetry
(from the cluster settings) which can be subtracted from each other
to reach the stated goal.

Here is an example use, using cockroach demo (nb: by default demo
uses an internal application_name):

```
$ cockroach demo

root> show cluster setting sql.metrics.cluster_init.timestamp;
sql.metrics.cluster_init.timestamp
--------------------------------------
                 1618666751186151817

root> show cluster setting sql.metrics.first_query.timestamp;
  sql.metrics.first_query.timestamp
-------------------------------------
                                  0

root> set application_name = 'foo';

root> show cluster setting sql.metrics.first_query.timestamp;
  sql.metrics.first_query.timestamp
-------------------------------------
                1618666852579067840

root> select 1;

root> -- observe: the first query timestamp does not change!

root> show cluster setting sql.metrics.first_query.timestamp;
  sql.metrics.first_query.timestamp
-------------------------------------
                1618666852579067840
```

Note also that the new first_query.timestamp value is approximate.
This is explained by the following comment in the source code:

// The timestamp of the first query that was not issued by
// an "internal" client (as determined by application_name).
//
// This is set to non-zero by any node receiving a query if the
// value is observed to still be zero.
// Meanwhile, as for every cluster setting, it is possible for this
// value to take a while to propagate to the rest of this cluster.
// Therefore, it is possible for multiple nodes to concurrently
// write to the cluster setting, when they receive their first
// client query within a short time frame (the setting propagation
// delay).
// We find this acceptable because this propagation delay
// is short (seconds) whereas the information we hope to extract
// from this timestamp need only be at minutes/hours granularity.

See the release note below for a descriptionof impact.

Release note (sql change): CockroachDB now stores the time at which
the SQL logical layer of a cluster was initialized in the cluster
setting sql.metrics.cluster_init.timestamp.It also stores the
approximate time at which the first query from a SQL client is
executed in the new cluster setting
sql.metrics.first_query.timestamp.

This makes it possible to compute the time to first query since
cluster initialization, by subtracting the two values.

Note that the value is approximate because when the initial clients
connect to multiple nodes simultaneously, it is possible for multiple
nodes to concurrently estimate that they received the “first
query”. The uncertainty in this approximation is limited by the
propagation delay of cluster settings, which is typically a dozen of
seconds or less across the entire cluster.

[cockroach-teamcity]

This change is 

[knz]

@jordanlewis heads up that I am delaying the implementation of unit testing until we reach some consensus about whether this approach is acceptable.

[knz]

See this comment for an alternate approach: #63815 (comment)

[knz]

superseded by #66427