Skip to content

Commit

Permalink
spanconfig: support rangefeeds for dynamic system tables
Browse files Browse the repository at this point in the history 
Fixes cockroachdb#73045.

We're running out of system table IDs (see cockroachdb#57531), and as a result
we're now introducing the notion of dynamic system IDs throughout the
system. Previously KV baked-in the assumption of static system IDs at
two points:
- When deciding to allow rangefeeds on a given range;
- When enforcing strict GC TTL;

It did so by decoding the range's key span and comparing against the
hard-coded maximum system ID, all to determine whether the range in
question contained system tables. If so, we allowed rangefeeds to be
declared over it, and also did not enforce strict GC TTL (only really
applies to user tables). This way of doing things does not compose with
dynamically allocated system table IDs. With arbitrary, possibly
non-contiguous IDs, we don't have the convenient key-comparison
properties to rely on.

To that end, we use the span configs infrastructure to to delegate
control of whether rangefeeds are enabled over a given range and whether
strict GC is enforced. This scheme allows SQL code to declare "system
table configs" over arbitrary schemas, and have KV still respect it.
This PR does not expose these span config settings as part of zone
configs -- there's no need to (though we could in the future). To
account for the asynchronous nature of the span configs infra, we need
to ensure that ranges without an available config default to enabling
rangefeeds.

Release note: None
  • Loading branch information
@irfansharif
irfansharif committed Jan 6, 2022
1 parent 4b14bdb commit a475ea7
Show file tree
Hide file tree
Showing 19 changed files with 686 additions and 543 deletions.
94 changes: 88 additions & 6 deletions pkg/ccl/spanconfigccl/spanconfigcomparedccl/testdata/basic
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ ALTER TABLE db.t1 CONFIGURE ZONE USING num_voters = 5;
# range's contents whenever anything in it changes.
# - The span configs infrastructure doesn't, at least for now, for
# inter-operability with the gossip-backed system.
# That said, the span configs infrastructure is used to drive whether
# rangefeeds are enabled on the system database tables. It also controls
# whether strict GC is enforced. For that reason we expect to find different
# span configs from each ("range system" vs. "database system").

configs version=legacy offset=4 limit=3
----
Expand All @@ -33,17 +37,17 @@ configs version=current offset=4 limit=3
----
...
/System/"tse" range system
/Table/SystemConfigSpan/Start range system
/Table/11 range system
/Table/SystemConfigSpan/Start database system (host)
/Table/11 database system (host)
...

# Both subsystems observe splits for the tables created above.

configs version=current offset=41
----
...
/Table/46 range system
/Table/47 range system
/Table/46 database system (host)
/Table/47 database system (host)
/Table/56 num_replicas=7 num_voters=5
/Table/57 num_replicas=7

Expand All @@ -55,10 +59,88 @@ configs version=legacy offset=41
/Table/56 num_replicas=7 num_voters=5
/Table/57 num_replicas=7

# Both subsystems are identical with respect to exposed configs (including for
# pseudo table IDs).
# Both subsystems differ slightly with respect to exposed configs ("range
# system" vs. "database system" as described earlier). This only applies to
# tables in the system database, excluding pseudo table IDs.

diff
----
--- gossiped system config span (legacy)
+++ span config infrastructure (current)
@@ -4,42 +4,42 @@
/System/tsd range default
/System/"tse" range system
-/Table/SystemConfigSpan/Start range system
-/Table/11 range system
-/Table/12 range system
-/Table/13 range system
-/Table/14 range system
-/Table/15 range system
+/Table/SystemConfigSpan/Start database system (host)
+/Table/11 database system (host)
+/Table/12 database system (host)
+/Table/13 database system (host)
+/Table/14 database system (host)
+/Table/15 database system (host)
/Table/16 range system
/Table/17 range system
/Table/18 range system
-/Table/19 range system
-/Table/20 range system
-/Table/21 range system
+/Table/19 database system (host)
+/Table/20 database system (host)
+/Table/21 database system (host)
/Table/22 range system
-/Table/23 range system
-/Table/24 range system
-/Table/25 ttl_seconds=600 num_replicas=5
-/Table/26 range system
-/Table/27 ttl_seconds=600 num_replicas=5
-/Table/28 range system
+/Table/23 database system (host)
+/Table/24 database system (host)
+/Table/25 ttl_seconds=600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
+/Table/26 database system (host)
+/Table/27 ttl_seconds=600 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
+/Table/28 database system (host)
/Table/29 range system
-/NamespaceTable/30 range system
-/NamespaceTable/Max range system
-/Table/32 range system
-/Table/33 range system
-/Table/34 range system
-/Table/35 range system
-/Table/36 range system
-/Table/37 range system
+/NamespaceTable/30 database system (host)
+/NamespaceTable/Max database system (host)
+/Table/32 database system (host)
+/Table/33 database system (host)
+/Table/34 database system (host)
+/Table/35 database system (host)
+/Table/36 database system (host)
+/Table/37 database system (host)
/Table/38 range system
-/Table/39 range system
-/Table/40 range system
-/Table/41 range system
-/Table/42 range system
-/Table/43 range system
-/Table/44 range system
-/Table/45 ttl_seconds=7200 num_replicas=5
-/Table/46 range system
-/Table/47 range system
+/Table/39 database system (host)
+/Table/40 database system (host)
+/Table/41 database system (host)
+/Table/42 database system (host)
+/Table/43 database system (host)
+/Table/44 database system (host)
+/Table/45 ttl_seconds=7200 ignore_strict_gc=true num_replicas=5 rangefeed_enabled=true
+/Table/46 database system (host)
+/Table/47 database system (host)
/Table/56 num_replicas=7 num_voters=5
/Table/57 num_replicas=7

# vim:ft=diff
115 changes: 62 additions & 53 deletions pkg/ccl/spanconfigccl/spanconfigcomparedccl/testdata/multitenant
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,30 +12,37 @@ initialize tenant=11
----

# Before kicking starting off tenant reconciliation, we should find seed
# configs for the newly initialized tenants. As yet, there are no differences
# between the subsystems.
# configs for the newly initialized tenants. As yet, there are no (unexpected)
# differences between the subsystems.

configs version=current offset=43
----
...
/Tenant/10 range default
/Tenant/11 range default

diff
diff offset=73
----
--- gossiped system config span (legacy)
+++ span config infrastructure (current)
...
/Tenant/10 range default
/Tenant/11 range default

reconcile tenant=11
----

# As soon as tenant-11 starts reconciling, we should observe more fine-grained
# span configs within its keyspan. This isn't true for the legacy system.

configs version=current offset=43 limit=5
----
...
/Tenant/10 range default
/Tenant/11 range default
/Tenant/11/Table/4 range default
/Tenant/11/Table/5 range default
/Tenant/11/Table/6 range default
/Tenant/11 database system (tenant)
/Tenant/11/Table/4 database system (tenant)
/Tenant/11/Table/5 database system (tenant)
/Tenant/11/Table/6 database system (tenant)
...

configs version=legacy offset=43
Expand All @@ -44,20 +51,21 @@ configs version=legacy offset=43
/Tenant/10 range default
/Tenant/11 range default

diff limit=10
diff offset=73 limit=10
----
--- gossiped system config span (legacy)
+++ span config infrastructure (current)
@@ -44,3 +44,36 @@
...
/Tenant/10 range default
/Tenant/11 range default
+/Tenant/11/Table/4 range default
+/Tenant/11/Table/5 range default
+/Tenant/11/Table/6 range default
+/Tenant/11/Table/7 range default
+/Tenant/11/Table/11 range default
+/Tenant/11/Table/12 range default
+/Tenant/11/Table/13 range default
-/Tenant/11 range default
+/Tenant/11 database system (tenant)
+/Tenant/11/Table/4 database system (tenant)
+/Tenant/11/Table/5 database system (tenant)
+/Tenant/11/Table/6 database system (tenant)
+/Tenant/11/Table/7 database system (tenant)
+/Tenant/11/Table/11 database system (tenant)
+/Tenant/11/Table/12 database system (tenant)
+/Tenant/11/Table/13 database system (tenant)
...

# Sanity check that new tenant tables show up correctly.
Expand All @@ -69,46 +77,47 @@ CREATE TABLE db.t2();
ALTER TABLE db.t1 CONFIGURE ZONE using num_replicas = 42, gc.ttlseconds = 1000;
----

diff
diff offset=73
----
--- gossiped system config span (legacy)
+++ span config infrastructure (current)
@@ -44,3 +44,38 @@
...
/Tenant/10 range default
/Tenant/11 range default
+/Tenant/11/Table/4 range default
+/Tenant/11/Table/5 range default
+/Tenant/11/Table/6 range default
+/Tenant/11/Table/7 range default
+/Tenant/11/Table/11 range default
+/Tenant/11/Table/12 range default
+/Tenant/11/Table/13 range default
+/Tenant/11/Table/14 range default
+/Tenant/11/Table/15 range default
+/Tenant/11/Table/19 range default
+/Tenant/11/Table/20 range default
+/Tenant/11/Table/21 range default
+/Tenant/11/Table/23 range default
+/Tenant/11/Table/24 range default
+/Tenant/11/Table/25 range default
+/Tenant/11/Table/26 range default
+/Tenant/11/Table/27 range default
+/Tenant/11/Table/28 range default
+/Tenant/11/NamespaceTable/30 range default
+/Tenant/11/NamespaceTable/Max range default
+/Tenant/11/Table/32 range default
+/Tenant/11/Table/33 range default
+/Tenant/11/Table/34 range default
+/Tenant/11/Table/35 range default
+/Tenant/11/Table/36 range default
+/Tenant/11/Table/37 range default
+/Tenant/11/Table/39 range default
+/Tenant/11/Table/40 range default
+/Tenant/11/Table/41 range default
+/Tenant/11/Table/42 range default
+/Tenant/11/Table/43 range default
+/Tenant/11/Table/44 range default
+/Tenant/11/Table/46 range default
-/Tenant/11 range default
+/Tenant/11 database system (tenant)
+/Tenant/11/Table/4 database system (tenant)
+/Tenant/11/Table/5 database system (tenant)
+/Tenant/11/Table/6 database system (tenant)
+/Tenant/11/Table/7 database system (tenant)
+/Tenant/11/Table/11 database system (tenant)
+/Tenant/11/Table/12 database system (tenant)
+/Tenant/11/Table/13 database system (tenant)
+/Tenant/11/Table/14 database system (tenant)
+/Tenant/11/Table/15 database system (tenant)
+/Tenant/11/Table/19 database system (tenant)
+/Tenant/11/Table/20 database system (tenant)
+/Tenant/11/Table/21 database system (tenant)
+/Tenant/11/Table/23 database system (tenant)
+/Tenant/11/Table/24 database system (tenant)
+/Tenant/11/Table/25 database system (tenant)
+/Tenant/11/Table/26 database system (tenant)
+/Tenant/11/Table/27 database system (tenant)
+/Tenant/11/Table/28 database system (tenant)
+/Tenant/11/NamespaceTable/30 database system (tenant)
+/Tenant/11/NamespaceTable/Max database system (tenant)
+/Tenant/11/Table/32 database system (tenant)
+/Tenant/11/Table/33 database system (tenant)
+/Tenant/11/Table/34 database system (tenant)
+/Tenant/11/Table/35 database system (tenant)
+/Tenant/11/Table/36 database system (tenant)
+/Tenant/11/Table/37 database system (tenant)
+/Tenant/11/Table/39 database system (tenant)
+/Tenant/11/Table/40 database system (tenant)
+/Tenant/11/Table/41 database system (tenant)
+/Tenant/11/Table/42 database system (tenant)
+/Tenant/11/Table/43 database system (tenant)
+/Tenant/11/Table/44 database system (tenant)
+/Tenant/11/Table/46 database system (tenant)
+/Tenant/11/Table/56 ttl_seconds=1000 num_replicas=42
+/Tenant/11/Table/57 range default

Expand Down
Loading

0 comments on commit a475ea7

Please sign in to comment.