spanconfigsqltranslator: populate protected_timestamps in SpanConfig #74803

adityamaru · 2022-01-13T16:52:18Z

This change teaches the SQLTranslator to hydrate the SpanConfigs
for a table with protected timestamps that apply to that table.

Concretely, this change initializes a spanconfig.ProtectedTimestampStateReader
in the txn in which the translation is taking place, thereby
providing a transactional view of the system.protected_ts_records
table. After generating the span configurations based on the zone
configurations that apply to the table, we hydrate the newly
introduced protected_timestamps field on each span configuration
with all the protected timestamps that apply to this table. This
includes protected timestamp records that directly target this
table, as well as records targetting the table's parent database.
This information is obtained from the ProtectedTimestampStateReader
mentioned above.

Additionally, this change modifies StartTenant to allow secondary
tenants to interact with the protected timestamp subsystem using a
"real" protectedts.Provider provided the migration
EnableProtectedTimestampsForTenant has run.

For testing purposes, this change teaches the data driven framework
of two additional commands protect and release.

Informs: #73727

Release note: None

cockroach-teamcity · 2022-01-13T16:52:28Z

This change is

adityamaru · 2022-01-13T16:53:13Z

First two commits are from #74737.

adityamaru · 2022-01-20T18:27:37Z

pkg/roachpb/span_config.proto

+  // ProtectedTimestamps captures all the protected timestamp records that apply
+  // to the range. The timestamp values represent the timestamps after which
+  // data will be protected from GC, if the protection succeeds.
+  repeated util.hlc.Timestamp protected_timestamps = 10 [(gogoproto.nullable) = false];


this is subject to change in #74765

pkg/ccl/spanconfigccl/spanconfigcomparedccl/datadriven_test.go

pkg/ccl/spanconfigccl/spanconfigreconcilerccl/datadriven_test.go

pkg/ccl/spanconfigccl/spanconfigsqltranslatorccl/datadriven_test.go

pkg/spanconfig/spanconfigsqltranslator/sqltranslator.go

pkg/spanconfig/spanconfigtestutils/utils.go

adityamaru · 2022-01-24T14:33:15Z

@arulajmani @irfansharif this should be RFAL!

irfansharif

LGTM mod comments below.

pkg/ccl/spanconfigccl/spanconfigsqltranslatorccl/datadriven_test.go

pkg/ccl/spanconfigccl/spanconfigsqltranslatorccl/testdata/protectedts

pkg/server/testserver.go

pkg/ccl/spanconfigccl/spanconfigsqltranslatorccl/datadriven_test.go

arulajmani

Reviewed 2 of 22 files at r7, 19 of 22 files at r8, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @adityamaru, @ajwerner, @dt, and @irfansharif)

pkg/clusterversion/cockroach_versions.go, line 386 at r8 (raw file):

		Version: roachpb.Version{Major: 21, Minor: 2, Internal: 48},
	},
	{

To confirm, it's okay for us to do this now because we haven't released any betas?

pkg/spanconfig/spanconfigsqltranslator/sqltranslator.go, line 376 at r8 (raw file):

		// Copy the ProtectionPolicies that apply to the table's SpanConfig onto its
		// SubzoneSpanConfig.
		subzoneSpanConfig.GCPolicy.ProtectionPolicies = tableSpanConfig.GCPolicy.ProtectionPolicies[:]

nit: would it be clearer if you pulled the right hand side of the assignment in a separate variable and used it to assign the ProtectionPolicies wherever you need to?

irfansharif · 2022-01-25T15:52:27Z

To confirm, it's okay for us to do this now because we haven't released any betas?

Yes, but also because we don't support upgrades from alphas/betas to prod clusters. Re-ordering these versions is safe any time before the actual release itself (not even the branch cut).

pkg/ccl/spanconfigccl/spanconfigsqltranslatorccl/datadriven_test.go

irfansharif · 2022-01-25T15:54:46Z

pkg/sql/logictest/testdata/logic_test/crdb_internal_tenant

@@ -11,6 +11,8 @@ SELECT node_id, name FROM crdb_internal.leases ORDER BY name
 0  eventlog
 0  jobs
 0  locations
+0  protected_ts_meta


BTW, what's this about?

I'm not a 100% sure, but a little digging led me to #69008 where we switched to leasing all system descriptors except a few in a deny list. The PTS tables are not in this deny list so I guess we expect to see them here?

pkg/spanconfig/spanconfigsqltranslator/protectedts_state_reader.go

adityamaru · 2022-01-25T20:13:33Z

nit: would it be clearer if you pulled the right hand side of the assignment in a separate variable and used it to assign the ProtectionPolicies wherever you need to?

Discussed offline, going to keep it as is for now since it avoids an extra copy.

This change teaches the SQLTranslator to hydrate the SpanConfigs for a table with protected timestamps that apply to that table. Concretely, this change initializes a spanconfig.ProtectedTimestampStateReader in the txn in which the translation is taking place, thereby providing a transactional view of the system.protected_ts_records table. After generating the span configurations based on the zone configurations that apply to the table, we hydrate the newly introduced protected_timestamps field on each span configuration with all the protected timestamps that apply to this table. This includes protected timestamp records that directly target this table, as well as records targetting the table's parent database. This information is obtained from the ProtectedTimestampStateReader mentioned above. Additionally, this change modifies StartTenant to allow secondary tenants to interact with the protected timestamp subsystem using a "real" protectedts.Provider provided the migration EnableProtectedTimestampsForTenant has run. For testing purposes, this change teaches the data driven framework of two additional commands protect and release. Informs: cockroachdb#73727 Release note: None

adityamaru · 2022-01-26T00:45:03Z

Bazel timeout seems unrelated.
TFTRs!

bors r=irfansharif,arulajmani

craig · 2022-01-26T01:44:30Z

Build succeeded:

GitHub CI (Cockroach)

adityamaru requested review from ajwerner, arulajmani, irfansharif and a team January 13, 2022 16:52

adityamaru requested review from a team as code owners January 13, 2022 16:52

adityamaru requested review from dt and otan and removed request for a team January 13, 2022 16:52

adityamaru requested review from a team and srosenberg and removed request for a team and otan January 13, 2022 16:52

adityamaru mentioned this pull request Jan 13, 2022

protectedts: rework protected timestamp storage for multi-tenant model #73727

Closed

23 tasks

adityamaru force-pushed the full-translate-contd branch from 84786e3 to 1118d9e Compare January 20, 2022 18:25

adityamaru commented Jan 20, 2022

View reviewed changes

adityamaru force-pushed the full-translate-contd branch from 1118d9e to 2e87a4b Compare January 20, 2022 18:32

irfansharif reviewed Jan 20, 2022

View reviewed changes

adityamaru force-pushed the full-translate-contd branch from 2e87a4b to 733eb94 Compare January 21, 2022 23:24

blathers-crl bot requested a review from irfansharif January 21, 2022 23:24

adityamaru force-pushed the full-translate-contd branch from 733eb94 to 57f3502 Compare January 22, 2022 21:16

irfansharif approved these changes Jan 24, 2022

View reviewed changes

irfansharif removed the request for review from srosenberg January 24, 2022 15:06

adityamaru force-pushed the full-translate-contd branch 2 times, most recently from 34db9a9 to 29a4a03 Compare January 25, 2022 14:58

arulajmani approved these changes Jan 25, 2022

View reviewed changes

irfansharif approved these changes Jan 25, 2022

View reviewed changes

adityamaru force-pushed the full-translate-contd branch from 29a4a03 to 50619f0 Compare January 25, 2022 20:13

adityamaru mentioned this pull request Jan 25, 2022

server: TestValidRoles failed #75483

Closed

adityamaru force-pushed the full-translate-contd branch from 50619f0 to 7c63710 Compare January 25, 2022 21:49

craig bot merged commit 8f15635 into cockroachdb:master Jan 26, 2022

adityamaru deleted the full-translate-contd branch January 26, 2022 14:59

irfansharif mentioned this pull request Jan 27, 2022

sql/execstats: TestTraceAnalyzer data race #75546

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

spanconfigsqltranslator: populate protected_timestamps in SpanConfig #74803

spanconfigsqltranslator: populate protected_timestamps in SpanConfig #74803

adityamaru commented Jan 13, 2022 •

edited

Loading

cockroach-teamcity commented Jan 13, 2022

adityamaru commented Jan 13, 2022

adityamaru Jan 20, 2022

adityamaru commented Jan 24, 2022

irfansharif left a comment

arulajmani left a comment

irfansharif commented Jan 25, 2022

irfansharif Jan 25, 2022

adityamaru Jan 25, 2022

adityamaru commented Jan 25, 2022

adityamaru commented Jan 26, 2022

craig bot commented Jan 26, 2022

spanconfigsqltranslator: populate protected_timestamps in SpanConfig #74803

spanconfigsqltranslator: populate protected_timestamps in SpanConfig #74803

Conversation

adityamaru commented Jan 13, 2022 • edited Loading

cockroach-teamcity commented Jan 13, 2022

adityamaru commented Jan 13, 2022

adityamaru Jan 20, 2022

Choose a reason for hiding this comment

adityamaru commented Jan 24, 2022

irfansharif left a comment

Choose a reason for hiding this comment

arulajmani left a comment

Choose a reason for hiding this comment

irfansharif commented Jan 25, 2022

irfansharif Jan 25, 2022

Choose a reason for hiding this comment

adityamaru Jan 25, 2022

Choose a reason for hiding this comment

adityamaru commented Jan 25, 2022

adityamaru commented Jan 26, 2022

craig bot commented Jan 26, 2022

adityamaru commented Jan 13, 2022 •

edited

Loading