Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

changefeedccl: protect system.comments and system.zones #130622

Conversation

stevendanna
Copy link
Collaborator

The catalog reader reads from system.descriptor, system.comments, and system.zones when reading a table descriptor from disk. We were only protecting system.descriptors.

Further, the test here previously wasn't testing anything because it was only forcing the GC-threshold to 1 second before read timestamp it was using.

Informs #128806

Release note (bug fix): Fix bug that could prevent a CHANGEFEED from being able to resume after being paused for prolonged period of time.

@cockroach-teamcity
Copy link
Member

This change is Reviewable

Copy link
Collaborator

@andyyang890 andyyang890 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @stevendanna)


pkg/ccl/changefeedccl/protected_timestamps.go line 49 at r1 (raw file):

	// We protect system.descriptors because a changefeed needs all of the history
	// of table descriptors to version data.
	tablesToProtect := make(descpb.IDs, 0, targets.NumUniqueTables()+1)

fyi might want to consider updating this number


pkg/ccl/changefeedccl/protected_timestamps.go line 74 at r1 (raw file):

		return nil
	})
	addTablePrefix(keys.DescriptorTableID)

should we also add it here?


pkg/ccl/changefeedccl/protected_timestamps_test.go line 510 at r1 (raw file):

		row := sqlDB.QueryRow(t, fmt.Sprintf("SELECT range_id FROM [SHOW RANGES FROM TABLE %s.%s]", tableName, databaseName))
		var rangeID int64
		row.Scan(&rangeID)

Could there ever be more than one range?


pkg/ccl/changefeedccl/protected_timestamps_test.go line 513 at r1 (raw file):

		refreshPTSReaderCache(s.Clock().Now(), tableName, databaseName)
		t.Logf("enqueuing range %d for mvccGC", rangeID)
		sqlDB.Exec(t, `SELECT crdb_internal.kv_enqueue_replica($1, 'mvccGC', true)`, rangeID)

Wanted to double check my understanding: this would trigger a manual synchronous GC on the range for the tables and we would expect it to not affect the tables that are protected by the PTS?


pkg/ccl/changefeedccl/protected_timestamps_test.go line 520 at r1 (raw file):

	sqlDB.Exec(t, "ALTER TABLE foo ADD COLUMN d STRING")
	time.Sleep(2 * time.Second)
	// If you want to GC all sytem tables:

nit: s/sytem/system/

The catalog reader reads from system.descriptor, system.comments, and
system.zones when reading a table descriptor from disk. We were only
protecting system.descriptors.

Further, the test here previously wasn't testing anything because it
was only forcing the GC-threshold to 1 second before read timestamp it
was using.

Informs cockroachdb#128806

Release note (bug fix): Fix bug that could prevent a CHANGEFEED from
being able to resume after being paused for prolonged period of time.
@stevendanna stevendanna force-pushed the ssd/shame-if-anything-happened-to-those-nice-comments-you-have-there branch from 6501995 to e315bd3 Compare September 16, 2024 07:52
Copy link
Collaborator Author

@stevendanna stevendanna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andyyang890)


pkg/ccl/changefeedccl/protected_timestamps.go line 49 at r1 (raw file):

Previously, andyyang890 (Andy Yang) wrote…

fyi might want to consider updating this number

Done.


pkg/ccl/changefeedccl/protected_timestamps.go line 74 at r1 (raw file):

Previously, andyyang890 (Andy Yang) wrote…

should we also add it here?

Done. Nice catch!


pkg/ccl/changefeedccl/protected_timestamps_test.go line 510 at r1 (raw file):

Previously, andyyang890 (Andy Yang) wrote…

Could there ever be more than one range?

In production absolutely. In this test, it would be very surprising.


pkg/ccl/changefeedccl/protected_timestamps_test.go line 513 at r1 (raw file):

Previously, andyyang890 (Andy Yang) wrote…

Wanted to double check my understanding: this would trigger a manual synchronous GC on the range for the tables and we would expect it to not affect the tables that are protected by the PTS?

Yes, this will run the replica through the MVCC GC queue synchronously. The true as the final argument is for skipShouldQueue which means we will alway run the queue's process callback which is what handles running the GC process.

The GC might affect tables even with the PTS record in place, but we expect that the gc threshold to not advance past the PTS. Thus we expect any read at ts to proceed even after sleeping for a little bit.

@stevendanna stevendanna marked this pull request as ready for review September 16, 2024 12:29
@stevendanna stevendanna requested a review from a team as a code owner September 16, 2024 12:29
@stevendanna stevendanna removed the request for review from a team September 16, 2024 12:47
Copy link
Collaborator

@andyyang890 andyyang890 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: Thanks!

Reviewed 2 of 2 files at r2, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @stevendanna)

@stevendanna stevendanna added backport-23.2.x Flags PRs that need to be backported to 23.2. backport-24.1.x Flags PRs that need to be backported to 24.1. backport-24.2.x Flags PRs that need to be backported to 24.2 labels Sep 18, 2024
@stevendanna
Copy link
Collaborator Author

bors r=andyyang890

@craig craig bot merged commit f09d877 into cockroachdb:master Sep 18, 2024
23 checks passed
asg0451 added a commit to asg0451/cockroach that referenced this pull request Sep 25, 2024
Adds `system.role_members` to the list of system
tables that are protected by changefeeds.

Fixes: cockroachdb#128806
See also: cockroachdb#130622

Release note (bug fix): Fix a bug which could
result in changefeeds using cdc queries failing
due to a system table being GC'd.
asg0451 added a commit to asg0451/cockroach that referenced this pull request Sep 25, 2024
Adds `system.role_members` to the list of system
tables that are protected by changefeeds.

Fixes: cockroachdb#128806
See also: cockroachdb#130622

Release note (bug fix): Fix a bug which could
result in changefeeds using cdc queries failing
due to a system table being GC'd.
asg0451 added a commit to asg0451/cockroach that referenced this pull request Sep 27, 2024
Adds `system.role_members` to the list of system
tables that are protected by changefeeds.

Informs: cockroachdb#128806
See also: cockroachdb#130622

Release note (bug fix): Fix a bug which could
result in changefeeds using cdc queries failing
due to a system table being GC'd.
craig bot pushed a commit that referenced this pull request Sep 27, 2024
131027: changefeedccl: protect system.role_members using protected timestamps r=rharding6373 a=asg0451

Adds `system.role_members` to the list of system
tables that are protected by changefeeds.

Informs: #128806
See also: #130622

Release note (bug fix): Fix a bug which could
result in changefeeds using cdc queries failing
due to a system table being GC'd.


Co-authored-by: Miles Frankel <[email protected]>
cthumuluru-crdb pushed a commit to cthumuluru-crdb/cockroach that referenced this pull request Oct 1, 2024
Adds `system.role_members` to the list of system
tables that are protected by changefeeds.

Informs: cockroachdb#128806
See also: cockroachdb#130622

Release note (bug fix): Fix a bug which could
result in changefeeds using cdc queries failing
due to a system table being GC'd.
stevendanna pushed a commit that referenced this pull request Oct 1, 2024
Adds `system.role_members` to the list of system
tables that are protected by changefeeds.

Informs: #128806
See also: #130622

Release note (bug fix): Fix a bug which could
result in changefeeds using cdc queries failing
due to a system table being GC'd.
stevendanna pushed a commit that referenced this pull request Oct 1, 2024
Adds `system.role_members` to the list of system
tables that are protected by changefeeds.

Informs: #128806
See also: #130622

Release note (bug fix): Fix a bug which could
result in changefeeds using cdc queries failing
due to a system table being GC'd.
stevendanna pushed a commit to stevendanna/cockroach that referenced this pull request Oct 1, 2024
Adds `system.role_members` to the list of system
tables that are protected by changefeeds.

Informs: cockroachdb#128806
See also: cockroachdb#130622

Release note (bug fix): Fix a bug which could
result in changefeeds using cdc queries failing
due to a system table being GC'd.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-23.2.x Flags PRs that need to be backported to 23.2. backport-24.1.x Flags PRs that need to be backported to 24.1. backport-24.2.x Flags PRs that need to be backported to 24.2
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants