Delete all legacy sharding related code #10278

mattlord · 2022-05-11T21:20:48Z

Description

This removes all code related to Legacy Sharding — which began its march toward deprecation and removal in Vitess 6.0 with the introduction of VReplication — the tooling and methods to shard, reshard, and diff keyspaces before VReplication.

Related Issue(s)

Follow-up to: Ensure all legacy sharding commands are clearly deprecated #10281
Fixes: Task: delete legacy sharding code #9219
- Fixes: Delete SplitClone #10276
- Fixes: Delete LegacySplitClone #10277
Related to: [14.0]: Tracking issue - flaky tests #10620

Checklist

"Backport me!" label has been added if this change should be backported
Tests were added or are not required
Documentation: Remove documentation related to legacy sharding in 15.0+ website#1090

Signed-off-by: Matt Lord <[email protected]>

mattlord · 2022-07-06T00:32:04Z

Thank you for the thorough review, @ajm188 ! I know that this one was difficult as it's so many lines changed (mostly removed) that it's easy to miss something so the fine-tooth comb used is very much appreciated. I believe that I addressed all of your comments here: 778b7d1

Please let me know if I missed something though.

ajm188

LGTM!

deepthi

Wow, this is going to be so good :)
Thank you for taking it on. Just a few minor comments, otherwise LGTM

go/flags/endtoend/vttablet.txt

deepthi · 2022-07-06T17:24:00Z

go/test/endtoend/cluster/vtctld_process.go

-		// hard-code these two soon-to-be deprecated drain values.
-		"--wait_for_drain_sleep_rdonly", "1s",
-		"--wait_for_drain_sleep_replica", "1s",


are these flags now deleted? or only deprecated?

They're now deleted. They were marked as deprecated before but this end2end test comment wasn't updated: inhttps://github.com/vitessio/vitess/blob/23727bfee12607250da769180fab8064b073dad1/go/vt/wrangler/keyspace.go#L48-L52

go/test/endtoend/tabletmanager/tablet_health_test.go

Signed-off-by: Matt Lord <[email protected]>

mattlord · 2022-07-07T16:30:16Z

Thanks, @deepthi ! I believe that I addressed all of your comments here: a9ba0cf

Please let me know if I missed something though.

deepthi · 2022-07-07T18:00:08Z

Thanks, @deepthi ! I believe that I addressed all of your comments here: a9ba0cf

Please let me know if I missed something though.

Everything looks good. I wish we could have a party to celebrate 🥳

* Delete all legacy sharding related code Signed-off-by: Matt Lord <[email protected]> * Move used until functions from initialsharding to cluster Signed-off-by: Matt Lord <[email protected]> * Remove vtctl commands Signed-off-by: Matt Lord <[email protected]> * Kill vtworker and SetKeyspaceServedFrom cmd Signed-off-by: Matt Lord <[email protected]> * WaitForDrain related stragglers Signed-off-by: Matt Lord <[email protected]> * Legacy local straggler workflow Signed-off-by: Matt Lord <[email protected]> * Get rid of SetKeyspaceShardingInfo & wait for drain stragglers Signed-off-by: Matt Lord <[email protected]> * Remove vtworker stragglers Signed-off-by: Matt Lord <[email protected]> * Update throttlerservice protobuf Signed-off-by: Matt Lord <[email protected]> * Rename test 24, add hashicorp vault test to it (now mysql_server_vault) Signed-off-by: Matt Lord <[email protected]> * Remove last mentiones of legacy sharding in vtctl Signed-off-by: Matt Lord <[email protected]> * remove binlog_use_v3_resharding_mode Signed-off-by: Matt Lord <[email protected]> * Address review comments Signed-off-by: Matt Lord <[email protected]> * Address review comments Signed-off-by: Matt Lord <[email protected]> * Correct vtgate help output Signed-off-by: Matt Lord <[email protected]>

…test + compressors (#119) * Delete all legacy sharding related code (vitessio#10278) * Delete all legacy sharding related code Signed-off-by: Matt Lord <[email protected]> * Move used until functions from initialsharding to cluster Signed-off-by: Matt Lord <[email protected]> * Remove vtctl commands Signed-off-by: Matt Lord <[email protected]> * Kill vtworker and SetKeyspaceServedFrom cmd Signed-off-by: Matt Lord <[email protected]> * WaitForDrain related stragglers Signed-off-by: Matt Lord <[email protected]> * Legacy local straggler workflow Signed-off-by: Matt Lord <[email protected]> * Get rid of SetKeyspaceShardingInfo & wait for drain stragglers Signed-off-by: Matt Lord <[email protected]> * Remove vtworker stragglers Signed-off-by: Matt Lord <[email protected]> * Update throttlerservice protobuf Signed-off-by: Matt Lord <[email protected]> * Rename test 24, add hashicorp vault test to it (now mysql_server_vault) Signed-off-by: Matt Lord <[email protected]> * Remove last mentiones of legacy sharding in vtctl Signed-off-by: Matt Lord <[email protected]> * remove binlog_use_v3_resharding_mode Signed-off-by: Matt Lord <[email protected]> * Address review comments Signed-off-by: Matt Lord <[email protected]> * Address review comments Signed-off-by: Matt Lord <[email protected]> * Correct vtgate help output Signed-off-by: Matt Lord <[email protected]> * go fmt Signed-off-by: Tim Vaillancourt <[email protected]> * Remove v2 resharding fields (vitessio#10409) * cleanup: remove sharding_column_name and sharding_column_type Signed-off-by: Arvind Murty <[email protected]> * cleanup: remove sharding_column_name and sharding_column_type Signed-off-by: Arvind Murty <[email protected]> * cleanup: remove sharding_column_name and sharding_column_type Signed-off-by: Arvind Murty <[email protected]> * cleanup: remove sharding_column_name and sharding_column_type Signed-off-by: Arvind Murty <[email protected]> * generate vtadmin files Signed-off-by: Arvind Murty <[email protected]> * cleanup: remove sharding_column_name and sharding_column_type from vtadmin Signed-off-by: Arvind Murty <[email protected]> * Merge from main Signed-off-by: Arvind Murty <[email protected]> * Fix bad merge conflict resolution Signed-off-by: Tim Vaillancourt <[email protected]> * Fix missing 'sharding' import Signed-off-by: Tim Vaillancourt <[email protected]> * Fix bad conflict resolution in go/test/endtoend/cellalias/cell_alias_test.go Signed-off-by: Tim Vaillancourt <[email protected]> * Backup/Restore: add support for external compressors and decompressors (vitessio#10558) * change to support an external decompressor Signed-off-by: Renan Rangel <[email protected]> Signed-off-by: Rameez Sajwani <[email protected]> * add external compressor support + builtin additional compressors Signed-off-by: Renan Rangel <[email protected]> Signed-off-by: Rameez Sajwani <[email protected]> * wrap external compressor/decompressor Signed-off-by: Renan Rangel <[email protected]> Signed-off-by: Rameez Sajwani <[email protected]> * go mod tidy + comments Signed-off-by: Renan Rangel <[email protected]> Signed-off-by: Rameez Sajwani <[email protected]> * add copyright notices Signed-off-by: Renan Rangel <[email protected]> Signed-off-by: Rameez Sajwani <[email protected]> * add support for builtin engine Signed-off-by: Renan Rangel <[email protected]> Signed-off-by: Rameez Sajwani <[email protected]> * Adding test case for buckup compression Signed-off-by: Rameez Sajwani <[email protected]> * Fixing unit test and run mod tidy Signed-off-by: Rameez Sajwani <[email protected]> * Removing unwanted unit tests Signed-off-by: Rameez Sajwani <[email protected]> * Increase timeout of backup tests Signed-off-by: Rameez Sajwani <[email protected]> * fixing linter errors Signed-off-by: Rameez Sajwani <[email protected]> * Change test logic to accomodate running selective tests Signed-off-by: Rameez Sajwani <[email protected]> * removing lint warning Signed-off-by: Rameez Sajwani <[email protected]> * fixing test failure Signed-off-by: Rameez Sajwani <[email protected]> * Removing un-necessary test Signed-off-by: Rameez Sajwani <[email protected]> * Fixing code review feeback Signed-off-by: Rameez Sajwani <[email protected]> * Change builtinEngine to consider 'auto' decompressor Signed-off-by: Rameez Sajwani <[email protected]> * fixing Upgrade/Downgrade test Signed-off-by: Rameez Sajwani <[email protected]> * Fix type & add summary under release notes Signed-off-by: Rameez Sajwani <[email protected]> * Fixing typos in summary Signed-off-by: Rameez Sajwani <[email protected]> * Fixing flag name typos Signed-off-by: Rameez Sajwani <[email protected]> Co-authored-by: Renan Rangel <[email protected]> Co-authored-by: Renan Rangel <[email protected]> * Add MySQL 8 Support to Backup Tests (vitessio#10691) * Add support for MySQL 8.0 in backup tests Signed-off-by: Matt Lord <[email protected]> * Add 8.0 workflow Signed-off-by: Matt Lord <[email protected]> * whitespace Signed-off-by: Matt Lord <[email protected]> * Use vtctldclient SetKeyspaceDurabilityPolicy to manage semi-sync This needed to be done after the shard was setup in order to satisfy the semantic assumptions related to semi-sync in the tests. Signed-off-by: Matt Lord <[email protected]> * Remove extraneous changes Signed-off-by: Matt Lord <[email protected]> * We need lz4 for TestXtrabackupStreamWithlz4Compression Signed-off-by: Matt Lord <[email protected]> * Try using Percona Repo for MySQL 8 to align mysqld and xtrabackup versions Signed-off-by: Matt Lord <[email protected]> * Specify stream type everywhere Signed-off-by: Matt Lord <[email protected]> * Remove repeated server install Signed-off-by: Matt Lord <[email protected]> * Moar... Signed-off-by: Matt Lord <[email protected]> * Move vtctlbackup test to 8.0 Signed-off-by: Matt Lord <[email protected]> * Rename vtbackup test and move to MySQL 8 Signed-off-by: Matt Lord <[email protected]> * Split the xbstream tests so the workflow doesn't time out Otherwise it was going over the 10min limit and getting killed. Signed-off-by: Matt Lord <[email protected]> * Use MySQL 8 compat method for setting passwords Signed-off-by: Matt Lord <[email protected]> * Test increasing timeout at another level Signed-off-by: Matt Lord <[email protected]> * Don't use the init passwords file with 8.0 mysqlctl doesn't start... This is likely due to the change in initialization behavior with MySQL 8.0 as it goes though two phases and you can't simply start up mysqld and pass it data, it has to initialize and restart first. Signed-off-by: Matt Lord <[email protected]> * Fix incorrect password update statement for vt_repl user Signed-off-by: Rohit Nayak <[email protected]> * Bump timeouts for 8.0 backup tests Signed-off-by: Matt Lord <[email protected]> * Bump it more :( Signed-off-by: Matt Lord <[email protected]> * Increase backup/restore timeout in backup_utils Signed-off-by: Matt Lord <[email protected]> * Apply new 8.0 template everywhere Signed-off-by: Matt Lord <[email protected]> * Fix bugs around how the compression flags were getting passed Signed-off-by: Matt Lord <[email protected]> * Use 45m timeout for the workflow, 30m for the run. Signed-off-by: Matt Lord <[email protected]> * These changes were no longer needed so limiting diff. Signed-off-by: Matt Lord <[email protected]> * Explicitly skip new linter check Signed-off-by: Matt Lord <[email protected]> Co-authored-by: Rohit Nayak <[email protected]> * Fix test file merge issues Signed-off-by: Tim Vaillancourt <[email protected]> * Go 1.18.7 to fix error Signed-off-by: Tim Vaillancourt <[email protected]> * Fix lint Signed-off-by: Tim Vaillancourt <[email protected]> * Fix vttablet.txt Signed-off-by: Tim Vaillancourt <[email protected]> * Remove sharding_column_name and sharding_column_type from vtctld2 (vitessio#10459) * Remove sharding_column_name and sharding_column_type Signed-off-by: notfelineit <[email protected]> * Run make web_build to compile production files Signed-off-by: notfelineit <[email protected]> * Remove vars from test Signed-off-by: notfelineit <[email protected]> * Remove sharding column type select box test from web test Signed-off-by: notfelineit <[email protected]> --------- Signed-off-by: Matt Lord <[email protected]> Signed-off-by: Tim Vaillancourt <[email protected]> Signed-off-by: Arvind Murty <[email protected]> Signed-off-by: notfelineit <[email protected]> Co-authored-by: Matt Lord <[email protected]> Co-authored-by: Arvind Murty <[email protected]> Co-authored-by: Rameez Sajwani <[email protected]> Co-authored-by: Renan Rangel <[email protected]> Co-authored-by: Renan Rangel <[email protected]> Co-authored-by: Rohit Nayak <[email protected]> Co-authored-by: Frances Thai <[email protected]>

…test + compressors (#119) (#124) * Delete all legacy sharding related code (vitessio#10278) * Delete all legacy sharding related code * Move used until functions from initialsharding to cluster * Remove vtctl commands * Kill vtworker and SetKeyspaceServedFrom cmd * WaitForDrain related stragglers * Legacy local straggler workflow * Get rid of SetKeyspaceShardingInfo & wait for drain stragglers * Remove vtworker stragglers * Update throttlerservice protobuf * Rename test 24, add hashicorp vault test to it (now mysql_server_vault) * Remove last mentiones of legacy sharding in vtctl * remove binlog_use_v3_resharding_mode * Address review comments * Address review comments * Correct vtgate help output * go fmt * Remove v2 resharding fields (vitessio#10409) * cleanup: remove sharding_column_name and sharding_column_type * cleanup: remove sharding_column_name and sharding_column_type * cleanup: remove sharding_column_name and sharding_column_type * cleanup: remove sharding_column_name and sharding_column_type * generate vtadmin files * cleanup: remove sharding_column_name and sharding_column_type from vtadmin * Merge from main * Fix bad merge conflict resolution * Fix missing 'sharding' import * Fix bad conflict resolution in go/test/endtoend/cellalias/cell_alias_test.go * Backup/Restore: add support for external compressors and decompressors (vitessio#10558) * change to support an external decompressor * add external compressor support + builtin additional compressors * wrap external compressor/decompressor * go mod tidy + comments * add copyright notices * add support for builtin engine * Adding test case for buckup compression * Fixing unit test and run mod tidy * Removing unwanted unit tests * Increase timeout of backup tests * fixing linter errors * Change test logic to accomodate running selective tests * removing lint warning * fixing test failure * Removing un-necessary test * Fixing code review feeback * Change builtinEngine to consider 'auto' decompressor * fixing Upgrade/Downgrade test * Fix type & add summary under release notes * Fixing typos in summary * Fixing flag name typos * Add MySQL 8 Support to Backup Tests (vitessio#10691) * Add support for MySQL 8.0 in backup tests * Add 8.0 workflow * whitespace * Use vtctldclient SetKeyspaceDurabilityPolicy to manage semi-sync This needed to be done after the shard was setup in order to satisfy the semantic assumptions related to semi-sync in the tests. * Remove extraneous changes * We need lz4 for TestXtrabackupStreamWithlz4Compression * Try using Percona Repo for MySQL 8 to align mysqld and xtrabackup versions * Specify stream type everywhere * Remove repeated server install * Moar... * Move vtctlbackup test to 8.0 * Rename vtbackup test and move to MySQL 8 * Split the xbstream tests so the workflow doesn't time out Otherwise it was going over the 10min limit and getting killed. * Use MySQL 8 compat method for setting passwords * Test increasing timeout at another level * Don't use the init passwords file with 8.0 mysqlctl doesn't start... This is likely due to the change in initialization behavior with MySQL 8.0 as it goes though two phases and you can't simply start up mysqld and pass it data, it has to initialize and restart first. * Fix incorrect password update statement for vt_repl user * Bump timeouts for 8.0 backup tests * Bump it more :( * Increase backup/restore timeout in backup_utils * Apply new 8.0 template everywhere * Fix bugs around how the compression flags were getting passed * Use 45m timeout for the workflow, 30m for the run. * These changes were no longer needed so limiting diff. * Explicitly skip new linter check * Fix test file merge issues * Go 1.18.7 to fix error * Fix lint * Fix vttablet.txt * Remove sharding_column_name and sharding_column_type from vtctld2 (vitessio#10459) * Remove sharding_column_name and sharding_column_type * Run make web_build to compile production files * Remove vars from test * Remove sharding column type select box test from web test --------- Signed-off-by: Matt Lord <[email protected]> Signed-off-by: Tim Vaillancourt <[email protected]> Signed-off-by: Arvind Murty <[email protected]> Signed-off-by: notfelineit <[email protected]> Co-authored-by: Matt Lord <[email protected]> Co-authored-by: Arvind Murty <[email protected]> Co-authored-by: Rameez Sajwani <[email protected]> Co-authored-by: Renan Rangel <[email protected]> Co-authored-by: Renan Rangel <[email protected]> Co-authored-by: Rohit Nayak <[email protected]> Co-authored-by: Frances Thai <[email protected]>

mattlord added Type: Internal Cleanup Component: VReplication release notes labels May 11, 2022

mattlord requested a review from rohit-nayak-ps May 11, 2022 21:23

mattlord added this to the v15.0 milestone May 11, 2022

This was referenced May 11, 2022

Ensure all legacy sharding commands are clearly deprecated #10281

Merged

Task: delete legacy sharding code #9219

Closed

Delete all legacy sharding related code

ee4db48

Signed-off-by: Matt Lord <[email protected]>

mattlord force-pushed the delete_legacy_sharding branch 2 times, most recently from 36778fc to c7a9364 Compare July 3, 2022 05:44

Move used until functions from initialsharding to cluster

1fae23b

Signed-off-by: Matt Lord <[email protected]>

mattlord force-pushed the delete_legacy_sharding branch from c7a9364 to 1fae23b Compare July 3, 2022 05:45

Remove vtctl commands

edf6c86

Signed-off-by: Matt Lord <[email protected]>

mattlord force-pushed the delete_legacy_sharding branch from 900a85d to edf6c86 Compare July 3, 2022 06:09

Kill vtworker and SetKeyspaceServedFrom cmd

142772c

Signed-off-by: Matt Lord <[email protected]>

mattlord force-pushed the delete_legacy_sharding branch from 48c486c to 142772c Compare July 3, 2022 06:31

mattlord added 3 commits July 3, 2022 02:39

WaitForDrain related stragglers

8142f2f

Signed-off-by: Matt Lord <[email protected]>

Legacy local straggler workflow

79a1c0d

Signed-off-by: Matt Lord <[email protected]>

Get rid of SetKeyspaceShardingInfo & wait for drain stragglers

845f333

Signed-off-by: Matt Lord <[email protected]>

mattlord force-pushed the delete_legacy_sharding branch from 7d5cde6 to 845f333 Compare July 3, 2022 07:34

Remove vtworker stragglers

1204118

Signed-off-by: Matt Lord <[email protected]>

mattlord force-pushed the delete_legacy_sharding branch from 39e8b52 to 1204118 Compare July 3, 2022 08:18

mattlord requested a review from deepthi July 3, 2022 08:18

mattlord added 2 commits July 3, 2022 04:42

Update throttlerservice protobuf

2a58893

Signed-off-by: Matt Lord <[email protected]>

Rename test 24, add hashicorp vault test to it (now mysql_server_vault)

360acad

Signed-off-by: Matt Lord <[email protected]>

mattlord added Component: Build/CI Component: VReplication and removed Component: VReplication labels Jul 3, 2022

mattlord marked this pull request as ready for review July 3, 2022 15:04

mattlord requested a review from ajm188 as a code owner July 3, 2022 15:04

Address review comments

778b7d1

Signed-off-by: Matt Lord <[email protected]>

mattlord requested a review from ajm188 July 6, 2022 00:32

ajm188 approved these changes Jul 6, 2022

View reviewed changes

deepthi approved these changes Jul 6, 2022

View reviewed changes

deepthi added the NeedsWebsiteDocsUpdate What it says label Jul 6, 2022

deepthi mentioned this pull request Jul 6, 2022

Remove legacy healthcheck files and structures #10542

Merged

3 tasks

frouioui removed the release notes label Jul 7, 2022

mattlord added 3 commits July 7, 2022 12:13

Merge remote-tracking branch 'origin/main' into delete_legacy_sharding

2ffff0b

Signed-off-by: Matt Lord <[email protected]>

Address review comments

a9ba0cf

Signed-off-by: Matt Lord <[email protected]>

Correct vtgate help output

05eaef3

Signed-off-by: Matt Lord <[email protected]>

mattlord mentioned this pull request Jul 7, 2022

Remove documentation related to legacy sharding in 15.0+ vitessio/website#1090

Merged

mattlord merged commit a1f1f12 into vitessio:main Jul 7, 2022

mattlord deleted the delete_legacy_sharding branch July 7, 2022 17:52

GuptaManan100 mentioned this pull request Jul 8, 2022

Remove removed flag in vttablet planetscale/vitess-operator#281

Merged

mattlord mentioned this pull request Aug 23, 2022

Add SKIP check to workflows removed in v15 #11083

Merged

3 tasks

mattlord mentioned this pull request Oct 27, 2022

[cherry-pick]fix anchors for release notes and summary (#11578) #11590

Merged

3 tasks

mattlord mentioned this pull request Jan 12, 2023

Remove Dead Legacy Workflow Manager Code #12085

Merged

3 tasks

timvaillancourt mentioned this pull request Aug 16, 2023

slack-vitess-r14.0.5: several vtorc backports from v15 slackhq/vitess#118

Closed

4 tasks

timvaillancourt mentioned this pull request Aug 17, 2023

slack-vitess-r14.0.5: remove legacy sharding, add MySQL 8.x backup test + compressors slackhq/vitess#119

Merged

4 tasks

rohit-nayak-ps mentioned this pull request Jun 13, 2024

CI Bug: Rename shard name back to match existing workflow file for vreplication_migrate_vdiff2_convert_tz #16148

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Delete all legacy sharding related code #10278

Delete all legacy sharding related code #10278

mattlord commented May 11, 2022 •

edited

Loading

mattlord commented Jul 6, 2022 •

edited

Loading

ajm188 left a comment

deepthi left a comment

deepthi Jul 6, 2022

mattlord Jul 7, 2022

mattlord commented Jul 7, 2022

deepthi commented Jul 7, 2022

Delete all legacy sharding related code #10278

Delete all legacy sharding related code #10278

Conversation

mattlord commented May 11, 2022 • edited Loading

Description

Related Issue(s)

Checklist

mattlord commented Jul 6, 2022 • edited Loading

ajm188 left a comment

Choose a reason for hiding this comment

deepthi left a comment

Choose a reason for hiding this comment

deepthi Jul 6, 2022

Choose a reason for hiding this comment

mattlord Jul 7, 2022

Choose a reason for hiding this comment

mattlord commented Jul 7, 2022

deepthi commented Jul 7, 2022

mattlord commented May 11, 2022 •

edited

Loading

mattlord commented Jul 6, 2022 •

edited

Loading