Move towards MySQL 8.0 as the default template generation #11153

dbussink · 2022-09-01T08:50:45Z

This upgrades the remaining things to Ubuntu 20.04 and makes MySQL 8.0 the default we run tests against. We still have tests for MySQL 5.7 but those are now explicitly opted into.

This should finish up the Ubuntu 20.04 upgrade and also makes things easier for the future when we need to upgrade again.

It is also marked as back port since we want to keep this updated on back ports as well or otherwise we're be blocked in the somewhat near future when 18.04 is deprecated. We probably want to look at changing the default maybe back there to 5.7 (to minimize other test changes that happened in previous PRs like #11118) but that's a separate step to look at during back porting.

Related Issue(s)

Part of the upgrade already started with #11026 but keeping things sensible in the CI generation scripts is much easier with this upgrade we think.

Checklist

"Backport me!" label has been added if this change should be backported
Tests were added or are not required
Documentation was added or is not required

This upgrades the remaining things to Ubuntu 20.04 and makes MySQL 8.0 the default we run tests against. We still have tests for MySQL 5.7 but those are now explicitly opted into. This should finish up the Ubuntu 20.04 upgrade and also makes things easier for the future when we need to upgrade again. Signed-off-by: Dirkjan Bussink <[email protected]>

vitess-bot · 2022-09-01T08:50:47Z

…neration Signed-off-by: deepthi <[email protected]>

…for 8.0 Signed-off-by: deepthi <[email protected]>

…neration Signed-off-by: deepthi <[email protected]>

… 45 mins Signed-off-by: deepthi <[email protected]>

…nly need this for vtorc, but I've made the change to the template so all tests get it. Signed-off-by: deepthi <[email protected]>

…neration Signed-off-by: deepthi <[email protected]>

deepthi · 2022-09-19T21:08:00Z

test/ci_workflow_gen.go

@@ -121,15 +120,14 @@ var (
 		"vreplication_cellalias",


Line 117: do we still need a separate mysql80 test?

test/ci_workflow_gen.go

.github/workflows/cluster_endtoend_vtorc_mysql57.yml

GuptaManan100

Everything else looks good to me!

GuptaManan100 · 2022-09-21T09:29:25Z

go/test/endtoend/vtorc/readtopologyinstance/main_test.go

@@ -93,7 +93,7 @@ func TestReadTopologyInstanceBufferable(t *testing.T) {
 	assert.Contains(t, primaryInstance.InstanceAlias, "zone1")
 	assert.NotEqual(t, 0, primaryInstance.ServerID)
 	assert.Greater(t, len(primaryInstance.ServerUUID), 10)
-	assert.Contains(t, primaryInstance.Version, "5.7")
+	assert.Contains(t, primaryInstance.Version, "8.0")


We can use assert.Regexp(t, [58].[07].*, replicaInstance.Version) instead. This will make this test work with both 5.7 and 8.0

Signed-off-by: deepthi <[email protected]>

…neration Signed-off-by: deepthi <[email protected]>

Signed-off-by: deepthi <[email protected]>

deepthi · 2022-09-27T02:38:56Z

I've addressed the feedback, and except for the known issue with vtbackup_transform, tests are passing.
When we merge this, we also need to remove a bunch of required checks and add new ones.
I'm inclined to wait until all of the WIP flag PRs are merged and then do one more merge from main into this PR before we merge it.

…80-default-ci-template-generation Signed-off-by: Rameez Sajwani <[email protected]>

Signed-off-by: Rameez Sajwani <[email protected]>

mattlord

❤️

vitess-bot · 2022-10-03T21:05:17Z

I was unable to backport this Pull Request to the following branches: release-14.0, release-13.0.

…1153) * Move towards MySQL 8.0 as the default template generation This upgrades the remaining things to Ubuntu 20.04 and makes MySQL 8.0 the default we run tests against. We still have tests for MySQL 5.7 but those are now explicitly opted into. This should finish up the Ubuntu 20.04 upgrade and also makes things easier for the future when we need to upgrade again. Signed-off-by: Dirkjan Bussink <[email protected]> * CI: rename shard vtorc_8.0 to vtorc_5.7, change expected test output for 8.0 Signed-off-by: deepthi <[email protected]> * CI: increase timeout for 8.0 tests on the actual test step from 30 to 45 mins Signed-off-by: deepthi <[email protected]> * CI: increase timeout to 45 minutes for mysql57 tests too. We really only need this for vtorc, but I've made the change to the template so all tests get it. Signed-off-by: deepthi <[email protected]> * CI: fix vtorc test to work with both 5.7 and 8.0 Signed-off-by: deepthi <[email protected]> * CI: missed docker flag in mysql57 template, one more fix to vtorc test Signed-off-by: deepthi <[email protected]> * removing spaces from pb file Signed-off-by: Rameez Sajwani <[email protected]> * removing spaces in pb file part 2 Signed-off-by: Rameez Sajwani <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: deepthi <[email protected]> Signed-off-by: Rameez Sajwani <[email protected]> Co-authored-by: deepthi <[email protected]> Co-authored-by: Rameez Sajwani <[email protected]> Signed-off-by: Rameez Sajwani <[email protected]>

…1153) * Move towards MySQL 8.0 as the default template generation This upgrades the remaining things to Ubuntu 20.04 and makes MySQL 8.0 the default we run tests against. We still have tests for MySQL 5.7 but those are now explicitly opted into. This should finish up the Ubuntu 20.04 upgrade and also makes things easier for the future when we need to upgrade again. Signed-off-by: Dirkjan Bussink <[email protected]> * CI: rename shard vtorc_8.0 to vtorc_5.7, change expected test output for 8.0 Signed-off-by: deepthi <[email protected]> * CI: increase timeout for 8.0 tests on the actual test step from 30 to 45 mins Signed-off-by: deepthi <[email protected]> * CI: increase timeout to 45 minutes for mysql57 tests too. We really only need this for vtorc, but I've made the change to the template so all tests get it. Signed-off-by: deepthi <[email protected]> * CI: fix vtorc test to work with both 5.7 and 8.0 Signed-off-by: deepthi <[email protected]> * CI: missed docker flag in mysql57 template, one more fix to vtorc test Signed-off-by: deepthi <[email protected]> * removing spaces from pb file Signed-off-by: Rameez Sajwani <[email protected]> * removing spaces in pb file part 2 Signed-off-by: Rameez Sajwani <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: deepthi <[email protected]> Signed-off-by: Rameez Sajwani <[email protected]> Co-authored-by: deepthi <[email protected]> Co-authored-by: Rameez Sajwani <[email protected]>

…11619) * Move towards MySQL 8.0 as the default template generation (#11153) * Move towards MySQL 8.0 as the default template generation This upgrades the remaining things to Ubuntu 20.04 and makes MySQL 8.0 the default we run tests against. We still have tests for MySQL 5.7 but those are now explicitly opted into. This should finish up the Ubuntu 20.04 upgrade and also makes things easier for the future when we need to upgrade again. Signed-off-by: Dirkjan Bussink <[email protected]> * CI: rename shard vtorc_8.0 to vtorc_5.7, change expected test output for 8.0 Signed-off-by: deepthi <[email protected]> * CI: increase timeout for 8.0 tests on the actual test step from 30 to 45 mins Signed-off-by: deepthi <[email protected]> * CI: increase timeout to 45 minutes for mysql57 tests too. We really only need this for vtorc, but I've made the change to the template so all tests get it. Signed-off-by: deepthi <[email protected]> * CI: fix vtorc test to work with both 5.7 and 8.0 Signed-off-by: deepthi <[email protected]> * CI: missed docker flag in mysql57 template, one more fix to vtorc test Signed-off-by: deepthi <[email protected]> * removing spaces from pb file Signed-off-by: Rameez Sajwani <[email protected]> * removing spaces in pb file part 2 Signed-off-by: Rameez Sajwani <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: deepthi <[email protected]> Signed-off-by: Rameez Sajwani <[email protected]> Co-authored-by: deepthi <[email protected]> Co-authored-by: Rameez Sajwani <[email protected]> * fixing template files to default to mysql80 Signed-off-by: Rameez Sajwani <[email protected]> * changing update statement to alter Signed-off-by: Rameez Sajwani <[email protected]> * Avoid deadlocks related to 0 receiver behavior (#10132) * Prevent deadlocks related to 0 receiver behavior Signed-off-by: Matt Lord <[email protected]> * Update test tables to use poller_idx Signed-off-by: Matt Lord <[email protected]> * Minor changes after mutex usage review in message manager + cache Signed-off-by: Matt Lord <[email protected]> * Use atomics for receiver count and messages pending Signed-off-by: Matt Lord <[email protected]> * Don't take exclusive lock when in fast path Signed-off-by: Matt Lord <[email protected]> * Update tests to use the new recommended message table structure See: vitessio/website#1015 Signed-off-by: Matt Lord <[email protected]> * Correct tests Signed-off-by: Matt Lord <[email protected]> * Update e2e test to use new recommended table structure Signed-off-by: Matt Lord <[email protected]> * Fix TestMessageStreamingPlan test Signed-off-by: Matt Lord <[email protected]> * Fix godriver/TestStreamMessaging test Signed-off-by: Matt Lord <[email protected]> * Split streamMu into streamProcessingMu and lastPollPositionMu Signed-off-by: Matt Lord <[email protected]> * Poller cannot take main lock w/o having X stream processing lock Signed-off-by: Matt Lord <[email protected]> * Improve the comments a bit Signed-off-by: Matt Lord <[email protected]> * Hold the main mutex during Add This is for safe concurrency with the last receiver unsubscribing Signed-off-by: Matt Lord <[email protected]> * Changes after pair reviewing with Sugu Signed-off-by: Matt Lord <[email protected]> * Use my GitHub handle for the self reference Signed-off-by: Matt Lord <[email protected]> Signed-off-by: Rameez Sajwani <[email protected]> * remove unwanted sleep Signed-off-by: Rameez Sajwani <[email protected]> * Fix failures Signed-off-by: Rameez Sajwani <[email protected]> * Fix backup tests Signed-off-by: Rameez Sajwani <[email protected]> * fix vtgate_gen4 error Signed-off-by: Rameez Sajwani <[email protected]> * upgrading to mysql 8.0 using vinalla mysql Signed-off-by: Rameez Sajwani <[email protected]> * pin to specific version 8.0.25 Signed-off-by: Rameez Sajwani <[email protected]> * Try to pin mysql version to 8.0.25 using tar file Signed-off-by: Rameez Sajwani <[email protected]> * Fix bug in last commit for 8.0.25 Signed-off-by: Rameez Sajwani <[email protected]> * fixing template files to use mysql8.0.25 Signed-off-by: Rameez Sajwani <[email protected]> * fixing community version Signed-off-by: Rameez Sajwani <[email protected]> * removing all mysql version before installation Signed-off-by: Rameez Sajwani <[email protected]> * moving cluster 11 to self host Signed-off-by: Rameez Sajwani <[email protected]> * trying different combination since mysql is not getting installed correctly Signed-off-by: Rameez Sajwani <[email protected]> * use tar instead of deb file Signed-off-by: Rameez Sajwani <[email protected]> * remove mysql stop statement Signed-off-by: Rameez Sajwani <[email protected]> * setting vt_mysql_root Signed-off-by: Rameez Sajwani <[email protected]> * moving export to right place Signed-off-by: Rameez Sajwani <[email protected]> * change template to accomodate tar file download Signed-off-by: Rameez Sajwani <[email protected]> * fixing mysql80 cluster Signed-off-by: Rameez Sajwani <[email protected]> * Adding mysql to the path env variable Signed-off-by: Rameez Sajwani <[email protected]> * fixing shardedpitr_tls test Signed-off-by: Rameez Sajwani <[email protected]> * fix upgrade-downgrade test Signed-off-by: Rameez Sajwani <[email protected]> * adjust epected AUTO_INCREMENT value Signed-off-by: Shlomi Noach <[email protected]> * removed 'expected_table_structure' files because there are different outputs in mysql57 and in mysql80 Signed-off-by: Shlomi Noach <[email protected]> * adding mysql version in workflow logs Signed-off-by: Rameez Sajwani <[email protected]> * move to utuntu 18 for upgrade-downgrade test Signed-off-by: Rameez Sajwani <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: deepthi <[email protected]> Signed-off-by: Rameez Sajwani <[email protected]> Signed-off-by: Matt Lord <[email protected]> Signed-off-by: Shlomi Noach <[email protected]> Co-authored-by: Dirkjan Bussink <[email protected]> Co-authored-by: deepthi <[email protected]> Co-authored-by: Matt Lord <[email protected]> Co-authored-by: Shlomi Noach <[email protected]>

* Move towards MySQL 8.0 as the default template generation (#11153) * Move towards MySQL 8.0 as the default template generation This upgrades the remaining things to Ubuntu 20.04 and makes MySQL 8.0 the default we run tests against. We still have tests for MySQL 5.7 but those are now explicitly opted into. This should finish up the Ubuntu 20.04 upgrade and also makes things easier for the future when we need to upgrade again. Signed-off-by: Dirkjan Bussink <[email protected]> * CI: rename shard vtorc_8.0 to vtorc_5.7, change expected test output for 8.0 Signed-off-by: deepthi <[email protected]> * CI: increase timeout for 8.0 tests on the actual test step from 30 to 45 mins Signed-off-by: deepthi <[email protected]> * CI: increase timeout to 45 minutes for mysql57 tests too. We really only need this for vtorc, but I've made the change to the template so all tests get it. Signed-off-by: deepthi <[email protected]> * CI: fix vtorc test to work with both 5.7 and 8.0 Signed-off-by: deepthi <[email protected]> * CI: missed docker flag in mysql57 template, one more fix to vtorc test Signed-off-by: deepthi <[email protected]> * removing spaces from pb file Signed-off-by: Rameez Sajwani <[email protected]> * removing spaces in pb file part 2 Signed-off-by: Rameez Sajwani <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: deepthi <[email protected]> Signed-off-by: Rameez Sajwani <[email protected]> Co-authored-by: deepthi <[email protected]> Co-authored-by: Rameez Sajwani <[email protected]> * changing update statement to alter Signed-off-by: Rameez Sajwani <[email protected]> * Adding mysql 8.0.25 Signed-off-by: Rameez Sajwani <[email protected]> * fix junit Signed-off-by: Rameez Sajwani <[email protected]> * fixing failures Signed-off-by: Rameez Sajwani <[email protected]> * fixing more unit tests Signed-off-by: Rameez Sajwani <[email protected]> * fix selfhost test Signed-off-by: Rameez Sajwani <[email protected]> * Fixing selfhosted workflows Signed-off-by: Rameez Sajwani <[email protected]> * fixing vreplication_v2 Signed-off-by: Rameez Sajwani <[email protected]> * removing cluster 12 and 18 from selfhosted Signed-off-by: Rameez Sajwani <[email protected]> * manual fixing some workflows Signed-off-by: Rameez Sajwani <[email protected]> * making vreplication_v2 exactly like upstream Signed-off-by: Rameez Sajwani <[email protected]> * Moving all the workflow to latest mysql version (vanilla) Signed-off-by: Rameez Sajwani <[email protected]> * remove port restrictions Signed-off-by: Rameez Sajwani <[email protected]> * tmp fix for vreplication_v2 Signed-off-by: Rameez Sajwani <[email protected]> * making vreplication_v2 to previous version Signed-off-by: Rameez Sajwani <[email protected]> * correcting typo Signed-off-by: Rameez Sajwani <[email protected]> * mvoing vreplication_v2 to selfhost Signed-off-by: Rameez Sajwani <[email protected]> * change docker image to mysql80 Signed-off-by: Rameez Sajwani <[email protected]> * removing limited source from vrepl Signed-off-by: Rameez Sajwani <[email protected]> * revert back last commit changes Signed-off-by: Rameez Sajwani <[email protected]> * test: fix aggregation test Signed-off-by: Harshit Gangal <[email protected]> * removing ptrtls Signed-off-by: Rameez Sajwani <[email protected]> * remove pitrTls tests Signed-off-by: Rameez Sajwani <[email protected]> * changing source/destination to mysql8.0 for vreplication Signed-off-by: Rameez Sajwani <[email protected]> * remove unwanted workflows Signed-off-by: Rameez Sajwani <[email protected]> * adding timeout to stress test Signed-off-by: Rameez Sajwani <[email protected]> * feeback through code review Signed-off-by: Rameez Sajwani <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: deepthi <[email protected]> Signed-off-by: Rameez Sajwani <[email protected]> Signed-off-by: Harshit Gangal <[email protected]> Co-authored-by: Dirkjan Bussink <[email protected]> Co-authored-by: deepthi <[email protected]> Co-authored-by: Harshit Gangal <[email protected]>

dbussink added Component: Build/CI Type: CI/Build Backport to: release-14.0 labels Sep 1, 2022

dbussink requested a review from frouioui September 1, 2022 08:50

dbussink mentioned this pull request Sep 1, 2022

Upgrade vtorc CI #11124

Closed

3 tasks

deepthi mentioned this pull request Sep 8, 2022

CI: Move all ubuntu-18.04 to ubuntu-20.04 #11026

Closed

3 tasks

deepthi added 6 commits September 12, 2022 16:04

Merge branch 'main' into dbussink/move-mysql80-default-ci-template-ge…

5d40d68

…neration Signed-off-by: deepthi <[email protected]>

CI: rename shard vtorc_8.0 to vtorc_5.7, change expected test output …

23cf9a2

…for 8.0 Signed-off-by: deepthi <[email protected]>

Merge branch 'main' into dbussink/move-mysql80-default-ci-template-ge…

b7300b6

…neration Signed-off-by: deepthi <[email protected]>

CI: increase timeout for 8.0 tests on the actual test step from 30 to…

49c4408

… 45 mins Signed-off-by: deepthi <[email protected]>

CI: increase timeout to 45 minutes for mysql57 tests too. We really o…

8a7eb2b

…nly need this for vtorc, but I've made the change to the template so all tests get it. Signed-off-by: deepthi <[email protected]>

Merge branch 'main' into dbussink/move-mysql80-default-ci-template-ge…

c041fb4

…neration Signed-off-by: deepthi <[email protected]>

deepthi requested review from rohit-nayak-ps and mattlord September 19, 2022 21:06

deepthi reviewed Sep 19, 2022

View reviewed changes

deepthi added this to the v15.0 milestone Sep 20, 2022

deepthi reviewed Sep 20, 2022

View reviewed changes

.github/workflows/cluster_endtoend_vtorc_mysql57.yml Outdated Show resolved Hide resolved

GuptaManan100 reviewed Sep 21, 2022

View reviewed changes

deepthi added 2 commits September 26, 2022 09:23

CI: fix vtorc test to work with both 5.7 and 8.0

76ce692

Signed-off-by: deepthi <[email protected]>

Merge branch 'main' into dbussink/move-mysql80-default-ci-template-ge…

85117e8

…neration Signed-off-by: deepthi <[email protected]>

deepthi marked this pull request as ready for review September 26, 2022 17:22

deepthi requested a review from shlomi-noach as a code owner September 26, 2022 17:22

CI: missed docker flag in mysql57 template, one more fix to vtorc test

529bfac

Signed-off-by: deepthi <[email protected]>

rsajwani mentioned this pull request Oct 3, 2022

Release of v15.0.0-RC1 #11425

Closed

17 tasks

rsajwani added 2 commits October 3, 2022 11:42

Merge remote-tracking branch 'upstream/main' into dbussink/move-mysql…

66e95a2

…80-default-ci-template-generation Signed-off-by: Rameez Sajwani <[email protected]>

removing spaces from pb file

6c09ae2

Signed-off-by: Rameez Sajwani <[email protected]>

removing spaces in pb file part 2

575abf3

Signed-off-by: Rameez Sajwani <[email protected]>

mattlord approved these changes Oct 3, 2022

View reviewed changes

deepthi merged commit e93e197 into vitessio:main Oct 3, 2022

deepthi deleted the dbussink/move-mysql80-default-ci-template-generation branch October 3, 2022 21:04

rsajwani mentioned this pull request Oct 4, 2022

Release of v15.0.0 #11426

Closed

100 tasks

frouioui added the Backport to: release-15.0 label Oct 13, 2022

frouioui mentioned this pull request Oct 14, 2022

Move CI workflow to use latest community version of mysql 8.0 #11493

Merged

3 tasks

This was referenced Nov 18, 2022

Move to MySQL 8.0 as the default template generation (#11153) #11769

Merged

Move towards MySQL 8.0 as the default template generation (#11153) #11619

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move towards MySQL 8.0 as the default template generation #11153

Move towards MySQL 8.0 as the default template generation #11153

dbussink commented Sep 1, 2022

vitess-bot bot commented Sep 1, 2022 •

edited by deepthi

Loading

deepthi Sep 19, 2022 •

edited

Loading

GuptaManan100 left a comment

GuptaManan100 Sep 21, 2022

deepthi commented Sep 27, 2022

mattlord left a comment

vitess-bot bot commented Oct 3, 2022

Move towards MySQL 8.0 as the default template generation #11153

Move towards MySQL 8.0 as the default template generation #11153

Conversation

dbussink commented Sep 1, 2022

Related Issue(s)

Checklist

vitess-bot bot commented Sep 1, 2022 • edited by deepthi Loading

Review Checklist

General

Bug fixes

Non-trivial changes

New/Existing features

Backward compatibility

deepthi Sep 19, 2022 • edited Loading

Choose a reason for hiding this comment

GuptaManan100 left a comment

Choose a reason for hiding this comment

GuptaManan100 Sep 21, 2022

Choose a reason for hiding this comment

deepthi commented Sep 27, 2022

mattlord left a comment

Choose a reason for hiding this comment

vitess-bot bot commented Oct 3, 2022

vitess-bot bot commented Sep 1, 2022 •

edited by deepthi

Loading

deepthi Sep 19, 2022 •

edited

Loading