ApplySchema: deprecate '--skip_preflight' flag #10716

shlomi-noach · 2022-07-17T16:28:53Z

Description

This PR deprecates the flag --skip_preflight in vtctlclient ApplySchema. The new behavior is as if --skip_preflight=true. That is, to always skip preflight. In fact, the preflight code is completely removed in this PR.

The Vitess team discussed deprecating --skip_preflight a long time ago, with the advent of Online DDL. Moreover, we have noticed that in production environments we always set the flag. And furthermore, the logic used by --skip_preflight is not on par with new schema change logic: Online DDL and schemadiff, and we've encountered scenarios where the logic was flawed.

As Vitess now recommends running migrations via Online DDL, we should move away from preflight checks.

Related Issue(s)

#6926

Checklist

"Backport me!" label has been added if this change should be backported
Tests were added or are not required
Documentation was added or is not required

Deployment Notes

Signed-off-by: Shlomi Noach <[email protected]>

vitess-bot · 2022-07-17T16:28:55Z

mattlord · 2022-07-17T17:46:20Z

I feel that deprecating the flag would be setting the default to true and adding a deprecation warning in the CLI and docs while removal would be removing the code done here along the the preflight tablet manager client code. The PR seems to be in the middle. Do we need to do a deprecation cycle? What's the value of keeping the flag if you can't set it?

Please keep in mind that I am glad we're getting rid of this and appreciate you working on that. 🙂

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach · 2022-07-20T14:06:15Z

Do we need to do a deprecation cycle? What's the value of keeping the flag if you can't set it?

@mattlord we have to have a deprecation cycle. So in v15 the user should get a warning (you're right! I did not add a warning log -- will fix), and in v16 we can remove the flag altogether.

You're also right that the easiest way forward would be to just take care of the flag, force it to true, then keep the rest of internal logic. However, we're doing some refactoring in tablet_executor.go to reduce footprint, and this would help in simplifying these efforts as it would remove some bulk of code.

UPDATE v16 is already released, so this work applies to v17 & v18.

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach · 2022-07-25T06:13:08Z

Added deprecation warning

Signed-off-by: Shlomi Noach <[email protected]>

ajm188 · 2022-07-25T10:24:59Z

go/cmd/vtctldclient/command/schema.go

@@ -141,7 +141,7 @@ func commandApplySchema(cmd *cobra.Command, args []string) error {
 		AllowLongUnavailability: applySchemaOptions.AllowLongUnavailability,
 		DdlStrategy:             applySchemaOptions.DDLStrategy,
 		Sql:                     parts,
-		SkipPreflight:           applySchemaOptions.SkipPreflight,
+		SkipPreflight:           true,


same comments I made in #10717 (comment), but for this flag

Comments addressed

Some feedback still needs to be addressed.

Signed-off-by: Shlomi Noach <[email protected]>

deepthi · 2022-07-27T20:45:29Z

go/vt/vtctl/vtctl.go

@@ -3110,7 +3112,7 @@ func commandApplySchema(ctx context.Context, wr *wrangler.Wrangler, subFlags *fl
 		AllowLongUnavailability: *allowLongUnavailability,
 		DdlStrategy:             *ddlStrategy,
 		Sql:                     parts,
-		SkipPreflight:           *skipPreflight,
+		SkipPreflight:           true,


Perhaps we should be deprecating the protobuf field for this as well? I'm not certain about that though - whether we do this now and deprecate the protobuf field in the next release.
@ajm188 what do you suggest?

i know i just said "yes" on the other PR, but i think we need to wait one more cycle, since old tablets will be consuming this field, which will take the zero value (false) instead of what we unconditionally pass here (true).

so deprecate flag + always pass true (v15) => remove flag + deprecate protobuf field + tablets ignore value and always skip-preflight (v16)

shlomi-noach · 2022-08-02T04:50:36Z

Cluster (vtgate_schema) seems to be consistently failing. Unsure yet how this PR affects it -- but it looks like it does.

Some feedback still to be addressed

github-actions · 2022-09-16T01:40:53Z

This PR is being marked as stale because it has been open for 30 days with no activity. To rectify, you may do any of the following:

Push additional commits to the associated branch.
Remove the stale label.
Add a comment indicating why it is not stale.

If no action is taken within 7 days, this PR will be closed.

shlomi-noach · 2022-09-18T05:14:16Z

This is not stale. I'm just stuck. Can't solve the CI issue.

shlomi-noach · 2022-10-13T06:51:49Z

Still stuck here.

github-actions · 2022-12-13T01:32:51Z

This PR is being marked as stale because it has been open for 30 days with no activity. To rectify, you may do any of the following:

Push additional commits to the associated branch.
Remove the stale label.
Add a comment indicating why it is not stale.

If no action is taken within 7 days, this PR will be closed.

github-actions · 2023-01-13T01:33:08Z

This PR is being marked as stale because it has been open for 30 days with no activity. To rectify, you may do any of the following:

Push additional commits to the associated branch.
Remove the stale label.
Add a comment indicating why it is not stale.

If no action is taken within 7 days, this PR will be closed.

github-actions · 2023-01-21T01:32:58Z

This PR was closed because it has been stale for 7 days with no activity.

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach · 2023-02-27T08:47:48Z

Fashionably late, this PR is ready for review!

dbussink

Yay code cleanup!

shlomi-noach · 2023-02-27T16:51:11Z

Current error in Upgrade Downgrade Testing Query Serving (Schema):

2023-02-27T15:36:44.2867788Z ERROR: Go version reported: go version go1.19.4 linux/amd64. Version 1.20.1+ required. See https://vitess.io/contributing/build-from-source for install instructions.
2023-02-27T15:36:44.2873075Z ##[error]Process completed with exit code 1.

shlomi-noach · 2023-02-28T07:09:45Z

Upgrade Downgrade Testing Query Serving (Schema) failure is a known issue to be fixed imminently; meanwhile merging this PR.

…#10716) (vitessio#1618) * ApplySchema: deprecate '--skip_preflight' flag (vitessio#10716) * ApplySchema: deprecate '-skip_preflight' flag Signed-off-by: Shlomi Noach <[email protected]> * warn if --skip_preflight is provided Signed-off-by: Shlomi Noach <[email protected]> * remove -skip_preflight from tests Signed-off-by: Shlomi Noach <[email protected]> * MarkDeprecated, fix usage Signed-off-by: Shlomi Noach <[email protected]> * remove skip_preflight test Signed-off-by: Shlomi Noach <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]> * remove SkipPreflight from tests Signed-off-by: Shlomi Noach <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]>

* Remove replication manager and run VTOrc in all e2e tests (#12149) * feat: remove replication manager Signed-off-by: Manan Gupta <[email protected]> * test: run VTOrc in all e2e tests (some exceptions) Signed-off-by: Manan Gupta <[email protected]> * docs: add summary notes changes Signed-off-by: Manan Gupta <[email protected]> * feat: deprecate disable-replication-manager flag Signed-off-by: Manan Gupta <[email protected]> * test: fix start vtorc function to be idempotent Signed-off-by: Manan Gupta <[email protected]> * docs: fix review comments Signed-off-by: Manan Gupta <[email protected]> * feat: fix port used by vtorc Signed-off-by: Manan Gupta <[email protected]> * test: really fix the ports issue Signed-off-by: Manan Gupta <[email protected]> * test: fix upgrade downgrade test issues with running vtorc Signed-off-by: Manan Gupta <[email protected]> * test: run ISP before starting VTOrc Signed-off-by: Manan Gupta <[email protected]> * feat: fix flakines in throttler test due to vtorc Signed-off-by: Manan Gupta <[email protected]> * test: fix flakiness in vreplication tests Signed-off-by: Manan Gupta <[email protected]> --------- Signed-off-by: Manan Gupta <[email protected]> * Allow manually kicking off CodeQL (#12200) * Move vtorc from go-sqlite3 to modernc.org/sqlite (#12214) * Move vtorc from go-sqlite3 to modernc.org/sqlite This moves vtorc from the go-sqlite3 library that uses CGO, to use modernc.org/sqlite which is a pure Go implementation. vtorc is the only component we have to build with CGO but it's causing pain for releases since we need to build it against an old Linux for linking against glibc. Using modernc.org/sqlite allows for using Go only again and makes all Vitess components buildable without CGO. In https://datastation.multiprocess.io/blog/2022-05-12-sqlite-in-go-with-and-without-cgo.html someone ran some basic benchmarks. It shows that the pure Go version can be twice as slow, but the usage of vtorc is very limited and we operate on small datasets, so I think the performance impact purely of a somewhat slower sqlite implementation is negligable. None of this is in a hot query serving path or anything like that, so I have little concern performance wise. Signed-off-by: Dirkjan Bussink <[email protected]> * Fix error handling in RowToArray Signed-off-by: Dirkjan Bussink <[email protected]> --------- Signed-off-by: Dirkjan Bussink <[email protected]> * Update the MySQL version used by our Docker images (#12054) * Change default MySQL version of lite and base to mysql80 instead of mysql57 Signed-off-by: Florent Poinsard <[email protected]> * Bump MySQL version to 8.0.31 in install_dependencies.sh Signed-off-by: Florent Poinsard <[email protected]> * addition of label to the mysql80 image Signed-off-by: Florent Poinsard <[email protected]> * apply review suggestions Signed-off-by: Florent Poinsard <[email protected]> * bring back mysql80 label Signed-off-by: Florent Poinsard <[email protected]> * add release summary Signed-off-by: Florent Poinsard <[email protected]> --------- Signed-off-by: Florent Poinsard <[email protected]> * Revert "Use ubuntu 20.04 for release builds (#12202)" (#12220) The underlying issue has now been addressed in: https://github.com/vitessio/vitess/pull/12214 This reverts commit 4b3b37d41ec08130d162b5115904108cbb3dc14b. Signed-off-by: Matt Lord <[email protected]> * Update release notes summary for the new default MySQL version (#12222) Signed-off-by: Florent Poinsard <[email protected]> * Revert changes made in #11960 (#12219) Signed-off-by: Florent Poinsard <[email protected]> * Bug fix: Cache filtered out tablets in topology watcher to avoid unnecessary GetTablet calls to topo (#12194) * check filter later in loadTablets. Add tests to confirm expected behavior Signed-off-by: Brian Ramos <[email protected]> * remove unnecessary assignment Co-authored-by: Florent Poinsard <[email protected]> Signed-off-by: Brian Ramos <[email protected]> * add some explanation around TestFilterByKeypsaceSkipsIgnoredTablets Signed-off-by: Brian Ramos <[email protected]> * simplify filter check logic as well as major test cleanup, using testify where possible Signed-off-by: Brian Ramos <[email protected]> --------- Signed-off-by: Brian Ramos <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> * Update Go dependencies (#12215) * Update Go dependencies This updates the Go dependencies to the latest versions before the Vitess 16 freeze so we can start with an up to date list. Also updates vitess-mixin since Dependabot is reporting a bunch of things on that. Signed-off-by: Dirkjan Bussink <[email protected]> * Fix deprecated sets.String This is now available using generics. Signed-off-by: Dirkjan Bussink <[email protected]> * run make proto Signed-off-by: Florent Poinsard <[email protected]> * fix e2e test expectations Signed-off-by: Florent Poinsard <[email protected]> --------- Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> * Properly test healthcheck cache response handling. (#12226) Signed-off-by: Matt Lord <[email protected]> * vtgate advertised mysql server version to 8.0.31 (#11989) * change vtgate advertised mysql version to 8.0.31 Signed-off-by: Harshit Gangal <[email protected]> * refactor keeping mysql advertised version Signed-off-by: Harshit Gangal <[email protected]> * refactor sqlparser mysql version to use the mysql_server_version default Signed-off-by: Harshit Gangal <[email protected]> * test: update test setup Signed-off-by: Harshit Gangal <[email protected]> * test: updated test help output Signed-off-by: Harshit Gangal <[email protected]> * added release notes Signed-off-by: Harshit Gangal <[email protected]> * feat: remove panic since it is getting triggered from the sidecardb Signed-off-by: Manan Gupta <[email protected]> --------- Signed-off-by: Harshit Gangal <[email protected]> Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Manan Gupta <[email protected]> Co-authored-by: Andres Taylor <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> Co-authored-by: Manan Gupta <[email protected]> * schemadiff: fix scenario where no tables exist in schema and with just views reading from DUAL (#12189) * [Gen4] Fix lookup vindexes with `autocommit` enabled (#12172) * test: show that `autocommit` on lookup vindexes is broken when used after a savepoint is created Signed-off-by: Arthur Schreiber <[email protected]> * test: rework the test case to show that the problem lies with transactions Signed-off-by: Arthur Schreiber <[email protected]> * Correctly handle `autocommit` in the `VIndexLookup` primitive. Signed-off-by: Arthur Schreiber <[email protected]> * Copy the `logging` struct so autocommit queries show up correctly in `vexplain` output. Signed-off-by: Arthur Schreiber <[email protected]> --------- Signed-off-by: Arthur Schreiber <[email protected]> * Add TOC to the summary docs (#12225) * docs: add toc to the summary docs Signed-off-by: Manan Gupta <[email protected]> * feat: add link to newly added summary change Signed-off-by: Manan Gupta <[email protected]> --------- Signed-off-by: Manan Gupta <[email protected]> * feat: bump reported version of main (#12234) Signed-off-by: Manan Gupta <[email protected]> * Use consistent type for vreplication id (#12218) * Use consistent type for vreplication id The `_vt.vreplication` table has an `INT` type without `UNSIGNED`, so it maps to the Go `int32` type to have the exact same consistent range. This means that we should use `int32` consistently for everywhere that we use the VReplication identifier. We had int32, uint32 and int used pretty randomly for these which leads to CodeQL flagging these and how this was found. Instead this now converts everything to int32. It's also safe to change the protobuf, since the protobuf wire format for int32, uint32, int64 & uint64 is all compatible and actually uses the same variable int length format. The only exception would be if someone has a VReplication identifier that would not fit in `int32`, but well, this PR shows that in that case we already blow up in unexpected and unknown ways anyway. So it's better to have that explicitly error. Signed-off-by: Dirkjan Bussink <[email protected]> * Add support for all integer types for bind variables This ensures we cover all the Go int types to generate bind parameters from. Signed-off-by: Dirkjan Bussink <[email protected]> * Regenerate protos for vtadmin Signed-off-by: Dirkjan Bussink <[email protected]> * Fix test for using ParseInt Signed-off-by: Dirkjan Bussink <[email protected]> --------- Signed-off-by: Dirkjan Bussink <[email protected]> * Fix typecasting issue for workflow types (#12217) * Fix typecasting issue for workflow types This addresses a number of the issues identified by CodeQL around typecasting. We use 32 bit integers for these, so let's make sure we parse and cast then as those everywhere it matters. Signed-off-by: Dirkjan Bussink <[email protected]> * Ensure to write to int into query Signed-off-by: Dirkjan Bussink <[email protected]> --------- Signed-off-by: Dirkjan Bussink <[email protected]> * Use the correct uint32 type for the GTID file position (#12229) This is defined as uint32 at the protocol level, so better to use that type consistently and also parse it accordingly with `ParseUint` with the right size. Signed-off-by: Dirkjan Bussink <[email protected]> * Fix CodeQL identified issues (#12199) This addresses two classes of issues that CodeQL found. One is a not as strict as needed regexp for a domain match, where there's no literal `.` being matched. The other is a user controlled XSS since we run the template with arbitrary user input. Instead we validate if the throttler with that name actually exists before trying to execute the template. On error we also don't reflect the input to avoid the same XSS. Signed-off-by: Dirkjan Bussink <[email protected]> * Copy remaining query serving design docs (#12170) * Copy remaining query serving design docs Signed-off-by: Florent Poinsard <[email protected]> * Remove duplicate design docs Signed-off-by: Florent Poinsard <[email protected]> --------- Signed-off-by: Florent Poinsard <[email protected]> * Use a working MariaDB mirror for CI (#12253) Signed-off-by: Dirkjan Bussink <[email protected]> * Update the release instructions after v16 code freeze (#12245) Signed-off-by: Florent Poinsard <[email protected]> * onlineddl_vrepl suite: fix auto_increment flakyness (#12246) * onlineddl_vrepl suite: fix auto_increment flakyness Signed-off-by: Shlomi Noach <[email protected]> * delete all rows before events begin Signed-off-by: Shlomi Noach <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]> * Revert default MySQL 80 version to `8.0.30` (#12252) * Revert default MySQL 80 version Signed-off-by: Florent Poinsard <[email protected]> * upgrade mysql80 default to 8.0.30 Signed-off-by: Florent Poinsard <[email protected]> * Add release notes regarding MySQL upgrade path in the vitess-operator Signed-off-by: Florent Poinsard <[email protected]> * Remove unwanted upgrade path step Signed-off-by: Florent Poinsard <[email protected]> --------- Signed-off-by: Florent Poinsard <[email protected]> * Fix additional integer type handling (#12237) * Fix additional integer type handling The tablet uid can be a uint32 everywhere and we can parse it like that with pflag as well. This way we don't have to cast and change the valid range and get correct messages when it's out of range. As well we use a regular int for the port which better matches how Go itself tracks ports and removes a bunch of casts from places where we already using int. Lastly it also fixes some small cases of how we parse to match the type we store it in. Signed-off-by: Dirkjan Bussink <[email protected]> * Fix parsing of server id for Zookeeper Signed-off-by: Dirkjan Bussink <[email protected]> * Fix help output Signed-off-by: Dirkjan Bussink <[email protected]> --------- Signed-off-by: Dirkjan Bussink <[email protected]> * grpcvtgateconn: add Dial and deprecate DialWithOpts (#12210) Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Co-authored-by: Dirkjan Bussink <[email protected]> * Use consistent type port and replication state (#12248) * Use consistent type port and replication state These are used as int32 in the protobuf (and enum maps to int32 for replication state). This means we should also keep using int32 internally if easily possible. Signed-off-by: Dirkjan Bussink <[email protected]> * Fix assertions in test Signed-off-by: Dirkjan Bussink <[email protected]> --------- Signed-off-by: Dirkjan Bussink <[email protected]> * evalengine: new evaluation framework (#12247) * evalengine: new evaluation framework Signed-off-by: Vicent Marti <[email protected]> * evalengine: add cached_size.go Signed-off-by: Vicent Marti <[email protected]> * evalengine: simplify casts in NullSafeHashCodeInPlace Signed-off-by: Vicent Marti <[email protected]> * evalengine: fix static warnings Signed-off-by: Vicent Marti <[email protected]> * evalengine: remove duplicate branch Signed-off-by: Vicent Marti <[email protected]> * planbuilder: update for tuple bind variables Signed-off-by: Vicent Marti <[email protected]> * evalengine: fix error in tuple hashing Signed-off-by: Vicent Marti <[email protected]> * evalengine: fix LIKE simplification Signed-off-by: Vicent Marti <[email protected]> * evalengine: cleanup unused tests and panics Signed-off-by: Vicent Marti <[email protected]> --------- Signed-off-by: Vicent Marti <[email protected]> * Examples: Add lib functions to wait for shard states (#12239) * Add lib functions to wait for shard states Signed-off-by: Matt Lord <[email protected]> * Fixup region example too Signed-off-by: Matt Lord <[email protected]> * Address minor review comments Signed-off-by: Matt Lord <[email protected]> * Remove extraneous line change Signed-off-by: Matt Lord <[email protected]> * Refactor to remove duplicated files/code Also address shellcheck warnings. Signed-off-by: Matt Lord <[email protected]> * Fix tests that run in CI Signed-off-by: Matt Lord <[email protected]> * Use more robust location detection Signed-off-by: Matt Lord <[email protected]> * Fix backup tests Signed-off-by: Matt Lord <[email protected]> * Exit on shard healthy failure Signed-off-by: Matt Lord <[email protected]> * Remove unused dups and adjust client_test and Dockerfiles Signed-off-by: Matt Lord <[email protected]> --------- Signed-off-by: Matt Lord <[email protected]> * Download mariadb from vitess-resources (#12271) * Download mariadb from vitess-resources Signed-off-by: Florent Poinsard <[email protected]> * Use proper mariadb version increment in tests Signed-off-by: Florent Poinsard <[email protected]> * Use 10.10.3 instead of LTS Signed-off-by: Florent Poinsard <[email protected]> --------- Signed-off-by: Florent Poinsard <[email protected]> * Workflow Show: display rows copied (#12231) * Display rows copied in Workflow Show Signed-off-by: Rohit Nayak <[email protected]> * Fix vtctl tests Signed-off-by: Rohit Nayak <[email protected]> --------- Signed-off-by: Rohit Nayak <[email protected]> * sqlparser: Add parsing support to spatial POINT expression (#12198) * feat: add parsing support to spatial POINT expression Signed-off-by: Ayman161803 <[email protected]> * fix: address formatting issues Signed-off-by: Ayman161803 <[email protected]> * fix: address formatting issues Signed-off-by: Ayman <[email protected]> * feat: add canonical test cases for spatial point expression Signed-off-by: Ayman161803 <[email protected]> --------- Signed-off-by: Ayman161803 <[email protected]> Signed-off-by: Ayman <[email protected]> * fix dual table handling (#12204) * fix planner: be careful when merging subqueries We were not checking dependencies of the outer subquery correctly, before merging with a route. This lead to invalid plans being produced. Signed-off-by: Andres Taylor <[email protected]> * fix planner: use tables on lhs when merging subqs When calculating if everything we need is available before merging routes, we now take into account anything coming from the LHS of any joins we are under. Signed-off-by: Andres Taylor <[email protected]> * test: add test merging subquery with join Signed-off-by: Andres Taylor <[email protected]> * use LHS from the top of the current node when planning RHSs Signed-off-by: Florent Poinsard <[email protected]> * added comments Signed-off-by: Andres Taylor <[email protected]> --------- Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> * Subquery bugfix (#12254) * fix planner: be careful when merging subqueries We were not checking dependencies of the outer subquery correctly, before merging with a route. This lead to invalid plans being produced. Signed-off-by: Andres Taylor <[email protected]> * fix planner: use tables on lhs when merging subqs When calculating if everything we need is available before merging routes, we now take into account anything coming from the LHS of any joins we are under. Signed-off-by: Andres Taylor <[email protected]> * test: add test merging subquery with join Signed-off-by: Andres Taylor <[email protected]> * use LHS from the top of the current node when planning RHSs Signed-off-by: Florent Poinsard <[email protected]> * added comments Signed-off-by: Andres Taylor <[email protected]> --------- Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> * feat: fix release script (#12284) Signed-off-by: Manan Gupta <[email protected]> * feat: copy release notes from release-16 (#12287) Signed-off-by: Manan Gupta <[email protected]> * Remove parallelization from tests that mutate shared state (#12257) Signed-off-by: Andrew Mason <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> * fix dual table handling (#12292) * fix dual table handling Signed-off-by: Andres Taylor <[email protected]> * update test expectations Signed-off-by: Andres Taylor <[email protected]> --------- Signed-off-by: Andres Taylor <[email protected]> * fix: transaction_isolation to be applied at session level (#12281) Signed-off-by: Harshit Gangal <[email protected]> * fix linter warnings (#12286) * fix linter warnings Signed-off-by: Andres Taylor <[email protected]> * add the same checks in CI as pre-commit Signed-off-by: Andres Taylor <[email protected]> * ci: disable revive by default Signed-off-by: Vicent Marti <[email protected]> --------- Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Vicent Marti <[email protected]> * go/vt/mysqlctl: add configurable read buffer to builtin backups (#12073) * go/vt/mysqlctl: add configurable read buffer to builtin backups Signed-off-by: Max Englander <[email protected]> * pr review: use uint Signed-off-by: Max Englander <[email protected]> * doc/releasenotes: v16 --builtinbackup-file-read-buffer-size and --builtinbackup-file-write-buffer-size Signed-off-by: Max Englander <[email protected]> * update flags txt Signed-off-by: Max Englander <[email protected]> * make part of v17 Signed-off-by: Max Englander <[email protected]> * Update doc/releasenotes/17_0_0_summary.md Co-authored-by: Deepthi Sigireddi <[email protected]> Signed-off-by: Max Englander <[email protected]> * improve wording -block Signed-off-by: Max Englander <[email protected]> --------- Signed-off-by: Max Englander <[email protected]> Signed-off-by: Max Englander <[email protected]> Co-authored-by: Deepthi Sigireddi <[email protected]> * VReplication Last Error: retry error if it happens after timeout (#12114) * Retry error if it happens after timeout Signed-off-by: Rohit Nayak <[email protected]> * Address review comments Signed-off-by: Rohit Nayak <[email protected]> * Address review comments Signed-off-by: Rohit Nayak <[email protected]> * Improve unit tests Signed-off-by: Rohit Nayak <[email protected]> --------- Signed-off-by: Rohit Nayak <[email protected]> * fix: added null safe operator precendence rule (#12297) Signed-off-by: Harshit Gangal <[email protected]> * Fix aggregation on outer joins (#12298) * fix: aggregation coming from right hand side of left join check for null Signed-off-by: Harshit Gangal <[email protected]> * test: added a sum test Signed-off-by: Harshit Gangal <[email protected]> * addressed review comments Signed-off-by: Harshit Gangal <[email protected]> --------- Signed-off-by: Harshit Gangal <[email protected]> * Change SQL error codes to use explicit type (#12249) This ErrorCode type maps to a uint16 to match what MySQL has as the range and parses things accordingly as well. With the explicit type we know better when we use the specific error code. Signed-off-by: Dirkjan Bussink <[email protected]> * Fix typecasting alerts inside vtgate engine (#12238) CodeQL reported a few cases in the vtgate engine where we can simplify and make the typecasting more correct. We were double casting for the eval as well which isn't useful, so reduce this to it doing one for the hash code. Signed-off-by: Dirkjan Bussink <[email protected]> * Move to independent sets package (#12251) This moves to a separate sets package inside Vitess so we don't break version pinning for k8s like in the Vitess operator. It mostly copies the existing implementation, but simplifies and cleans it up to only copy what we actually use. Signed-off-by: Dirkjan Bussink <[email protected]> * SchemaDiff: normalize boolean columns to tinyint(1), and, add TableQualifier hint to force adding a db qualifier to the alter diff. (#12133) * Normalize boolean columns to tinyint(1) Signed-off-by: Rohit Nayak <[email protected]> * Properly handle invalid defaults provided for boolean columns Signed-off-by: Rohit Nayak <[email protected]> * Use TableQualifier hint to decide whether to use the db qualifier for the tables in create/alter scripts Signed-off-by: Rohit Nayak <[email protected]> * Keep current behavior as default and require TableQualifierDeclared to force use of the 'to' desired schema's qualifier Signed-off-by: Rohit Nayak <[email protected]> * Fix incorrect renaming of enum in comments Signed-off-by: Rohit Nayak <[email protected]> * Additional tests for boolean/tinyint(1) equivalence Signed-off-by: Rohit Nayak <[email protected]> * Add tests for create/drop with table qualifiers Signed-off-by: Rohit Nayak <[email protected]> --------- Signed-off-by: Rohit Nayak <[email protected]> * Allow zero (in) date when setting up internal _vt schema (#12262) * Allow zero (in) date when setting up internal _vt schema Signed-off-by: Shlomi Noach <[email protected]> * modify test sto include @@sql_mod query support Signed-off-by: Shlomi Noach <[email protected]> * Allow zero (in) date when setting up internal _vt schema Signed-off-by: Shlomi Noach <[email protected]> * modify test sto include @@sql_mod query support Signed-off-by: Shlomi Noach <[email protected]> * Fix test failures Signed-off-by: Rohit Nayak <[email protected]> * Fix failing tests. Minor refactor Signed-off-by: Rohit Nayak <[email protected]> * change sql_mode t omost permissive (empty) Signed-off-by: Shlomi Noach <[email protected]> * setPermissiveSQLMode Signed-off-by: Shlomi Noach <[email protected]> * fixes per review Signed-off-by: Shlomi Noach <[email protected]> * Add missing mock query Signed-off-by: Rohit Nayak <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]> Signed-off-by: Rohit Nayak <[email protected]> Co-authored-by: Rohit Nayak <[email protected]> * maintainers: add dbussink and rework some sections (#12329) Signed-off-by: Vicent Marti <[email protected]> * TableFilter: fix excluded RE listing (#12318) Signed-off-by: Shlomi Noach <[email protected]> * Internal refactor: LastError as a public struct (#12321) * Refactor LastError as a public struct Signed-off-by: Shlomi Noach <[email protected]> * wording Signed-off-by: Shlomi Noach <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]> * evalengine: it's time for JSON! (#12274) * collations: add max width to charsets Signed-off-by: Vicent Marti <[email protected]> * evalengine: add support for JSON built-ins Signed-off-by: Vicent Marti <[email protected]> * evalengine: support JSON_ARRAY and JSON_OBJECT Signed-off-by: Vicent Marti <[email protected]> * evalengine: JSON object sorting like MySQL Signed-off-by: Vicent Marti <[email protected]> * evalengine: add support for CAST(val AS JSON) Signed-off-by: Vicent Marti <[email protected]> * evalengine: support TO_BASE64 and FROM_BASE64 Signed-off-by: Vicent Marti <[email protected]> * evalengine: support JSON_DEPTH Signed-off-by: Vicent Marti <[email protected]> * evalengine: support JSON_LENGTH Signed-off-by: Vicent Marti <[email protected]> * evalengine: fix codegen Signed-off-by: Vicent Marti <[email protected]> * evalengine: do not wrap `json.Value` Signed-off-by: Vicent Marti <[email protected]> * sizegen: fix external type generation Signed-off-by: Vicent Marti <[email protected]> * evalengine: support JSON_CONTAINS_PATH Signed-off-by: Vicent Marti <[email protected]> * evalengine: support JSON_KEYS Signed-off-by: Vicent Marti <[email protected]> * evalengine: fix linter warnings Signed-off-by: Vicent Marti <[email protected]> * evalengine: fix evalToJSON semantics Signed-off-by: Vicent Marti <[email protected]> * evalengine: fix error wrapping Signed-off-by: Vicent Marti <[email protected]> * evalengine: update sizegen Signed-off-by: Vicent Marti <[email protected]> * evalengine: @systay review feedback Signed-off-by: Vicent Marti <[email protected]> * evalengine: oops! fix bad else/if Signed-off-by: Vicent Marti <[email protected]> --------- Signed-off-by: Vicent Marti <[email protected]> * evalengine: fix bugs with decimal rounding (#12360) Signed-off-by: Vicent Marti <[email protected]> * VReplication: ignore GC tables in schema analysis (#12320) * TableFilter: fix excluded RE listing Signed-off-by: Shlomi Noach <[email protected]> * extract GCTableNameExpression Signed-off-by: Shlomi Noach <[email protected]> * VReplication: exclude GC tables when reading schema Signed-off-by: Shlomi Noach <[email protected]> * wrap schema analyzis errors for better tracability Signed-off-by: Shlomi Noach <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]> * Add parsing support for linestring constructor (#12299) * feat: add parsing support to linestring constructor Signed-off-by: Ayman161803 <[email protected]> * test: add canonical tests Signed-off-by: Ayman161803 <[email protected]> * fix: fix formatting issues Signed-off-by: Ayman161803 <[email protected]> * chore: increment shift/reduce conflicts to 4 Signed-off-by: Ayman161803 <[email protected]> * fix: fix shift/reduce conflict by adding precendence Signed-off-by: Ayman161803 <[email protected]> * fix: update yacc grammar to replace []pointExpr with Exprs Signed-off-by: Ayman161803 <[email protected]> * fix: update format function for linestringExpr and select_test_cases Signed-off-by: Ayman161803 <[email protected]> --------- Signed-off-by: Ayman161803 <[email protected]> * testutils: use a rows parser for test comparisons (#12138) * tesutils: use a rows parser for test comparisons Signed-off-by: Vicent Marti <[email protected]> * utils: add license Signed-off-by: Vicent Marti <[email protected]> * moved parse_rows to sqltypes to avoid import cycle Signed-off-by: Andres Taylor <[email protected]> * concatenate: fix test Signed-off-by: Vicent Marti <[email protected]> * sqltypes: fix comparison error with duplicate rows Signed-off-by: Vicent Marti <[email protected]> * vexplain: fix undeterministic test comparisons Signed-off-by: Vicent Marti <[email protected]> --------- Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Andres Taylor <[email protected]> Co-authored-by: Andres Taylor <[email protected]> * vtbackup, mysqlctl: detailed backup and restore metrics (#11979) * vtbackup, mysqlctl: detailed backup and restore metrics While backup stats are exposed via `servenv` (which is great!) ideally we would like more fine-grained stats on: * Download last backup. * Decompress last backup. * Catchup to primary with replication. * Compress new backup. * Upload new backup. Signed-off-by: Max Englander <[email protected]> * move fakemysqldaemon.go to mysqlctl package Signed-off-by: Max Englander <[email protected]> * go/vt/mysqlctl: add backup/restore stats tests Signed-off-by: Max Englander <[email protected]> * address pr comments round 1 Signed-off-by: Max Englander <[email protected]> * checkpoint Signed-off-by: Max Englander <[email protected]> * reorganize metric names Signed-off-by: Max Englander <[email protected]> * track bytes processed in backup/restore Signed-off-by: Max Englander <[email protected]> * address pr comments: remove 'this file' from comments Signed-off-by: Max Englander <[email protected]> * Update go/vt/mysqlctl/backupstats/stats.go Co-authored-by: Deepthi Sigireddi <[email protected]> Signed-off-by: Max Englander <[email protected]> * Update go/vt/mysqlctl/backupstats/stats.go Co-authored-by: Deepthi Sigireddi <[email protected]> Signed-off-by: Max Englander <[email protected]> * Update go/vt/mysqlctl/backupstats/stats.go Co-authored-by: Deepthi Sigireddi <[email protected]> Signed-off-by: Max Englander <[email protected]> * pr review: improve comments Signed-off-by: Max Englander <[email protected]> * make stats more expvars-friendly Signed-off-by: Max Englander <[email protected]> * -backupstorage.Parameterizer, +backupstorage.BackupStorage.WithParams Signed-off-by: Max Englander <[email protected]> * remove deleted field Signed-off-by: Max Englander <[email protected]> * Update go/ioutil/meter.go Signed-off-by: Max Englander <[email protected]> * Update go/ioutil/meter.go Signed-off-by: Max Englander <[email protected]> * Update go/ioutil/meter.go Signed-off-by: Max Englander <[email protected]> * -constants.go +release notes Signed-off-by: Max Englander <[email protected]> * add stats example to release notes Signed-off-by: Max Englander <[email protected]> --------- Signed-off-by: Max Englander <[email protected]> Signed-off-by: Max Englander <[email protected]> Co-authored-by: Deepthi Sigireddi <[email protected]> * VReplication: workflows with multiple streams fail to start on multi-primary setups (#12228) * Handle auto_increment_increment step > 1 while creating multiple vreplication streams Signed-off-by: Rohit Nayak <[email protected]> * Enhance e2e test for validating auto_increment step logic Signed-off-by: Rohit Nayak <[email protected]> * Add new query to mock db Signed-off-by: Rohit Nayak <[email protected]> * Add missing mock for TestEngineExec Signed-off-by: Rohit Nayak <[email protected]> * Cleanup Signed-off-by: Rohit Nayak <[email protected]> * Fix goimports Signed-off-by: Rohit Nayak <[email protected]> * Address review comments Signed-off-by: Rohit Nayak <[email protected]> * Use uint16 for auto increment step Signed-off-by: Rohit Nayak <[email protected]> * Update comments Signed-off-by: Rohit Nayak <[email protected]> --------- Signed-off-by: Rohit Nayak <[email protected]> * sqlparser/schemadiff: normalize CAST type case (#12372) Signed-off-by: Shlomi Noach <[email protected]> * add database name to _vt.views table (#12374) Signed-off-by: Harshit Gangal <[email protected]> Co-authored-by: Harshit Gangal <[email protected]> * [vtgate planner] Routing & Merging refactor (#12197) * start of extracting routing logic out from Route into an interface Signed-off-by: Andres Taylor <[email protected]> * wip - milestone - first plan-test green Signed-off-by: Andres Taylor <[email protected]> * wip - single table routes work fine Signed-off-by: Andres Taylor <[email protected]> * wip - sequence tables passing Signed-off-by: Andres Taylor <[email protected]> * wip - all tests in from_cases.json passing Signed-off-by: Andres Taylor <[email protected]> * wip - more work on merge logic Signed-off-by: Andres Taylor <[email protected]> * wip - need to switch the routes as well Signed-off-by: Andres Taylor <[email protected]> * subquery merging Signed-off-by: Harshit Gangal <[email protected]> * dual routing Signed-off-by: Harshit Gangal <[email protected]> * wip - make sure to keep inner/outer in the right place Signed-off-by: Andres Taylor <[email protected]> * make sure to not forget already seen predicates when merging sharded routes Signed-off-by: Andres Taylor <[email protected]> * add merged subqueries to the merged field Signed-off-by: Andres Taylor <[email protected]> * dual subquery and none routing change Signed-off-by: Harshit Gangal <[email protected]> * handle merging subqueries on IN comparisons Signed-off-by: Andres Taylor <[email protected]> * recalculate routing after merging subquery Signed-off-by: Andres Taylor <[email protected]> * rename field to make it easier to grokk Signed-off-by: Andres Taylor <[email protected]> * better merging logic when merging non-sharded tables Signed-off-by: Andres Taylor <[email protected]> * when merging subqueries, we must also copy predicates Signed-off-by: Andres Taylor <[email protected]> * copy keyspace when producing NoneRouting Signed-off-by: Andres Taylor <[email protected]> * implement Routing methods in TargetedRouting and update OpCode outside of shardedrouting Signed-off-by: Florent Poinsard <[email protected]> * Fix targeted routing update params and rest of the dml cases Signed-off-by: Florent Poinsard <[email protected]> * Enhanced UpdateRoutingLogic function to detect constant null and None routes Signed-off-by: Florent Poinsard <[email protected]> * clean up info schema route mergeing Signed-off-by: Andres Taylor <[email protected]> * handle reference tables with alternates Signed-off-by: Andres Taylor <[email protected]> * update remaining plan_tests Signed-off-by: Andres Taylor <[email protected]> * unify unsharded and reference routing in the same logic Signed-off-by: Andres Taylor <[email protected]> * clean up merging.go Signed-off-by: Andres Taylor <[email protected]> * build the alternate routes with the correct keyspace Signed-off-by: Andres Taylor <[email protected]> * minor fixes after self-review Signed-off-by: Andres Taylor <[email protected]> * when missing current keyspace, any valid keyspace can be used Signed-off-by: Andres Taylor <[email protected]> * final small changes. I promise Signed-off-by: Andres Taylor <[email protected]> * review feedback Signed-off-by: Andres Taylor <[email protected]> * tidy up method after review feedback Signed-off-by: Andres Taylor <[email protected]> * more cleanup - fix goland warnings in new files Signed-off-by: Andres Taylor <[email protected]> * add keyspace information to sequence routing Signed-off-by: Andres Taylor <[email protected]> * more cleanups from PR review Signed-off-by: Andres Taylor <[email protected]> * more cleanup Signed-off-by: Andres Taylor <[email protected]> --------- Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Harshit Gangal <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Co-authored-by: Harshit Gangal <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> * This PR bumps the vitess resrouces release increment from v2 to v4 (#12272) Signed-off-by: Florent Poinsard <[email protected]> * Switch to using new Go 1.19 CRL parser (#12315) `x509.ParseCRL` is deprecated, we should use `x509.ParseRevocationList` instead which is new in Go 1.19. Signed-off-by: Dirkjan Bussink <[email protected]> * Online DDL: remove legacy "stowaway table" logic (#12288) * Online DDL: remove legacy stowaway table logic Signed-off-by: Shlomi Noach <[email protected]> * removed column from table definition Signed-off-by: Shlomi Noach <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]> * Fix for USING when column names not lower cased (#12378) Signed-off-by: Andres Taylor <[email protected]> * Fix bug in vtexplain around JOINs (#12376) Signed-off-by: Andres Taylor <[email protected]> * fix help text for backup_storage_number_blocks (#12258) Signed-off-by: Rameez Sajwani <[email protected]> * Fix additional typecasting alerts (#12309) * Fix additional typecasting alerts This should solve the remaining valid typecasting alerts from CodeQL. Signed-off-by: Dirkjan Bussink <[email protected]> * Fix additional typecasting issues Ran through more of the ParseInt / ParseUint / Atoi calls manually that CodeQL didn't identify and found a bunch more inconsistencies that should be fixed. Signed-off-by: Dirkjan Bussink <[email protected]> --------- Signed-off-by: Dirkjan Bussink <[email protected]> * Fix additional casting bugs in the sqlparser This fixes a bunch of additional canonical casing bugs in the parser. See https://github.com/vitessio/vitess/issues/12392 for the list of issues it addresses. Signed-off-by: Dirkjan Bussink <[email protected]> * Fixes for local docker environment (#12382) Signed-off-by: Andy Lim <[email protected]> * [main] Schema RPC to fetch table/view definition (#12375) (#12397) * feat: proto changes to add new rpc Signed-off-by: Harshit Gangal <[email protected]> * feat: vttablet changes in queryservice to implement the new rpc Signed-off-by: Harshit Gangal <[email protected]> * added test on tabletserver Signed-off-by: Harshit Gangal <[email protected]> * feat: use the new schema rpc in vtgate to get updated view definition Signed-off-by: Harshit Gangal <[email protected]> * remove error from getschemaresponse and change the enum name Signed-off-by: Harshit Gangal <[email protected]> * updated proto files for vtadmin Signed-off-by: Harshit Gangal <[email protected]> * fix e2e test Signed-off-by: Harshit Gangal <[email protected]> --------- Signed-off-by: Harshit Gangal <[email protected]> Co-authored-by: Harshit Gangal <[email protected]> * Bump golang.org/x/net from 0.5.0 to 0.7.0 (#12390) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.5.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](https://github.com/golang/net/compare/v0.5.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Move to Go 1.19 atomics (#12391) * Move to Go 1.19 atomics This change moves to Go 1.19 atomics in favor of the version we had in the `sync2` package. The version in that package predates the availability of the new atomics, but for the future we want to use the Go provided ones. On top of that, we can also use the semaphore from `x/sync` instead of a custom one in `sync2`. This cleans up a bunch of additional code from `sync2`. Signed-off-by: Dirkjan Bussink <[email protected]> * Don't assume the value is a duration This can be any float, so we need to store the float bits in an atomic uint64 instead (as Go has no atomic floats). Signed-off-by: Dirkjan Bussink <[email protected]> * Log actual error if acquiring semaphore fails Signed-off-by: Dirkjan Bussink <[email protected]> * Fix overlooked metrics threshold case Signed-off-by: Dirkjan Bussink <[email protected]> --------- Signed-off-by: Dirkjan Bussink <[email protected]> * Update additional Go dependencies (#12401) Dependabot does pick up these go.mod files but somehow fails to open PRs for them to update things that are vulnerable. So doing it manually here. Signed-off-by: Dirkjan Bussink <[email protected]> * Fix another mixin dep (#12403) Dependabot is still warning about this one and missed this in #12401 Signed-off-by: Dirkjan Bussink <[email protected]> * Remove a bunch of debug logging (#12404) This removes a bunch of debug logging that is set for higher log levels. The problem is that many of these are logging user provided values directly which is something CodeQL also alerts on. If someone needs this kind of additional logging level, I think it suffices to modify the code locally to test something rather than always carrying these high log level entries. Signed-off-by: Dirkjan Bussink <[email protected]> * Stop launchable integration for unit tests (#12386) * feat: stop launchable integration for unit tests Signed-off-by: Manan Gupta <[email protected]> * feat: fix identation for the unit tests Signed-off-by: Manan Gupta <[email protected]> * feat: also add name to the unit tests Signed-off-by: Manan Gupta <[email protected]> --------- Signed-off-by: Manan Gupta <[email protected]> * Makefile: use buildx when GOOS/GOARCH are specified (#12081) Signed-off-by: Max Englander <[email protected]> * Auto refresh tables on vtgate Status Page (#12335) * Component: vtgate, Auto Update tables in Status For Vtgate page Signed-off-by: jun <[email protected]> * rename function name Signed-off-by: jun <[email protected]> * move tableRefreshRate to servenv.RegisterFlags() Signed-off-by: jun <[email protected]> * Replace JQuery with Vanilla javascript && set default value 0 as non-refreshing && update e2e .txt flags Signed-off-by: jun <[email protected]> * update go/flags/endtoend/*.txt Signed-off-by: jun <[email protected]> * update e2e flags txt files Signed-off-by: jun <[email protected]> * update --table-refresh-interval tag with proper position Signed-off-by: jun <[email protected]> * remove extra space in mysqlctl.txt vtorc.txt repush Signed-off-by: jun <[email protected]> --------- Signed-off-by: jun <[email protected]> Co-authored-by: jun <[email protected]> * (re)Formalize SQLError in VReplication, add underlying wrap/unwrap functionality (#12327) * (re)Formalize SQLError in VReplication, add underlying wrap/unwrap functionality Signed-off-by: Shlomi Noach <[email protected]> * update test to match new UnwrapAll() behavior Signed-off-by: Shlomi Noach <[email protected]> * do not unwrap Signed-off-by: Shlomi Noach <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]> * Online DDL: improve retry of vreplication errors with `vitess` `ALTER TABLE` migrations (#12323) * Refactor LastError as a public struct Signed-off-by: Shlomi Noach <[email protected]> * Online DDL: use LastError for retrying vreplication streams Signed-off-by: Shlomi Noach <[email protected]> * differentiate between a temrinally failed workflow and one which 'has error' and can be retried Signed-off-by: Shlomi Noach <[email protected]> * wording Signed-off-by: Shlomi Noach <[email protected]> * simplified hasError() Signed-off-by: Shlomi Noach <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]> * Fixing `onlineddl_vrepl` flakiness, and adding more tests (#12325) * TableFilter: fix excluded RE listing Signed-off-by: Shlomi Noach <[email protected]> * extract GCTableNameExpression Signed-off-by: Shlomi Noach <[email protected]> * VReplication: exclude GC tables when reading schema Signed-off-by: Shlomi Noach <[email protected]> * wrap schema analyzis errors for better tracability Signed-off-by: Shlomi Noach <[email protected]> * Refactor LastError as a public struct Signed-off-by: Shlomi Noach <[email protected]> * Online DDL: use LastError for retrying vreplication streams Signed-off-by: Shlomi Noach <[email protected]> * refactor UpdateThrottlerTopoConfig Signed-off-by: Shlomi Noach <[email protected]> * onlineddl_vrepl tests: - using topo-based throttler config - break down of throttle-based PRS test - adding --postpone-completion -based PRS test - fixing wait for VReplicatoin status Signed-off-by: Shlomi Noach <[email protected]> * differentiate between a temrinally failed workflow and one which 'has error' and can be retried Signed-off-by: Shlomi Noach <[email protected]> * (re)Formalize SQLError in VReplication, add underlying wrap/unwrap functionality Signed-off-by: Shlomi Noach <[email protected]> * differentiate between a temrinally failed workflow and one which 'has error' and can be retried Signed-off-by: Shlomi Noach <[email protected]> * wording Signed-off-by: Shlomi Noach <[email protected]> * simplified hasError() Signed-off-by: Shlomi Noach <[email protected]> * do not unwrap Signed-off-by: Shlomi Noach <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]> * Cleanup vtorc ssl package (#12423) This is not used for anything anymore, so let's remove it. Signed-off-by: Dirkjan Bussink <[email protected]> * SidecarDB Init: don't fail on schema init errors (#12328) * Don't fail on schema init errors Signed-off-by: Rohit Nayak <[email protected]> * Fix debug var attribute name Signed-off-by: Rohit Nayak <[email protected]> * Fix typo in debug var Signed-off-by: Rohit Nayak <[email protected]> * Address review comment Signed-off-by: Rohit Nayak <[email protected]> * Address review comments Signed-off-by: Rohit Nayak <[email protected]> --------- Signed-off-by: Rohit Nayak <[email protected]> * [main] Examples: Correct VTAdmin Discovery File Path And Add Check (#12415) (#12430) * Examples: correct VTAdmin discovery file path and add check Signed-off-by: Matt Lord <[email protected]> * Remove jq dependency Signed-off-by: Matt Lord <[email protected]> * vtadmin examples scripts hardcode localhost Signed-off-by: Matt Lord <[email protected]> * Fix errant expectation Signed-off-by: Matt Lord <[email protected]> * Adjust healthy shard indicator Signed-off-by: Matt Lord <[email protected]> --------- Signed-off-by: Matt Lord <[email protected]> Co-authored-by: Matt Lord <[email protected]> * sqlparser: Tools cleanup (#12407) * tools: fix compilation checks in ast helpers Signed-off-by: Vicent Marti <[email protected]> * sqlparser: simplify generation Signed-off-by: Vicent Marti <[email protected]> * tools: make sizegen more resilient Signed-off-by: Vicent Marti <[email protected]> * tools: fix formatting of all generated code Signed-off-by: Vicent Marti <[email protected]> * tools: fix call Signed-off-by: Vicent Marti <[email protected]> * tools: review feedback Signed-off-by: Vicent Marti <[email protected]> --------- Signed-off-by: Vicent Marti <[email protected]> * Add configuration option for minimum TLS version (#12424) * Add configuration option for minimum TLS version We were hardcoding this to TLS 1.0 which is not considered safe anymore these days and was called out by CodeQL as a security issue. Instead we should default to TLS 1.2 which we do everywhere else, and allow for an option to change it if needed so that we're safe by default here. Signed-off-by: Dirkjan Bussink <[email protected]> * Add release notes item Signed-off-by: Dirkjan Bussink <[email protected]> --------- Signed-off-by: Dirkjan Bussink <[email protected]> * VReplication: Use MariaDB Compat JSON Functions (#12434) This is needed to better support importing from MariaDB into Vitess/MySQL. Signed-off-by: Matt Lord <[email protected]> Co-authored-by: Matt Lord <[email protected]> * Remove are-you-alive example (#12432) Signed-off-by: Florent Poinsard <[email protected]> * schemadiff: AlterTableAlgorithmStrategy (#12442) * schemadiff: AlterTableAlgorithmStratgy Signed-off-by: Shlomi Noach <[email protected]> * minor rename Signed-off-by: Shlomi Noach <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]> * Add release notes summary for views (#12422) (#12425) * Add release notes summary for support views * add link to the documentation * simplify views release notes --------- Signed-off-by: Florent Poinsard <[email protected]> * [main] Change `GetSchema` RPC to return `CreateView` instead of `SelectStmt` (#12421) (#12441) * Changed getSchemaRPC to return CreateView instead of SelectStmt Signed-off-by: Florent Poinsard <[email protected]> * test: fix expectation Signed-off-by: Harshit Gangal <[email protected]> --------- Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Harshit Gangal <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> Co-authored-by: Harshit Gangal <[email protected]> * Add 128 bit hashing for the evalengine (#12428) * evalengine: 128-bit hashes This adds support for 128 bit hashes. With 128 bit hashes, we have enough entropy so we can remove the need to compare values after hashing. Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * evalengine: Add support for 128 bit hashing This adds a null safe hashing function for 128 bit hashing and tests it. There's no usage of this yet, but that will be extended in the future. Signed-off-by: Dirkjan Bussink <[email protected]> * Address review feedback Signed-off-by: Dirkjan Bussink <[email protected]> * Ensure to always initialize hasher Signed-off-by: Dirkjan Bussink <[email protected]> --------- Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Co-authored-by: Vicent Martí <[email protected]> * feat: update the pull request template (#12443) Signed-off-by: Manan Gupta <[email protected]> * evalengine: use 128 bit hashing internally (#12452) * evalengine: use 128 bit hashing internally Signed-off-by: Vicent Marti <[email protected]> * evalengine: implement hashable for hashable types Signed-off-by: Vicent Marti <[email protected]> * evalengine: update sizegen Signed-off-by: Vicent Marti <[email protected]> * evalengine: fix error with unsupported collations Signed-off-by: Vicent Marti <[email protected]> --------- Signed-off-by: Vicent Marti <[email protected]> * Fix Vtop example and release script (#12457) * examples: should use mysql80 property for vitess image Signed-off-by: deepthi <[email protected]> Signed-off-by: Manan Gupta <[email protected]> * Use MySQL80 in all examples/operator YAML files Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Manan Gupta <[email protected]> * feat: update release script to update vtadmin image version too Signed-off-by: Manan Gupta <[email protected]> * feat: fix vtop example script to use vtctldclient instead Signed-off-by: Manan Gupta <[email protected]> * feat: update operator.yaml to the latest version Signed-off-by: Manan Gupta <[email protected]> * feat: fix yaml files to not pass deprecated parameters Signed-off-by: Manan Gupta <[email protected]> * feat: revert changes made for testing Signed-off-by: Manan Gupta <[email protected]> --------- Signed-off-by: deepthi <[email protected]> Signed-off-by: Manan Gupta <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Co-authored-by: deepthi <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> * feat: add steps on how to merge during code-freeze (#12444) Signed-off-by: Manan Gupta <[email protected]> * [main] GetSchema rpc to streaming api (#12447) (#12459) * GetSchema rpc to streaming api Signed-off-by: Harshit Gangal <[email protected]> * addressed review comment Signed-off-by: Harshit Gangal <[email protected]> --------- Signed-off-by: Harshit Gangal <[email protected]> Signed-off-by: Manan Gupta <[email protected]> Co-authored-by: Harshit Gangal <[email protected]> Co-authored-by: Manan Gupta <[email protected]> * feat: don't keep data for upgrade-downgrade tests (#12465) Signed-off-by: Manan Gupta <[email protected]> Co-authored-by: Manan Gupta <[email protected]> * Create Views allowed for same keyspace (#12409) (#12454) * restrict views to single keyspace * feat: also create/alter and drop views on MySQL for information-schema-support * feat: do the DMLs for alter and create of view in a transaction so that they can be rolled back in case of MySQL failure * ignore view changes by schema tracker in vttablet when views are enabled * test: fix test expectation * Correct syntax error in defer func of E2E test --------- Signed-off-by: Harshit Gangal <[email protected]> Signed-off-by: Manan Gupta <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Co-authored-by: Manan Gupta <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> * Add port to enable vtadmin access. (#12467) Signed-off-by: Andy Lim <[email protected]> * Fix scalar aggregation engine primitive for column truncation (#12468) (#12473) * fix: scalar aggregation truncation * test: added scalar aggr engine unit test * remove onecase change --------- Signed-off-by: Harshit Gangal <[email protected]> Co-authored-by: Harshit Gangal <[email protected]> * Fix `SHOW VSCHEMA TABLES` tests using v17 vtgate that expected `dual` (#12381) (#12466) * Fix SHOW VSCHEMA TABLES tests using v17 vtgate * Fix tests expectations --------- Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Manan Gupta <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> * examples schema: demi-idempotency via CREATE TABLE IF NOT EXISTS (#12453) * examples schema: demi-idempotency via CREATE TABLE IF NOT EXISTS Signed-off-by: Shlomi Noach <[email protected]> * idempotency: if not exists for all tables under examples/ Signed-off-by: Shlomi Noach <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]> * evalengine: add more coercion paths to Hash128 (#12463) Signed-off-by: Vicent Marti <[email protected]> * [misc] Delete `automation*` protos (#12449) * Remove unused automation,automationservice protos and codegen Signed-off-by: Andrew Mason <[email protected]> * Release notes Signed-off-by: Andrew Mason <[email protected]> --------- Signed-off-by: Andrew Mason <[email protected]> * BugFix: Unsharded query using a derived table and a dual table (#12484) (#12491) * test: add failing test for a query in an unsharded keyspace using a derived table and reference table * feat: fix logic to check if a single unsharded keyspace is used * feat: fix test output --------- Signed-off-by: Manan Gupta <[email protected]> * Make upgrade downgrade job names unique (#12485) Signed-off-by: Florent Poinsard <[email protected]> * Add nvm and node to vtadmin-up.sh (#12439) * Add nvm and node Signed-off-by: notfelineit <[email protected]> * Update script with PR recommendations Signed-off-by: notfelineit <[email protected]> * Push up local changes Signed-off-by: notfelineit <[email protected]> * Update examples/common/scripts/vtadmin-up.sh Signed-off-by: Frances Thai <[email protected]> * Update examples/common/scripts/vtadmin-up.sh Signed-off-by: Frances Thai <[email protected]> --------- Signed-off-by: notfelineit <[email protected]> Signed-off-by: Frances Thai <[email protected]> * Format vtctldclient GetKeyspace output using cli.MarshalJSON. (#12495) Signed-off-by: Matt Lord <[email protected]> * ApplySchema: deprecate '--skip_preflight' flag (#10716) * ApplySchema: deprecate '-skip_preflight' flag Signed-off-by: Shlomi Noach <[email protected]> * warn if --skip_preflight is provided Signed-off-by: Shlomi Noach <[email protected]> * remove -skip_preflight from tests Signed-off-by: Shlomi Noach <[email protected]> * MarkDeprecated, fix usage Signed-off-by: Shlomi Noach <[email protected]> * remove skip_preflight test Signed-off-by: Shlomi Noach <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]> * streamlog: make generic (#12494) * streamlog: make generic Signed-off-by: Vicent Marti <[email protected]> * gofmt Signed-off-by: Vicent Marti <[email protected]> --------- Signed-off-by: Vicent Marti <[email protected]> * feat: copy release notes from v16 (#12516) Signed-off-by: Manan Gupta <[email protected]> * Upgrade to `go1.20.1` (#12399) * Upgrade to go1.20.1 Signed-off-by: Florent Poinsard <[email protected]> * Bump golangci-lint version Signed-off-by: Florent Poinsard <[email protected]> * Apply linter suggestions Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: deepthi <[email protected]> * fix TestTLSClientVerifyIdentity Signed-off-by: Florent Poinsard <[email protected]> * Revert unwanted change in server_test Signed-off-by: Florent Poinsard <[email protected]> * handle readtablet error Signed-off-by: Florent Poinsard <[email protected]> * skip TestFuzz Signed-off-by: Florent Poinsard <[email protected]> * Unskip TestFuzz and fix issue Signed-off-by: Florent Poinsard <[email protected]> --------- Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: deepthi <[email protected]> Co-authored-by: deepthi <[email protected]> * Fix `vtctldclient`'s Root command to return an error on unknown command (#12481) * Add a Run func to `vtctldclient`'s Root command to return an error on unknown command Closes #12480. Signed-off-by: Andrew Mason <[email protected]> * Add test Signed-off-by: Andrew Mason <[email protected]> * flags test data Signed-off-by: Andrew Mason <[email protected]> --------- Signed-off-by: Andrew Mason <[email protected]> * Flakes: Use new healthy shard check in vreplication e2e tests (#12502) * Use new healthy shard check in vreplication e2e tests This is needed because checking that there's a primary tablet for the shard in vtgate's healtcheck is no longer a reliable indicator that the shard has a healthy serving primary, because now a primary needs to initialize its sidecar database and wait for that to replicate via semi-sync before it becomes serving and can proceed to perform normal functions. So this delay could cause test flakiness if you required a healthy shard before continuing with the test. Signed-off-by: Matt Lord <[email protected]> * Try to address unit test race flakes around log size They looked like this: WARNING: DATA RACE Write at 0x000005bf9b60 by goroutine 27141: github.com/spf13/pflag.newUint64Value() /home/runner/go/pkg/mod/github.com/spf13/[email protected]/uint64.go:9 +0x5a github.com/spf13/pflag.(*FlagSet).Uint64Var() /home/runner/go/pkg/mod/github.com/spf13/[email protected]/uint64.go:45 +0x55 vitess.io/vitess/go/vt/log.RegisterFlags() /hom…

ApplySchema: deprecate '-skip_preflight' flag

c7857c1

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach added Type: Internal Cleanup Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: Query Serving Component: TabletManager labels Jul 17, 2022

shlomi-noach requested a review from deepthi July 17, 2022 16:28

shlomi-noach requested review from ajm188 and notfelineit as code owners July 17, 2022 16:28

shlomi-noach mentioned this pull request Jul 17, 2022

ApplySchema: deprecate '--allow_long_unavailability' flag #10717

Merged

3 tasks

Merge branch 'main' into apply-schema-deprecate-skip-preflight

4e7cdbf

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach added 2 commits July 24, 2022 10:45

Merge branch 'main' into apply-schema-deprecate-skip-preflight

6c63668

Signed-off-by: Shlomi Noach <[email protected]>

warn if --skip_preflight is provided

5c80d8d

Signed-off-by: Shlomi Noach <[email protected]>

remove -skip_preflight from tests

92b84bc

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach requested review from harshit-gangal, systay and frouioui as code owners July 25, 2022 06:28

ajm188 reviewed Jul 25, 2022

View reviewed changes

deepthi previously approved these changes Jul 25, 2022

View reviewed changes

shlomi-noach added 2 commits July 27, 2022 16:07

Merge branch 'main' into apply-schema-deprecate-skip-preflight

1e9d657

Signed-off-by: Shlomi Noach <[email protected]>

MarkDeprecated, fix usage

aadaa8e

Signed-off-by: Shlomi Noach <[email protected]>

deepthi previously approved these changes Jul 27, 2022

View reviewed changes

deepthi reviewed Jul 27, 2022

View reviewed changes

shlomi-noach removed the Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. label Sep 18, 2022

rohit-nayak-ps self-requested a review as a code owner November 10, 2022 22:08

github-actions bot added the Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. label Dec 13, 2022

shlomi-noach removed the Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. label Dec 13, 2022

github-actions bot added the Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. label Jan 13, 2023

github-actions bot closed this Jan 21, 2023

shlomi-noach reopened this Feb 27, 2023

shlomi-noach added 2 commits February 27, 2023 08:04

merge main, resolve conflict

5db8806

Signed-off-by: Shlomi Noach <[email protected]>

remove skip_preflight test

2b2cfd5

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach requested a review from a team February 27, 2023 08:47

shlomi-noach removed the Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. label Feb 27, 2023

dbussink approved these changes Feb 27, 2023

View reviewed changes

vmg approved these changes Feb 27, 2023

View reviewed changes

shlomi-noach merged commit 607a9a4 into vitessio:main Feb 28, 2023

shlomi-noach deleted the apply-schema-deprecate-skip-preflight branch February 28, 2023 07:09

shlomi-noach mentioned this pull request May 25, 2023

ApplySchema vttablet RPC: allow special characters in table and column names #13054

Merged

4 tasks

ajm188 mentioned this pull request Aug 1, 2023

[vtctldclient] flags need to be defined to be deprecated #13681

Merged

4 tasks

shlomi-noach mentioned this pull request Aug 23, 2023

[onlineddl] retry and cleanup #13830

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ApplySchema: deprecate '--skip_preflight' flag #10716

ApplySchema: deprecate '--skip_preflight' flag #10716

shlomi-noach commented Jul 17, 2022

vitess-bot bot commented Jul 17, 2022 •

edited

Loading

mattlord commented Jul 17, 2022

shlomi-noach commented Jul 20, 2022 •

edited

Loading

shlomi-noach commented Jul 25, 2022

ajm188 Jul 25, 2022

shlomi-noach Jul 25, 2022

shlomi-noach Jul 27, 2022

deepthi Jul 27, 2022

ajm188 Jul 28, 2022

shlomi-noach commented Aug 2, 2022

github-actions bot commented Sep 16, 2022

shlomi-noach commented Sep 18, 2022

shlomi-noach commented Oct 13, 2022

github-actions bot commented Dec 13, 2022

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 21, 2023

shlomi-noach commented Feb 27, 2023

dbussink left a comment

shlomi-noach commented Feb 27, 2023

shlomi-noach commented Feb 28, 2023

ApplySchema: deprecate '--skip_preflight' flag #10716

ApplySchema: deprecate '--skip_preflight' flag #10716

Conversation

shlomi-noach commented Jul 17, 2022

Description

Related Issue(s)

Checklist

Deployment Notes

vitess-bot bot commented Jul 17, 2022 • edited Loading

Review Checklist

General

If a new flag is being introduced:

If a workflow is added or modified:

Bug fixes

Non-trivial changes

New/Existing features

Backward compatibility

mattlord commented Jul 17, 2022

shlomi-noach commented Jul 20, 2022 • edited Loading

shlomi-noach commented Jul 25, 2022

ajm188 Jul 25, 2022

Choose a reason for hiding this comment

shlomi-noach Jul 25, 2022

Choose a reason for hiding this comment

shlomi-noach Jul 27, 2022

Choose a reason for hiding this comment

deepthi Jul 27, 2022

Choose a reason for hiding this comment

ajm188 Jul 28, 2022

Choose a reason for hiding this comment

shlomi-noach commented Aug 2, 2022

github-actions bot commented Sep 16, 2022

shlomi-noach commented Sep 18, 2022

shlomi-noach commented Oct 13, 2022

github-actions bot commented Dec 13, 2022

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 21, 2023

shlomi-noach commented Feb 27, 2023

dbussink left a comment

Choose a reason for hiding this comment

shlomi-noach commented Feb 27, 2023

shlomi-noach commented Feb 28, 2023

vitess-bot bot commented Jul 17, 2022 •

edited

Loading

shlomi-noach commented Jul 20, 2022 •

edited

Loading