sqlparser: fix binary unary operator bug #940
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This fixes issue #798 where a construct like "a - -b" produced incorrect code. I also found a related bug where a construct like "- -a" also produced incorrect code. This is also a potential vulnerability because "--" is treated as a comment.
|
LGTM |
sougou
added a commit
that referenced
this pull request
Jul 30, 2015
sqlparser: fix binary unary operator bug
notfelineit
pushed a commit
to planetscale/vitess
that referenced
this pull request
Nov 16, 2022
* Add Boost on top of latest-14.0 This adds Boost on top of the latest-14.0 branch which is what we'll be running soon in PSDB production and where we want to run Boost as well. It squashes the entire history. Signed-off-by: Dirkjan Bussink <[email protected]> * Always stream results for upqueries as they can be large (#884) This sets the query options as well to olap mode just in case. We don't need to run any DML and only read data, so that is safe as well. With streaming we can better handle cases with very large result sets. We still build the set in memory though in the Boost nodes but we need memory anyway if we need to process and store the intermediate results for large numbers of results here. Signed-off-by: Dirkjan Bussink <[email protected]> * integration: add more test cases Signed-off-by: Vicent Marti <[email protected]> * boost: port TestDoubleShuffle (#919) Signed-off-by: Matt Layher <[email protected]> * Swiss tables (#917) * Swiss tables Replace the internal Go maps with concurrent Swiss tables. They are much faster and concurrent safe for the use cases we have. The concurrency is achieved by swapping two swiss tables and having a multi step delete process to avoid reading freed memory. Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * swiss: fix tests Signed-off-by: Vicent Marti <[email protected]> * swiss: remove unused code Signed-off-by: Vicent Marti <[email protected]> * go mod: remove unused dependencies Signed-off-by: Vicent Marti <[email protected]> * swiss: cleanup and refactoring Signed-off-by: Vicent Marti <[email protected]> * vstream: fix race when writing gtid Signed-off-by: Vicent Marti <[email protected]> * swiss: fix crash & leaks Signed-off-by: Vicent Marti <[email protected]> * state: remove interface for State Signed-off-by: Vicent Marti <[email protected]> * swiss: fix recursive frees in Rows Signed-off-by: Vicent Marti <[email protected]> * Add comments for offheap implementation Signed-off-by: Dirkjan Bussink <[email protected]> Co-authored-by: Vicent Marti <[email protected]> * Boost/races1 (#922) * boost: Go 1.19 formatting Signed-off-by: Matt Layher <[email protected]> * domain: pass Domain context in constructor to avoid race Signed-off-by: Matt Layher <[email protected]> * domain: fix tracing Signed-off-by: Vicent Marti <[email protected]> * domain: fix race in upqueries Signed-off-by: Vicent Marti <[email protected]> * memorytopo: fix race in election Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Matt Layher <[email protected]> * offheap: test cmalloc misuse panics (#924) Signed-off-by: Matt Layher <[email protected]> * Cleanup unused CI jobs for now for Boost (#931) Signed-off-by: Dirkjan Bussink <[email protected]> * workflows: run only boost tests (#940) Signed-off-by: Vicent Marti <[email protected]> * fix API after rebase (#941) Signed-off-by: Vicent Marti <[email protected]> * common: temporarily represent ColumnSet as a string (#942) Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Matt Layher <[email protected]> * domain: first pass at removing d.ctx (#938) * domain: first pass at removing d.ctx Signed-off-by: Matt Layher <[email protected]> * domain: incorporate review feedback, move trace completion Signed-off-by: Matt Layher <[email protected]> * proto: add Packet.id for tracing Signed-off-by: Matt Layher <[email protected]> * boostrpc: pass trace ID in Packet rather than metadata Signed-off-by: Matt Layher <[email protected]> * trace: packet IDs are always int32 Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Vicent Marti <[email protected]> * offheap: print pointer with %#x to silence vet (#943) Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Matt Layher <[email protected]> * Add double shuffle test with external bases (#923) * Add double shuffle test with external bases Signed-off-by: Dirkjan Bussink <[email protected]> * Trigger a replay from an external base if data is missing Signed-off-by: Dirkjan Bussink <[email protected]> * planner: change external base domain placement Signed-off-by: Vicent Marti <[email protected]> * Fix the replay path for lookups with a different type but equal value The problem here is that we had maps using a boostpb.Row that would be different for different types (say int32(1) vs. int64(1)). Those are equivalent though and we should handle those correctly. This adds a new RowSet that uses the proper hashing to see these values as equivalent and we handle them properly. Signed-off-by: Dirkjan Bussink <[email protected]> * external: remove graphviz Signed-off-by: Vicent Marti <[email protected]> * debug: cleanup row formatting Signed-off-by: Vicent Marti <[email protected]> * processing: tweak keys in Miss Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Vicent Marti <[email protected]> * grouped: use hashes for grouping (#946) Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * Setup additional test as test with external base (#947) This adds the albums test as an external base test as well. This is the last one using a recipe before we can start removing those from the non external tests. Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * boost: dbg helpers (#952) * boost: implement debug helpers Signed-off-by: Vicent Marti <[email protected]> * dbg: improve logging performance Signed-off-by: Vicent Marti <[email protected]> * dbg: fix linter warnings Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * Always enforce setting up an external base (#960) * Always enforce setting up an external base We don't want to have cases where we're using SQL based recipes and don't use an external base. This panic() enforces this now. We remove the tests that use this since we have the same tests in the external base tests already so we can keep using those. Signed-off-by: Dirkjan Bussink <[email protected]> * Update tests with an external source for golden MIR tests Signed-off-by: Dirkjan Bussink <[email protected]> * Cleanup unused code bits Signed-off-by: Dirkjan Bussink <[email protected]> * Remove create table handling from recipe This is done separately with loading the schema from the topo. Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * boost: VSchema Loading (#968) * controller: allow passing a vschema Signed-off-by: Vicent Marti <[email protected]> * integration: simplify tests Signed-off-by: Vicent Marti <[email protected]> * server: vschema loading Signed-off-by: Vicent Marti <[email protected]> * srvtopo: fix watch crash during shutdown Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * integration: remove graphviz dependency (#973) Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * Lookup an `INT64(0)` key when the list of `keynames` contains an empty string (#976) * Lookup a nil key when the list of keynames contains an empty string Signed-off-by: Florent Poinsard <[email protected]> * Use INT64(0) for the bogokey when key is empty Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> * Fix graphviz quote (#986) * fix graphviz Signed-off-by: Andres Taylor <[email protected]> * Fix tests Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Co-authored-by: Andres Taylor <[email protected]> * vstream: fix gtid being empty (#987) Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * boost: implement eviction (#981) * boost: implement eviction Signed-off-by: Vicent Marti <[email protected]> * eviction: fix linter warnings Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * proto: Cleanup proto conversions (#993) * flownode: cleanup proto conversions Signed-off-by: Vicent Marti <[email protected]> * flownode: more conversion cleanup Signed-off-by: Vicent Marti <[email protected]> * integration: always check domain serialization Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * Reproduction and possible fix for materialized packet bug (#997) * reproduction and possible fix for materialized packet bug Signed-off-by: Vicent Marti <[email protected]> * Implement logic to optionally ignore message packets The changes here implement the ideas discussed already in previous commits on this branch. It introduces the Vstream packet type and creates that from vstream objects. Additionally it changes the filtering to still pass down messages, but to set the ignored flag on packets we have seen before in replays for upqueries. Signed-off-by: Dirkjan Bussink <[email protected]> * Add additional test with multiple partial materialized views Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Co-authored-by: Dirkjan Bussink <[email protected]> * boost planner: first operator building blocks (#969) * boost: VSchema Loading (#968) * controller: allow passing a vschema Signed-off-by: Vicent Marti <[email protected]> * integration: simplify tests Signed-off-by: Vicent Marti <[email protected]> * server: vschema loading Signed-off-by: Vicent Marti <[email protected]> * srvtopo: fix watch crash during shutdown Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * boost planner: first operator building blocks Signed-off-by: Andres Taylor <[email protected]> * added some logic for converting the operator tree to mir Signed-off-by: Andres Taylor <[email protected]> * refactor: move graphviz rendering Signed-off-by: Andres Taylor <[email protected]> * add projectedColumns and bogokey to operator to mir output Signed-off-by: Florent Poinsard <[email protected]> * added the leaf node Signed-off-by: Andres Taylor <[email protected]> * add querysignature to operators Signed-off-by: Andres Taylor <[email protected]> * feat: add pushDownPredicate rewriting Signed-off-by: Andres Taylor <[email protected]> * feat: add logic so signatures and hashes are the same between the ops and qg planner Signed-off-by: Andres Taylor <[email protected]> * Support Joins and pushing down predicates Signed-off-by: Florent Poinsard <[email protected]> * feat: enable pushdown to either join side Signed-off-by: Andres Taylor <[email protected]> * various small fixes Signed-off-by: Andres Taylor <[email protected]> * feat: first query passes without table prefixes Signed-off-by: Andres Taylor <[email protected]> * feat: handle explicit joins in the FROM clause Signed-off-by: Andres Taylor <[email protected]> * test: make it possible to see the operator tree using graphviz Signed-off-by: Andres Taylor <[email protected]> * imports Signed-off-by: Andres Taylor <[email protected]> * Generate the incorporator golden test output Signed-off-by: Florent Poinsard <[email protected]> * Use JSON format for the incorporator golden tests Signed-off-by: Florent Poinsard <[email protected]> * Use vindex columns in the table operator Signed-off-by: Florent Poinsard <[email protected]> * remove unused field Signed-off-by: Andres Taylor <[email protected]> * update golden tests to make sense. added jsonability to the operators Signed-off-by: Andres Taylor <[email protected]> * add view to the operator tree Signed-off-by: Andres Taylor <[email protected]> * Propagate needed columns throughout the operator tree Signed-off-by: Florent Poinsard <[email protected]> * Add columns to the view and join operators Signed-off-by: Florent Poinsard <[email protected]> * updated golden tests Signed-off-by: Andres Taylor <[email protected]> * Add support for group by Signed-off-by: Florent Poinsard <[email protected]> * Implement MarshalJSON for GroupBy Signed-off-by: Florent Poinsard <[email protected]> * Extracted the operator planning methods into a single one Signed-off-by: Florent Poinsard <[email protected]> * Compare the operator output with the incorporator expectation + added new tables to the vschema Signed-off-by: Florent Poinsard <[email protected]> * feat: add query signature to operators which supports hash on unordered predicates and table names Signed-off-by: Manan Gupta <[email protected]> * test: add failing test for merging code Signed-off-by: Manan Gupta <[email protected]> * move the graphviz visit logic to outside the addToGraph method Signed-off-by: Andres Taylor <[email protected]> * clean up Signed-off-by: Andres Taylor <[email protected]> * update golden tests again Signed-off-by: Andres Taylor <[email protected]> * Addition of nolint comments to convert_op Signed-off-by: Florent Poinsard <[email protected]> * Fix and skip merge_test Signed-off-by: Florent Poinsard <[email protected]> * Applied review suggestions Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Manan Gupta <[email protected]> Co-authored-by: Vicent Martí <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> Co-authored-by: Manan Gupta <[email protected]> * simplify the planner (#1010) Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Andres Taylor <[email protected]> * .github: restore PlanetScale PR docker build (#1008) Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Matt Layher <[email protected]> * boost: Test against MySQL fork for testing (#1020) * Test against MySQL fork for Boost testing Signed-off-by: Dirkjan Bussink <[email protected]> * Enable track gtid for our fork Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * flownode: ignore packets in Reader (#1044) The bug fixed by @dbussink in #997 is incomplete: for that PR, we ensure that nodes with partially materialized states do not process ignored packets. However, a `Reader` is a node that can have a partially materialized state, and yet the state is not reachable from the Domain's state map. We need to explicitly check whether the `Reader` itself is partially materialized and ignore the packet if so. Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * vtctld: stub ApplyBoostRecipe RPC (#1034) * vtctld: stub ApplyBoostRecipe RPC Signed-off-by: Matt Layher <[email protected]> * vtctld: wire up ApplyRecipe Signed-off-by: Vicent Marti <[email protected]> * fix linter warnings Signed-off-by: Vicent Marti <[email protected]> * grpcvtctldserver: remove test Signed-off-by: Vicent Marti <[email protected]> * vtctld: fix boost initialization Signed-off-by: Vicent Marti <[email protected]> * vtgate: fix Boost initialization Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Vicent Marti <[email protected]> * vtboost: wire up to servenv with healthchecks (#1053) * vtboost: wire up to servenv with healthchecks Signed-off-by: Vicent Marti <[email protected]> * endtoend: randomize ports Signed-off-by: Vicent Marti <[email protected]> * vtboost: remove sleep Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * Go from Operator to Flow (#1003) * refactor the operators so we can build the flow nodes without going through mir Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> * test: make sure all integration_external tests have vschema Signed-off-by: Andres Taylor <[email protected]> * refactor: extract an incorporator interface Signed-off-by: Andres Taylor <[email protected]> * build base nodes with primary keys Signed-off-by: Andres Taylor <[email protected]> * Use Reader node when dealing with a View Op and connect edges of a NodeReuse Signed-off-by: Florent Poinsard <[email protected]> * make sure that the view/reader get the corrent name Signed-off-by: Andres Taylor <[email protected]> * move the tableId to the ReuseNode and not the Table Signed-off-by: Andres Taylor <[email protected]> * help the node be in sync with operators around columns Signed-off-by: Andres Taylor <[email protected]> * don't change table names Signed-off-by: Andres Taylor <[email protected]> * Lookup AggrFunc indexes in the projection Signed-off-by: Florent Poinsard <[email protected]> * add bogokey is no parameter is given Signed-off-by: Andres Taylor <[email protected]> * refactor column handling Signed-off-by: Andres Taylor <[email protected]> * Support groupby with no grouping column Signed-off-by: Florent Poinsard <[email protected]> * Remove non-required projections Signed-off-by: Florent Poinsard <[email protected]> * make it possible for Column to have multiple expressions Signed-off-by: Andres Taylor <[email protected]> * made offsets work on joins Signed-off-by: Andres Taylor <[email protected]> * tiny refactor Signed-off-by: Andres Taylor <[email protected]> * refactor union handling Signed-off-by: Andres Taylor <[email protected]> * prepare for supporting derived tables Signed-off-by: Andres Taylor <[email protected]> * Planning of derived tables and fixed small aggregation issues Signed-off-by: Florent Poinsard <[email protected]> * make sure to use Add instead of append for all columns Signed-off-by: Andres Taylor <[email protected]> * uniquefy column outputs Signed-off-by: Andres Taylor <[email protected]> * cleanups Signed-off-by: Andres Taylor <[email protected]> * Updated the incorporator tests to only use the operator Signed-off-by: Florent Poinsard <[email protected]> * add evalengine support to projection Signed-off-by: Andres Taylor <[email protected]> * remove faulty tests Signed-off-by: Andres Taylor <[email protected]> * undo regression on aggregation planning Signed-off-by: Andres Taylor <[email protected]> * projection: fix typechecking Signed-off-by: Vicent Marti <[email protected]> * gomod: cleanup Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Andres Taylor <[email protected]> Co-authored-by: Vicent Marti <[email protected]> * make projections follow the input order (#1056) * make projections follow the input order Signed-off-by: Andres Taylor <[email protected]> * cleanups Signed-off-by: Andres Taylor <[email protected]> * clean up proto Signed-off-by: Andres Taylor <[email protected]> * stop using nil for projections Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Andres Taylor <[email protected]> * controller: remove old incorporator (#1059) Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Andres Taylor <[email protected]> * boost: New Recipe API (#1063) * [wip] recipe API Signed-off-by: Vicent Marti <[email protected]> * boost: wip recipe API Signed-off-by: Vicent Marti <[email protected]> * recipe: new installation API Signed-off-by: Vicent Marti <[email protected]> * controller: new API Signed-off-by: Vicent Marti <[email protected]> * boost: remove vtboost-stress Signed-off-by: Vicent Marti <[email protected]> * vtboost: proto fixes Signed-off-by: Vicent Marti <[email protected]> * boost: change the way protos work Signed-off-by: Vicent Marti <[email protected]> * Generate vtadmin proto Signed-off-by: Florent Poinsard <[email protected]> * boost: remove query by offset and name Signed-off-by: Vicent Marti <[email protected]> * boost: fix recipe linter warnings Signed-off-by: Vicent Marti <[email protected]> * gomod: cleanup Signed-off-by: Vicent Marti <[email protected]> * boost: add ResetQueries Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> * Use UUIDs to differentiate Boost Clusters in the topology server (#1064) * Addition of UUIDs for each boost cluster Signed-off-by: Florent Poinsard <[email protected]> * Use unique name to name the UUID in the tests Signed-off-by: Florent Poinsard <[email protected]> * Remove enable_boost and use the UUID in the end to end tests Signed-off-by: Florent Poinsard <[email protected]> * have a single flag to deal with the cluster UUID Signed-off-by: Florent Poinsard <[email protected]> * rename the description of the enable_boost vtgate flag Signed-off-by: Florent Poinsard <[email protected]> * Add a default value to the boost cluster uuid flag Signed-off-by: Florent Poinsard <[email protected]> * Privatize the boost flag Signed-off-by: Florent Poinsard <[email protected]> * simplify the use of boostClusterUUID in the topo watcher Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> * Use Gen4 end-to-end tests to test Boost (#994) * Removed the use of testing.TB in the helper methods of the E2E tests Signed-off-by: Florent Poinsard <[email protected]> * Addition of TestSimpleQuery test Signed-off-by: Florent Poinsard <[email protected]> * Intial addition of the generation tool Signed-off-by: Florent Poinsard <[email protected]> * Write file with new generated boost content Signed-off-by: Florent Poinsard <[email protected]> * Remove the AST parsing, handle multiple same selects and add the boost read stats assertion Signed-off-by: Florent Poinsard <[email protected]> * Create and generate new E2E packages Signed-off-by: Florent Poinsard <[email protected]> * Generate the main_test.go file automatically Signed-off-by: Florent Poinsard <[email protected]> * Copy .sql and .json file instead of doing a symbolic link Signed-off-by: Florent Poinsard <[email protected]> * Add missing imports to generated test files Signed-off-by: Florent Poinsard <[email protected]> * Remove duplicated views Signed-off-by: Florent Poinsard <[email protected]> * Generate the entire queries E2E folder Signed-off-by: Florent Poinsard <[email protected]> * Formatted the go/test/endtoend/vtgate/queries/... packages Signed-off-by: Florent Poinsard <[email protected]> * Generated the boost E2E tests Signed-off-by: Florent Poinsard <[email protected]> * Changed the main E2E template to read multiple SQL schemas Signed-off-by: Florent Poinsard <[email protected]> * Use vtgateProcess instead of clusterInstance Signed-off-by: Florent Poinsard <[email protected]> * Use the AST parser to parse and generate the new E2E tests Signed-off-by: Florent Poinsard <[email protected]> * Sanitize generate_e2e.go Signed-off-by: Florent Poinsard <[email protected]> * Handle clusterInstance in generated e2e tests Signed-off-by: Florent Poinsard <[email protected]> * Apply format on the generated main files Signed-off-by: Florent Poinsard <[email protected]> * Revert unwanted changes on the order by tests Signed-off-by: Florent Poinsard <[email protected]> * Run boost e2e tests on all the boost packages Signed-off-by: Florent Poinsard <[email protected]> * Removed unwanted change in TestAggregateTypes Signed-off-by: Florent Poinsard <[email protected]> * Revert unwanted changes to vitess OSS Signed-off-by: Florent Poinsard <[email protected]> * Added comment in the e2e generation tool to ease readability Signed-off-by: Florent Poinsard <[email protected]> * Format Gen4 end-to-end tests (#11089) * Format Gen4 end-to-end tests Signed-off-by: Florent Poinsard <[email protected]> * Fixed schemaSQL embed in union e2e test Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> * run the boost CI e2e tests only on endtoend/ root Signed-off-by: Florent Poinsard <[email protected]> * Fix TestInvalidDateTimeTimestampVals linting issues (#11098) Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> * Use vtparams instead of clusterInstance in TestNormalizeAllFields (#11102) Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> * Better test generation Signed-off-by: Florent Poinsard <[email protected]> * start using error instead of panic Signed-off-by: Andres Taylor <[email protected]> * Remove duplicate main test var declaration Signed-off-by: Florent Poinsard <[email protected]> * Parse comments Signed-off-by: Florent Poinsard <[email protected]> * more checks for unsupported constructs Signed-off-by: Andres Taylor <[email protected]> * switch more panics to errors; remove code for unsupported constructs Signed-off-by: Andres Taylor <[email protected]> * really check for nil Signed-off-by: Andres Taylor <[email protected]> * addition of defer stmts in tests Signed-off-by: Florent Poinsard <[email protected]> * skip tests if boost does not support a query Signed-off-by: Andres Taylor <[email protected]> * Create the boost cluster only once per test package Signed-off-by: Florent Poinsard <[email protected]> * Fix build issues Signed-off-by: Florent Poinsard <[email protected]> * More exlusive list of E2E tests, addition of panic recovery and better view generation Signed-off-by: Florent Poinsard <[email protected]> * Revert unwanted changes Signed-off-by: Florent Poinsard <[email protected]> * skip currently unsupported queries Signed-off-by: Andres Taylor <[email protected]> * don't use fmt.sprintf when not needed Signed-off-by: Andres Taylor <[email protected]> * fix recipe application Signed-off-by: Vicent Marti <[email protected]> * Fix TestVTBoostStartupAndCleanup Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Andres Taylor <[email protected]> Co-authored-by: Vicent Marti <[email protected]> * Add two new workflows for boost to check generated content is correct (#1073) * Addition of a workflow to ensure the incorporator golden tests are running and are up to date Signed-off-by: Florent Poinsard <[email protected]> * Addition of a workflow to ensure the e2e tests are in sync Signed-off-by: Florent Poinsard <[email protected]> * install goimports in generate e2e test Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> * Boost/having (#1067) * add support for HAVING Signed-off-by: Andres Taylor <[email protected]> * refactor Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Andres Taylor <[email protected]> * boost: use multiple aggregations (#1076) Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> * boost: Allow removing individual queries (#1072) * boost: implement individual recipe removal Signed-off-by: Vicent Marti <[email protected]> * vtboost: add ExternalId Signed-off-by: Vicent Marti <[email protected]> * recipe: fix storing external ID Signed-off-by: Vicent Marti <[email protected]> * controller: recipe ptr Signed-off-by: Vicent Marti <[email protected]> * conv: check for tables Signed-off-by: Vicent Marti <[email protected]> * recipe: sort bases too Signed-off-by: Vicent Marti <[email protected]> * recipe: deprecate Public Signed-off-by: Vicent Marti <[email protected]> * update the endtoend tests Signed-off-by: Vicent Marti <[email protected]> * recipe: make public_id a string Signed-off-by: Vicent Marti <[email protected]> * fix external tests Signed-off-by: Vicent Marti <[email protected]> * operators: fix test Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * proto/vtboost: support removal of more than one query at once (#1080) Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Matt Layher <[email protected]> * boost: remove TestHaving WithVschema option from bad merge * Boost/union (#1079) * reintroduce UNION Signed-off-by: Andres Taylor <[email protected]> * re-introduce tests and fix rebase problems Signed-off-by: Andres Taylor <[email protected]> * refactor - rename struct Signed-off-by: Andres Taylor <[email protected]> * disallow UNION DISTINCT Signed-off-by: Andres Taylor <[email protected]> * clean ups Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Andres Taylor <[email protected]> * drpc: add connection helpers including pooling (#1083) * drpc: add connection helpers including pooling Signed-off-by: Vicent Marti <[email protected]> * drpc: make pool generic Signed-off-by: Vicent Marti <[email protected]> * drpc: add panic recovery Signed-off-by: Vicent Marti <[email protected]> * drpc: debug test Signed-off-by: Vicent Marti <[email protected]> * drpc: randomize port numbers in tests to avoid collissions Signed-off-by: Vicent Marti <[email protected]> * fuxxx Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * boost: add `-boost_hostname` (#1088) Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * boost: Distributed eviction (#1089) * boost: eviction planner Signed-off-by: Vicent Marti <[email protected]> * wip Signed-off-by: Vicent Marti <[email protected]> * wip Signed-off-by: Vicent Marti <[email protected]> * controller: enable distributed eviction Signed-off-by: Vicent Marti <[email protected]> * server: enable period eviction Signed-off-by: Vicent Marti <[email protected]> * state: add config for eviction period Signed-off-by: Vicent Marti <[email protected]> * boost: cleanup Signed-off-by: Vicent Marti <[email protected]> * gomod: tidy Signed-off-by: Vicent Marti <[email protected]> * memmap: use proper atomic types Signed-off-by: Vicent Marti <[email protected]> * rowstore: fix test Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * boost: Cluster UUID (#1093) * boost: do not expose `boost_cluster_uuid` in vtgate/vtctld Signed-off-by: Vicent Marti <[email protected]> * boost: reset connections when changing cluster Signed-off-by: Vicent Marti <[email protected]> * boosttopo: expose an `enable_boost` flag Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * boost: Improve recipe management (#1097) * recipe: more resilient to errors Signed-off-by: Vicent Marti <[email protected]> * dbg: dumper * controller: lazy loading of external bases Signed-off-by: Vicent Marti <[email protected]> * controller: fix lazy table reuse Signed-off-by: Vicent Marti <[email protected]> * incorporator: refactor integration tests Signed-off-by: Vicent Marti <[email protected]> * remove graphviz popups Signed-off-by: Andres Taylor <[email protected]> * support for more aggregating over complex expressions Signed-off-by: Andres Taylor <[email protected]> * integration: add missing test file Signed-off-by: Vicent Marti <[email protected]> * boost: linter warnings Signed-off-by: Vicent Marti <[email protected]> * integration: update golden tests Signed-off-by: Vicent Marti <[email protected]> * actions: remove unused golden test conversion Signed-off-by: Vicent Marti <[email protected]> * boosttopo: fix recipe propagation Signed-off-by: Vicent Marti <[email protected]> * don't share slices Signed-off-by: Andres Taylor <[email protected]> * boosttopo: gofmt Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Andres Taylor <[email protected]> Co-authored-by: Andres Taylor <[email protected]> * boost: Graphviz v2 (#1098) * graphviz: show also memory evictions Signed-off-by: Vicent Marti <[email protected]> * boost: fixes to graphviz graphs and memory eviction Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * go/boost: use hyphens for new vtboost flags (#1100) Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Matt Layher <[email protected]> * boost: View Query Improvements (#1101) * worker: allow single query requests Signed-off-by: Vicent Marti <[email protected]> * boost: add hitrate tracking to the view client Signed-off-by: Vicent Marti <[email protected]> * reader: fix variable capture Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * go/boost: start drpc listener immediately (#1107) Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Matt Layher <[email protected]> * boost: View metrics (#1106) * boost: cleanup view & table clients Signed-off-by: Vicent Marti <[email protected]> * boost: add view metrics to the client and server Signed-off-by: Vicent Marti <[email protected]> * test: fix generation with new helpers Signed-off-by: Vicent Marti <[email protected]> * endtoend: check in more tests Signed-off-by: Vicent Marti <[email protected]> * endtoend: do not generate DML tests Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * boost: add support for a standalone NOP planner (#1108) Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * feat: support count star in boost (#1110) Signed-off-by: Manan Gupta <[email protected]> Signed-off-by: Manan Gupta <[email protected]> Co-authored-by: Manan Gupta <[email protected]> * boost: stop SELECT * queries (#1111) * stop SELECT * queries Signed-off-by: Andres Taylor <[email protected]> * clean up errors Signed-off-by: Andres Taylor <[email protected]> * planner: improve structured error handling Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Vicent Marti <[email protected]> * refactor offsetbinding to interface method (#1117) Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Andres Taylor <[email protected]> * Add more complex state tracking for Boost clusters (#1115) * Add more complex state tracking for Boost clusters A single UUID doesn't cut it for what we want to track. This adds the more expanded state of available clusters so we can track things for when we have draining etc. for example. Doesn't functionally change anything yet, except for using the new format. Signed-off-by: Dirkjan Bussink <[email protected]> * Use a callback to save the data with retries Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * boost: Simple Tables (#1119) * storage: replace swiss3 with a normal hash map Signed-off-by: Vicent Marti <[email protected]> * boost: remove hash from offheap rows Signed-off-by: Vicent Marti <[email protected]> * offheap: use a nil pointer for empty rows Signed-off-by: Vicent Marti <[email protected]> * backlog: fix tests Signed-off-by: Vicent Marti <[email protected]> * lrstore: linter Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * state: return evicted keys instead of values (#1120) * state: return evicted keys instead of values Signed-off-by: Vicent Marti <[email protected]> * integration: do not deadlock when panicking during Graphviz Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * boost: TopK (`ORDER BY` + `LIMIT`) and DISTINCT (#1116) * add TopK Signed-off-by: Andres Taylor <[email protected]> * put TopK on top of projection so we can use the bogokey as a paramter if nothing else is available Signed-off-by: Andres Taylor <[email protected]> * should return after setting the error Signed-off-by: Andres Taylor <[email protected]> * add DISTINCT Signed-off-by: Andres Taylor <[email protected]> * integration: do not test UNION DISTINCT Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Vicent Marti <[email protected]> * Boost: Fix SELECT DISTINCT and add UNION DISTINCT (#1121) * make distinct great again Signed-off-by: Andres Taylor <[email protected]> * enable UNION DISTINCT Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Andres Taylor <[email protected]> * [Boost] Clean up of errors (#1125) * replace some errors with typed errors Signed-off-by: Andres Taylor <[email protected]> * use better errors in the boostplan and operators packages Signed-off-by: Florent Poinsard <[email protected]> * Simplify structured errors Removes a few function indirections and message generation which we don't need. Providing the parsed statement for a non supported query also makes it easy for a caller to check if they want to do more specific things with the actual type or not etc. Signed-off-by: Dirkjan Bussink <[email protected]> * More specific error messages Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> Co-authored-by: Dirkjan Bussink <[email protected]> * Add explicit error message for missing column references (#1126) Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * Implement additional multi cluster tracking (#1128) This implements the logic to track clusters and their materializations across different clusters with different uuids. Warming is not yet implemented but will be in a follow up. Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * Use separate primary key error type (#1135) This one is different enough since we want to know the explicit keyspace and table name here, that it's better to have a separate error type. Otherwise we have no way to communicate the actual table name which is the most important aspect here. Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * vtgate: use a session variable to toggle Boost on/off per connection (#1133) * vtgate: use a session variable to toggle Boost on/off per connection Signed-off-by: Vicent Marti <[email protected]> * vtadmin: update protos Signed-off-by: Vicent Marti <[email protected]> * boost: reduce potential conflicts Signed-off-by: Vicent Marti <[email protected]> * sysvars: cleanup Signed-off-by: Vicent Marti <[email protected]> * sysvars: move into its own file Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * Keep track of table usage in the planner (#1134) * Keep track of table usage in the planner Signed-off-by: Florent Poinsard <[email protected]> * Return the table report through the TestIncorporator Signed-off-by: Andres Taylor <[email protected]> * off-topic: remove create view Signed-off-by: Andres Taylor <[email protected]> * test: sort columns before checking Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Andres Taylor <[email protected]> Co-authored-by: Andres Taylor <[email protected]> * incorporator: return table report in the QFP (#1139) Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * Fix union and normalize end-to-end planner tests (#1132) * Fix the normalize e2e by producing better errors for select * Signed-off-by: Florent Poinsard <[email protected]> * Fix the union e2e tests Signed-off-by: Florent Poinsard <[email protected]> * Use proper errors for dual tables Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> * calculate column usage before adding new nodes to the query (#1142) Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Andres Taylor <[email protected]> * boost: switch endtoend test back to UUID (#1144) Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Matt Layher <[email protected]> * Reuse nodes in single query (#1141) * Added node, op and column equality Signed-off-by: Andres Taylor <[email protected]> * Add NodeReuser and test Signed-off-by: Andres Taylor <[email protected]> * make it possible to re-use the same nodes inside a single query Signed-off-by: Andres Taylor <[email protected]> * use PlanContext instead of SemTable Signed-off-by: Andres Taylor <[email protected]> * move in DDL into the PlanContext Signed-off-by: Andres Taylor <[email protected]> * review feedback Signed-off-by: Andres Taylor <[email protected]> * fix bug that was stopping node reuse Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Andres Taylor <[email protected]> * [boost] Add a PRIMARY KEY to the aggregation E2E tests (#1147) * Add a PRIMARY KEY to the aggregation E2E tests (#11459) (#1138) Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> * Regen the boost e2e tests Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> * vtgate: keep track of Boost state in the sysvars (#1146) * vtgate: keep track of Boost state in the sysvars Signed-off-by: Vicent Marti <[email protected]> * vtgate: cleanup system variable handling Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * Keep track of the select * expansion in boost (#1151) Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> * Allow [SELECT *] again (#1153) Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Andres Taylor <[email protected]> * boost: async recipe API (#1154) * topo: move recipes in our topo Signed-off-by: Vicent Marti <[email protected]> * topo: use a new architecture for topology and recipe application Signed-off-by: Vicent Marti <[email protected]> * topo: improve the recipe change API Signed-off-by: Vicent Marti <[email protected]> * boost: fix linter warnings Signed-off-by: Vicent Marti <[email protected]> * endtoend: fix generation Signed-off-by: Vicent Marti <[email protected]> * boost: split topo packages Signed-off-by: Vicent Marti <[email protected]> * recipe: move back into its own package Signed-off-by: Vicent Marti <[email protected]> * endtoend: increase sleep period Signed-off-by: Vicent Marti <[email protected]> * common: add LICENSE Signed-off-by: Vicent Marti <[email protected]> * topo: fix watcher race Signed-off-by: Vicent Marti <[email protected]> * boost: return a list of cluster states Signed-off-by: Vicent Marti <[email protected]> * topo: rename VersionNumber Signed-off-by: Vicent Marti <[email protected]> * topo: smaller diff for VersionNumber Signed-off-by: Vicent Marti <[email protected]> * controller: better error reporting in ControllerState Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * boostclient: expose SaveError type for gRPC error codes (#1157) Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Matt Layher <[email protected]> * proto: BoostPutRecipe RPC (#1158) * proto: BoostPutRecipe RPC Signed-off-by: Matt Layher <[email protected]> * controller: implement PutRecipe API Signed-off-by: Vicent Marti <[email protected]> * gen: fix code generation Signed-off-by: Vicent Marti <[email protected]> * topo: fix APIs Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Vicent Marti <[email protected]> * Fix planner according to the aggregation end-to-end planner tests (#1127) * fix the aggregation tests Signed-off-by: Florent Poinsard <[email protected]> * Fail when there is an aggregation in the where clause Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> * boost: 128-bit hashes (#1159) * boost: use a 128 bit hasher Signed-off-by: Vicent Marti <[email protected]> * backlog: pass in a hasher Signed-off-by: Vicent Marti <[email protected]> * wip Signed-off-by: Vicent Marti <[email protected]> * wip Signed-off-by: Vicent Marti <[email protected]> * spookyhash Signed-off-by: Vicent Marti <[email protected]> * hash128: use metrohash Signed-off-by: Vicent Marti <[email protected]> * gomod: tidy Signed-off-by: Vicent Marti <[email protected]> * hash128: benchmark against sha256 Signed-off-by: Vicent Marti <[email protected]> * fix linter warnings Signed-off-by: Vicent Marti <[email protected]> * hash128: enable 128-bit hashing by default Signed-off-by: Vicent Marti <[email protected]> * hash: use the hasher more sparingly Signed-off-by: Vicent Marti <[email protected]> * vt: move hasher into its own package with Apache license Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * boost: return gRPC NotFound for topo lookups (#1167) * boost: return gRPC NotFound for topo lookups Signed-off-by: Matt Layher <[email protected]> * topo: prefix error constants; add Cause Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Vicent Marti <[email protected]> * Boost/outer to inner (#1118) * refactor to make it easier to read Signed-off-by: Andres Taylor <[email protected]> * turn outer join into inner join when possible Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Manan Gupta <[email protected]> Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Manan Gupta <[email protected]> * Implement query warmup logic (#1166) * Implement query warmup logic This replicates queries without blocking in a separate goroutine to any cluster that is in the warmup state. Signed-off-by: Dirkjan Bussink <[email protected]> * watcher: implement gradual failovers Signed-off-by: Vicent Marti <[email protected]> * Implement initial tests for failovers with watches Signed-off-by: Dirkjan Bussink <[email protected]> * topo: add integration tests Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Vicent Marti <[email protected]> * offheap: split out gomalloc for CGO_ENABLED=0 (#1170) Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Matt Layher <[email protected]> * added tests for outer-to-inner-conversion (#1171) Signed-off-by: Andres Taylor <[email protected]> Signed-off-by: Andres Taylor <[email protected]> * Track creation time of cluster (#1172) Either this can be explicitly passed in, or we generate a new timestamp based on the current time if nothing is given. Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * boost: add GetCluster by UUID RPC (#1174) Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Matt Layher <[email protected]> * boost: vstream improvements (#1173) * vstream: wip Signed-off-by: Vicent Marti <[email protected]> * worker: fix VStream implementation Signed-off-by: Vicent Marti <[email protected]> * worker: precompute external table field Signed-off-by: Vicent Marti <[email protected]> * vstream: fix table filtering Signed-off-by: Vicent Marti <[email protected]> * worker: add vstream metrics Signed-off-by: Vicent Marti <[email protected]> * integration: ensure vstream rows are being processed Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * Turn on the planner E2E tests in the CI (#1130) * Turn on the planner E2E tests Signed-off-by: Florent Poinsard <[email protected]> * Attempt at reducing data race Signed-off-by: Florent Poinsard <[email protected]> * Proper locking of cluster.CancelFunc Signed-off-by: Florent Poinsard <[email protected]> * Remove the call to EnableTracingOpt Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> * boost: more status codes and client tests (#1178) Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Matt Layher <[email protected]> * boost: hack to support hyphen and underscore flags (#1183) * boost: hack to support hyphen and underscore flags Signed-off-by: Matt Layher <[email protected]> * cmd: use flag.Vars to provide aliases for our CLI flags Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Vicent Marti <[email protected]> * boost: shard-aware vstream (#1182) * worker: use a srvtopo.Resolver for streaming Signed-off-by: Vicent Marti <[email protected]> * worker: fixups for vstream Signed-off-by: Vicent Marti <[email protected]> * endtoend: regenerate tests Signed-off-by: Vicent Marti <[email protected]> * endtoend: fix generation Signed-off-by: Vicent Marti <[email protected]> * integration: bring back ExternalGateway Signed-off-by: Vicent Marti <[email protected]> * domain: quiet noisy logging Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * topo: keep recipe version in proto (#1185) Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * boost: remove flags hack, remove default --cell arg (#1187) Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Matt Layher <[email protected]> * Listen on the same address for workers as global (#1191) We need to listen on addresses that are actually accessible to other nodes in the network or we end up publishing the 127.0.0.1 addresses and things don't actually work. Both for the domain to transfer messages across domains if we run with multiple nodes and for the reader so that we publish a correct reader address that vtgates can access. Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * boost: support migrations (#1193) * wip * wip: onlineddl Signed-off-by: Vicent Marti <[email protected]> * vstream: handle migrations in the vstream Signed-off-by: Vicent Marti <[email protected]> * boost: type-aware sharding Signed-off-by: Vicent Marti <[email protected]> * domain: fix lint warning Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Andres Taylor <[email protected]> * vstream: do not inject defaults as a placeholder (#1195) Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * Fix reading data from results where keys were included (#1194) * Fix reading data from results where keys were included This fixes the issue where a value returned from a reader would include the key if the key was not already in the result set. It is implemented by filtering out these additional keys from the column list when we store the value into the lookup table for the reader. Fixes a number of TODOs in the tests where this was asserted incorrectly before. Signed-off-by: Dirkjan Bussink <[email protected]> * flownode: debug extra reader columns Signed-off-by: Vicent Marti <[email protected]> * Update tests with fixed returned keys Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Vicent Marti <[email protected]> * boostplan: add logic for when to allocate new external bases (#1197) Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * Update Actions jobs to run on latest versions (#1200) This is already updated on the latest branch for vitess-private, so we also should update here to avoid deprecation errors. Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * Fix parsing of MySQL GTID positions (#1202) GTID positions are sets of GTIDs since there can be multiple as we have failovers etc. Before we were assuming only a single GTID set, not a set of sets. We replace here our protobuf type with a generic string to simplify things and rely much more on the Vitess `mysql` package with associated parsing of GTIDs to deal with everything. Somehow it magically works on the first try, so we're happy with it. Fixes the following error: ``` {"level":"error","ts":1666651712.112728,"caller":"worker/vstream.go:299","msg":"failed to parse gtid","worker_id":"4b51716b-1ae8-45f9-9165-4920085d808e","context":"EventProcessor","target.keyspace":"mike-boost","target.shard":"-","target.cell":"","gtid":"MySQL56/64ba1f57-5074-11ed-9eee-1ea9b4a2c7de:1-34982,8aabbf4f-5074-11ed-b225-aa23ce7e3ba2:1-20443,a6f1bf40-5073-11ed-9c0f-12a3889dc912:1-343402","error":"strconv.ParseUint: parsing \"34982,8aabbf4f-5074-11ed-b225-aa23ce7e3ba2:1-20443,a6f1bf40-5073-11ed-9c0f-12a3889dc912:1-343402\": invalid syntax","stacktrace":"vitess.io/vitess/go/boost/server/worker.(*EventProcessor).processTarget.func1\n\tvitess.io/vitess/go/boost/server/worker/vstream.go:299\nvitess.io/vitess/go/vt/vttablet/grpctabletconn.(*gRPCQueryClient).VStream\n\tvitess.io/vitess/go/vt/vttablet/grpctabletconn/conn.go:665\nvitess.io/vitess/go/vt/vttablet/queryservice.(*wrappedService).VStream.func1\n\tvitess.io/vitess/go/vt/vttablet/queryservice/wrapped.go:245\nvitess.io/vitess/go/vt/vtgate.(*TabletGateway).withRetry\n\tvitess.io/vitess/go/vt/vtgate/tabletgateway.go:333\nvitess.io/vitess/go/vt/vttablet/queryservice.(*wrappedService).VStream\n\tvitess.io/vitess/go/vt/vttablet/queryservice/wrapped.go:244\nvitess.io/vitess/go/boost/server/worker.(*EventProcessor).processTarget\n\tvitess.io/vitess/go/boost/server/worker/vstream.go:260\nvitess.io/vitess/go/boost/server/worker.(*EventProcessor).process.func1\n\tvitess.io/vitess/go/boost/server/worker/vstream.go:225"} ``` Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * addition of the schema tracking user flag in vtboost (#1201) * addition of the schema tracking user flag in vtboost Signed-off-by: Florent Poinsard <[email protected]> * require schema tracking to be enabled to start vtboost Signed-off-by: Florent Poinsard <[email protected]> * rename boost schema tracking flags to be unique Signed-off-by: Florent Poinsard <[email protected]> * Export vtgate schema tracking flags and use them in boost Signed-off-by: Florent Poinsard <[email protected]> Signed-off-by: Florent Poinsard <[email protected]> * topo: keep worker count in topology (#1204) * topo: keep worker count in topology Signed-off-by: Vicent Marti <[email protected]> * endtoend: fix tests Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * sqle: fix bad dependency Signed-off-by: Vicent Marti <[email protected]> * boost: MySQL-only GTID parsing (#1206) * vstream: simpler GTID parsing Signed-off-by: Vicent Marti <[email protected]> * integration: fix fake gtid parsing Signed-off-by: Vicent Marti <[email protected]> * boost: update cached_size Signed-off-by: Vicent Marti <[email protected]> * Fix GTID handling We need to keep this around outside of the loop since new events depend on a previously received GTID event. Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Co-authored-by: Dirkjan Bussink <[email protected]> * boost: Mike's Schema (#1214) * integration: reproduction test Signed-off-by: Vicent Marti <[email protected]> * flownode: return default column on identity projections Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> * boost: use hyphens for flags again (#1216) Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Matt Layher <[email protected]> * Add gtid mode flag for boost binary (#1215) * Make track gtid mode the default for when nothing else is set This ensures that for prod traffic we run with TRACK_GTID mode against the latest MySQL with our patch to avoid the otherwise possible race conditions. Signed-off-by: Dirkjan Bussink <[email protected]> * Add flag to configure GTID tracking mode We need to be able to set this for production pods as well, spefically to TRACK_GTID for PSDB where we run with our MySQL patch. Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * Add testing for multi cell setups and add logging (#1217) * Add testing for multi cell setups and add logging This adds a bit more logging when we have a failure and also when we start the external tablet gateway. Signed-off-by: Dirkjan Bussink <[email protected]> * Fix sending upquery to default shard name Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * Setup caller id for upqueries (#1226) * Setup caller id for upqueries This ensures that Boost uses a proper caller id when sending the upqueries. Signed-off-by: Dirkjan Bussink <[email protected]> * Fix ordering logic in GRPC server We need to send back an error code before we try to stream anything since if we stream anything, it's assumed that the stream setup is successful which isn't the case here. This was resulting in missing errors in production and that failures would lead to zero results. Signed-off-by: Dirkjan Bussink <[email protected]> * Add testing with ACLs and caller id This logic sets up a caller ID and enforces an ACL for callers. Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * boost: do not expose extra fields to the client (#1220) * boost: do not expose extra fields to the client Signed-off-by: Vicent Marti <[email protected]> * boost: fix log printf Signed-off-by: Matt Layher <[email protected]> * boost: panic stopgap Signed-off-by: Matt Layher <[email protected]> * Fix logging We should log here based on the number of columns, not on the number of rows. Signed-off-by: Dirkjan Bussink <[email protected]> * boost: overhaul how key schemas are sent to the client Signed-off-by: Vicent Marti <[email protected]> * topo: add missing files Signed-off-by: Vicent Marti <[email protected]> * endtoend: fix test delay Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Matt Layher <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Co-authored-by: Matt Layher <[email protected]> Co-authored-by: Dirkjan Bussink <[email protected]> * Cancel context before closing everything (#1228) This ensures that all watchers etc. are stopped before we close the connections to the topo. Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> * boost: Shard-aware upqueries (#1227) * boost: use a full Executor for upqueries Signed-off-by: Vicent Marti <[email protected]> * integration: refactor mock packages Signed-off-by: Vicent Marti <[email protected]> * Implement sharded upqueries for track GTID mode They now also work for the tracking GTID mode where we shove the session state into the query result. Signed-off-by: Dirkjan Bussink <[email protected]> * endtoend: remove autogenerated tests Signed-off-by: Vicent Marti <[email protected]> * endtoend: add username Signed-off-by: Vicent Marti <[email protected]> * boost: refactor end2end tests Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Co-authored-by: Dirkjan Bussink <[email protected]> * WIP: handle cases with similar query hash (#1232) * WIP: handle cases with similar query hash When two queries have the same normalized SQL, we still want to validate and check against both of them. Fixes #1224 Signed-off-by: Dirkjan Bussink <[email protected]> * endtoend: implement test Signed-off-by: Vicent Marti <[email protected]> * boost: fix protogen Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: Vicent Marti <[email protected]> * boost: Test Improvements (#1234) * end2end: wire up ACLs like in PlanetScale prod Signed-off-by: Vicent Marti <[email protected]> * boost: refactor test layout and helpers Signed-off-by: Vicent Marti <[email protected]> * end2end: fix bad rename Signed-off-by: Vicent Marti <[email protected]…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes issue #798
where a construct like "a - -b" produced incorrect code.
I also found a related bug where a construct like "- -a"
also produced incorrect code.
This is also a potential vulnerability because "--" is
treated as a comment.