Skip to content

Vitess 6 General Availability
Compare
Choose a tag to compare
@morgo morgo released this 29 Apr 14:18
· 43 commits to release-6.0 since this release
v6.0.20-20200429
f7fa695

Known Issues

  1. While running a builtin backup with large databases, you might sometimes get this error: use of closed network connection: cannot copy data: MultipartUpload: upload multipart failed.
    We believe this is caused by the newer aws-sdk-go version we upgraded to in 5.0. It might be this problem filed upstream.
  2. On xtrabackup backed up tablets on the new build, you might occasionally hit this panic:
    Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, caught panic during Backup: bufio: writer returned negative count from Write. This also seems to be related to aws-sdk-go, though we haven't traced it to a known issue.
  3. There is a known regression where the transaction pool timeout will not apply, and connections never expire. We are working on a fix for this issue, and will release an update shortly.

Incompatible Changes

This release includes the following changes, which may result in incompatibilities when upgrading from a previous release:

Changes to Local Examples

The local examples have been updated to use the new VReplication workflows for MoveTables and Resharding. This obsoletes the previous workflows of Vertical Split Clone and Horizontal Sharding. The earlier examples remain available in the “local.legacy” example directory for now. #6016 #6015 #6052 #6072 #6106 #6121

Change to Helm Chart

The helm chart has been updated to default to using Kubernetes as the Topology Server provider. This change was required to remove the dependency on etcd-operator, which is no longer under active support. #5989

Earlier Kubernetes Operator Retired

Support for the Vitess Kubernetes Operator has been removed due to a lack of maintainers. Adoption is understood to be low, but users are encouraged to migrate to using the Helm charts which continue to be supported, or PlanetScale’s Vitess Operator (provided by a third party) #5873

SQL Support: Keywords are no longer lowercase

Vitess no longer lower cases SQL keywords. In corner cases, this may affect applications which made specific use of SQL keywords, and then referred to a column in the result set #5897

SQL Support: Parenthesis are now removed from the AST

Vitess now simplifies expressions to remove parentheses from expressions. In corner cases, this may affect applications that use an expression that contains parens and then refer to it as a column in the result. #5906

Statement Based Replication, SplitQuery and UpdateStream

Code for the unsupported features of Statement-based replication (SBR), SplitQuery and UpdateStream has now been removed. Users using SBR are encouraged to switch to using Row-based replication instead. #5940 #5972 #5928

Messaging: First Class Citizen + API Improvements

The messaging feature has had several breaking changes and feature additions

  • Direct inserts are now supported: INSERT INTO msg_table SELECT col1, col2...
  • time_scheduled and time_created are no longer required nor automatically populated.
  • priority is a new required field. Messages with a lower priority will be processed first.
  • Message retention is now measured after time_acked instead of time_scheduled
  • New table options vt_min_backoff and vt_max_backoff set bounds on exponential backoff for message retries
  • 33% backoff jitter has been added for messages that are getting postponed to prevent thundering herds.

Users of the messages feature are encouraged to read through the full details in issue #5947 #5948 #6114

RPM and DEB Packages

Vitess binaries and examples are now installed in system standard paths, rather than the /vt directory #5836

Vitess Go Driver

Values of type sqltypes.Timestamp are now automatically converted to time.Time values
#5788

Simplified VTtablet Schema Load Process

The process of loading vttablet schema has been simplified, as many of the statistics loaded were expensive to gather but remained unused. This may lead to compatibility breakage in the use of the features: Sequences, VStreamer and Messages #5951

Planned Reparent

The implementation of Planned Reparent and Emergency Reparent have been improved, with more graceful handling of failure modes. While this change is functionality neutral, it is important that Vitess components are upgraded in the recommended order. #6050

Bugs Fixed

  • Binlog Client: Vitess components now randomize the server_id when acting as a MySQL replica. #5998

  • heartbeat: Prevent a scenario where tables could be created on a replica leading to errant transactions. Now tables are only created on the master with binlog ON, using the app_user not dba_user #6053

  • Messages: For the mysql protocol, buffered results coming from vitess are flushed a minimum of every 100ms, and configurable by mysql_server_flush_delay. This helps prevent against a case where intermittent streaming could delay messages from being sent. #5879

  • mysqlctl: Correctly escape and quote passwords that contain special characters. #5761

  • VReplication: Several improvements to failure handling and miscellaneous fixes #5753 #5792 #5801 #5842 #5935 #5905 #5845 #5942

  • Vtcompose: Add a check to make sure that files can be read #6024

  • Vtcompose: refactor vtcompose for improved testability #5994

  • Vindexes: Support for a region_json Vindex #6017 #5806 #5955

  • Vtctl: Before applying a VSchema, vtctl now correctly checks that the schema exists. #5931

  • Vtctl: Vtctl now exits early if the topo_global_root was set incorrectly #6119

  • Vtctl: EnsureVSchema and GetOrCreateShard now pass in the cell name. This helps prevent against corruption, which could occur from auto-rebuilding of cells automatically when creating a new shard or keyspace. #5930

  • VTExplain: An issue where incorrectly formatted comments could lead to a hang has been fixed. #5984

  • VTGate: Leading zeros were incorrectly dropped in microsecond timestamps when using prepared statements #5901

  • VTGate: Setting the transaction_mode in a mysql session is now case insensitive. #5963

  • VTGate: When updating consistent indexes, VTGate applies an optimization where it checks if anything really changed before applying an update. In the case that this check results in errors, VTGate will now skip the optimization and execute the update regardless. #5883

  • VTGate: Remove allocation of unused bytes #5874

  • VTGate: The gateway interface functionality has been simplified in anticipation of two new gateway implementations: TabletGateway and VTDirectGateway #5978 #5992

  • VTGate: The process of watching for VSchema updates from the topology server now recovers successfully if it receives an invalid update. This helps prevent against a scenario where a malformed vschema could take down all vtgate servers. #6051

  • VTGate: Previously VTGate performed health checks against all keyspaces, not just keyspaces listed in keyspaces_to_watch. Healthchecks are now limited correctly. #5815

  • VTGate: SQL Support: database() is now only rewritten against dual #5793

  • VTGate: Plan Cache is now disabled for statements with nil instructions #6045

  • VTGate: Make sure to only accept DDL that can be parsed, at least partially #6023

  • VTGate: Log when accepting a new connection fails #5881

  • VTGate: Prepared Statements did not correctly handle signed integers #5974

  • VTGate: Executor statistics did not correctly account for dividing by zero, leading to a potential panic. #5904

  • VTGate: The error message for unsupported DDL creation has been improved #6109

  • VTGate: The metric QueriesProcessedByTable was not correctly incremented with streaming queries processed by vtgate #6057

  • GRPC Client: The default keep alive and keep alive timeout values have been changed from unlimited to 10 seconds. This helps prevent cases where queries could hang while waiting for a failed pod or node. In addition, the defaultMaxMessageSize is now set to 16MB #5922 #3890

  • VTTablet: local_metadata is now populated after an agent has been initialized. This helps prevent against the ambiguous case where both -init_populate_metadata and -restore_from_backup are specified. #5783

  • VTTablet: A hang could occur when shutting down a tablet server when the Topo Server is unavailable. Tablet servers will now time out correctly. #5865

  • VTTablet Health Check: A warning is now logged when the vttablet StreamHealth RPC fails #5917

  • VTTablet Planned Reparent: A new context is now created for calling UndoDemoteMaster. This helps prevent a scenario where the existing context may have expired. #5841

  • VTTablet: A panic could occur when calling NewDBConnNoPool #6086

  • VTTablet: Queries of type STREAM_EXECUTE were inadvertently dropped from logging #6056

  • Zookeeper Topo: Trailing newlines are now ignored when specifying -topo_zk_auth_file
    #6030

  • Misc: Stats counters would erroneously print warnings to log files when adding a negative value. This warning did not consider the case where counters are reused. #5988

  • Misc: Binary log events with the special name of "mysql.rds_*" are now skipped. This helps improve compatibility with Amazon RDS flavors #5966

Functionality Added or Changed

  • Docker: sysstat and strace are now bundled in vitess/lite images #5814

  • Docker: new lite images based on Red Hat UBI 7 are now available #6100

  • Example Code: An are you alive example has been added #5646

  • Example Code: The demo application has been migrated from Python to Go #6022

  • Example Code: A set of notes and scripts on how to use Vitess with common frameworks #6034

  • Helm charts: now work with the latest Kubernetes version. An issue where the 308_final.yml script did not delete a tablet has also been fixed. #6048 #6083 #6075 #6074

  • Helm charts: Helm charts now enable proxy_tablets by default #6112

  • Java Client: The deprecated VTGateBlockingConn client has now been removed. The newer VTGateBlockingConnection client should be used instead. #5973

  • Get Started Guides: Examples now use the VReplication workflows MoveTables and Reshard instead of Vertical Split Clone and Horizontal Resharding. #6016 #6015 #6052 #6072 #6106

  • Local Example: Scripts that start services have been refactored to ./scripts/vttablet-up.sh, ./scripts/mysqlctl-up.sh and start only one daemon at a time. The motivation for doing this was to make the usage more closely mimic init scripts. #5808

  • Messaging: add new priority field #5967

  • Messaging: Setting both a minimum and maximum backoff time is now supported (rather than exponentially backing off forever) #5979

  • Messaging: Messaging has been refactored to use Row-based Replication vstreamer feed #5913

  • Messaging: A 33% Jitter has been added to postpone backoff #6092

  • Topology Service: A new Kubernetes native Topology Server is now available. #5703 #5957

  • vitessdriver: add ability to set custom gRPC dial options #5820

  • VReplication: Several Improvements to the stability of Vstreamer stability and minor bug fixed. #5878 #5926 #5891 #5804

  • VReplication: A debugging interface is now provided at /debug/vrlog #5889

  • VReplication: The workflows previously known as 'Migrate', 'MigrateReads' and 'MigrateWrites' have been renamed to MoveTables, SwitchReads and SwitchWrites respectively. This breaking change was made because VReplication was previously considered experimental. The new name disambiguates the use of 'Migrate'. These commands also now support a ‘Dry run’ option. #6018

  • VReplication: Filtered VReplication now supports int and varbinary filters for columns #5949

  • VStream: The Vstream now provides journal events, using a similar interface as VReplication. #5746

  • Vtctl: A -force flag has been added so that calls to DeleteCellInfo can still succeed after a topology server has been permanently shut down. #6013

  • Vtctl: A --skip-verify option has been added to CopySchemaShard. This helps in scenarios where source and destination use different versions of MySQL, and verifying the schema as identical may fail. #5265

  • Vtctld: The flag -proxy_tablets has been added. This allows vtctld act as a reverse proxy for all of the tablets it manages. This helps ease a pain point for Kubernetes users #6058 #6062

  • Vtctld: A new health check (/debug/status) has been added to check the status of tablet servers #6113

  • Vtctld: Specifying the cell when initializing vtctld is now optional. If you don't specify a cell, vtctld will choose one for you. #6111

  • VTGate: Improvements to SQL Support:

    • For compatibility with mysqldump, the syntax SET unique_checks=0 and SET sql_quote_show_create=1 is now passed but ignored #5846 #5856
    • The syntax SET SESSION transaction_read_only = 0 is now supported #5827
    • USE INDEX (without index name) #5811 #5763
    • SHOW INDEXES ..[EXTENDED] and SHOW KEYS #6006 #5936
    • SHOW CREATE TABLE with qualifier #5585
    • SHOW FULL COLUMNS #6078
    • SHOW CHARACTER SET WHERE #5710
    • GROUP_CONCAT() with a LIMIT is now permitted #5871
    • SELECT options can now appear in any order #6082
    • Correctly handle DML without a WHERE clause #6011
    • Support for DML scatter queries with lookup vindexes #5829
    • Function names are now formatted correctly #5918
    • Ensure correctness when using SCHEMA() and FOUND_ROWS() functions #5732
    • Ensure correctness of found rows #5924 #5923
    • Improved testing for the use of last_insert_id() and rollback #5887
    • Support for user defined variables #5921
    • Support for partial update of a composite owned Vindex #5875
    • Improve Update and Delete Query Routing for IN Clause #6116
    • Allow SET user defined variables #6093
    • The parser and executor have been incrementally refactored for both maintainability and generating query plans #5770 #5892 #5941 #5995 #6031 #6010 #6090 #5995 #5941 #6035 #6069 #6042 #6047 #6080 #6055 #6064 #6099 #6054 #6108 #6105 #5982 #5959 #5993 #6077 #6096

  • VTGate: When a batched insert fails, VTGate will now only return a list of problem rows that failed due to vindex.Verify errors. Rows that did not produce problems are now omitted. #5896

  • VTGate: The ability to set a default workload is now possible, via -mysql_default_workload #5888

  • VTGate: The previously deprecated v2 routing API has been removed. However, v2 routing is still supported in the limited context of explicitly targeting shards. In the context of explicitly targeting shards, autocommit is now supported and routing has become more flexible. #5962 #5909 #6040

  • VTGate: Client connections are now permitted to upgrade their transaction_mode (for example from MULTI to TWOPC) on a per-session basis #6049

  • VTTablet: The Vttablet Server has been refactored and componentized. This change is a precursor to upcoming features such as multiple schemas per tablet #6027 #6020 #6009 #5809 #6094 #6070

  • VTTablet: A new option named db_connect_timeout_ms has been added to specify the amount of time to wait when connecting to a MySQL Server over TCP. For backwards compatibility, the default is to hang waiting indefinitely. #5832

  • VTTablet Backup: Adding Azure Blob backup support #5674 #5937

  • VTTablet Backup: The option -xtrabackup_prepare_flags has been added to support passing flags to xtrabackup for the prepare phase #5954

  • VTTablet Backup: Previously deprecated flags gcs_backup_storage_project and lock_timeout have been removed #5748

  • VTTablet: A new option -enable-consolidator-replicas has been added to the query engine
    #5862

  • VTTablet: Error codes returned by the Tablet Server have been improved #6012

  • VTTablet: The option table-acl-config-reload-interval has been added to reload the file specified by -table-acl-config on a timer. This matches similar functionality already available in vtgate. #5925

  • VTTablet: Logging messages have been added during the init tablet phase #6004

  • VTTablet: Resource pools now provide more detailed statistics of wait time distribution #5727

  • VTTablet: Logging has been improved during init tasks. This helps improve debugging failed operations. #6004

  • Misc: The DBConfigs code has been refactored. This helps ensure proper usage of connection parameters for every mysql connection, and cleans up a previous source of bugs. #5833 #5819

  • Misc: All binaries distributed by Vitess now print their version to the log files on startup #5996 #6005

  • Misc: Timeouts are now set for all connection pools #6063

Build Environment Changes

Assets 5
Loading