Merge branch 'master' of https://github.com/vitessio/vitess into vtsh…

…ovel-poc

.github/ISSUE_TEMPLATE/question.md

@@ -6,4 +6,4 @@ about: If you have a question, please check out our other community resources in
 Issues on GitHub are intended to be related to bugs or feature requests, so we recommend using our other community resources instead of asking here.
 
 - [Vitess User Guide](https://vitess.io/user-guide/introduction/)
-- Any other questions can be asked in the community [Slack workspace](https://bit.ly/vitess-slack)
+- Any other questions can be asked in the community [Slack workspace](https://join.slack.com/t/vitess/shared_invite/enQtMzIxMDMyMzA0NzA1LTYxMjk2M2M2NjAwNGY0ODljY2E1MjBlZjRkMmZmNDVkZTBhNDUxNzNkOGM4YmEzNWEwOTE2NjJiY2QyZjZjYTE)

.github/workflows/e2e-test-cluster.yml

@@ -0,0 +1,41 @@
+name: e2e Test Cluster
+on: [push, pull_request]
+jobs:
+
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    steps:
+
+    - name: Set up Go
+      uses: actions/setup-go@v1
+      with:
+        go-version: 1.13
+
+    - name: Check out code
+      uses: actions/checkout@v1
+
+    - name: Get dependencies
+      run: |
+        sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget
+        sudo service mysql stop
+        sudo service etcd stop
+        sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
+        sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
+        go mod download
+
+    - name: Run bootstrap.sh
+      run: |
+        echo "Copying new bootstrap over location of legacy one."
+        cp .github/bootstrap.sh .
+        ./bootstrap.sh
+
+    - name: Build
+      run: |
+        GOBIN=$PWD/bin make build
+
+    - name: Run e2e test cluster
+      run: |
+        export PATH=$PWD/bin:$PATH
+        source ./dev.env
+        VTDATAROOT=/tmp/vtdataroot VTTOP=$PWD VTROOT=$PWD tools/e2e_test_cluster.sh

Makefile

@@ -104,6 +104,9 @@ unit_test_race: build
 e2e_test_race: build
 	tools/e2e_test_race.sh
 
+e2e_test_cluster: build
+	tools/e2e_test_cluster.sh
+
 .ONESHELL:
 SHELL = /bin/bash
 

README.md

@@ -30,7 +30,7 @@ Ask questions in the
 discussion forum.
 
 For topics that are better discussed live, please join the
-[Vitess Slack](https://bit.ly/vitess-slack) workspace.
+[Vitess Slack](https://join.slack.com/t/vitess/shared_invite/enQtMzIxMDMyMzA0NzA1LTYxMjk2M2M2NjAwNGY0ODljY2E1MjBlZjRkMmZmNDVkZTBhNDUxNzNkOGM4YmEzNWEwOTE2NjJiY2QyZjZjYTE) workspace.
 
 Subscribe to
 [[email protected]](https://groups.google.com/forum/#!forum/vitess-announce)

config/mycnf/master_mariadb104.cnf

@@ -0,0 +1,23 @@
+# This file is auto-included when MariaDB 10.4 is detected.
+
+# enable strict mode so it's safe to compare sequence numbers across different server IDs.
+gtid_strict_mode = 1
+innodb_stats_persistent = 0
+
+# Semi-sync replication is required for automated unplanned failover
+# (when the master goes away). Here we just load the plugin so it's
+# available if desired, but it's disabled at startup.
+#
+# If the -enable_semi_sync flag is used, VTTablet will enable semi-sync
+# at the proper time when replication is set up, or when masters are
+# promoted or demoted.
+
+# semi_sync has been merged into master as of mariadb 10.3 so this is no longer needed 
+#plugin-load = rpl_semi_sync_master=semisync_master.so;rpl_semi_sync_slave=semisync_slave.so
+
+# When semi-sync is enabled, don't allow fallback to async
+# if you get no ack, or have no slaves. This is necessary to
+# prevent alternate futures when doing a failover in response to
+# a master that becomes unresponsive.
+rpl_semi_sync_master_timeout = 1000000000000000000
+rpl_semi_sync_master_wait_no_slave = 1

dev.env

@@ -77,6 +77,12 @@ fi
 PKG_CONFIG_PATH=$(prepend_path "$PKG_CONFIG_PATH" "$VTROOT/lib")
 export PKG_CONFIG_PATH
 
+# According to https://github.com/etcd-io/etcd/blob/a621d807f061e1dd635033a8d6bc261461429e27/Documentation/op-guide/supported-platform.md,
+# currently, etcd is unstable on arm64, so ETCD_UNSUPPORTED_ARCH should be set.
+if [ "$(arch)" == aarch64 ]; then
+    export ETCD_UNSUPPORTED_ARCH=arm64
+fi
+
 # Useful aliases. Remove if inconvenient.
 alias gt='cd $GOTOP'
 alias pt='cd $PYTOP'

doc/V3HighLevelDesign.md

@@ -782,7 +782,7 @@ Recapitulating what we’ve covered so far:
 
 Once we start allowing joins and subqueries, we have a whole bunch of table aliases and relationships to deal with. We have to contend with name clashes, self-joins, as well as scoping rules. In a way, the vschema has acted as a static symbol table so far. But that’s not going to be enough any more.
 
-The core of the symbol table will contain a map whose key will be a table alias, and the elements will be [similar to the table in vschema](https://github.com/vitessio/vitess/blob/master/go/vt/vtgate/planbuilder/schema.go#L22). However, it will also contain a column list that will be built as the query is parsed.
+The core of the symbol table will contain a map whose key will be a table alias, and the elements will be [similar to the table in vschema](https://github.com/vitessio/vitess/blob/0b3de7c4a2de8daec545f040639b55a835361685/go/vt/vtgate/vindexes/vschema.go#L82). However, it will also contain a column list that will be built as the query is parsed.
 
 ### A simple example
 

doc/VitessQueues.md

@@ -79,7 +79,7 @@ capabilities, the usual horizontal sharding process can be used.
 
 Queue Tables are marked in the schema by a comment, in a similar way we detect
 Sequence Tables
-[now](https://github.com/vitessio/vitess/blob/master/go/vt/tabletserver/table_info.go#L37).
+[now](https://github.com/vitessio/vitess/blob/0b3de7c4a2de8daec545f040639b55a835361685/go/vt/vttablet/tabletserver/tabletserver.go#L138).
 
 When a tablet becomes a master, and there are Queue tables, it creates a
 QueueManager for each of them.

docker/lite/Dockerfile

@@ -40,16 +40,23 @@ RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-ins
     gnupg dirmngr ca-certificates wget libdbd-mysql-perl rsync libaio1 libatomic1 libcurl3 libev4 \
  && for i in $(seq 1 10); do apt-key adv --no-tty --recv-keys --keyserver keyserver.ubuntu.com 8C718D3B5072E1F5 && break; done \
  && echo 'deb http://repo.mysql.com/apt/debian/ stretch mysql-5.7' > /etc/apt/sources.list.d/mysql.list \
+ && for i in $(seq 1 10); do apt-key adv --no-tty --keyserver keys.gnupg.net --recv-keys 9334A25F8507EFA5 && break; done \
+ && echo 'deb http://repo.percona.com/apt stretch main' > /etc/apt/sources.list.d/percona.list && \
+    { \
+        echo debconf debconf/frontend select Noninteractive; \
+        echo percona-server-server-5.7 percona-server-server/root_password password 'unused'; \
+        echo percona-server-server-5.7 percona-server-server/root_password_again password 'unused'; \
+    } | debconf-set-selections \
  && apt-get update \
  && DEBIAN_FRONTEND=noninteractive \
     apt-get install -y --no-install-recommends \
       bzip2 \
       libmysqlclient20 \
       mysql-client \
       mysql-server \
- && wget https://www.percona.com/downloads/XtraBackup/Percona-XtraBackup-2.4.13/binary/debian/stretch/x86_64/percona-xtrabackup-24_2.4.13-1.stretch_amd64.deb \
- && dpkg -i percona-xtrabackup-24_2.4.13-1.stretch_amd64.deb \
- && rm -f percona-xtrabackup-24_2.4.13-1.stretch_amd64.deb \
+      libjemalloc1 \
+      libtcmalloc-minimal4 \
+      percona-xtrabackup-24 \
  && rm -rf /var/lib/apt/lists/* \
  && groupadd -r vitess && useradd -r -g vitess vitess
 

examples/compose/README.md

@@ -95,13 +95,18 @@ vitess/examples/compose$ ./client.sh
 
 ### Connect to vgate and run queries
 vtgate responds to the MySQL protocol, so we can connect to it using the default MySQL client command line.
-You can also use the `./lmysql.sh` helper script.  
 ```
 vitess/examples/compose$ mysql --port=15306 --host=127.0.0.1
-vitess/examples/compose$ ./lmysql.sh --port=15306 --host=127.0.0.1
 ```
 **Note that you may need to replace `127.0.0.1` with `docker ip` or `docker-machine ip`** 
 
+You can also use the `./lmysql.sh` helper script.
+```
+vitess/examples/compose$ ./lmysql.sh --port=15306 --host=<DOCKER_HOST_IP>
+```
+
+where `<DOCKER_HOST_IP>` is `docker-machine ip` or external docker host ip addr
+
 ### Play around with vtctl commands
 
 ```

go.mod

@@ -8,6 +8,7 @@ require (
 	github.com/armon/go-metrics v0.0.0-20190430140413-ec5e00d3c878 // indirect
 	github.com/aws/aws-sdk-go v0.0.0-20180223184012-ebef4262e06a
 	github.com/boltdb/bolt v1.3.1 // indirect
+	github.com/cespare/xxhash/v2 v2.1.1
 	github.com/cockroachdb/cmux v0.0.0-20170110192607-30d10be49292 // indirect
 	github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd // indirect
 	github.com/coreos/etcd v0.0.0-20170626015032-703663d1f6ed

go.sum

@@ -29,6 +29,9 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/boltdb/bolt v1.3.1 h1:JQmyP4ZBrce+ZQu0dY660FMfatumYDLun9hBCUVIkF4=
 github.com/boltdb/bolt v1.3.1/go.mod h1:clJnj/oiGkjum5o1McbSZDSLxVThjynRyGBgiAx27Ps=
+github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
+github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
+github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/circonus-labs/circonus-gometrics v2.3.1+incompatible/go.mod h1:nmEj6Dob7S7YxXgwXpfOuvO54S+tGdZdw9fuRZt25Ag=
 github.com/circonus-labs/circonusllhist v0.1.3/go.mod h1:kMXHVDlOchFAehlya5ePtbp5jckzBHf4XRpQvBOLI+I=
 github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=

go/pools/resource_pool.go

@@ -38,6 +38,9 @@ var (
 	// ErrTimeout is returned if a resource get times out.
 	ErrTimeout = errors.New("resource pool timed out")
 
+	// ErrCtxTimeout is returned if a ctx is already expired by the time the resource pool is used
+	ErrCtxTimeout = errors.New("resource pool context already expired")
+
 	prefillTimeout = 30 * time.Second
 )
 
@@ -198,7 +201,7 @@ func (rp *ResourcePool) get(ctx context.Context) (resource Resource, err error)
 	// If ctx has already expired, avoid racing with rp's resource channel.
 	select {
 	case <-ctx.Done():
-		return nil, ErrTimeout
+		return nil, ErrCtxTimeout
 	default:
 	}
 

go/pools/resource_pool_test.go

@@ -639,7 +639,7 @@ func TestExpired(t *testing.T) {
 		p.Put(r)
 	}
 	cancel()
-	want := "resource pool timed out"
+	want := "resource pool context already expired"
 	if err == nil || err.Error() != want {
 		t.Errorf("got %v, want %s", err, want)
 	}