-
Notifications
You must be signed in to change notification settings - Fork 204
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Signed-off-by: hallaroo <[email protected]> Signed-off-by: deepthi <[email protected]>
- Loading branch information
Showing
36 changed files
with
1,277 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
--- | ||
title: Advanced Configuration | ||
description: Frequently Asked Questions about Vitess | ||
docs_nav_disable_expand: false | ||
--- | ||
|
30 changes: 30 additions & 0 deletions
30
content/en/docs/faq/advanced-configuration/authentication.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
--- | ||
title: Authentication | ||
description: Frequently Asked Questions about Vitess | ||
weight: 1 | ||
--- | ||
|
||
## How do I set up MySQL authentication in Vitess? | ||
|
||
Vitess uses its own mechanism for managing users and their permissions through VTGate. As a result, the CREATE USER.... and GRANT... statements will not work if sent through VTGate. Instead VTGate takes care of authentication for requests, so you will need to add any users that should have access to the Keyspaces via command-line options to VTGate. | ||
|
||
The simplest way to configure users is via a static authentication method. You can define the users in a JSON formatted file or string. Then you can load this file into VTGate with the additional command line parameters. | ||
|
||
You will be able to configure the UserData string and add multiple passwords. For password format, Vitess supports the mysql_native_password hash format and you should always specify your passwords using this in a non-test or external environment. | ||
|
||
To see an example of how to configure the static authentication file and more information on the various options please read this [guide](https://vitess.io/docs/user-guides/configuration-advanced/user-management/#authentication). | ||
|
||
There are other authentication mechanisms that can be utilized including LDAP-based authentication and TLS client certificate-based authentication. | ||
|
||
## How do I configure user-level permissions in Vitess? | ||
|
||
If you need to enforce fine-grained access control in Vitess, you cannot use the normal MySQL GRANT system to give certain application-level MySQL users more or fewer permissions than others. This is because Vitess uses connection pooling with fixed MySQL users at the VTTablet level, and implements its own authentication at the VTGate level. | ||
|
||
Not all of the MySQL GRANT system has been implemented in Vitess. Authorization can be done via table-level ACLs. Individual users at the VTGate level can be assigned 3 levels of permissions. | ||
- Read (corresponding to read DML, e.g. SELECT) | ||
- Write (corresponding to write DML, e.g. INSERT, UPDATE, DELETE) | ||
- Admin (corresponding to DDL, e.g. ALTER TABLE) | ||
|
||
The tables to which the permissions apply can be enumerated or specified using a regular expression. | ||
|
||
Vitess authorization via ACLs is applied at the VTTablet level, as opposed to on VTGate, where authentication is enforced. There are a number of VTTablet command line parameters that control the behavior of ACLs. You can see examples and read more about the command line parameters and further configuration options [here](https://vitess.io/docs/user-guides/configuration-advanced/authorization/#vttablet-parameters-for-table-acls). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
--- | ||
title: Components | ||
description: Frequently Asked Questions about Vitess | ||
weight: 2 | ||
--- | ||
|
||
## How can I change MySQL server variables in Vitess? | ||
|
||
In general, if you want to apply global variables at the MySQL level, you have to do it through VTTablet. There are a few ways to do that in the operator, but we recommend that you use vtctldclient ExecuteFetchAsDba. | ||
|
||
For example if you want to temporarily switch `sync_binlog` off on the MySQL that is being managed by a tablet with alias `zone1-0000000100` you would perform the following: | ||
|
||
```sh | ||
$ vtctldclient -server localhost:15999 ExecuteFetchAsDba zone1-0000000100 "set global sync_binlog=0" | ||
``` | ||
|
||
This would show the following result after checking the variable: | ||
|
||
```sh | ||
$ vtctldclient -server localhost:15999 ExecuteFetchAsDba zone1-0000000100 "show variables like 'sync_binlog'"+---------------+-------+| Variable_name | Value |+---------------+-------+| sync_binlog | 0 |+---------------+-------+ | ||
``` | ||
|
||
## Examples of how to use Vitess components | ||
|
||
We have a couple of step through examples in Github [here](https://github.com/aquarapid/vitess_examples). Currently, these cover Operator Backup and Restore, Create Lookup Vindex, and VStream. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
--- | ||
title: Metrics | ||
description: Frequently Asked Questions about Vitess | ||
weight: 8 | ||
--- | ||
|
||
## What Grafana dashboards are available? | ||
|
||
There are a set of Grafana dashboards and Prometheus alerts available on the Vitess tree in GitHub [here](https://github.com/vitessio/vitess/tree/master/vitess-mixin). You can get some additional context on these dashboards [here](https://github.com/vitessio/vitess/pull/5609). | ||
|
||
## How can I implement user-level query logging? | ||
|
||
If you would like to differentiate metrics for a 'bad_user@their_machine' from a 'good_user@their_machine', rather than having both users appear to be the same user from the same server to MySQL you will need to use table ACLs. | ||
|
||
Vitess exports per-user stats on table ACLs. There are example usages of table ACLs demonstrated in the end-to-end tests. | ||
- The configuration of table ACLs can be found [here](https://github.com/vitessio/vitess/blob/master/go/vt/vttablet/endtoend/main_test.go#L174). | ||
- The tests that demonstrate how table ACLs work can be found [here](https://github.com/vitessio/vitess/blob/master/go/vt/vttablet/endtoend/acl_test.go). | ||
|
||
To locate the variables that enable the export of per-users stats you will need to look in `/debug/vars` for variables that start with `User`, like `UserTableQueryCount`. The export itself is a multi-dimensional export categorized by Table, User and Query Type. You can also find similar names exported as prometheus metrics. | ||
|
||
Analyzing these variables can enable you to quickly narrow down the root cause of an incident, as these stats are fine-grained. Once you've identified the table and query type, you can then drill into `/queryz` or `/debug/query_stats` to determine if the issue is a particular query. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
--- | ||
title: Vindex | ||
description: Frequently Asked Questions about Vitess | ||
weight: 6 | ||
--- | ||
|
||
## What is a secondary Vindex? How does it work? | ||
|
||
Secondary Vindexes are additional Vindexes against other columns of a table offering optimizations for WHERE clauses that do not use the Primary Vindex. Secondary Vindexes return a single or a limited set of keyspace IDs which will allow VTGate to only target shards where the relevant data is present. In the absence of a Secondary Vindex, VTGate would have to send the query to all shards (called a scatter query). | ||
|
||
It is important to note that Secondary Vindexes are only used for making routing decisions. The underlying database shards will need traditional indexes on those same columns, to allow efficient retrieval from the table on the underlying MySQL instances. | ||
|
||
MARKED NOT HELPFUL | ||
|
||
## How do I create a unique index for a column in Vitess? | ||
|
||
Unique index is a distinct MySQL option. For Vitess just normal MySQL DDL will do. You have a couple other options as well either to use `ApplySchema` or directly apply the index to MySQL. | ||
|
||
Please note this is different from a unique Vindex, as that enables sending queries to one specific shard rather than ensuring the uniqueness of a column. | ||
|
||
## How do I make a CreateLookupVindex? | ||
|
||
In addition to the [user guide](https://vitess.io/docs/user-guides/configuration-advanced/createlookupvindex/) on CreateLookupVindex we also have an example walkthrough [here](https://github.com/aquarapid/vitess_examples/tree/master/vindexes/createlookupvindex). | ||
|
||
This walkthrough demonstrates the syntax of a CreateLookupVindex how to make one, how to add it to a column, and how to verify that it was successfully added. | ||
|
||
## What is a LookupVindex and how does it work? | ||
|
||
CreateLookupVindex is a new VReplication workflow that was introduced in Vitess 6. It is used to create and backfill a lookup Vindex automatically for a table that already exists and that could already have a significant amount of data in it. | ||
|
||
The CreateLookupVindex process uses VReplication for the backfill process, until the lookup Vindex is “in sync”. Then the normal process for adding/deleting/updating rows in the lookup Vindex via the standard transactional flow when updating the “owner” table for the Vindex takes over. | ||
|
||
You can read more about how to make a CreateLookupVindex [here](https://vitess.io/docs/user-guides/configuration-advanced/createlookupvindex/). If you are unfamiliar with Vindexes we recommend that you first read the information [here](https://vitess.io/docs/reference/features/vindexes). | ||
|
||
MARKED NOT HELPFUL | ||
|
||
## Does the Primary Vindex need to match its Primary Key? | ||
|
||
It is not necessary that a Primary Vindex be the same as the Primary Key. In fact, there are many use cases where you would not want this. For example, if there are tables with one-to-many relationships, the Primary Vindex of the main table is likely to be the same as the Primary Key. | ||
|
||
However, if you want the rows of the secondary table to reside in the same shard as the parent row, the Primary Vindex for that table must be the foreign key that points to the main table. A typical example is a user and order table. | ||
|
||
In this case, the order table has the `user_id` as a foreign key to the `id` of the user table. The `order_id` may be the primary key for `order`, but you may still want to choose `user_id` as Primary Vindex, which will make a user's orders live in the same shard as the user. |
24 changes: 24 additions & 0 deletions
24
content/en/docs/faq/advanced-configuration/vreplication.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
--- | ||
title: VReplication | ||
description: Frequently Asked Questions about Vitess | ||
weight: 7 | ||
--- | ||
|
||
## What is semi-sync replication? | ||
|
||
Semi-sync replication enables you to prevent your primary from finishing replication until a replica confirms that it has received all the changes. Thus adding an extra guarantee that at least one other machine has copies of the data. | ||
|
||
This addresses the problem of a combination of lagging replication and network issues resulting in data loss. With semi-sync replication, even if you have network issues you shouldn’t lose your data. | ||
|
||
Please do note that when using semi-sync replication you will have to wait for your data to flow from the primary to the replica and then get a confirmation back to the primary. Thus each transaction may take longer. The length of time depends on how close network wise the replica is to the primary. | ||
|
||
## What is the typical replication lag in VReplication? | ||
|
||
|
||
VReplication is very fast, typically replication lag is below a second as long as your network is good. | ||
|
||
However, if there is a network partition, things can be delayed depending on your configuration. For anything transactional, we recommend always reading from the source table. This principle follows the same rule as recommending reading from primary instead of a replica. | ||
|
||
## Why would I use semi-sync replication? | ||
|
||
Semi-sync replication ensures higher levels of durability between the primary and at least one replica. You can read more about semi-sync replication here. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
--- | ||
title: Vschema | ||
description: Frequently Asked Questions about Vitess | ||
weight: 5 | ||
--- | ||
|
||
## How do you select your primary key for Vitess? | ||
|
||
It is important to choose a strong primary Vindex when creating your VSchema, so the qualities should you look at are the following: | ||
- Frequency in WHERE clause of queries | ||
- Uniqueness (of the mapping function) | ||
- This means that a vindex will map a column value to only one keyspace ID (or none at all) | ||
- Co-locating rows for joins and for single-shard transactions | ||
- This means using the same primary vindex for multiple tables, as all rows tied to the same primary index will automatically be located in the same shard due to the uniqueness property of the vindex map | ||
- High cardinality | ||
- This means producing a sufficiently large number of keyspace IDs, which will give you finer control for rebalancing load through resharding | ||
|
||
You can read more detail about how to select your primary key [here](https://vitess.io/blog/2019-02-07-choosing-a-vindex/). | ||
|
||
## How can you update or change your vschema? | ||
|
||
We recommend using ApplySchema and ApplyVSchema in order to make updates to schemas within Vitess. It is also important to note that you will need to update both your MySQL database schema as well as your VSchema. | ||
|
||
The [ApplySchema](https://vitess.io/docs/reference/programs/vtctl/#applyvschema) command applies a schema change to the specified keyspace on every primary tablet, running in parallel on all shards. Changes are then propagated to replicas. The ApplyVSchema command applies the specified VSchema to the keyspace. The VSchema can be specified as a string or in a file. You can read more about the process to use these commands [here](https://vitess.io/docs/reference/features/schema-management/#changing-your-schema). | ||
|
||
There are a few ways that changes can be made to your schemas within Vitess. If you don’t want to use ApplySchema you can read more about the different methods to make updates [here](https://vitess.io/docs/user-guides/schema-changes/). | ||
|
||
## Without a Vschema how can table and schema routing work? | ||
|
||
There are a couple of special cases for when you don’t have a VSchema in place. | ||
|
||
For example, if you add a table called foo to an unsharded keyspace called ks1 the following routing will enable you to access the table: | ||
1. USE ks1; select * from foo; | ||
2. From the unqualified schema using select * from ks1.foo; | ||
3. As long as you have only one keyspace, you can use select * from foo in anonymous mode | ||
|
||
However, if you have more than one keyspace you will not be able to access the table from the unqualified schema using select * from foo until you add the table to VSchema. | ||
|
||
For a sharded keyspace will not be able to access the table until you have a VSchema for it. However, you will be able to see it in show tables. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
--- | ||
title: Vtgate | ||
description: Frequently Asked Questions about Vitess | ||
weight: 4 | ||
--- | ||
|
||
## How do you use gRPC with vtgate? | ||
|
||
To do this you will need to use the Vitess MySQL Go client. You can find a Golang Database compatible gRPC driver [here](https://pkg.go.dev/vitess.io/vitess/go/vt/vitessdriver). For Java go [here](https://github.com/vitessio/vitess/tree/master/java). | ||
|
||
Once you have the appropriate driver you will need to add the `-service_map grpc-vtgateservice` VTGate flag and set the port `-grpc_port`. | ||
|
||
This runs on a standard gRPC interface, so if you want to directly use it you can follow the example below: | ||
|
||
```sh | ||
#!/usr/bin/env node | ||
import Debug from "debug"; | ||
import * as grpc from "grpc"; | ||
import {CallerID} from './proto/vitess/vtrpc_pb'; | ||
import {BoundQuery} from './proto/vitess/query_pb'; | ||
import {Session,ExecuteRequest,ExecuteResponse} from './proto/vitess/vtgate_pb'; | ||
import {VitessClient} from './proto/vitess/vtgateservice_grpc_pb'; | ||
const log = Debug("VtgateClient"); | ||
const client = new VitessClient("139.178.90.99:15306",grpc.credentials.createInsecure()); | ||
const SingleQuery = async () => { | ||
return new Promise((resolve,reject) => { | ||
const caller = new CallerID(); | ||
caller.setPrincipal("nodejs"); | ||
const session = new Session(); | ||
session.setTargetString("main"); | ||
const query = new BoundQuery(); | ||
query.setSql("SELECT * from main.sbtest1 where id=10"); | ||
const request = new ExecuteRequest() | ||
request.setSession(session); | ||
request.setQuery(query); | ||
request.setCallerId(caller); | ||
client.execute(request, (err: grpc.ServiceError | null, response: ExecuteResponse ) => { | ||
if( err != null ){ | ||
console.log(`[SingleQuery] Error: ${err.message}`) | ||
reject(err); return; | ||
} | ||
console.log(`[SingleQuery] Response: ${JSON.stringify(response.toObject())}`) | ||
resolve(response); | ||
}); | ||
}); | ||
} | ||
async function main() { | ||
client. | ||
console.log(`[main] Starting`); | ||
await SingleQuery(); | ||
} | ||
main().then((_) => _); | ||
``` | ||
## How does vtgate know which shard to route a query to? | ||
VTGate knows two things about your Vitess components: the Vschema and the schema of MySQL. | ||
This enables VTGate to look at the WHERE clause of the query and then route the queries to correct shards. VTGate is also aware of the sharding metadata, cluster state, required latency, and availability of tables, so it will only scatter the query across the shards it needs to use. |
Oops, something went wrong.