Skip to content

Commit

Permalink
tiproxy: add vip configurations (#18498)
Browse files Browse the repository at this point in the history
  • Loading branch information
Oreoxmt authored Aug 19, 2024
1 parent 2ec1371 commit eb946a8
Show file tree
Hide file tree
Showing 6 changed files with 45 additions and 13 deletions.
7 changes: 7 additions & 0 deletions config-templates/simple-tiproxy.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@ global:
ssh_port: 22
deploy_dir: "/tidb-deploy"
data_dir: "/tidb-data"
component_versions:
tiproxy: "v1.2.0"
server_configs:
tiproxy:
ha.virtual-ip: "10.0.1.10/24"
ha.interface: "eth0"

pd_servers:
- host: 10.0.1.1
Expand All @@ -23,6 +29,7 @@ tikv_servers:

tiproxy_servers:
- host: 10.0.1.11
- host: 10.0.1.12

monitoring_servers:
- host: 10.0.1.13
Expand Down
27 changes: 25 additions & 2 deletions tiproxy/tiproxy-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,9 @@ max-connections = 100
[api]
addr = "0.0.0.0:3080"

[log]
level = "info"
[ha]
virtual-ip = "10.0.1.10/24"
interface = "eth0"

[security]
[security.cluster-tls]
Expand Down Expand Up @@ -118,6 +119,28 @@ Configurations for the load balancing policy of TiProxy.
+ Possible values: `resource`, `location`, `connection`
+ Specifies the load balancing policy. For the meaning of each possible value, see [TiProxy load balancing policies](/tiproxy/tiproxy-load-balance.md#configure-load-balancing-policies).

### ha

High availability configurations for TiProxy.

#### `virtual-ip`

+ Default value: `""`
+ Support hot-reload: no
+ Specifies the virtual IP address in the CIDR format, such as `"10.0.1.10/24"`. In a cluster with multiple TiProxy instances, only one instance binds to the virtual IP. If this instance goes offline, another TiProxy instance will automatically bind to the IP, ensuring clients can always connect to an available TiProxy through the virtual IP.

> **Note:**
>
> - Virtual IP is only supported on Linux operating systems.
> - The Linux user running TiProxy must have permission to bind IP addresses.
> - The virtual IP and the IPs of all TiProxy instances must be within the same CIDR range.
#### `interface`

+ Default value: `""`
+ Support hot-reload: no
+ Specifies the network interface to bind the virtual IP to, such as `"eth0"`. The virtual IP will be bound to a TiProxy instance only when both [`ha.virtual-ip`](#virtual-ip) and `ha.interface` are set.

### `labels`

+ Default value: `{}`
Expand Down
2 changes: 1 addition & 1 deletion tiproxy/tiproxy-deployment-topology.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ TiProxy is a L7 proxy server for TiDB, which can balance connections and migrate
| TiDB | 3 | 16 VCore 32GB * 3 | 10.0.1.4 <br/> 10.0.1.5 <br/> 10.0.1.6 | Default port <br/> Global directory configuration |
| PD | 3 | 4 VCore 8GB * 3 | 10.0.1.1 <br/> 10.0.1.2 <br/> 10.0.1.3 | Default port <br/> Global directory configuration |
| TiKV | 3 | 16 VCore 32GB 2TB (nvme ssd) * 3 | 10.0.1.7 <br/> 10.0.1.8 <br/> 10.0.1.9 | Default port <br/> Global directory configuration |
| TiProxy | 1 | 4 VCore 8 GB * 1 | 10.0.1.11 | Default port <br/> Global directory configuration |
| TiProxy | 2 | 4 VCore 8 GB * 1 | 10.0.1.11 <br/> 10.0.1.12 | Default port <br/> Global directory configuration |
| Monitoring & Grafana | 1 | 4 VCore 8GB * 1 500GB (ssd) | 10.0.1.13 | Default port <br/> Global directory configuration |

### Topology templates
Expand Down
3 changes: 3 additions & 0 deletions tiproxy/tiproxy-grafana.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ TiProxy has four panel groups. The metrics on these panels indicate the current
- backend network break: fails to read from or write to the TiDB. This may be caused by a network problem or the TiDB server shutting down
- backend handshake fail: TiProxy fails to handshake with the TiDB server
- Goroutine Count: the number of Goroutines on each TiProxy instance
- Owner: the TiProxy instance that executes various tasks. For example, `10.24.31.1:3080 - vip` indicates that the TiProxy instance at `10.24.31.1:3080` is bound to a virtual IP. The tasks include the following:
- vip: binds a virtual IP
- metric_reader: reads monitoring data from TiDB servers

## Query-Summary

Expand Down
11 changes: 4 additions & 7 deletions tiproxy/tiproxy-load-balance.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,18 +16,15 @@ By default, TiProxy enables all policies with the following priorities:
5. Location-based load balancing: TiProxy prioritizes routing requests to the TiDB server geographically closest to TiProxy.
6. Connection count-based load balancing: when the connection count of a TiDB server is much higher than that of other TiDB servers, TiProxy migrates connections from that TiDB server to a TiDB server with fewer connections.

> **Note:**
>
> - Health-based, memory-based, and CPU-based load balancing policies depend on [Prometheus](https://prometheus.io). Ensure that Prometheus is available. Otherwise, these policies do not take effect.
> - To adjust the priorities of load balancing policies, see [Configure load balancing policies](#configure-load-balancing-policies).
To adjust the priorities of load balancing policies, see [Configure load balancing policies](#configure-load-balancing-policies).

## Status-based load balancing

TiProxy periodically checks whether a TiDB server is offline or shutting down using the SQL port and status port.

## Health-based load balancing

TiProxy determines the health of a TiDB server by querying its error count from Prometheus. When the health of a TiDB server is abnormal while others are normal, TiProxy migrates connections from that server to a healthy TiDB server, achieving automatic failover.
TiProxy determines the health of a TiDB server by querying its error count. When the health of a TiDB server is abnormal while others are normal, TiProxy migrates connections from that server to a healthy TiDB server, achieving automatic failover.

This policy is suitable for the following scenarios:

Expand All @@ -36,7 +33,7 @@ This policy is suitable for the following scenarios:

## Memory-based load balancing

TiProxy queries the memory usage of TiDB servers from Prometheus. When the memory usage of a TiDB server is rapidly increasing or reaching a high level, TiProxy migrates connections from that server to a TiDB server with lower memory usage, preventing unnecessary connection termination due to OOM. TiProxy does not guarantee identical memory usage across TiDB servers. This policy only takes effect when a TiDB server is at risk of OOM.
TiProxy queries the memory usage of TiDB servers. When the memory usage of a TiDB server is rapidly increasing or reaching a high level, TiProxy migrates connections from that server to a TiDB server with lower memory usage, preventing unnecessary connection termination due to OOM. TiProxy does not guarantee identical memory usage across TiDB servers. This policy only takes effect when a TiDB server is at risk of OOM.

When a TiDB server is at risk of OOM, TiProxy attempts to migrate all connections from it. Usually, if OOM is caused by runaway queries, ongoing runaway queries will not be migrated to another TiDB server for re-execution, because these connections can only be migrated after the transaction is complete.

Expand All @@ -48,7 +45,7 @@ This policy has the following limitations:

## CPU-based load balancing

TiProxy queries the CPU usage of TiDB servers from Prometheus and migrates connections from a TiDB server with high CPU usage to a server with lower usage, reducing overall query latency. TiProxy does not guarantee identical CPU usage across TiDB servers but ensures that the CPU usage differences are minimized.
TiProxy queries the CPU usage of TiDB servers and migrates connections from a TiDB server with high CPU usage to a server with lower usage, reducing overall query latency. TiProxy does not guarantee identical CPU usage across TiDB servers but ensures that the CPU usage differences are minimized.

This policy is suitable for the following scenarios:

Expand Down
8 changes: 5 additions & 3 deletions tiproxy/tiproxy-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ When a TiDB server performs scaling in or scaling out, if you use a common load

### Quick deployment

TiProxy is integrated into [TiUP](https://github.com/pingcap/tiup), [TiDB Operator](https://github.com/pingcap/tidb-operator), [TiDB Dashboard](/dashboard/dashboard-intro.md), and [Grafana](/tiproxy/tiproxy-grafana.md), which reduces the deployment, operation, and management costs.
TiProxy is integrated into [TiUP](https://github.com/pingcap/tiup), [TiDB Operator](https://github.com/pingcap/tidb-operator), [TiDB Dashboard](/dashboard/dashboard-intro.md), and [Grafana](/tiproxy/tiproxy-grafana.md), and supports built-in virtual IP management, reducing the deployment, operation, and management costs.

## User scenarios

Expand Down Expand Up @@ -91,7 +91,7 @@ This section describes how to deploy and change TiProxy using TiUP. For how to d
3. Configure the TiProxy instances.
To ensure the high availability of TiProxy, it is recommended to deploy at least two TiProxy instances. You can use hardware load balancers to distribute traffic to each TiProxy instance, or configure virtual IP to route the traffic to the available TiProxy instance.
To ensure the high availability of TiProxy, it is recommended to deploy at least two TiProxy instances and configure a virtual IP by setting [`ha.virtual-ip`](/tiproxy/tiproxy-configuration.md#virtual-ip) and [`ha.interface`](/tiproxy/tiproxy-configuration.md#interface) to route the traffic to the available TiProxy instance.

When selecting the model and number of TiProxy instances, consider the following factors:

Expand All @@ -106,12 +106,14 @@ This section describes how to deploy and change TiProxy using TiUP. For how to d

```yaml
component_versions:
tiproxy: "v1.0.0"
tiproxy: "v1.2.0"
server_configs:
tiproxy:
security.server-tls.ca: "/var/ssl/ca.pem"
security.server-tls.cert: "/var/ssl/cert.pem"
security.server-tls.key: "/var/ssl/key.pem"
ha.virtual-ip: "10.0.1.10/24"
ha.interface: "eth0"
```

4. Start the cluster.
Expand Down

0 comments on commit eb946a8

Please sign in to comment.