Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Multi-tenant and Sharded TSO #5895

Closed
60 tasks done
binshi-bing opened this issue Feb 2, 2023 · 4 comments
Closed
60 tasks done

Support Multi-tenant and Sharded TSO #5895

binshi-bing opened this issue Feb 2, 2023 · 4 comments
Assignees
@binshi-bing
Labels
type/feature-request Categorizes issue or PR as related to a new feature.

Comments

@binshi-bing
Copy link
Contributor

binshi-bing commented Feb 2, 2023 •
edited
Loading

Feature Request

Describe your feature request related problem

The current TSO solution is known for its poor scalability and single point of failure. As we are moving to 'Cloud' providing DBaaS and Serverless Computing, supporting multi-tenant is one of primary goals which brings more requirements to the TSO service, including:

  1. Horizontal scalability across tenants.
  2. High resilience and reliability across tenants, i.e., reduce Blast Radius

Describe the feature you'd like

By sharding TSO service across tenants, we aim to achieve the following goals:

  1. Provide unlimited horizontal scalability across all tenants.
  2. In all failure scenarios, there is no single point failure for all the tenants.
  3. Provide no-degraded service during rolling update deployment and normal operations.
  4. Keep compatibility with the existing features.

Big Picture

image

  1. A TSO node can manage multiple tenant groups.
  2. Each tenant group is an independent TSO service instance.
  3. Each tenant group has multiple replicas which are placed in different failure domains to provide HA.
  4. The following utility tools will be provided from command console, control plane and even web admin UI.
    • The balancer -- manages the sharding state of the TSO service via etcd.
    • The cluster membership manager -- manages membership of the TSO nodes.
    • The tenant group membership manager -- manages the mapping of tenant -> tenant group.

For more details, please refer to the RFC (TODO: add link to the RFC).

Milestone 1 - (goal) deliver single-group tso microservice (Completed)

All work are tracked in #5836

Milestone 2 - (goal) code complete for multi-group tso microservice (4/21/2023) (Completed)

Milestone 3 - (goal) deploy multi-group tso microservice in dev env (5/5/2023) (Note: May 1st - 3rd, holiday)

Milestone 4 - (goal) deploy multi-group tso microservice in staging env (5/29/2023)

（Milestone 4 landing plan) timeline breakdown

Milestone 5 - (goal) deploy multi-group tso microservice in prod env (ETA: N/A) (the exact time will be decided by Serverless service team)

Describe alternatives you've considered

Please see the "The Alternative Architectures Considered" in the RFC (TODO: add link to the RFC).

Teachability, Documentation, Adoption, Migration Strategy
@binshi-bing binshi-bing added the type/feature-request Categorizes issue or PR as related to a new feature. label Feb 2, 2023
This was referenced Feb 2, 2023
Merged
Merged
ti-chi-bot added a commit that referenced this issue Feb 9, 2023
@rleungx @ti-chi-bot
pkg: support service discovery (#5911)
2be26ff 
ref #5895

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
@rleungx rleungx mentioned this issue Mar 2, 2023
Open
18 tasks
@binshi-bing binshi-bing changed the title Support Tenant TSO Support Multi-tenant and Sharded TSO Mar 6, 2023
@nolouch nolouch mentioned this issue Mar 10, 2023
Closed
@binshi-bing binshi-bing mentioned this issue Mar 17, 2023
Merged
@JmPotato JmPotato mentioned this issue Mar 20, 2023
Closed
@rleungx rleungx mentioned this issue Mar 21, 2023
Merged
@JmPotato JmPotato mentioned this issue Mar 21, 2023
Merged
ti-chi-bot added a commit that referenced this issue Mar 22, 2023
@JmPotato @ti-chi-bot
client: refine serviceModeKeeper code (#6201)
738e15f 
ref #5895

Some code refinements for `serviceModeKeeper`.

Signed-off-by: JmPotato <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
ti-chi-bot added a commit that referenced this issue Mar 22, 2023
@rleungx @ti-chi-bot
*: use txn for saving timestamp (#6199)
27b9474 
ref #5895

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
@rleungx rleungx mentioned this issue Mar 22, 2023
Merged
This was referenced Mar 22, 2023
Closed
Merged
@binshi-bing
Copy link
Contributor Author

binshi-bing commented Mar 23, 2023 •
edited
Loading

Meeting notes of TSO wg sync-up (attendees: @rleungx , @lhy1024 , @hnes , @binshi-bing ):

  1. M1 retrospective
  • (Yutong) The previous decision was to change the persistent path even for the existing tenants. Propose to use the same path for the existing tenants which passed the internal discussion.
  • (Yutong) Didn't plan TSO Requests forwarding (by PD/API) in time which became blocker for API mode service development.
  • (Bin) I overlooked server proxing, TSO Requests forwarding (by PD/API), used in the migration and didn't plan in time
  • (Bin) Underestimate the work of decoupling service from embedded etcd.
  1. M2 kick-off

  • (on-going) Stabilize TSO microservice (Everyone. @rleungx will devise the roadmap for Productization -- introduce and assign work for E2E testing, Correctness testing, Perf Testing, Chaos Testing and other necessary testings)

  • TSO Service

    • (on-going) Make server side change to support multi-tenant and sharding.@bin Shi

    • Support TSO requests forwarding by PD/API service. @lhy1024

    • Add Client-side discovery for monolithic-service/microservice configuration, availability and state. @binshi-bing

    • on-going) Add TSO Restful APIs

  • TSO Service Management (Utils/Restful APIs/API Implementation: TSO Nodes, Tenant Groups, Tenants)

    • Balancer (assign keyspace groups to tso nodes)
    • Tenant groups membership management @binshi-bing
      • Map all existing tenants to special keyspace group
      • Support default policy to distribute tenants (Round-robin)
      • Define the mapping from tenant to tenant group with placement rule and location affinity being taken into consideration. @bin Shi
    • TSO Cluster membership management @binshi-bing

ti-chi-bot added a commit that referenced this issue Mar 23, 2023
@rleungx @ti-chi-bot
pkg: remove txn lock (#6208)
769de12 
ref #5895

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
@binshi-bing
Copy link
Contributor Author

After internal sync within the PD TSO Working Group, here is the roadmap to land on TSO microservice:

  1. Mar. 22th
    The single-keyspace-group version (w/o sharding) is ready for baking in the dev environment.
  2. Apr. 3rd
    Code complete for the version which supports multiple keyspace groups and sharding.
    The PD client and Keyspace management API are ready for integration with the serverless tier.3.
  3. Apr. 17th
    The multi-keyspace-group version (w/ sharding) is ready for baking in the dev environment.

@binshi-bing binshi-bing mentioned this issue Mar 23, 2023
Merged
ti-chi-bot pushed a commit that referenced this issue Mar 23, 2023
@binshi-bing @lhy1024
Serialize tso stream creation (#6211)
c67e90a 
ref #5895

Fix the problem described above by serializing the tso stream creation.

Signed-off-by: Bin Shi <[email protected]>

Co-authored-by: lhy1024 <[email protected]>
ti-chi-bot pushed a commit that referenced this issue Mar 27, 2023
@binshi-bing
Refactor TSO forward/dispatcher to be shared by both PD and TSO (#6175)
d67bf27 
ref #5895

Add general tso forward/dispatcher for independent pd(tso)/tso services and cross cluster forwarding.

Signed-off-by: Bin Shi <[email protected]>
ti-chi-bot pushed a commit that referenced this issue Mar 27, 2023
@binshi-bing
Add basic multi-keyspace-group management (#6214)
fa18b03 
ref #5895

Support basic functions of multi-keyspace-group management

Signed-off-by: Bin Shi <[email protected]>
This was referenced Mar 27, 2023
Closed
Closed
Closed
@rleungx rleungx mentioned this issue Mar 31, 2023
Merged
ti-chi-bot added a commit that referenced this issue Apr 4, 2023
@rleungx @ti-chi-bot
server: fix watch keyspace revision (#6251)
da43376 
ref #5895

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
@JmPotato JmPotato mentioned this issue Apr 4, 2023
Merged
ti-chi-bot pushed a commit that referenced this issue Apr 6, 2023
@JmPotato
tso, server: refine the TSO allocator manager parameters (#6269)
33daef0 
ref #5895

- Refine the TSO allocator manager parameters.
- Always run `tsoAllocatorLoop` to advance the Global TSO.

Signed-off-by: JmPotato <[email protected]>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@JmPotato @ti-chi-bot @rleungx
keyspace: add benchmarks for keyspace assignment patrol (tikv#6507)
c044b37 
ref tikv#5895

Add benchmarks for keyspace assignment patrol.

Signed-off-by: JmPotato <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @ti-chi-bot @rleungx
Print keyspace-group-id zap field in the tso log conditionally (tikv#…
48ea7a9 
…6514)

ref tikv#5895

To keep the logging info in on-premises clean, we only print keyspace-group-id zap field
for the non-default keyspace group id.

Signed-off-by: Bin Shi <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @rleungx
Fix tso server close stuck issue (tikv#6529)
ff7d54d 
ref tikv#5895, close tikv#6304

Rewrite TSO gPRC/HTTP server Close().

Signed-off-by: Bin Shi <[email protected]>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @rleungx
mcs, tso: change keyspace group primary path. (tikv#6526)
13e3e93 
ref tikv#5895

mcs, tso: change keyspace group primary path.

The path for non-default keyspace group primary election changes
from  "/ms/{cluster_id}/tso/{group}/primary" to "/ms/{cluster_id}/tso/keyspace_groups/election/{group}/primary".
Default keyspace group keeps /ms/{cluster_id}/tso/00000/primary.

Signed-off-by: Bin Shi <[email protected]>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @rleungx
Add TestUpgradingAPIandTSOClusters (tikv#6534)
8dfd12b 
ref tikv#5895

Add TestUpgradingAPIandTSOClusters to test the scenario that after we restart the API cluster
then restart the TSO cluster, the TSO service can still serve TSO requests normally.

Signed-off-by: Bin Shi <[email protected]>
rleungx added a commit to rleungx/pd that referenced this issue Aug 2, 2023
@rleungx @ti-chi-bot
mcs: add a test for starting tso server first (tikv#6535)
4f59ce5 
ref tikv#5895

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx added a commit to rleungx/pd that referenced this issue Aug 2, 2023
@rleungx
mcs: support pprof for microservices (tikv#6541)
b087d1a 
ref tikv#5895

Signed-off-by: Ryan Leung <[email protected]>
rleungx added a commit to rleungx/pd that referenced this issue Aug 2, 2023
@rleungx
*: fix some log issues (tikv#6568)
c5f208e 
ref tikv#5895, ref tikv#6390

Signed-off-by: Ryan Leung <[email protected]>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @rleungx
Improve tso proxy reliability (tikv#6585)
87ad946 
ref tikv#5895

Improve tso proxy reliability.

1. Add protection mechanisms to TSO Proxy.
    a. Throttle the concurrency of TSO Proxy streamings. Default 5000.
    b. If TSO Proxy didn't receive the TSO request from the client for 1 hour, close the stream.
2. Optimize forceLoad lock with RW lock.
3. Enable stress test.
4. Add deadline for API leader forwarding request to TSO service.
5. Make tso response channel more safely.
6. Move tso proxy stress test away from the test suite as it has impact on other test cases.
7. Fix grpc client connection pool (server side) resource leak problem.
8. Make MaxConcurrentTSOProxyStreamings (5000 as default) and TSOProxyClientRecvTimeout (1 hour as default) configurable.
9. Add metrics tsoProxyHandleDuration, tsoProxyBatchSize and tsoProxyForwardTimeoutCounter.

Signed-off-by: Bin Shi <[email protected]>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @rleungx
Add keyspace and keyspace group info to the time fallback log. (tikv#…
b3121b6 
…6581)

ref tikv#5895

Add keyspace and keyspace group info to the time fallback log to help debugging time fallback issue in multi-timeline scenario.

Signed-off-by: Bin Shi <[email protected]>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @rleungx
config, server: fault injection in TSO Proxy (tikv#6588)
023ec12 
ref tikv#5895

Add failure test cases.

Signed-off-by: Bin Shi <[email protected]>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @rleungx
mcs, tso: fix potential inconsistency caused by non-atomic applying k…
b39a544 
…eyspace movement state change in the persistent store (tikv#6596)

ref tikv#5895

fix potential inconsistency caused by non-atomic applying the state change in the persistent in the following cases:
1. Keyspace group split/merge
2. Keyspace movement across keyspace groups.

Signed-off-by: Bin Shi <[email protected]>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @rleungx
Add keyspace group info in the timestamp fallback log in the client. (t…
4860a5b 
…ikv#6654)

ref tikv#5895

Add keyspace group info in the timestamp fallback log in the client.

Signed-off-by: Bin Shi <[email protected]>
rleungx added a commit to rleungx/pd that referenced this issue Aug 2, 2023
@rleungx @ti-chi-bot
mcs: add log for finishing split (tikv#6656)
58bd4b3 
ref tikv#5895

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@JmPotato @ti-chi-bot @rleungx
client: fix the keyspace ID RW race inside tsoServiceDiscovery (tikv#…
7e5f64a 
…6657)

ref tikv#5895

Fix the keyspace ID RW race inside `tsoServiceDiscovery`.

Signed-off-by: JmPotato <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @ti-chi-bot @rleungx
Add more debugging info to time fallback log. (tikv#6700)
f5c6c35 
ref tikv#5895

Add more debugging info to time fallback log.
[2023/06/27 10:50:54.196 -07:00] [PANIC] [tso_dispatcher.go:764] ["[tso] timestamp fallback"] 
[dc-location=global] [keyspace=4294967295]
[last-ts="(1687888254152, 1)"] [cur-ts="(1687888254052, 2)"] 
[last-tso-server=127.0.0.1:3380] [cur-tso-server=127.0.0.1:3380]
[last-keyspace-group-in-request=0] [cur-keyspace-group-in-request=0] 
[last-keyspace-group-in-response=0] [cur-keyspace-group-in-response=0] 
[last-response-received-at=2023/06/27 10:50:54.195 -07:00]
[cur-response-received-at=2023/06/27 10:50:54.196 -07:00]

Signed-off-by: Bin Shi <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx added a commit to rleungx/pd that referenced this issue Aug 2, 2023
@rleungx @ti-chi-bot
tools: support querying keyspace group by state (tikv#6706)
58b87f2 
ref tikv#5895

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx added a commit to rleungx/pd that referenced this issue Aug 2, 2023
@rleungx
tools: support get all groups (tikv#6714)
cf7b547 
ref tikv#5895, ref tikv#6706

Signed-off-by: Ryan Leung <[email protected]>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@JmPotato @ti-chi-bot @rleungx
tso: implement groupSplitPatroller to speed up the split process (tik…
3d8f8d1 
…v#6736)

ref tikv#5895, close tikv#6696

Implement `groupSplitPatroller` to speed up the split process.

Signed-off-by: JmPotato <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @ti-chi-bot @rleungx
tso: support multi-keyspace, fault injection and keyspace-name in pd-…
cf0240c 
…tso-bench (tikv#6608)

ref tikv#5895

support multi-keyspace, fault injection and keyspace-name in pd-tso-bench

Signed-off-by: Bin Shi <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@lhy1024 @ti-chi-bot @rleungx
*: move keyspace group primary path code to key_path.go (tikv#6755)
5aa7748 
ref tikv#5895

Signed-off-by: lhy1024 <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @rleungx
Add keyspace group info in the timestamp fallback log in the client. (t…
35785c3 
…ikv#6654)

ref tikv#5895

Add keyspace group info in the timestamp fallback log in the client.

Signed-off-by: Bin Shi <[email protected]>
rleungx added a commit to rleungx/pd that referenced this issue Aug 2, 2023
@rleungx @ti-chi-bot
mcs: add log for finishing split (tikv#6656)
f14794d 
ref tikv#5895

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@JmPotato @ti-chi-bot @rleungx
client: fix the keyspace ID RW race inside tsoServiceDiscovery (tikv#…
02ee803 
…6657)

ref tikv#5895

Fix the keyspace ID RW race inside `tsoServiceDiscovery`.

Signed-off-by: JmPotato <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @ti-chi-bot @rleungx
Add more debugging info to time fallback log. (tikv#6700)
902fc32 
ref tikv#5895

Add more debugging info to time fallback log.
[2023/06/27 10:50:54.196 -07:00] [PANIC] [tso_dispatcher.go:764] ["[tso] timestamp fallback"] 
[dc-location=global] [keyspace=4294967295]
[last-ts="(1687888254152, 1)"] [cur-ts="(1687888254052, 2)"] 
[last-tso-server=127.0.0.1:3380] [cur-tso-server=127.0.0.1:3380]
[last-keyspace-group-in-request=0] [cur-keyspace-group-in-request=0] 
[last-keyspace-group-in-response=0] [cur-keyspace-group-in-response=0] 
[last-response-received-at=2023/06/27 10:50:54.195 -07:00]
[cur-response-received-at=2023/06/27 10:50:54.196 -07:00]

Signed-off-by: Bin Shi <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx added a commit to rleungx/pd that referenced this issue Aug 2, 2023
@rleungx @ti-chi-bot
tools: support querying keyspace group by state (tikv#6706)
7c6cc4f 
ref tikv#5895

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx added a commit to rleungx/pd that referenced this issue Aug 2, 2023
@rleungx
tools: support get all groups (tikv#6714)
2f5eccb 
ref tikv#5895, ref tikv#6706

Signed-off-by: Ryan Leung <[email protected]>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@JmPotato @ti-chi-bot @rleungx
tso: implement groupSplitPatroller to speed up the split process (tik…
837661b 
…v#6736)

ref tikv#5895, close tikv#6696

Implement `groupSplitPatroller` to speed up the split process.

Signed-off-by: JmPotato <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @ti-chi-bot @rleungx
tso: support multi-keyspace, fault injection and keyspace-name in pd-…
dbcf74b 
…tso-bench (tikv#6608)

ref tikv#5895

support multi-keyspace, fault injection and keyspace-name in pd-tso-bench

Signed-off-by: Bin Shi <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@JmPotato JmPotato mentioned this issue Aug 30, 2023
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@binshi-bing binshi-bing

Labels
type/feature-request Categorizes issue or PR as related to a new feature.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@binshi-bing