Fault Tolerance for Regions in distribute mode

[MichaelScofield]

What problem does the new feature solve?

Fault tolerance for regions is a critical feature for high availability(HA) in distribute mode, which we currently lack of.

This issue tracks the sub-tasks that gradually achieve region HA in distribute mode.

What does the feature do?

Sub-tasks:

• RFC: docs: region failover RFC #1139 : rendered

WAL:

• Remote WAL: [Proposal] Distributed write-ahead logging #626 (Not planned here. Made some compromise on strict data reliability.)

In Datanode:

• open/close region (table) upon receiving heartbeat instructions from Meta

  • heartbeat response handler feat(datanode): iImplement the heartbeat response handler #1547
  • open region feat: implement OpenTableHandler #1567
  • close region feat: implement CloseRegionHandler #1569

• Local global WAL for testing purpose.

In Meta:

• failure detector feat: phi accrual failure detector #1200
• failover executor with distributed procedure feat: region failover procedure #1558

In Frontend:

• heartbeat mechanism to deal with region failover Establish a heartbeat mechanism between meta and frontend #1189
  • refactor: move the common part of the heartbeat response handler to common #1627
  • feat: add invalidate table cache handler #1633
  • feat: add invalidate cache step #1658
• properly handles read/write to accommodate with region failover
  • test: add write test for region failover #1673

Testing:

• with madsim
• with argo workflow

Implementation challenges

No response

[killme2008]

Can we write an RFC for this feature? It's really important.

[MichaelScofield]

will do

…

2023年3月6日 11:00，dennis zhuang ***@***.***> 写道： Can we write an RFC for this feature? It's really important. — Reply to this email directly, view it on GitHub <#1126 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHR2D7CQ2KMX5ZTGTDYO3DW2VHOHANCNFSM6AAAAAAVQRS5T4>. You are receiving this because you were assigned.

[MichaelScofield]

testing (chaos mesh, argo workflow, operator etc) will be tracked in new issues