Skip to content
This repository has been archived by the owner on Jun 23, 2022. It is now read-only.

feat(bulk-load): cancel bulk load #531

Merged
merged 2 commits into from
Jul 6, 2020
Merged

Conversation

hycdong
Copy link
Contributor

@hycdong hycdong commented Jul 6, 2020

This pull request is about cancel bulk load by client.

  1. client send cancel/force_cancel request to meta server feat(bulk-load): pause bulk load part1 - add control_bulk_load rpc and start to pause bulk load #514
  2. meta server handle cancel bulk load
    • cancel bulk load is only allowed when current bulk load status is downloading or paused, while force bulk load has no limit, however, force bulk load may lead to consistency problem, some data may have ingested while some data may not (in function on_control_bulk_load)
    • update app bulk load status to canceled (in function update_app_status_on_remote_storage_reply)
    • update each partition status to canceled, and if old status is paused, should set should_send_request = true to resend bulk_load_request, for other status, request will be resent in function on_partition_bulk_load_reply feat(bulk-load): bulk load download part5 - meta handle download status and progress #482
  3. replica server receive bulk_load_request during cancel
    • when replica receive request whose status is canceled, will call function handle_bulk_load_finish to cleanup bulk load (in function do_bulk_load)
    • secondary reports cleaned up flag to primary (in function report_bulk_load_states_to_primary)
    • primary reports group cleaned up flag to meta server (in function report_bulk_load_states_to_meta)
  4. meta server handle bulk_load_response during cancel

Special explanations:

  • It is possible for is_group_bulk_load_context_cleaned_up is not set in function handle_bulk_load_finish, the possible condition:
    • app and partition bulk load status is downloading, meta send request to replica, replica server report download progress to meta server, at this time, bulk load is canceled, app bulk load status has been updated to cancel, meta will call function handle_bulk_load_finish, and is_group_bulk_load_context_cleaned_up flag is not set in this response, meta will do nothing with this response, and resend bulk_load_request to replica

As a result, I update related code in function handle_bulk_load_finish.

@hycdong hycdong marked this pull request as ready for review July 6, 2020 04:48
@acelyc111 acelyc111 merged commit 60761bb into XiaoMi:master Jul 6, 2020
@hycdong hycdong deleted the cancel_bulk_load branch July 7, 2020 01:15
hycdong added a commit to hycdong/rdsn that referenced this pull request Jul 8, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants