Skip to content
This repository has been archived by the owner on Jun 23, 2022. It is now read-only.

feat(bulk-load): bulk load download part4 - replica report download status and progress #479

Merged
merged 2 commits into from
May 29, 2020

Conversation

hycdong
Copy link
Contributor

@hycdong hycdong commented May 28, 2020

The whole bulk load download process is like:

  1. meta server set app and all partitions bulk load status is downloading feat(bulk-load): meta server start bulk load #454
  2. meta server send request to primary to download files feat(bulk-load): meta server send bulk load request #457
  3. primary download files feat(bulk-load): bulk load download part1 - replica start download #465 feat(bulk-load): bulk load download part2 - replica download files #471 feat(bulk-load): bulk load download part3 - replica parse metadata, verify files and update progress #475
  4. primary broadcast request to secondary feat(bulk-load): group bulk load request in replica group #460
  5. secondary download files feat(bulk-load): bulk load download part1 - replica start download #465 feat(bulk-load): bulk load download part2 - replica download files #471 feat(bulk-load): bulk load download part3 - replica parse metadata, verify files and update progress #475
  6. secondary report download progress and status to primary
  7. primary report group download progress and status to meta server
  8. meta receive download progress
    • if all replicas of the group finish download all files, meta set this partition status as downloaded, and if all partitions are downloaded, app status will be set as downloaded, otherwise, meta will resend request to primary

This pull request is about replica report download status and progress, step 6 and step 7 above.

@hycdong hycdong changed the title feat(bulk-load): bulk load download part4 - replica report download progress feat(bulk-load): bulk load download part4 - replica report download status and progress May 28, 2020
@hycdong hycdong marked this pull request as ready for review May 28, 2020 06:25
neverchanje
neverchanje previously approved these changes May 28, 2020
src/dist/replication/lib/bulk_load/replica_bulk_loader.cpp Outdated Show resolved Hide resolved
src/dist/replication/lib/bulk_load/replica_bulk_loader.cpp Outdated Show resolved Hide resolved
}

total_progress /= _replica->_primary_states.membership.max_replica_count;
ddebug_replica("total download progress = {}%", total_progress);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about use float type?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

total_progress will report to meta server in bulk_load_response, defined in thrift, it seems that we don't use float in thrift before, and I think it is okay to define it as integer.

Comment on lines +502 to +505
case bulk_load_status::BLS_DOWNLOADING:
case bulk_load_status::BLS_DOWNLOADED:
bulk_load_state.__set_download_progress(_download_progress.load());
bulk_load_state.__set_download_status(_download_status.load());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We always report the two type of status, so why separate to two types?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean that why not combine download progress and status into one structure? I used to define a structure called download_states before, but I finally separate it, it will make code more complex, besides, download_progress and download_status is more clearer than download_states.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants