[HUDI-7645] Optimize BQ sync tool for MDT #11065
Conversation
There was a problem hiding this comment.
@wombatu-kun : Please take a look at the comment and let me know your thoughts.
| Stream<HoodieBaseFile> allLatestBaseFiles; | ||
| if (useFileListingFromMetadata) { | ||
| LOG.info("Fetching all base files from MDT."); | ||
| allLatestBaseFiles = fsView.getLatestBaseFiles(); |
There was a problem hiding this comment.
Looks like fsView.getLatestBaseFiles() only returns already loaded file-groups in the view so some partitions may not be loaded at all. Can you check if this is your intended behavior ?
There was a problem hiding this comment.
Yes, you are right, fsView.getLatestBaseFiles() only returns already loaded file-groups in the view. But I don't see any other approach to load all latest files in one call to HoodieMetadataFileSystemView/HoodieTableMetadata. It would be great if you or @nsivabalan (as reporter of this task) give me some advice.
There was a problem hiding this comment.
i've found fsView.loadAllPartitions() to load all partitions in one call, and now all file-groups are loaded in the view before getting latest base files by fsView.getLatestBaseFiles().
wombatu-kun
force-pushed
the
HUDI-7645_Optimize_BQ_sync_tool_for_MDT
branch
from
d417a41
to
9464340
Compare
| Stream<HoodieBaseFile> allLatestBaseFiles; | ||
| if (useFileListingFromMetadata) { | ||
| LOG.info("Fetching all base files from MDT."); | ||
| allLatestBaseFiles = fsView.getLatestBaseFiles(); |
Change Logs
Looks like in BQ sync, we are polling fsview for latest files sequentially for every partition.
When MDT is enabled, we could load all partitions in one call.
Impact
none
Risk level (write none, low medium or high below)
none
Documentation Update
none
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist