Implement parallelism for the distributedLoad command #9895

liuhongtong · 2019-09-16T07:55:30Z

distributedLoad traverses the specified path, distributes a load job of a file to job master and waits the job completed one by one, that is a serial process. So the performance may be not acceptable.
Now distributedLoad traverses the specified path, distributes a batch load job of files to job master and distributes a new job if one job completed, that is a intercurrent process.

Signed-off-by: liuhongtong [email protected]

Fix #9791

Signed-off-by: liuhongtong <[email protected]>

alluxio-bot · 2019-09-16T07:55:35Z

Automated checks report:

AmplabJenkins build check: PENDING
- We were not able to detect AmplabJenkins test results on this PR. Status will update when testing completes.
Commits associated with Github account: PASS
PR title follows the conventions: PASS

Some checks failed. Please fix the reported issues and reply 'alluxio-bot, check this please' to re-run checks.

gpang · 2019-09-16T17:59:09Z

@liuhongtong Thanks for this improvement! Could you provide some more information in the PR description? Thanks!

AmplabJenkins · 2019-09-16T18:59:20Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-09-16T18:59:22Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/5532/
Test FAILed.

calvinjia

@liuhongtong Thanks for the substantial improvement to this call. Do you mind taking a look at PersistCommand and seeing if that pattern can apply to distributedload?

Unrelated to this PR - We have a few implementations of multi-threaded CLI and it would be good to consolidate them into a single framework.

shell/src/main/java/alluxio/cli/fs/command/DistributedLoadCommand.java

apc999 · 2019-09-17T06:28:33Z

shell/src/main/java/alluxio/cli/fs/command/DistributedLoadCommand.java

+          .hasArg(true)
+          .desc("number of replicas to have for each block of the loaded file")
+          .build();
+  public static final Option THREAD_OPTION =


nit: THREADS_OPTION to be consistent

Do you mean const value like C language?

apc999 · 2019-09-17T06:28:46Z

shell/src/main/java/alluxio/cli/fs/command/DistributedLoadCommand.java

@@ -35,7 +43,28 @@
 */
 @ThreadSafe
 public final class DistributedLoadCommand extends AbstractFileSystemCommand {
-  private static final String REPLICATION = "replication";
+  public static final Option REPLICATION =


nit: REPLCIATION_OPTION to be consistent

shell/src/main/java/alluxio/cli/fs/command/DistributedLoadCommand.java

liuhongtong · 2019-09-17T06:42:59Z

@gpang distributedLoad traverses the specified path, distributes a load job of a file to job master and waits the job completed one by one, that is a serial process. So the performance may be not acceptable.
Now distributedLoad traverses the specified path, distributes a batch load job of files to job master and distributes a new job if one job completed, that is a intercurrent process.

Signed-off-by: liuhongtong <[email protected]>

liuhongtong · 2019-09-17T08:37:02Z

Loading 5TiB data from a big HDFS cluster with 30 Alluxio worker & job worker.
While there is no other I/O in Alluxio.
NIC: 10000Mb/s

load test

date; alluxio fs load /ns1_lht_test_alluxio; date
Sun Aug 25 17:13:07 CST 2019
Aug 26, 2019 7:18:42 AM
about 14h 5m

distributedLoad test

date; alluxio fs distributedLoad /ns1_lht_test_alluxio; date
Mon Aug 26 08:57:36 CST 2019
Mon Aug 26 18:07:15 CST 2019
about 9h

new distributedLoad test

default (128 threads)
time alluxio fs distributedLoad /ns1_lht_test_alluxio/
real	5m11.984s

256 threads
time alluxio fs distributedLoad /ns1_lht_test_alluxio/ -thread 256
real	4m29.423s

512 threads
time alluxio fs distributedLoad /ns1_lht_test_alluxio/ -thread 512
real	4m7.610s

AmplabJenkins · 2019-09-17T08:42:25Z

Merged build finished. Test PASSed.

AmplabJenkins · 2019-09-17T08:42:26Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/5554/
Test PASSed.

liuhongtong · 2019-09-17T08:50:11Z

@apc999 @calvinjia updated. PTAL. Thanks.

alluxio-bot · 2019-09-17T11:58:28Z

Automated checks report:

AmplabJenkins build check: PASS
Commits associated with Github account: PASS
PR title follows the conventions: PASS

All checks passed!

calvinjia · 2019-09-17T18:30:45Z

@liuhongtong Could you open a github issue for consolidating optionally multi-threaded CLIs to a general framework?

liuhongtong · 2019-09-18T02:37:54Z

@calvinjia OK. I would like to open a new issue and consolidate a general framework for multi-threaded CLIs.

Signed-off-by: liuhongtong <[email protected]>

liuhongtong · 2019-09-18T03:26:05Z

New issue: #9905

AmplabJenkins · 2019-09-18T03:39:19Z

Merged build finished. Test PASSed.

AmplabJenkins · 2019-09-18T03:39:20Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/5576/
Test PASSed.

Signed-off-by: liuhongtong <[email protected]>

AmplabJenkins · 2019-09-18T08:15:48Z

Merged build finished. Test PASSed.

AmplabJenkins · 2019-09-18T08:15:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/5577/
Test PASSed.

AmplabJenkins · 2019-09-18T17:47:29Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-09-18T17:47:30Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/5582/
Test FAILed.

AmplabJenkins · 2019-09-19T02:47:48Z

Merged build finished. Test PASSed.

AmplabJenkins · 2019-09-19T02:47:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/5601/
Test PASSed.

apc999 · 2019-09-20T06:59:24Z

alluxio-bot, merge this please

liuhongtong · 2019-09-22T04:12:07Z

@apc999 Thanks for merging this pr.

distributedLoad traverses the specified path, distributes a load job of a file to job master and waits the job completed one by one, that is a serial process. So the performance may be not acceptable. Now distributedLoad traverses the specified path, distributes a batch load job of files to job master and distributes a new job if one job completed, that is a intercurrent process. Signed-off-by: liuhongtong <[email protected]> Fix Alluxio#9791 pr-link: Alluxio#9895 change-id: cid-f7ece37fda7dd3bd2a6c38783b77edc410e22fab

Optimize distributedLoad Alluxio#9791

9b5ed4d

Signed-off-by: liuhongtong <[email protected]>

apc999 changed the title ~~Optimize distributedLoad #9791~~ Optimize performance of distributedLoad Sep 16, 2019

apc999 self-requested a review September 16, 2019 17:46

apc999 requested a review from calvinjia September 16, 2019 23:01

calvinjia requested changes Sep 16, 2019

View reviewed changes

apc999 requested changes Sep 17, 2019

View reviewed changes

update after first review

84faeb4

Signed-off-by: liuhongtong <[email protected]>

liuhongtong changed the title ~~Optimize performance of distributedLoad~~ Optimize performance of distributedLoad #9791 Sep 17, 2019

alluxio-bot assigned LuQQiu Sep 17, 2019

gpang changed the title ~~Optimize performance of distributedLoad #9791~~ Implement parallelism for the distributedLoad command Sep 17, 2019

calvinjia approved these changes Sep 17, 2019

View reviewed changes

modify error output

e80f594

Signed-off-by: liuhongtong <[email protected]>

decrease getStatus

9fc9081

Signed-off-by: liuhongtong <[email protected]>

Update DistributedLoadCommand.java

800669b

Update DistributedLoadCommand.java

cc68732

apc999 self-requested a review September 20, 2019 06:58

apc999 approved these changes Sep 20, 2019

View reviewed changes

alluxio-bot merged commit f5b70fd into Alluxio:master Sep 20, 2019

bf8086 mentioned this pull request Oct 1, 2019

Implement parallelism for the distributedLoad command apc999/alluxio#3

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement parallelism for the distributedLoad command #9895

Implement parallelism for the distributedLoad command #9895

liuhongtong commented Sep 16, 2019 •

edited

Loading

alluxio-bot commented Sep 16, 2019

gpang commented Sep 16, 2019

AmplabJenkins commented Sep 16, 2019

AmplabJenkins commented Sep 16, 2019

calvinjia left a comment

apc999 Sep 17, 2019

liuhongtong Sep 17, 2019

apc999 Sep 17, 2019

liuhongtong commented Sep 17, 2019

liuhongtong commented Sep 17, 2019 •

edited

Loading

AmplabJenkins commented Sep 17, 2019

AmplabJenkins commented Sep 17, 2019

liuhongtong commented Sep 17, 2019

alluxio-bot commented Sep 17, 2019

calvinjia commented Sep 17, 2019

liuhongtong commented Sep 18, 2019

liuhongtong commented Sep 18, 2019

AmplabJenkins commented Sep 18, 2019

AmplabJenkins commented Sep 18, 2019

AmplabJenkins commented Sep 18, 2019

AmplabJenkins commented Sep 18, 2019

AmplabJenkins commented Sep 18, 2019

AmplabJenkins commented Sep 18, 2019

AmplabJenkins commented Sep 19, 2019

AmplabJenkins commented Sep 19, 2019

apc999 commented Sep 20, 2019

liuhongtong commented Sep 22, 2019

Implement parallelism for the distributedLoad command #9895

Implement parallelism for the distributedLoad command #9895

Conversation

liuhongtong commented Sep 16, 2019 • edited Loading

alluxio-bot commented Sep 16, 2019

gpang commented Sep 16, 2019

AmplabJenkins commented Sep 16, 2019

AmplabJenkins commented Sep 16, 2019

calvinjia left a comment

Choose a reason for hiding this comment

apc999 Sep 17, 2019

Choose a reason for hiding this comment

liuhongtong Sep 17, 2019

Choose a reason for hiding this comment

apc999 Sep 17, 2019

Choose a reason for hiding this comment

liuhongtong commented Sep 17, 2019

liuhongtong commented Sep 17, 2019 • edited Loading

AmplabJenkins commented Sep 17, 2019

AmplabJenkins commented Sep 17, 2019

liuhongtong commented Sep 17, 2019

alluxio-bot commented Sep 17, 2019

calvinjia commented Sep 17, 2019

liuhongtong commented Sep 18, 2019

liuhongtong commented Sep 18, 2019

AmplabJenkins commented Sep 18, 2019

AmplabJenkins commented Sep 18, 2019

AmplabJenkins commented Sep 18, 2019

AmplabJenkins commented Sep 18, 2019

AmplabJenkins commented Sep 18, 2019

AmplabJenkins commented Sep 18, 2019

AmplabJenkins commented Sep 19, 2019

AmplabJenkins commented Sep 19, 2019

apc999 commented Sep 20, 2019

liuhongtong commented Sep 22, 2019

liuhongtong commented Sep 16, 2019 •

edited

Loading

liuhongtong commented Sep 17, 2019 •

edited

Loading