Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Low performance of distributedLoad #9791

Closed
liuhongtong opened this issue Aug 26, 2019 · 2 comments · Fixed by #9895
Closed

Low performance of distributedLoad #9791

liuhongtong opened this issue Aug 26, 2019 · 2 comments · Fixed by #9895
Labels
needs-info waiting on issue opener to respond priority-medium type-bug This issue is about a bug

Comments

@liuhongtong
Copy link
Contributor

liuhongtong commented Aug 26, 2019 •
edited by apc999
Loading

Alluxio Version:
2.0.0

Describe the bug
I have contrasted the performance of load and distributedLoad with 5TiB data. The size of file is 600MiB. The cluster has 30 nodes

\# date; alluxio fs load /ns1_lht_test_alluxio; date
Sun Aug 25 17:13:07 CST 2019
......
Aug 26, 2019 7:18:42 AM

It takes 14 hours and 5 minutes.

\# date; alluxio fs distributedLoad /ns1_lht_test_alluxio; date
Mon Aug 26 08:57:36 CST 2019
......
Mon Aug 26 18:07:15 CST 2019

It takes 9 hours and 10 minutes.

There is no enormous improvement for distributedLoad.
If this is a bug or any optimized configuration is available?

To Reproduce
refer the section above

Expected behavior
distributedLoad can support high performance.

Urgency
N/A
@liuhongtong liuhongtong added the type-feature This issue is a feature request label Aug 26, 2019
@apc999 apc999 added type-bug This issue is about a bug and removed type-feature This issue is a feature request labels Aug 26, 2019
@apc999 apc999 changed the title How to improve the performance of distributedLoad Low performance of distributedLoad Aug 27, 2019
@calvinjia calvinjia added the needs-info waiting on issue opener to respond label Aug 29, 2019
@calvinjia
Copy link
Contributor

@liuhongtong Are you loading multiple 600MB files for a total of 5TB data? Also, can you verify if all the nodes are being utilized (you can generally tell by all the workers having Alluxio space used).

@apc999
Copy link
Contributor

apc999 commented Sep 11, 2019

@calvinjia 5TB data in total, each file is 600MB.
@liuhongtong can you confirm the second question from @calvinjia ?

liuhongtong added a commit to liuhongtong/alluxio that referenced this issue Sep 16, 2019
@liuhongtong
Optimize distributedLoad Alluxio#9791
9b5ed4d 
Signed-off-by: liuhongtong <[email protected]>
@liuhongtong liuhongtong mentioned this issue Sep 16, 2019
Merged
alluxio-bot pushed a commit that referenced this issue Sep 20, 2019
@liuhongtong @alluxio-bot
Implement parallelism for the distributedLoad command
f5b70fd 
distributedLoad traverses the specified path, distributes a load job of
a file to job master and waits the job completed one by one, that is a
serial process. So the performance may be not acceptable.
Now distributedLoad traverses the specified path, distributes a batch
load job of files to job master and distributes a new job if one job
completed, that is a intercurrent process.

Signed-off-by: liuhongtong <[email protected]>

Fix #9791

pr-link: #9895
change-id: cid-f7ece37fda7dd3bd2a6c38783b77edc410e22fab
bf8086 pushed a commit to bf8086/alluxio that referenced this issue Oct 1, 2019
@liuhongtong @bf8086
Implement parallelism for the distributedLoad command
3d43b67 
distributedLoad traverses the specified path, distributes a load job of
a file to job master and waits the job completed one by one, that is a
serial process. So the performance may be not acceptable.
Now distributedLoad traverses the specified path, distributes a batch
load job of files to job master and distributes a new job if one job
completed, that is a intercurrent process.

Signed-off-by: liuhongtong <[email protected]>

Fix Alluxio#9791

pr-link: Alluxio#9895
change-id: cid-f7ece37fda7dd3bd2a6c38783b77edc410e22fab
@bf8086 bf8086 mentioned this issue Oct 1, 2019
Merged
apc999 pushed a commit to apc999/alluxio that referenced this issue Oct 2, 2019
@liuhongtong @apc999
Implement parallelism for the distributedLoad command
5ab45e3 
distributedLoad traverses the specified path, distributes a load job of
a file to job master and waits the job completed one by one, that is a
serial process. So the performance may be not acceptable.
Now distributedLoad traverses the specified path, distributes a batch
load job of files to job master and distributes a new job if one job
completed, that is a intercurrent process.

Signed-off-by: liuhongtong <[email protected]>

Fix Alluxio#9791

pr-link: Alluxio#9895
change-id: cid-f7ece37fda7dd3bd2a6c38783b77edc410e22fab
apc999 pushed a commit to apc999/alluxio that referenced this issue Oct 2, 2019
@liuhongtong @apc999
Implement parallelism for the distributedLoad command
9b9c858 
distributedLoad traverses the specified path, distributes a load job of
a file to job master and waits the job completed one by one, that is a
serial process. So the performance may be not acceptable.
Now distributedLoad traverses the specified path, distributes a batch
load job of files to job master and distributes a new job if one job
completed, that is a intercurrent process.

Signed-off-by: liuhongtong <[email protected]>

Fix Alluxio#9791

pr-link: Alluxio#9895
change-id: cid-f7ece37fda7dd3bd2a6c38783b77edc410e22fab
apc999 pushed a commit to apc999/alluxio that referenced this issue Oct 23, 2019
@liuhongtong @apc999
Implement parallelism for the distributedLoad command
201424a 
distributedLoad traverses the specified path, distributes a load job of
a file to job master and waits the job completed one by one, that is a
serial process. So the performance may be not acceptable.
Now distributedLoad traverses the specified path, distributes a batch
load job of files to job master and distributes a new job if one job
completed, that is a intercurrent process.

Signed-off-by: liuhongtong <[email protected]>

Fix Alluxio#9791

pr-link: Alluxio#9895
change-id: cid-f7ece37fda7dd3bd2a6c38783b77edc410e22fab
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
needs-info waiting on issue opener to respond priority-medium type-bug This issue is about a bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Implement parallelism for the distributedLoad command liuhongtong/alluxio
3 participants
@apc999 @calvinjia @liuhongtong