[SPARK-1957] [WIP] Pluggable Diskstore for BlockManager

[colorant]

No description provided.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15271/

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[witgo]

@colorant
This is a big changes. Can you explain this change reason?

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15274/

[colorant]

Hi @witgo This is aim to aligned with the spark pluggable storage roadmap. say support different storage media across different existing / emerging hardware. ( for now, say you could put data on SSD for performance, and would fall back to HDD when SSD is full etc. )

And talked with @pwendell a little bit on this before. He suggested that they (Databricks) will come up with a API redesign for pluggable storage. At the same time we can try to add a disk store layer firstly to verify some ideas.

[colorant]

Hi, jira here:

https://issues.apache.org/jira/browse/SPARK-1957

[colorant]

And the implementation here is try to minimize the impact to current spark code base. Especially the shuffle part of code as described in the jira. which might be improved later together with shuffle framework itself. currently, Just try to make the wheel runs.

And there are improvements needed on the quota control part of logic. Say enhance the quota control to make it work on path/disk bases instead of store bases as a whole to reduce the chance of out of disk space issue ( though as whole the store's quotation is not overrun yet) etc. And would benefit from catching varous out of disk space exception etc.

However, if the disk space is not a issue. I think this one just works fine. And I also have tried to make sure that if not extra config is done on diskstore. the general data flow path is almost identical as the current approaching, overhead is neglec table.

[colorant]

@pwendell would you mind take look on this one ;)

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15748/

[colorant]

Hi @andrewor14 , other than #1209 , also this one is related to BM, could you also take a look on the general idea ? I know the code need a rebase to the latest code, but I am seek for a general feedback about the ideas ;)

[andrewor14]

@colorant Can you file a JIRA to describe what your'e trying to achieve here? Also, how is this related to #1209? If this is still relevant, can you up merge this to master and resolve the conflicts?

[colorant]

@andrewor14, JIRA ticket is: https://issues.apache.org/jira/browse/SPARK-1957. And #1209 is about to make the data format more general for BM. which also modified a lot of files related in this PR. While, yes, since rxin is pushing ManagedBuffer for data transfer, That PR might not be necessary, I will close that one if this is true.

[andrewor14]

Can you add it to the title?

[pwendell]

Thit is useful as a prototype, I'd prefer to close this issue as an active review. We can use this as a starting point if we revisit the internal interfaces here.

[liyezhang556520]

Hi @pwendell , We (I and @colorant) are still following this issue, it will be nice for spark to support the pluggable storage, since it can take advantage of the total memory hierarchy (2LM, NVM, SSD, HDD, etc.). @colorant talked with you about API design for pluggable storage, and we'd like to know your point of view on this, and what we can do to keep this going on? And we can make a discussion on JIRA. Thanks.