[SPARK-938][doc] Add OpenStack Swift support #2298
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is initial documentation describing how to integrate Spark with Swift. This commit contains documentation for stand alone cluster. Next patches will contain details how to integrate Swift in other deployment of Spark.
Documentation how to integrate Spark with Openstack Swift.
Conflicts: docs/openstack-integration.md
Conflicts: docs/openstack-integration.md
SPARK-938 - Openstack Swift object storage support
Bring the branch up to date and fixed some documentation typos.
SPARK-938 - Openstack Swift object storage support
|
QA tests have started for PR 2298 at commit
|
| Spark's support for Hadoop InputFormat allows it to process data in OpenStack Swift using the | ||
| same URI formats as in Hadoop. You can specify a path in Swift as input through a | ||
| URI of the form <code>swift://container.PROVIDER/path</code>. You will also need to set your | ||
| Swift security credentials, through <code>core-sites.xml</code> or via |
|
QA tests have finished for PR 2298 at commit
|
|
|
||
| # Configuration Parameters | ||
|
|
||
| Create <code>core-site.xml</code> and place it inside <code>/spark/conf</code> directory. |
There was a problem hiding this comment.
minor - but could this say conf/ rather than /spark/conf... we usually don't assume the name of the root folder (in many cases it e.g. has a version)
|
Great thanks LGTM Reynold. I'll pull this in. |
asfgit
pushed a commit
that referenced
this pull request
Sep 8, 2014
See compiled doc at http://people.apache.org/~rxin/tmp/openstack-swift/_site/storage-openstack-swift.html This is based on #1010. Closes #1010. Author: Reynold Xin <[email protected]> Author: Gil Vernik <[email protected]> Closes #2298 from rxin/openstack-swift and squashes the following commits: ff4e394 [Reynold Xin] Two minor comments from Patrick. 279f6de [Reynold Xin] core-sites -> core-site dfb8fea [Reynold Xin] Updated based on Gil's suggestion. 846f5cb [Reynold Xin] Added a link from overview page. 0447c9f [Reynold Xin] Removed sample code. e9c3761 [Reynold Xin] Merge pull request #1010 from gilv/master 9233fef [Gil Vernik] Fixed typos 6994827 [Gil Vernik] Merge pull request #1 from rxin/openstack ac0679e [Reynold Xin] Fixed an unclosed tr. 47ce99d [Reynold Xin] Merge branch 'master' into openstack cca7192 [Gil Vernik] Removed white spases from pom.xml 99f095d [Reynold Xin] Pending openstack changes. eb22295 [Reynold Xin] Merge pull request #1010 from gilv/master 39a9737 [Gil Vernik] Spark integration with Openstack Swift c977658 [Gil Vernik] Merge branch 'master' of https://github.com/gilv/spark 2aba763 [Gil Vernik] Fix to docs/openstack-integration.md 9b625b5 [Gil Vernik] Merge branch 'master' of https://github.com/gilv/spark eff538d [Gil Vernik] SPARK-938 - Openstack Swift object storage support ce483d7 [Gil Vernik] SPARK-938 - Openstack Swift object storage support b6c37ef [Gil Vernik] Openstack Swift support (cherry picked from commit eddfedd) Signed-off-by: Patrick Wendell <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See compiled doc at
http://people.apache.org/~rxin/tmp/openstack-swift/_site/storage-openstack-swift.html
This is based on #1010. Closes #1010.