Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add example that reads a local file, writes to a DFS path provided by th... #3347

Closed
wants to merge 12 commits into from
Closed

Add example that reads a local file, writes to a DFS path provided by th... #3347

wants to merge 12 commits into from

Conversation

rnowling
Copy link
Contributor

...e user, reads the file back from the DFS, and compares word counts on the local and DFS versions. Useful for verifying DFS correctness.

@rnowling
Add example that reads a local file, writes to a DFS path provided by…
0c89558 
… the user, reads the file back from the DFS, and compares word counts on the local and DFS versions. Useful for verifying DFS correctness.
@SparkQA
Copy link

SparkQA commented Nov 18, 2014

Test build #23561 has started for PR 3347 at commit 0c89558.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 18, 2014

Test build #23561 has finished for PR 3347 at commit 0c89558.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23561/
Test PASSed.

andrewor14
andrewor14 reviewed Nov 19, 2014
View reviewed changes
examples/src/main/scala/org/apache/spark/examples/DFSReadWriteTest.scala
import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.SparkContext._
import scala.io.Source._
import java.io.File
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

import order should be java > scala > 3rd party packages > spark

@rnowling
Copy link
Contributor Author

@andrewor14 , thanks for the comments! I believe I fixed everything except for the changing the name of the example. I wanted some more feedback.

I wrote the example to test reading and writing to a DFS. It does so by comparing the result of word count on a local file to word count on the file after copying to the DFS. I'm using it to make sure that DFSs are configured properly and accessible by all nodes.

Do you still want me to drop the test suffix? Thanks!

@SparkQA
Copy link

SparkQA commented Nov 20, 2014

Test build #23672 has started for PR 3347 at commit b0ef9ea.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 20, 2014

Test build #23672 has finished for PR 3347 at commit b0ef9ea.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23672/
Test PASSed.

@SparkQA
Copy link

SparkQA commented Nov 20, 2014

Test build #23675 has started for PR 3347 at commit af8ccb7.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 20, 2014

Test build #23675 has finished for PR 3347 at commit af8ccb7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23675/
Test PASSed.

@rnowling
Copy link
Contributor Author

rnowling commented Dec 8, 2014

@andrewor14 Could you take a second look when you get a chance? Thanks!

@andrewor14
Copy link
Contributor

I think the latest changes look fine. @pwendell any thoughts?

@rnowling
Copy link
Contributor Author

@andrewor14 @pwendell Have you had a chance to look at this? Thanks!

@rnowling
Copy link
Contributor Author

rnowling commented Jan 5, 2015

@andrewor14 @pwendell Happy New Year! Ping on this PR? Thanks!

@SparkQA
Copy link

SparkQA commented Feb 2, 2015

Test build #26532 has started for PR 3347 at commit af8ccb7.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 2, 2015

Test build #26532 has finished for PR 3347 at commit af8ccb7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26532/
Test PASSed.

@srowen
Copy link
Member

srowen commented May 18, 2015

@andrewor14 if you like the change I think you can merge it; no other comments here. Otherwise i think this has timed out.

andrewor14
andrewor14 reviewed Jun 18, 2015
View reviewed changes
examples/src/main/scala/org/apache/spark/examples/DFSReadWriteTest.scala

sc.stop()

if(localWordCount == dfsWordCount) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: space after if, here and everywhere.

@andrewor14
Copy link
Contributor

@rnowling sorry for slipping on this. I'll merge this into master after addressing the latest comments myself. Thanks for your patience.

@asfgit asfgit closed this in a985803 Jun 19, 2015
@viirya viirya mentioned this pull request Jun 19, 2015
Closed
@rnowling
Copy link
Contributor Author

Thanks @andrewor14 !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@rnowling @SparkQA @AmplabJenkins @andrewor14 @srowen