Skip to content

Commit

Permalink
[SPARK-19324][SPARKR] Spark VJM stdout output is getting dropped in S…
Browse files Browse the repository at this point in the history
…parkR

## What changes were proposed in this pull request?

This affects mostly running job from the driver in client mode when results are expected to be through stdout (which should be somewhat rare, but possible)

Before:
```
> a <- as.DataFrame(cars)
> b <- group_by(a, "dist")
> c <- count(b)
> sparkR.callJMethod(c$countjc, "explain", TRUE)
NULL
```

After:
```
> a <- as.DataFrame(cars)
> b <- group_by(a, "dist")
> c <- count(b)
> sparkR.callJMethod(c$countjc, "explain", TRUE)
count#11L
NULL
```

Now, `column.explain()` doesn't seem very useful (we can get more extensive output with `DataFrame.explain()`) but there are other more complex examples with calls of `println` in Scala/JVM side, that are getting dropped.

## How was this patch tested?

manual

Author: Felix Cheung <[email protected]>

Closes #16670 from felixcheung/rjvmstdout.

(cherry picked from commit a7ab6f9)
Signed-off-by: Shivaram Venkataraman <[email protected]>
  • Loading branch information
felixcheung authored and shivaram committed Jan 27, 2017
1 parent 4002ee9 commit 9a49f9a
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 4 deletions.
11 changes: 8 additions & 3 deletions R/pkg/R/utils.R
Original file line number Diff line number Diff line change
Expand Up @@ -756,12 +756,17 @@ varargsToJProperties <- function(...) {
props
}

launchScript <- function(script, combinedArgs, capture = FALSE) {
launchScript <- function(script, combinedArgs, wait = FALSE) {
if (.Platform$OS.type == "windows") {
scriptWithArgs <- paste(script, combinedArgs, sep = " ")
shell(scriptWithArgs, translate = TRUE, wait = capture, intern = capture) # nolint
# on Windows, intern = F seems to mean output to the console. (documentation on this is missing)
shell(scriptWithArgs, translate = TRUE, wait = wait, intern = wait) # nolint
} else {
system2(script, combinedArgs, wait = capture, stdout = capture)
# http://stat.ethz.ch/R-manual/R-devel/library/base/html/system2.html
# stdout = F means discard output
# stdout = "" means to its console (default)
# Note that the console of this child process might not be the same as the running R process.
system2(script, combinedArgs, stdout = "", wait = wait)
}
}

Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_Windows.R
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ test_that("sparkJars tag in SparkContext", {
if (.Platform$OS.type != "windows") {
skip("This test is only for Windows, skipped")
}
testOutput <- launchScript("ECHO", "a/b/c", capture = TRUE)
testOutput <- launchScript("ECHO", "a/b/c", wait = TRUE)
abcPath <- testOutput[1]
expect_equal(abcPath, "a\\b\\c")
})

0 comments on commit 9a49f9a

Please sign in to comment.