Return warnings for futures with plan multicore

[russellpierce]

Consider:

v %<=% {Sys.sleep(3);warning("Warning");10} %plan% lazy
v
v %<=% {Sys.sleep(3);warning("No warning?");10} %plan% multicore
v

Unlike errors, warnings are not passed forward under the multicore plan.

Also, amazing package. I wish I'd discovered it sooner.

[HenrikBengtsson]

Thanks, I'm glad you like it.

Good point about warnings not being passed on for multicore futures. This idea would probably also apply to any class of conditions. In order to solve this in a generic way, I also have to consider cases where other backends are used to resolve futures, e.g. distributed evaluation on clusters (e.g. via BatchJobs). And if so, should there be an option to specify what types of conditions should be passed on/filtered out?

Many questions, but I'll certainly look into this. If you have more thoughts about this, I'm all ears.

[russellpierce]

It is only an opinion, but it seems to me that if one were doing standard eager evaluation then one would have to engage in filtering of conditions one didn't want. So, I think I'd leave filtering out conditions to the user and pass on all conditions that can be passed.

[HenrikBengtsson]

Detailed examples for this issue (ran in separate R sessions just in case):

Eager futures:

> library("future")
> v %<=% { message("begin"); warning("bang!"); message("done"); TRUE } %plan% eager
begin
done
Warning message:
In eval(expr, envir, enclos) : bang!
> warnings()
Warning message:
In eval(expr, envir, enclos) : bang!
> v
[1] TRUE
> warnings()
Warning message:
In eval(expr, envir, enclos) : bang!

Lazy futures:

> library("future")
> v %<=% { message("begin"); warning("bang!"); message("done"); TRUE } %plan% lazy
> warnings()
NULL
> v
begin
done
[1] TRUE
Warning message:
In eval(expr, envir, enclos) : bang!
> warnings()
Warning message:
In eval(expr, envir, enclos) : bang!

As expected, note how warnings() returns on the generated warning even before future variable v is acquired.

Multicore futures:

> library("future")
> v %<=% { message("begin"); warning("bang!"); message("done"); TRUE } %plan% multicore
begin
> done
warnings()
Warning message:
In eval(expr, envir, enclos) : bang!
> v
[1] TRUE
> warnings()
Warning message:
In eval(expr, envir, enclos) : bang!

As expected, note how warnings() returns on the generated warning even before future variable v is acquired.

From the above, I conclude that the issues is not that warnings are not recorded, but they are simply not printed when using multicore futures.

[HenrikBengtsson]

More examples for illustrating what is possible and not.

Warnings turned into errors automatically are properly captured in all cases:

> library("future")
> options(warn=2L)  ## Convert warnings to errors

> e %<=% { message("begin"); warning("bang!"); message("done"); TRUE } %plan% eager                       begin
> e
Error in eval(expr, envir, enclos) : (converted from warning) bang!

> l %<=% { message("begin"); warning("bang!"); message("done"); TRUE } %plan% lazy
> l
begin
Error in eval(expr, envir, enclos) : (converted from warning) bang!

> m %<=% { message("begin"); warning("bang!"); message("done"); TRUE } %plan% multicore
> begin
m
Error in eval(expr, env) : (converted from warning) bang!

[HenrikBengtsson]

BTW, instead of generating errors, one can force immediate output of warnings using options(warn=1L). For example:

> library("future")
> options(warn=1L)  ## Output warning messages as they occur

> m %<=% { message("begin"); warning("bang!"); message("done"); TRUE } %plan% multicore
> begin
Warning in eval(expr, env) : bang!
done

> m
[1] TRUE

@drknexus, is the above sufficient/what you're looking for?

Note that capturing warnings using try-catch mechanisms is a whole 'nother game.

[DarwinAwardWinner]

There's some possible solutions to this issue here: http://stackoverflow.com/questions/4948361/how-do-i-save-warnings-and-errors-as-output-from-a-function

[HenrikBengtsson]

Thanks @DarwinAwardWinner.

Just for the record of this issue/thread: Contrary to errors, which we wish to propagate immediately (unless explicitly captured by explicit code), warnings and other conditions need to be captured but non-interrupting. This is for instance why tryCatch(expr, warning=...) does not work, because it also interrupts expr with the warning. Instead one has to go down a bit lower and use withCallingHandlers() and invokeRestart() as suggested in the link by @DarwinAwardWinner.

With the above in place, the question is what should happen when we capture warnings? One idea I can imagine is that any conditions captured by the future f will be re-signaled as soon as value(f) is called. This is how it works for errors currently. So, one could imagine the same for warnings. The tricky part is that contrary to errors, we also want to return the value of the future expression. That is, internally we need to "transfer back" both the value and any captured conditioned. (For errors, it is currently the same as the returned error object). Thus, when transferring back (value, conditions), we need to make sure to split them up. This should be doable within value().

Somewhat related to this idea of capturing conditions and re-signal them in the calling process, is the idea of automatically capturing standard output and error (Issue #67) and re-output them. The same idea applies to capturing plot commands (which is doable in R >= 3.3.0), but that would require much more work, because I don't think it is easy to do this for more than one plot at the time.

Any feedback on this related to design, implementation, use cases, potential pit falls or plain wishes is appreciated.

[russellpierce]

I think options(warn=1L) will provide the behavior I was expecting for multicore, i.e. roughly consistent with eager futures.

As for what is optimal... that's as you say a bit more murky. On one hand, one wants to immediately know when the code in the future isn't behaving quite as expected. On the other hand, the warning is inherently relevant to the value that is being returned from the future.

Between the two options, for my part, I lean towards a preference of re-signaling the error/messages as soon as value(f) is called.

[HenrikBengtsson]

For the record, there's a related discussion in #PR17122 titled 'mclapply drops warnings when options(warn=0)' (reported on 2016-07-20).

[burchill]

@HenrikBengtsson Pardon my confusion, but in version 1.9.0, signalled conditions still aren't collected, right? I can't seem to find anything in the docs about how to use result(), which seems like it is aimed to take care of this problem? Or maybe it's just supposed to capture the output?

If it's true that signalled conditions still aren't collected, I've written some code that I've modified from my own personal code-base that might be useful. I've uploaded a gist of it here.

In short, the condition_collector function collects all conditions signalled in the expression, muffling warnings and messages when possible (e.g., invoking the restarts "muffleWarning" and "muffleMessage" if it can) and stopping after it encounters (and collects) an error. It also collects custom conditions. It then returns a list of the value of the evaluated expression, and sorted messages, warnings, errors, and miscellaneous conditions. The signal_conditions function then goes through and signals all the conditions that were collected, and returns the value of the expression. The idea being that expressions would be wrapped in the condition_collector function before being sent off to a future, and after being returned, signal_conditions would then raise everything.

If it's the case that this sort of thing has already been taken care of (and I've done all this for naught, which would be my life in a nutshell), could you refer me to the relevant functions/arguments?

Thanks!

[HenrikBengtsson]

Thanks so much for this. I'm a bit swamped now, and will be away from my compute for several days, but I'll certainly look into your proposal because it looks like it fits into the "missing features that c/should be added" list.

PS. Correct, this is not supported in future 1.9.0 - the introduction of result() is meant to allow for "sending" this type of richer results back from workers.

[HenrikBengtsson]

FYI, in the develop branch of future, to become future 1.11.0, conditions (including message:s and warning:s) are now also relayed (in addition to stdout output). For example,

> library(future)
> plan(multisession)

> x %<-% {
+   cat("Some output\n")
+   message("hello")
+   warning("whoops")
+   message("world")
+   42
+ }

> x
Some output
hello
world
[1] 42
Warning message:
In eval(quote({ : whoops

> x
[1] 42

For early access, install as:

remotes::install_github("HenrikBengtsson/future@develop")