spawn on multiple computer nodes

[mictadlo]

Hi, any plans to support spawn on multiple computer nodes similar like Julia ( http://docs.julialang.org/en/latest/manual/parallel-computing/#parallel-map-and-loops ) does it:

require("count_heads")

a = @spawn count_heads(100000000)
b = @spawn count_heads(100000000)
fetch(a)+fetch(b)

The above code perform some trials on two machines, and add together the results.

nheads = @parallel (+) for i=1:100000000
  int(randbool())
end

The above computation is automatically distributed across all available compute nodes, and the result, reduced by summation (+), is returned at the calling node.

I read about raspberry pi supercomputer (
http://www.tomshardware.com/news/Raspberry-Pi-Supercomputer-Legos-Linux,17596.html ) and just wondering whether Rust could do the same?

Mic

[Kimundi]

It will probably not be included as part of the core language (rust is more geared towards single process parallelism), but something like that should be possible with library code. The hard part will be sending arbitrary closures/functions, but even that should ultimately be possible with some kind of compiletime reflexion, (de)serialization, and jit compiliation on the other end. Of course, whether that's as performant as a language that supports it natively is another question...

[bstrie]

Naively, I feel like it should be possible to extend the existing task scheduler API such that instead of only being able to say things like "spawn one task for each system core" you could also say "spawn tasks across all available compute nodes".

[bstrie]

To clarify, by "compute nodes" I'm including the possibility of heterogenous distributed compute clusters. I originally thought the Julia documentation was referring to these, but it looks like it's really just referring to scaling across multiple cores.

[metajack]

flatpipes already enables part of this. Probably the first thing that is needed to build something like this is to create something akin to Erlang's distribution protocol and node finding.

[nikomatsakis]

I imagine one could build a cross-process spawning framework. You
would not want to use ~fn() closures to define the task bodies but
rather extern "Rust" fn() functions, which carry no closure
state. All communication would then be done via pipes that would
serialize/deserialize behind the scenes. Anyway, I basically agree
this is something we (or someone in the Rust community, hint hint)
will build as a library and not as part of the core language.

[bstrie]

Tagging this as an interesting project in case anyone feels inspired.

[mictadlo]

Implementation of Erlang/OTP node in Go ( https://github.com/goerlang/node ) another project which use it is https://github.com/goerlang/eclus . Here are even parts from erlang written in GO ( https://github.com/goerlang ). Could maybe be converted to Rust.

[toddaaro]

Added a few tags. This would be great to have. I also think this will need a fairly tight coupling with the runtime to do well. A naive approach is to just extend workstealing to allow steals accross physical machines, as long as the "work being stolen" is appropriately flagged. It would be interesting to provide good "hooks" in the runtime to do this without it being "part" of the runtime, but we aren't ready to think about that yet.

[catamorphism]

Bug triage. An interesting project, but not a blocker.

[mictadlo]

Here is a talk about Cloud Haskell http://skillsmatter.com/podcast/home/haskell-cloud/js-4179

[mictadlo]

This project http://www.gocircuit.org/ extends the reach of Go's linguistic environment to multi-host/multi-process applications.

[flaper87]

Visiting for triage. Besides the comments and suggestions here, nothing else seems to have been done.

[thestinger]

This would be an interesting project, but it's not in the scope of the standard libraries. There are far too many design compromises involved to provide a standard solution to distributed computing. It would also imply cryptography (TLS) in the standard library, which has been rejected. Go dropped the experimental support for netchan from the standard library because the problem space is too complex to tackle there.