Cloud.Parallel is limited to 1000 jobs

[eiriktsarpalis]

Running the following code:

let test = [ for i in 1 .. 2000 -> cloud { return i } ] |> Cloud.Parallel 
runtime.CreateProcess test

will result the result the computation in never completing. Checking the process info:

 Process                                                                                                                                                                   

 Name                        Process Id    State  Completed  Execution Time             Jobs            Result Type  Start Time                    Completion Time 
 ----                        ----------    -----  ---------  --------------             ----            -----------  ----------                    --------------- 
       913b3e6761f54e848235c88558d13c42  Running  False      00:09:49.9545292    0 /   0 /   9 / 18001  int []       17/3/2015 12:22:50 μμ +00:00  N/A             

Jobs : Active / Faulted / Completed / Total

The number of created jobs is suspect. It greatly exceeds the expected number (which should be 2001).

Running the same computation with a different fault policy

runtime.CreateProcess(foo, faultPolicy = FaultPolicy.NoRetry)

will result in a fault exception.

 Process                                                                                                                                                                              

 Name                        Process Id      State  Completed  Execution Time            Jobs           Result Type  Start Time                    Completion Time            
 ----                        ----------      -----  ---------  --------------            ----           -----------  ----------                    ---------------            
       16613c6990f04b04852bd5bfd80a326b  Completed  True       00:00:58.6470000    1 /   0 /   0 /   1  int []       17/3/2015 12:52:33 μμ +00:00  03/17/2015 12:53:32 +00:00 

[isaacabraham]

Does it work if you pipe the same number of tasks through DivideAndConquer?

[eiriktsarpalis]

Yes, we found that this relates to size restrictions when batch enqueueing to the service bus. Will have a fix posted soon.

[krontogiannis]

Batch enqueues have been constrained to 1K, until we can guarantee that more enqueues can be done atomically.

[dsyme]

I just tried this and it is still giving an exception (in the 0.6.10 client I am using)

System.ArgumentOutOfRangeException: Limit reached.
Parameter name: computations
   at Cloud.Parallel(seq<Cloud<Int32>> computations)
--- End of stack trace from previous location where exception was thrown ---
   at <StartupCode$MBrace-Azure-Client>[email protected](Unit _arg4) in C:\workspace\MBrace.Azure\src\MBrace.Azure.Client\Process.fs:line 133
   at [email protected](a a)
   at MBrace.Core.Internals.ExceptionDispatchInfoUtils.Async.RunSync[T](FSharpAsync`1 workflow, FSharpOption`1 cancellationToken) in c:\Users\eirik\Development\mbrace\MBrace.Core\src\MBrace.Core\Utils\ExceptionDispatchInfo.fs:line 136
   at <StartupCode$FSI_0034>.$FSI_0034.main@()
Stopped due to error

[dsyme]

(The exception is currently expected given that batch enqueues are limited to 1000)