Returning unscheduled Tasks

[fredrikekre]

I noticed that some async_* routines return unscheduled task. I am just curious about this choice (@task over @async) since I have not seen this pattern elsewhere. I can understand that it gives the user a choice of when to schedule these tasks, but it seems like you would in 99% of cases schedule them right away. Is there a practical reason when you would want to delay the scheduling?

Also, in e.g. the test_fem_sa.jl]() example, this task

PartitionedArrays.jl/test/test_fem_sa.jl

Lines 74 to 76 in d5224f7

	# Start an assembly of the COO Vectors,
	# i.e., send and add the triplets (i,j,v) to the part that owns i.
	t = async_assemble!(I,J,V,rows)

is not scheduled until later

PartitionedArrays.jl/test/test_fem_sa.jl

Lines 103 to 104 in d5224f7

	# Wait the assembly of the coo vectors to finish
	map_parts(wait∘schedule,t)

I was confused by the comment there since the task is not actually started yet. Would it be better to use this pattern in this example (and elsewhere):

t = map_parts(schedule, async_assemble!(I, J, V, rows))

# do the other stuff here

 map_parts(wait, t)

[fverdugo]

I noticed that some async_* ..

Yes, there is a reason.This is one of the parts that is not yet well resolved as you pointed out. So, any new idas will be welcomed!

The main reason is that the mpi process will block as soon as you schedule the task. This is a quite tricky problem. See the discussion here

https://discourse.julialang.org/t/how-to-combine-mpi-non-blocking-isend-irecv-and-julia-tasks/52524

for details.

MPI.jl has an open PR that could help to improve the API at the PartitionedArrays.jl side: JuliaParallel/MPI.jl#452

[fredrikekre]

Thanks for the pointers.

So right now there is not really a point in using async_assemble! instead of assemble!, right? E.g. in the example I linked above, there it nothing on the lines between that needs to happen before scheduling the task?

[fverdugo]

So right now there is not really a point in using async_assemble! instead of assemble!, right?

There is a point. When you call async_assemble! the MPI communication begins right away and you can overlap computations until you call map_parts(wait∘schedule,t). Note that this schedules and waits for a Julia task that wrapps the MPI wait. But the MPI communication itself started before.

[fredrikekre]

Ah I see now that not everything is wrapped in the returned @task, e.g. this:

PartitionedArrays.jl/src/Interfaces.jl

Line 2479 in d5224f7

gi_rcv, gj_rcv, v_rcv = map_parts(setup_rcv,part,parts_rcv,rows.partition,coo_values)

is happening before async_assemble! returns, and this:

PartitionedArrays.jl/src/Interfaces.jl

Line 2509 in d5224f7

t4 = map_parts(setup_snd,t1,t2,t3,part,rows.partition,gi_snd,gj_snd,v_snd,coo_values)

is not scheduled until later. So async_assemble! is partly asynchronous.

[fverdugo]

This discussion is not relevant anymore. Since v0.3.0, the library always returns scheduled tasks.