You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, when computing something like xp.add(a, b), a is computed first, followed by b, followed by the add operation.
We could get more parallelism by computing a and b at the same time, since they don't depend on each other.
(This is true for the Lithops and Modal DagExecutors which both rely on map_unordered, but not for BeamDagExecutor, which can already compute a and b in parallel since it delegates to a Beam DAG.)
The text was updated successfully, but these errors were encountered:
For Modal we could use the same idea as AsyncPythonDagExecutor to merge async streams for parallelism.
Lithops isn't async so that won't work. One idea I had was to compose multiple map calls into one by tagging the function and the inputs so that each input is processed by the relevant function.
Currently, when computing something like
xp.add(a, b)
,a
is computed first, followed byb
, followed by theadd
operation.We could get more parallelism by computing
a
andb
at the same time, since they don't depend on each other.(This is true for the Lithops and Modal
DagExecutor
s which both rely onmap_unordered
, but not forBeamDagExecutor
, which can already computea
andb
in parallel since it delegates to a Beam DAG.)The text was updated successfully, but these errors were encountered: