[Support] Add parallel_for support to run a loop in parallel

[jcf94]

As has mentioned in #5962.
We would like a runtime implementation of parallel loop to speed up some thread safe loops.

Take some API reference from https://docs.microsoft.com/en-us/cpp/parallel/concrt/parallel-algorithms?view=vs-2019#parallel_for. I've tried to add an implementation of parallel_for_each as well, but finding that the const reference style of tvm::Array seems not very compatible with this API.

cc @tqchen @merrymercy @FrozenGene @junrushao1994

[jroesch]

cc @tkonolige

[tkonolige]

I think I'm a little late to this discussion, but what is the reason for having our own thread pool/parallel_for implementation? OpenMP is already an optional dependency, we could use it when available.

I can understand that having our own implementation means we don't have to depend on another library, but it also means we need to maintain it and add features as we need them (though this doesn't seem like much code).

[tqchen]

@jcf94 it would be great if we can simplify the parallel for implementation, e.g. std::thread has pretty low launching overhead, and we can likely drop the threadpool and start std::thread on each parallel_for round. As long as the function cost is not high, it should be a simpler implementation.

[jcf94]

@jcf94 it would be great if we can simplify the parallel for implementation, e.g. std::thread has pretty low launching overhead, and we can likely drop the threadpool and start std::thread on each parallel_for round. As long as the function cost is not high, it should be a simpler implementation.

I've tried a simpler implementation using std::thread, have a look if this is better. 😄

[tqchen]

Thanks @jcf94 please look into the CI issue. Also please do a benchmark to see if the new implementation will/will not affect perf, since it is only fine for larger functions and we might still need pool for very fine grained parallelism

[jcf94]

Thanks @jcf94 please look into the CI issue. Also please do a benchmark to see if the new implementation will/will not affect perf, since it is only fine for larger functions and we might still need pool for very fine grained parallelism

Had some simple benchmark on the two implementations, the current one even works better in large loop size(since each threads' workload is pre-defined by the partitioner).
Currently Ansor's requirement on the parallel_for is just used on some thread independent tasks, this simple implementation is enough to work well. I'm not sure if some tasks with fine grained parallelism will benefit on a pool implementation.

[jcf94]

@merrymercy @tqchen I'm thinking that do we need a interface to pass the parapllel_for to Python? In my understanding, the python multi-threading may have some performance issues caused by GIL.

[FrozenGene]

@merrymercy @tqchen I'm thinking that do we need a interface to pass the parapllel_for to Python? In my understanding, the python multi-threading may have some performance issues caused by GIL.

for this, multi processing maybe enough...

one final comment unaddressed

[tqchen]

Thanks @jcf94 @merrymercy @jroesch @ @tkonolige !