make tbb threads configurable

[jiafuzha]

Description

make TBB threads configurable so that other resource managers or frameworks, like Spark, can manage resource when use DAAL. What we did is DAAL on Spark. The typical way to run workload on Spark is,

• break large input into splits
• one task per split
• assign one or more CPU cores (cpu_per_task, configurable in Spark) for each task (split)
• In each worker node, CPU cores are distributed to different executors running on the same node. And each executor could run different number of tasks simultaneously. For example, a executor has 8 cores and cpu_per_task is 2. Then executor can run 4 tasks as same time.
• Each task invokes DAAL for some algorithms. We don’t’ want DAAL for single task occupy all CPU cores whilst there are other tasks running.

As you can see, it’s the reason why we make TBB threads configurable.

Changes proposed in this pull request:

• load native library only once
• number of TBB threads is configured via system property, "tbb-threads", like below.
  java -Dtbb-threads=10 ...

[dr-pain]

address comments please

[napetrov]

make test

[napetrov]

Precommit failures associated with problems on host and not changes in this PR

[PivovarA]

all Ok. the number of tbb threads is configurable and can be set automatically.

[averbukh]

What are reasons to read the environment variable in the library instead of an application? Do you need something additionally in the DAAL to assign the number of threads via Java API?

[jiafuzha]

What are reasons to read the environment variable in the library instead of an application? Do you need something additionally in the DAAL to assign the number of threads via Java API?

cause the number of cores for each task is configurable in Spark. We want the TBB threads to be changed accordingly when cores per task changed. Thus we read it from the configurable env variable. A new Java API is an alternative. The similar logic for configurable TBB threads is still needed.

[averbukh]

What are reasons to read the environment variable in the library instead of an application? Do you need something additionally in the DAAL to assign the number of threads via Java API?

cause the number of cores for each task is configurable in Spark. We want the TBB threads to be changed accordingly when cores per task changed. Thus we read it from the configurable env variable. A new Java API is an alternative. The similar logic for configurable TBB threads is still needed.

I'm OK with changing of the number of cores/threads, but doing it via environment variable introduces latent behavior for every user of the library. What do you need additionally in the Environment class to perform that you want?

[averbukh]

I'm OK with changing of the number of cores/threads, but doing it via environment variable introduces latent behavior for every user of the library

[jiafuzha]

Guys,

I went through some related code and found it's not an easy task for me to add APIs for all algorithms. Task KMeans as example. The entry point is SampleKMeans.scala. In this class, we need to make changes as below,
1, read threads configuration, spark.daal.tbb-threads, from the SparkConf object.
2, change the train method of KMeans.scala to add parameter of tbb-threads.
3, Inside computeInit method, I see InitDistributedStep2Master, InitDistributedStep1Local and more compute methods. I cannot figure out where to set tbb-threads.

And for other algorithms, we need all these steps too. So please suggest.

[averbukh]

Guys,

I went through some related code and found it's not an easy task for me to add APIs for all algorithms. Task KMeans as example. The entry point is SampleKMeans.scala. In this class, we need to make changes as below,
1, read threads configuration, spark.daal.tbb-threads, from the SparkConf object.
2, change the train method of KMeans.scala to add parameter of tbb-threads.
3, Inside computeInit method, I see InitDistributedStep2Master, InitDistributedStep1Local and more compute methods. I cannot figure out where to set tbb-threads.

And for other algorithms, we need all these steps too. So please suggest.

Looks like you don't need to do item 2. All you need is to set the number of threads for Environment class. See this Java example for that please.

[PivovarA]

Hello @zjf2012
I don't like the idea of ​​adding new parameters to KMeans.scala (and to other algorithms), since we want our spark samples to have a spark.mllib-like interfaces.
You solve the issue that DAAL occupies all the cpu cores that may be needed for other tasks.
In this case, there are several ways to solve this problem:

1. You can extract number of cores per executor value from Spark Conf and set the correct number of tbb threads using Environment class.
2. You can fix this problem by editing Spark Conf and setting parameters properly.

[averbukh]

Do the approach with setting of the number of threads via Environment class (how it done in the Java example I mentioned) works for you?

[averbukh]

No activity from contributor - I suggest to close PR in week.
Also, after discussion with @PivovarA , I propose to add number of thread assignment based on Spark configuration into Spark samples as separate task.