OpenBLAS uses too many make processes while building

[eschnett]

While installing Julia 0.4.5 on Stampede at TACC, I received this message from the system administrators:

You are still running too many tasks in the make, please 
STOP performing so many compile tasks at one time, you
are causing excessive load on the logins.  Thanks, TACC Admins
eschnett 18646 18633  0 12:07 pts/366  00:00:00 make -j4
eschnett 33142  9819  1 12:16 pts/366  00:00:00 make -j 16 GOTOBLAS_MAKEFILE= -C kernel TARGET_CORE=NEHALEM kernel

It seems the problem occurred while Julia was building OpenBLAS. It also seems that OpenBLAS builds with 16 processes (the number of cores on the login node), although I told Julia to use at most 4 processes (see the first make line quoted above).

Using many processes is fine when building on a laptop or private workstation, but on shared resources, one needs to be less possessive.

It seems that these lines in OpenBLAS's getarch.c are responsible for this:

#if NO_PARALLEL_MAKE==1
    printf("MAKE += -j 1\n");
#else
#ifndef OS_WINDOWS
    printf("MAKE += -j %d\n", get_num_cores());
#endif

Is there a chance that OpenBLAS could not add a -j option if MFLAGS already contains one?

[martin-frbg]

Which version of OpenBLAS are you building ? Checking the commit history of getarch.c one finds that
a parameter MAKE_NB_JOBS was added to Makefile.rule (and getarch.c of course) on 28 Dec 2015 (which would mean after 0.2.15, but latest is 0.2.17 already)

[eschnett]

This is apparently OpenBLAS 0.2.15. (I thought before that it was the tip of the development branch.) I'll check 0.2.17.

[eschnett]

0.2.17 still doesn't quite do the right thing. It is possible to manually choose the number of processes, but what I'm looking for is honouring the user's -j option.

This patch does the trick for me:

diff --git a/Makefile.system b/Makefile.system
index b89f60e..2dbdad0 100644
--- a/Makefile.system
+++ b/Makefile.system
@@ -139,6 +139,10 @@ NO_PARALLEL_MAKE=0
 endif
 GETARCH_FLAGS  += -DNO_PARALLEL_MAKE=$(NO_PARALLEL_MAKE)

+ifdef MAKE_NO_J
+GETARCH_FLAGS += -DMAKE_NO_J=$(MAKE_NO_J)
+endif
+
 ifdef MAKE_NB_JOBS
 GETARCH_FLAGS += -DMAKE_NB_JOBS=$(MAKE_NB_JOBS)
 endif
diff --git a/getarch.c b/getarch.c
index f9c49e6..dffad70 100644
--- a/getarch.c
+++ b/getarch.c
@@ -1012,6 +1012,7 @@ int main(int argc, char *argv[]){
 #endif
 #endif

+#ifndef MAKE_NO_J
 #ifdef MAKE_NB_JOBS
     printf("MAKE += -j %d\n", MAKE_NB_JOBS);
 #elif NO_PARALLEL_MAKE==1
@@ -1021,6 +1022,7 @@ int main(int argc, char *argv[]){
     printf("MAKE += -j %d\n", get_num_cores());
 #endif
 #endif
+#endif

     break;

This introduces a new make option MAKE_NO_J, that ensures that no -j N option is added to the make command. This makes make honour what the user specified.

[brada4]

Would be more practical to ask julia to add something to OPENBLAS_BUILD_OPTS to obey own make job count...
Stampede web page says it is SANDYBRIDGE but you build for NEHALEM and dont use AVX

[eschnett]

I want to call make without any -j options, because this way, make will communicate with all other make commands currently running to build Julia, and will globally ensure that at most 4 processes are running at one time. If I build OpenBLAS explicitly with make -j4, then there will be 4 make processes for OpenBLAS, plus up to three for other parts of Julia.

I'd like to be able to do that, but if OpenBLAS unconditionally adds a -j option I can't.

I'm not choosing any particular build options for OpenBLAS. I expect this to build a generic version that supports a large number of Intel CPU architectures.

[xianyi]

@eschnett ,

@jeromerobert submitted the patch about make MAKE_NB_JOBS=0 to delete -j in OpenBLAS.