Cannot run parallel make

[andr2000]

3rd party libraries and Arduino libraries are two different makefile rules both trying to fetch sub-modules if make runs in parallel mode, e.g. -j option specified. This results in errors like:
"error: could not lock config file .git/config: File exists"
This can be worked around by either running make without "-j" or putting ".NOTPARALLEL" target [1],
effectively disabling the parallel build, but fetching seems to need a fix.

[1] https://www.gnu.org/software/make/manual/html_node/Special-Targets.html

[slaff]

Recent versions if git have -j option for parallel execution: https://git-scm.com/docs/git-submodule. Or we can use the gnu parallel tool to parallelize and speed-up the build. Have to check if it is available under windows, Mac OS X and FreeBSD though.

@andr2000 Parallelizing the build sounds like a good idea to me - if you want yo can prepare a PR and send it for review.

[andr2000]

Well, I know about the "-j" option, but the problem is that it won't help
What we have here is we run multiple "git submodule update..." commands in parallel and each effectively running with "-j 1", while "-j" option would only make sense if we have a single "git submodule update", e.g. the rules which make troubles are (at least) "third-party/%:" and "Libraries/%:".
IOW, you can read those as "start git submodule update... for all submodules matching "third-party/%:" and "Libraries/%:" in parallel and let them fight"
So, the way forward is to disable parallel execution of these rules:

1. is not easily possible at least for GNU make (see .NOTPARALLEL: - applies to all makefile at once)
2. it seems that rules should be rewritten with parallelism in mind

[mikee47]

I can confirm that this is a problem on Windows builds as well. My workaround is to use regular make for dist-clean and initially pulling in repos. Thereafter make clean and make -j is generally OK, though sometimes gets strange file truncated errors which I suspect are related to the quality of the Windows compiler. I also note that compiling under Linux (even on a VM) is considerably faster than Windows, but that's a well-known issue and not something we can really help (it's a MinGW thing).

So I concur with @andr2000.

[andr2000]

Well, it does need altering the makefile and making lots of changes,
you can have an idea here

[slaff]

@andr2000 Why don't you send the changes as PR which will be marked as experimental? (just add "Experimental" in the title) This way I can test it under Linux, Mac OS X and FreeBSD and others can test it on the different flavours of Windows.

[andr2000]

Because the changes are WIP and not yet complete

[slaff]

Should be fixed with the merging of #1724.

[slaff]

PR #1724 is merged! @andr2000 Please check if the parallel build works for you. If not, feel free to re-open this issue.