Unknown Family of Compilers (Build Failure)

[jpc2350]

Hi Guys,

I'm know trying to build Python using EasyBuild and I'm getting this error when using the Dummy Toolchain, Primarily when trying to tie in NumPY:

Unknown family of compilers being used: DUMMY

Is there anyway to get around this using EasyBuild but without using the GCC/Intel Toolchains?

Thanks

[boegel]

@jpc2350 If you prefer using your system compiler, you could install a wrapper module for it via the GCC-system.eb easyconfig.

For numpy specifically, that will only get you so far though, since you also need a BLAS/LAPACK library in that case.

In general, it's better in the long term is you build a toolchain (e.g. foss/2017a) first with EasyBuild, and then build software on top of that.
Using system compilers, tools & libraries is possible, but probably requires more effort than is worth it...

Any particular reason you are reluctant to install a toolchain with EasyBuild?

[jpc2350]

Hi @boegel, Unfortunately in this case, it's not been my decision. I will take a look at the wrapper module thanks.

[jpc2350]

Hi @boegel, quick question - Are we able to use a toolchain without having the modulefile named with the toolchain in it?

[boegel]

@jpc2350 Yes, you can use a custom module naming scheme and just strip out the toolchain part. I suspect you may run into problems at some point if you do that though, since mixing things that were built with different toolchains would lead to problems.
If you stick to building things with one particular toolchain (e.g. foss/2017a), that may work.

There are a couple of module naming schemes implemented in https://github.com/easybuilders/easybuild-framework/tree/master/easybuild/tools/module_naming_scheme, the default being EasyBuildMNS.

To use a custom module naming scheme, you need to provide an implementation in for example mymns.py, and let EasyBuild pick it up via --include-module-naming-schemes=path/to/mymns.py.

[jpc2350]

Hi @boegel, Ok thanks - I'll take a look at that and see what we can do.

Also, how do I build a tool with a GCC toolchain, where the modules exist (without the toolchain extension) or where there are no recipe files that have the toolchain extension?

I'm using the --use-existing-modules option, but that doesn't seem to work?

[ocaisa]

The --use-existing-modules only functions with the --minimal-toolchains option (and from what you have described would also require --add-dummy-to-minimal-toolchains). See http://easybuild.readthedocs.io/en/latest/Manipulating_dependencies.html#using-minimal-toolchains-for-dependencies for a more complete explanation.

Your use case is not entirely clear to me though, and I have my doubts it will do what you might be expecting it to (i.e, it only works with EB generated module files).

[ocaisa]

Regarding not including toolchain extensions in the module naming scheme, using a hierarchical naming scheme can achieve this for you. If you want to do this in a flat naming scheme then you are explicitly removing uniqueness (zlib/1.2.8 built with foss has the same module name as one built with intel) which of course can lead to problems since this is against the philosophy of generating a coherent software stack.

[jpc2350]

Hi @ocaisa thanks, So at the moment, the decision has been made to use the system compiler - GCC 5.x (packaged with OS). So at this point, we're only running with that single toolchain which is the 'dummy' one. I have however, installed GCCcore-6.4.0 to fix the issue with NumPY and Perl.

[ocaisa]

I have a pair of open PRs (that will hopefully be merged very soon) that help to cover your type of situation (and still maintain the general philosophy)
easybuilders/easybuild-easyblocks#1106
easybuilders/easybuild-easyconfigs#4136
...but they won't help with the module naming scheme, for which I would recommend a hierarchical scheme with a set of preloaded modules (the ones that extend the path) so that the user can directly see all modules from the outset.

[jpc2350]

So I'm still getting a NumPY build issue - It's not the GCCcore problem (as I've now added that) - I'm not convinced it's an EasyBuild problem - but the tests come back as problematic (which NumPY confirm is acceptable) but the problem appears to be that the test fails and although a normal build will still work - the easybuild fails on this point.

Finished processing dependencies for numpy==1.12.1
== 2017-10-04 09:26:36,453 run.py:525 INFO parse_log_for_error msg: Command used: export PYTHONPATH=/tmp/eb-MCURSJ/tmpVF85HL/lib/python2.7/site-packages:$PYTHONPATH && /tools/ebpoc/apps/Python/2.7.13-GCCcore-6.4.0/bin/python setup.py install --prefix=/tmp/eb-MCURSJ/tmpVF85HL
== 2017-10-04 09:26:36,453 run.py:527 INFO parse_log_for_error (some may be harmless) regExp (?<Unable to render embedded object: File ([(,-]) not found.[(,-]|.?\w) found:
_configtest.c:6: error: undefined reference to 'exp'
collect2: error: ld returned 1 exit status
_configtest.c:6: error: undefined reference to 'exp'
collect2: error: ld returned 1 exit status
== 2017-10-04 09:26:36,453 run.py:484 WARNING Found 4 errors in command output (output: _configtest.c:6: error: undefined reference to 'exp', collect2: error: ld returned 1 exit status, _configtest.c:6: error: undefined reference to 'exp', collect2: error: ld returned 1 exit status)
== 2017-10-04 09:26:36,453 run.py:179 INFO running cmd: export PYTHONPATH=/tmp/eb-MCURSJ/tmpVF85HL/lib/python2.7/site-packages:$PYTHONPATH && /tools/ebpoc/apps/Python/2.7.13-GCCcore-6.4.0/bin/python -m timeit -n 3 -r 3 -s "import numpy; x = numpy.random.random((1000, 1000))" "numpy.dot(x, x.T)"
== 2017-10-04 09:26:54,121 build_log.py:157 ERROR EasyBuild crashed with an error (at ?:124 in init): Time for 1000x1000 matrix dot product: 1600 msec >= 500 msec => ERROR (at easybuild/easyblocks/n/numpy.py:269 in test_step)
== 2017-10-04 09:26:54,121 easyblock.py:2650 WARNING build failed (first 300 chars): Time for 1000x1000 matrix dot product: 1600 msec >= 500 msec => ERROR
== 2017-10-04 09:26:54,122 easyblock.py:287 INFO Closing log for application name Python version 2.7.13

[boegel]

@jpc2350 Did you figure things out (since you closed this issue)?

[jpc2350]

At the moment it's still plaguing me, I've got a few other identified issues - but I thought it would be better raising them as separate cases.