[WIP/RFP] Secondary subtoolchains #2234
Conversation
| from easybuild.toolchains.fft.intelfftw import IntelFFTW | ||
| from easybuild.toolchains.linalg.intelmkl import IntelMKL | ||
|
|
||
|
|
||
| class Iomkl(Iompi, IntelMKL, IntelFFTW): | ||
| class Iomkl(Iompi, Iimkl, IntelMKL, IntelFFTW): |
There was a problem hiding this comment.
Since you're inheriting from Iimkl you don't need to also inherit from IntelMKL, IntelFFTW (they're already there)
There was a problem hiding this comment.
+1, but this may have impact on how toolchains are verified by EasyBuild (not sure though)
|
What does the toolchain hierarchy output look like after this? |
|
For me, I'm ok with the concept but I do think you should be verifying that the underlying toolchains of |
|
For toolchain output you mean this using |
| @@ -134,6 +134,71 @@ def cache_aware_func(toolchain): | |||
|
|
|||
| return cache_aware_func | |||
|
|
|||
| def get_subtoolchain_in_deps(parsed_ec, subtoolchain_name, current_tc_name): | |||
There was a problem hiding this comment.
@bartoldeman This is begging for a dedicated unit test, to verify that it works as intended (and doesn't get broken in the future somehow)...
| # only retain candidates that match subtoolchain name | ||
| cands = [c for c in cands if c['name'] == subtoolchain_name] | ||
| if isinstance(subtoolchain_name, list): | ||
| for name in subtoolchain_name[1:]: |
There was a problem hiding this comment.
please clarify with a comment why you are skipping the 1st subtoolchain?
|
@bartoldeman This gives you a flat list, i.e. not a DAG, which basically means there's an order implied between That's not necessarily a big issue though, since which one you pick may not matter too much, and having a hit for both may actually not happen in practice: software that doesn't depend on MPI would only have an easyconfig with So, I think this is OK. I assume this work as intended for you? Do you have some practical examples? |
|
Hello Kenneth, Alan, thanks for the feedback! Yes this is working as intended for us. One example I have is mpi4py (iomkl), which depends on numpy or in our case the scipy stack module (iimkl). I won't have time to work on this this week though. It will have to wait until next week or the week there after. Our nix+easybuild stack is moving into production so these are exciting but busy times. Bart |
This PR replaces easybuilders#2234. subtoolchains are now searched using a breadth-first-search, which allows multiple subtoolchains per toolchain. Some subtoolchains such as golf and golfc are now marked OPTIONAL since they do not have to exist as modules, and can be skipped. Since those optional subtoolchains cannot usually be found via a dependency search (as toolchain easyconfigs do not typically have subtoolchains as dependencies) try to search for subtoolchain/version with the same version as the parent. E.g. goolfc/2.6.10 searches for golfc/2.6.10, and golfc/2.6.10 searches for golf/2.6.10 and gcccuda/2.6.10.
This PR replaces easybuilders#2234. subtoolchains are now searched using a breadth-first-search, which allows multiple subtoolchains per toolchain. Some subtoolchains such as golf and golfc are now marked OPTIONAL since they do not have to exist as modules, and can be skipped. Since those optional subtoolchains cannot usually be found via a dependency search (as toolchain easyconfigs do not typically have subtoolchains as dependencies) try to search for subtoolchain/version with the same version as the parent. E.g. goolfc/2.6.10 searches for golfc/2.6.10, and golfc/2.6.10 searches for golf/2.6.10 and gcccuda/2.6.10.
With the introduction of the iimkl (Intel + MKL without MPI) toolchain in our software stack it could not be a subtoolchain with the current infrastructure. In fact the subtoolchain structure becomes a DAG:
This patchset changes iomkl to have two subtoolchains. However it does not implement a DAG, as it assumes that the sub-subtoolchains of iomkl (the subtoolchains of iimkl and iompi, that is, iccifort) are identical.
This is sufficient for our setup but feels a bit quick and dirty. Please let me know what you think.