Albany no longer building with spack on Anvil #12
Comments
|
Note that we're using a relatively old gcc (8.2.0). |
|
There are newer compilers and libraries available. I can try those and see if they work okay. We usually try to stay in sync with E3SM but we don't have to. |
|
As expected, I'm seeing the same error with |
|
I can't find a more up-to-date set of compilers and MPI on Anvil to test with. |
|
Thanks for checking all of this @xylar . I am sad to say that 'internal compiler error' typically means there is a compiler bug, at least in my experience (and people say it online too: https://stackoverflow.com/questions/12405606/what-do-internal-compiler-error-messages-mean-and-what-can-i-do ). @jewatkins , @mperego , @bartgol , @kliegeois - is this consistent with your experience? I will say that gcc-8.2.0 is pretty old at this point... it would be interesting to see what happens with newer compilers. I'm not sure how I can help with this. I could build a gcc-8.2.0 compiler on one of my machines and see if the error happens for me or not, to try to reproduce it. That's the only thing that comes to mind now... |
|
In my experience, ICEs are very susceptible to the most ridicolous microchanges. The way I usually tackle them is by isolating the code that gives the ICE, and try to manipulate it slightly (add parentheses, remove const qualifiers, reorder sums,...). If that doesn't work, I try to move the function that causes issues to a separate cpp file. All of these should have no impact from the syntax point of view, but ICEs are usually a failure of the compiler to correctly parse, so they are legit solutions. @xylar , I would try to modify the line in some silly way, like or Whatever works to get the compiler to generate the correct parsing tree. |
|
@bartgol : it sounds like you're suggesting to try to reproduce the problem and come up with a sort of hack to avoid the compiler error? I could try building gcc-8.2.0 to see if I get the same error. Please stay tuned. Other suggestions are welcome... |
|
Well, I don't suggest to try to reproduce it. I suggest to try some mods like those on the machine that has the issue. You can, of course, try to reproduce, but I would not be surprised if, even with the same compilers, you don't get it on another machine. |
|
Okay, thank you both. I don't think it's worth investigating the specific error or fiddling with that line of code. I'll try compiling new compilers, MPI and the whole gamut and we'll see how it goes. |
|
Ok. I actually tried compiling with a gcc-8.2.0 compiler but got other errors not related to this. The compiler was from a module on one of our clusters and I think it requires other modules to be loaded to compile certain libraries, which I'd have to figure out. Please do let us know if you see this error with other newer compilers. |
|
I've got a fix for this in MPAS-Dev/compass#539. I think we're good. I needed to build OpenMPI with Spack (using a newer compiler) along with all the dependencies. So basically treating Anvil as an unsupported machine. But that works. |
|
Glad to hear you have a workaround @xylar ! |
|
@ikalash and @matthewhoffman, and update on this (as I noted in MPAS-Dev/compass#539). It seems that MPI from spack works okay on a single node but jobs on multiple nodes just hang. I don't really have the time or expertise to debug the MPI build any further. So this means we're currently without a way to support MALI on Anvil, which is a shame. I don't think our project is high enough priority to put pressure on the LCRC folks to upgrade the version of Gnu and MPI for us. Suggestions are welcome but I'm starting to think we have to just drop Anvil support of MALI. |
|
Interesting. I am not sure I have any suggestions for this... I can add this to the Agenda for the Albany meeting this Tuesday. If you'd like to discuss it, please try to join. |
|
We have asked for a new set of compilers, MPI libraries, hdf5, netcdf-c netcdf-fortran and parallel-netcdf for both gnu and intel on Anvil. We don't know how long that might take but we hope that it will solve the issue. @jewatkins was involved in deciding which versions of packages (particularly compilers and OpenMPI) we requested, so we hope we'll be in good shape. |
|
I think we'll let Albany on Anvil go. It doesn't seem like there's much of a push to update compilers there. |
|
Makes sense to me. |
@matthewhoffman and I are seeing the following when we build on Albany with Gnu and OpenMPI:
Spack error message
The change seems to be somewhat recent -- thing worked as of January 12th.
Here is what I ran:
Bash script
Here is the yaml file:
dev_compass_1_2_0-alpha_4_gnu_openmpi_albany.yaml
The text was updated successfully, but these errors were encountered: