-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unpin or update many packages (mostly Python) in configs/common/packages.yaml
, fix S4 site config
#1384
Unpin or update many packages (mostly Python) in configs/common/packages.yaml
, fix S4 site config
#1384
Conversation
…e-configs' of https://github.com/climbfuji/spack-stack into feature/unpin_update_common_packages
…eature/unpin_update_common_packages
…eature/unpin_update_common_packages
…eature/unpin_update_common_packages
…eature/unpin_update_common_packages
configs/common/packages.yaml
configs/common/packages.yaml
…eature/unpin_update_common_packages
All, the NEPTUNE tests passed. Please start testing with UFS, JEDI, ... we need this PR in as soon as possible for spack-stack 1.9.0 (downstream PRs depend on it). Thanks! |
@climbfuji I'll test JEDI/Skylab. |
Thanks very much @srherbener! You should expect problems with the shapely update (@ericlingerfelt may know more), but hopefully nothing else. |
I'm getting concretize errors like this on several platforms:
This particular message is from S4. Does this need another PR to be merged, or an update for the spack submodule commit hash? Or perhaps pilot error. I did the following (on S4) before attempting to do the spack-stack build:
Is that the issue? Thanks! |
Definitely no miniconda, those times are long gone |
Thanks @climbfuji - much appreciated! |
…nal curl/openssl, use external qt
@srherbener |
…shes with the 'require' block below
configs/common/packages.yaml
configs/common/packages.yaml
, fix S4 site config
Thanks @climbfuji! I am building jedi-bundle now, and will run ctests and skylab. |
Note, I just submitted a PR (#1420) to correct the documentation. |
jedi-bundle looks good, I'm running skylab now. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concretizes fine on personal machine, concretizes and builds UWM deps fine on Hera.
Yay! Thanks for testing :-) |
…ustomization of common packages for Derecho (#1409) 1. Update esmf from 8.8.0b06 to 8.8.0b10 2. Simplify/clean up modification of common packages in Derecho site config (move from openblas to intel-oneapi-mkl, since openblas no longer builds - neither 0.3.24 nor 0.3.28) 3. Configure LLVM 19.1.4 for Bounty; this requires updating openblas from 0.3.24 to 0.3.28, which we cannot do globally yet (waiting for #1384); there is also a build issue for ip that we can't work around, for now one has to manually replace ip with sp in spack-ext/path/to/my/env/package.py.
…eature/unpin_update_common_packages
…l and pin zstd to 1.5.6
Testing skylab on S4 has been difficult. There is something about the flow that keeps inviting the login resource monitor to kill ecflow. We tried several changes to essentially run the flow in series, but ecflow still gets killed. I think we found an actual issue with a couple tasks based on matplotlib, but the matplotlib version didn't change (from 1.8.0). @fabiolrdiniz took a look and seemed to think that the matplotlib usage was not right, and was feeling that we need to look into that and address it with fixing the tasks. It's not clear if shapely has anything to do with matplotlib tasks mentioned above, and seems unlikely, but we also saw a plotting task fail (which likely is affected by shapely). This is expected given the issue with the old shapely version. It seems that the forecasting and variational tasks are intact and working. I think things are working well enough, that it makes sense to go ahead and merge this PR and then move forward fixing the issues we found. Do you agree @fabiolrdiniz? |
Thanks for tagging me, @srherbener. I'm a little concerned about the tasks being killed on S4. What we were speculating about exceeding the number of tasks running on the login node seems not to be the problem (since we set the workflow to run serially). I wonder if the same test works fine with another spack-stack version. I agree that the matplotlib-related crashes may reveal an issue with how we plot figures. However, what is not clear to me is how confident we are with the functionality of that package. Is it capable of creating simple plots like the one below? #!/usr/bin/env python
import matplotlib.pyplot as plt
import numpy as np
x = np.array([1, 2, 3, 4])
y = x*2
plt.plot(x, y)
x1 = [2, 4, 6, 8]
y1 = [3, 5, 7, 9]
plt.plot(x1, y1, '-.')
plt.xlabel("X-axis data")
plt.ylabel("Y-axis data")
plt.title('multiple plots')
plt.tight_layout()
plt.savefig('test.png', format='png', dpi=300)
plt.close() |
@fabiolrdiniz I just checked and the matplotlib version has been 3.7.4 since at least spack-stack-1.7.0. Perhaps something upstream from the plotting routine has changed (eg, messages from the forecast or variational apps) and the tasks with matplotlib need to be adjusted? |
I'm building spack-stack with this PR feature branch, and corresponding spack submodule, on my Mac. I'll try testing jedi-bundle and skylab there and see what I get. |
@fabiolrdiniz, I tried this script in my spack-stack-1.8.0 environment and my newly built environment from this PR on my Mac. In both cases the script worked and I got this result: |
With my new env based on this PR, jedi-bundle built successfully. I got a 39 ctest failures but that isn't too out of the ordinary for the Mac. Also, all of the fv3 and mpas tests pass and only two soca test failures: one due to a floating point exception the other due to a tolerance mismatch after the applicate completed successfully. All of the other test failures were ufo and saber tests. I'm trying skylab now. |
Thanks for checking, @srherbener! By any chance, have you tried to run one of the experiments with your laptop build? If yes, did it fail similarly to S4? Thanks again! |
Sorry, only now I saw your previous message about SkyLab. We were writing messages at the same time. Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Skylab on my Mac is working! It successfully completed the gfs-3dfgat-c12 experiment and made it through the first cycle successfully in the skylab-atm-land-small experiment.
Wohoo, this is great news. Thanks very much for your efforts testing this PR. I'll merge now. This is going to unlock a lot of updates for spack-stack-1.9.0. Thanks again everyone for testing and reviewing. |
Summary
In preparation for spack-stack-1.9.0, this PR unpins or updates several packages in
configs/common/packages.yaml
(mostly Python packages). Most notably,py-shapely
(@ericlingerfelt FYI) andpy-numpy
(@DavidHuber-NOAA FYI) are updated.The
py-numpy
update may require bug fixes with the Intel classic compiler that @DavidHuber-NOAA worked on and that are currently under review in spack develop (see #1276).Included is an update of the S4 site config, which had several flaws that prevented building and testing this PR.
Testing
neptune-dev
template (@climbfuji)Applications affected
All.
Systems affected
None directly.
Dependencies
[email protected]
and add conflict for[email protected]
with[email protected]:
spack#484Issue(s) addressed
Resolves #1065
Working towards #1329
Checklist
All dependency PRs/issues have been resolved and this PR can be merged.