GSD tests are failing on big-endian architectures #1829
Comments
|
@rathann just so I understand your preferred solution: Thus you would like |
|
Yes, if that's possible. Otherwise I'll probably have to stop shipping MDAnalysis built for big-endian arches. |
|
Can we flag gsd tests as known failures on big endian? Ie can we detect
this from inside python.
…On Wed, 14 Mar 2018, 12:19 rathann, ***@***.***> wrote:
Yes, if that's possible. Otherwise I'll probably have to stop shipping
MDAnalysis built for big-endian arches.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#1829 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AI0jB4xYeaA57KK1xnATBtivJ_vFUUOuks5teQq6gaJpZM4So0tr>
.
|
|
Tests are one thing, but the other would be if @rathann ships builds of MDAnalysis that are known to be broken. That would be bad and I don't think anybody wants that.
I would like MDA to be useable on most HPC so big-endian builds should be possible.
If we make gsd optional again then we need to protect the gsd import --- perhaps at the format registration stage? For the tests we can just skip if gsd is not present.
…--
Oliver Beckstein
email: [email protected]
Am Mar 14, 2018 um 05:56 schrieb Richard Gowers ***@***.***>:
Can we flag gsd tests as known failures on big endian? Ie can we detect
this from inside python.
|
|
@orbeckst yeah - we could not import GSD on that architecture. Something like: # coordinates/__init__.py
if not BIG_ENDIAN:
from . import GSD
else:
warnings.warn('GSD not available because of incompatibility')This should make MDA blissfully unaware of GSD as a format keyword. But with this the install process could be the same for both. If we made gsd optional as a requirement then it wouldn't be installed by default, which means you'd get annoying error messages put in by us like |
|
I would just do a pipRequiring condaI don't know about conda packages – can they be made architecture-aware (@kain88-de ?). Managing dependencies...I don't like tying MDAnalysis to other supporting packages that might not be supported as well as the main dependencies or MDA itself. We should think of a way to mitigate these problems. In this case I am rather prepared to downgrade |
|
Yes, that's my understanding, but I don't think @rathann wants to ship a package that is known to be non-functional on the arch. (I wouldn't) |
|
By the way, for users to install every dependency we can use For instance, right now using My main issue is then what to do with conda packages. We could make that it installs a broken gsd as a dependency (the conda package of gsd will happily install itself) and then check ourselves if this is a broken gsd, disable it. When gsd raises its own error then we can test for that. |
|
The thing is that there are many ways to install a package. Even if it is not available as a RPM because @rathann knowns it does not work, it can still be installed with pip. Also, just basing the error message on if GSD is installed or not means the message will state that the package is not available (pushing the user to install a package that is broken for its arch) and not that it will not work. |
|
The way to implement @richardjgowers suggestion seems to be: BIG_ENDIAN = (sys.byteorder == 'big') |
Preparing an alternative big-endian file that is read on the appropriate platform might be an easier solution. @rathann could start with a patch for the package system and we can then look to include it. |
|
It doesn't work in the way @kain88-de described above. |
According to the author, GSD is not cross-platform compatible. Since the format isn't intended for sharing trajectories. Making the library an optional dependency won't change that. The question is if one can still read/write files when staying on the same computer, independent of its architecture. |
|
Thanks for the explanation @kain88-de. So the first question to answer is if gsd can be built successfully on big-endian and if it is able to process GSD trajectories on big-endian. (In our case "process" would mean to be able to read a gsd trajectory that was produced on big-endian and write a trajectory that can also be read on the same machine.) If this is the case then I am coming around to the idea for MDA to have a little-endian and a big-endian test trajectory and we would then at test-time decide which one to present to our GSD tests. (We might even be able to have our GSDReader check if a trajectory was produced on big- or little endian and then fail gracefully, at least until GSD itself implements the check.) Does anyone have access to a big-endian machine to
|
|
@orbeckst I have a ppc64 VM and can do it, but I'm going to need some step by step instructions for producing a trajectory. |
|
I don't know how to create a native GSD trajectory, @rcortini produced the original test trajectory – can you chime in? MDAnalysis does not have a GSD writer but one could use the gsd.hoomd library directly. But before generating some random trajectories I'd rather investigate if there's a way to re-create our
|
|
@rcortini can you briefly tell us how you created the HOOMD gsd trajectory that we use as a test file in MDAnalysis? Thanks! |
|
Hi, sorry for the late response. Here is a piece of code that will produce an almost identical trajectory. It requires the installation of the HOOMD-blue python package. from __future__ import division
import hoomd
import hoomd.md
# init HOOMD-blue
hoomd.context.initialize("");
# init particles in the system
uc = hoomd.lattice.unitcell(N = 1,
a1 = [10.8, 0, 0],
a2 = [0, 1.2, 0],
a3 = [0, 0, 1.2],
dimensions = 3,
position = [[0,0,0]],
type_name = ['R'],
mass = [1.0],
moment_inertia = [[0,
1/12*1.0*8**2,
1/12*1.0*8**2]],
orientation = [[1, 0, 0, 0]]);
system = hoomd.init.create_lattice(unitcell=uc, n=[2,18,18]);
system.particles.types.add('A');
# define rigid bodies
rigid = hoomd.md.constrain.rigid();
rigid.set_param('R',
types=['A']*8,
positions=[(-4,0,0),(-3,0,0),(-2,0,0),(-1,0,0),
(1,0,0),(2,0,0),(3,0,0),(4,0,0)]);
rigid.create_bodies()
# define particle interactions
nl = hoomd.md.nlist.cell()
lj = hoomd.md.pair.lj(r_cut=2**(1/6), nlist=nl)
lj.set_params(mode='shift')
lj.pair_coeff.set(['R', 'A'], ['R', 'A'], epsilon=1.0, sigma=1.0)
# define integrator
hoomd.md.integrate.mode_standard(dt=0.005)
rigid = hoomd.group.rigid_center();
hoomd.md.integrate.langevin(group=rigid, kT=1.0, seed=42);
# specify trajectory output
hoomd.dump.gsd("example.gsd",
period=2e3,
group=hoomd.group.all(),
overwrite=True);
# run simulation
hoomd.run(4e3);This is basically the example of the "Rigid Rods" in the |
|
Many thanks. (Sorry for the long delay.) I think we should pick up this issue for 0.19.x.
|
|
This sounds like an upstream thing, we don't actually read the GSD file directly |
|
Oh I see, we want two different gsd files to test... |
|
FWIW, it is now possible to test the |
|
@tylerjereddy how does one do big-endian tests on Travis CI? Do you have a link to docs? – Are you referring to the Arm64 support https://blog.travis-ci.com/2019-10-07-multi-cpu-architecture-support ? |
|
See here: https://blog.travis-ci.com/2019-11-12-multi-cpu-architecture-ibm-power-ibm-z The NumPy PR doing this a few weeks ago: numpy/numpy#14907 Note that we may have mixed results considering we are much higher up the ecosystem pyramid (some of our deps may have to be built from source each time on that arch until the conda/pypi binaries become available with newer standards, etc.; aggressive caching might alleviate that after the initial build, maybe) |
|
Confirming tests are still failing in v2.7.0 on a big-endian system (s390x). Debian has an s390x porterbox. You could request access if you'd like to dig into it more closely. |
|
Thanks @drew-parsons It might be worth mentioning that GSD is no longer a required component of MDAnalysis, if it helps get a Debian package out, I would suggest not including it as an optional dependency. |
|
True. I've already deactivated gsd for s390x so the current errors I'm seeing are something else. Perhaps I should file a separate bug report, but I was thinking to keep the big-endian (or at least s390x) errors in one place. The summary of errors I'm referring to is |
Expected behaviour
All tests pass.
Actual behaviour
Code to reproduce the behaviour
I reported this upstream and it looks like big-endian arches are not supported. Could GSD be made optional again (like in 0.16?)
Currently version of MDAnalysis:
0.17.0
The text was updated successfully, but these errors were encountered: