hanging while reading in gromacs trr trajectory

[dkonstan]

Expected behavior

quick loading of GROMACS universe with trr trajectory into MDAnalysis

Actual behavior

it hangs indefinitely in some internal process and is resistant to Ctrl^C so the process is something very internal probably. DCD format etc works fine with same topology.

Code to reproduce the behavior

uni = MDAnalysis.Universe("topol.top", "some_gromacs_trajectory.trr", topology_format="ITP")

Current version of MDAnalysis

• Which version are you using? 2.2.0
• Which version of Python (python -V)? 3.9
• Which operating system? Linux (cluster I think CentOS)

[richardjgowers]

@dkonstan how large is the trajectory in question? There is an index of frames built on first load (for trr format) and this might be taking too long. Does a much smaller trajectory file work as expected?

[dkonstan]

You are right, it is a VERY large file (~800 GB). However, these huge trajectories load fast in other formats. I don't see why TRR needs to have a routine that loops over the whole thing. Is it something specific about the TRR format? Right now, I am simply converting TRR to DCD using mdtraj (I'm sorry to use a competitor hehe) and then using MDAnalysis straightforwardly, but it would be nice not to have to do this.

…

On Wed, Aug 24, 2022 at 6:20 PM Richard Gowers ***@***.***> wrote: @dkonstan <https://github.com/dkonstan> how large is the trajectory in question? There is an index of frames built on first load (for trr format) and this might be taking too long. Does a much smaller trajectory file work as expected? — Reply to this email directly, view it on GitHub <#3789 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLHQP2X5AHY5V5Q2KT6X6TV22N3HANCNFSM57QVGQIA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[richardjgowers]

The TRR format doesn't allow seeking (jumping to a random frame) natively, so we build an index to be able to seek. Obviously this isn't working well for you. It would be nice if we could make this index building lazy (note frame offsets as they are read) or be able to disable it entirely for super large files like this.

[orbeckst]

Why do we have to build the index for TRR with iterating? TRR has a fixed frame size so we should be able to compute the index.

(XTC is a different issue.)

@dkonstan in general it’s important for MDA to have the index because that is the only feasible approach for us to guarantee fast random frame access for all trajectory formats while not reading the whole trajectory into memory.

[orbeckst]

@richardjgowers disabling the index will likely break fundamental assumptions about how we handle trajectories.

[richardjgowers]

@orbeckst maybe TRR isn't compressed but I think it still allows different strides in position force and velocity reporting... but I think yes maybe there's an analytical solution to seeking.

[orbeckst]

Hm, yes, you are right. You never know what you find in a TRR step.

[jbarnoud]

It would be nice if we could make this index building lazy (note frame offsets as they are read)

I just opened #3793 in that direction.

[dkonstan]

Got it, thank you! Yes that would be great.

…

On Wed, Aug 24, 2022 at 7:00 PM Richard Gowers ***@***.***> wrote: The TRR format doesn't allow seeking (jumping to a random frame) natively, so we build an index to be able to seek. Obviously this isn't working well for you. It would be nice if we could make this index building lazy (note frame offsets as they are read) or be able to disable it entirely for super large files like this. — Reply to this email directly, view it on GitHub <#3789 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLHQP5QKSOWK5AS6KJC2ILV22SPDANCNFSM57QVGQIA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[orbeckst]

Would it be useful to have some indication that index building is happening, such as a progressbar or at least a message?