How do I merge a large number of predictions into a project without a lot of clicking?

[davorvr]

Hi, I'm performing training and prediction on an HPC cluster, which produces upwards of 200 .slp files. When I download them back to my own computer, which has a different path to videos (prefix changes from "/root/mnt/..." to "/home/davor/...", the rest of the path remains the same), I get this window for each file:

Followed by this:

Is there a way to do this for for all files at once, maybe via the command line or the Python API? Thanks!

[github-actions[bot]]

Thank you for starting a new discussion!

We appreciate your input and will review it soon.

Warning

A friendly reminder that this is a public forum. Please be cautious when clicking links, downloading files, or running scripts posted by others.

• Always verify the credibility of links and code.
• Avoid running scripts or installing files from untrusted sources.
• If you're unsure, ask for clarification before proceeding.

Stay safe and happy SLEAPing!

Best regards,
The Team

[talmo]

Hi @davorvr,

For sure! The easiest way would be to use sleap-io, our standalone library for working with SLP files exactly for this use case. Files saved with sleap-io will work seamlessly with SLEAP and vice-versa.

I'd recommend installing it in a new environment if possible, but let us know if it gives you any issues if you prefer to do it in the sleap env. It has both pip (recommended if doing it in the sleap environment) and conda (recommended if doing it in its own environment) packages.

We recently added some convenient functionality for exactly this situation via Labels.replace_filenames. For example:

import sleap_io as sio

labels = sio.load_file("labels.v001.slp")

# Fix paths using prefixes.
labels.replace_filenames(prefix_map={
    "D:/data/sleap_projects": "/home/user/sleap_projects",
    "C:/Users/sleaper/Desktop/test": "/home/user/sleap_projects",
})

labels.save("labels.v002.slp")

You can also give it a list of filenames (labels.replace_filenames(new_filenames=["new_path/video.mp4"])) or a dictionary mapping old filenames to new filenames (labels.replace_filenames(filename_map={"old_path/video.mp4": "new_path/video.mp4"})).

Let us know if that works for you or if you have any questions!

Cheers,

Talmo

[davorvr]

Thanks, the sleap-io tool looks great and it'll be a big help! Using it, I came across two things:

When trying to import one of the .slp prediction output files using sio.load_file, I'm getting a "permission denied" error. This is because the cluster I'm running SLEAP on is using a fakeroot container which mounts my home directory with all the necessary files under a directory within /root. It errors out here: https://github.com/talmolab/sleap-io/blob/main/sleap_io/io/video.py#L87 with a PermissionError:

PermissionError: [Errno 13] Permission denied: '/root/mnt/banana/2/NOR/Test/rotated_GX010150_right_top_upd.MP4'

Would it be possible to open up a labels file "lazily", without sleap-io trying to test for the existence of the video? Should I open an issue on the sleap-io repository?

And would it be possible to do this on a finished analysis file? This would allow me to run everything from training to generating analysis files on the cluster.

[talmo]

Hi @davorvr,

Hmmm, interesting, this usually works fine with videos that don't exist, but you're not the first to run into weird I/O errors when working with more exotic filesystems (we did too). This is a new one though

Do you mind opening an issue over in that repo with a full console log and a snippet from the offending code?

Re: analysis files: do you mean the HDF5 files we export in SLEAP ("Analysis HDF5") or just the final SLP files with predictions? (The former no, the latter yes.)

[davorvr]

Thanks, submitted: talmolab/sleap-io#116