Added function to subset labels

[aperkes]

We're using sleap for a fun little project to check how neural nets perform on different substrates, and we realized we were going to need to subset the labels into a training and test set in order to check the accuracy. It seems like this would be a really helpful function to be built into the GUI, or at least the python module. It wasn't that hard to figure out how to write a script that would do it (copied below), and I thought it might be a nice additional feature, if it doesn't break anything to be included.

Thanks for all your continued work on SLEAP. It is, as always, just a joy to use.

import sleap
import sys
import numpy as np

## Load in the file. Make copies for train and test
in_file = sys.argv[1]
labels = sleap.load_file(in_file)
labels_train = sleap.load_file(in_file)
labels_test = sleap.load_file(in_file)

## get train/test split
n_labels = len(labels)
n_test = int(n_labels * 0.25)
n_train = n_labels - n_test

## Get indices for train test
np.random.seed(42)

random_indices = np.random.permutation(list(range(n_labels)))

indices_train = sorted(random_indices[:n_train])
indices_test = sorted(random_indices[n_train:])

## update labels with new indices
labels_train.labeled_frames = [labels_train.labeled_frames[i] for i in indices_train]
labels_test.labeled_frames = [labels_test.labeled_frames[i] for i in indices_test]

#import pdb;pdb.set_trace()
## Save updated slp files.
labels_train.save(in_file.replace('.slp','.train.slp'))
labels_test.save(in_file.replace('.slp','.test.slp'))

[gitttt-1234]

Hi @aperkes,

While we don't have this feature in the GUI, we do have .split() method from the sleap.io.dataset.Labels object. Here's a colab notebook to split and save a .slp file using this function.
https://colab.research.google.com/drive/15kct4dJw-PYvM41HYVE82qD7SVbiMRkM#scrollTo=xgc6MFtN-gYQ

Thanks!

Divya

[gitttt-1234]

Update from subgroup discussion:

This should be better exposed as a GUI feature, however, it needs to come before training happens in the core workflow.

Suggested PR:

1. Move the dataset validation fraction split fields from the model-specific tabs to the main tab.
2. Add an optional field to specify the test split.

Mechanics to generate the test split are already in sleap.nn.training, as well as evaluation metrics generation.

As a second PR, it would be nice to extend functionality in the evaluation metrics GUI to allow user to specify the split that is being visualized (defaulting to test, then validation depending on which is available) so you can see training/val/test. It would also be nice while we're poking around in there to have a way to export the metrics figure(s) to PDF and maybe CSV or something.

[aperkes]

Oh thanks! That's a very handy function and I'm embarrassed that I missed that in the docs. Thank you for the quick response!

…

On Fri, Aug 23, 2024 at 10:46 AM DivyaSesh ***@***.***> wrote: Update from subgroup discussion: This should be better exposed as a GUI feature, however, it needs to come before training happens in the core workflow. Suggested PR: 1. Move the dataset validation fraction split fields from the model-specific tabs to the main tab. 2. Add an optional field to specify the test split. Mechanics to generate the test split are already in sleap.nn.training, as well as evaluation metrics generation. As a second PR, it would be nice to extend functionality in the evaluation metrics GUI to allow user to specify the split that is being visualized (defaulting to test, then validation depending on which is available) so you can see training/val/test. It would also be nice while we're poking around in there to have a way to export the metrics figure(s) to PDF and maybe CSV or something. — Reply to this email directly, view it on GitHub <#1914 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRNK6QYWAWQJNT67S4ZNULZS5YPBAVCNFSM6AAAAABMYK7RJKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANBTGMZDSMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[talmo]

Another option I'll point to for future reference is sleap-io, our standalone package for working with SLP files that has a similar functionality:

import sleap_io as sio

# Load source labels.
labels = sio.load_file("labels.v001.slp")

# Make splits and export with embedded images.
labels.make_training_splits(n_train=0.8, n_val=0.1, n_test=0.1, save_dir="split1", seed=42)

# Splits will be saved as self-contained SLP package files with images and labels.
labels_train = sio.load_file("split1/train.pkg.slp")
labels_val = sio.load_file("split1/val.pkg.slp")
labels_test = sio.load_file("split1/test.pkg.slp")