Multiple termite petries!

[transkriptase]

Hello,

I use Sleap to track termites in a 9 cm petri dish (approximately 50 termites in one petri dish), and I'm wondering if you could help me with a few things. First, if I want to record and track multiple petri dishes at the same time (4 petri dishes), do you think Sleap would be efficient for that? My second question is about camera recommendations. Do you have any suggestions for an industrial camera or specific camera features? I tried using a phone camera with 60 fps, but I would prefer a system where I can integrate a camera if possible. I would be very grateful for your suggestions.

Thank you very much.

[talmo]

Hi @transkriptase,

Happy to help!

First, if I want to record and track multiple petri dishes at the same time (4 petri dishes), do you think Sleap would be efficient for that?

Yep. You'll have a much easier time if you can split the video so that there's only 1 petri dish in each video file, but technically it'll work with all of them in view. #1388 is a good example of another user with a similar question if you want to check out that thread as well.

My second question is about camera recommendations. Do you have any suggestions for an industrial camera or specific camera features? I tried using a phone camera with 60 fps, but I would prefer a system where I can integrate a camera if possible.

A key criteria for camera selection is the resolution per pixel that you get with your camera and field-of-view. The way you'll calculate the minimum resolution required is by determining the physical size of the smallest feature you want to track and dividing ~3-5 pixels by that length to get the minimum pixels per millimeter that you need.

For example, assuming your termites are around 3 mm, but you want to track their limbs which maybe are around 0.2 mm, then 5 pixels / 0.2 mm = 25 pixels / mm. If the arena they're in is ~10 body lengths = 30 mm, then you need a camera that can capture images that are at least 30 mm * 25 pixels / mm = 750 px (height and width).

Of course, you want to make sure that the images are not blurry at the distance you need to have it away from the subjects ("minimum working distance"), which may be tunable if you have an adjustable lens.

As far as frame rate goes, this depends a lot on the behaviors you care about. For social behaviors and coarse locomotion, usually 30-60 FPS is pretty good since most coarse invertebrate movements aren't much faster than ~15 Hz, but if you want to capture fine-scale kinematics, you might need to go as high as 150 FPS. Most importantly though, is that you want to reduce motion blur. This is achieved, even at lower FPS rates, by reducing the exposure time on the camera. ~5 ms or less is considered great, but keep in mind that this means fewer photons are captured by the sensor, so the image will appear darker and noisier. You can compensate for this by adding more illumination to the arena (though doing so can be tricky while avoiding specular reflections and excessive heating).

Control of these parameters sometimes isn't possible on conventional cameras, especially phone ones. The most commonly used cameras for behavior monitoring are "machine vision cameras", which allow for control of parameters like exposure and have great quantum efficiency (i.e., how much illumination you need to get good images). The most widely used ones are Baslers and FLIR cameras, but the specific model really depends on your use case. You can check out the ones we used in the SLEAP paper as a starting point, but I'd recommend looking at the termite literature to see what other labs have been using.

Let us know if you have any questions!

Cheers,

Talmo

[transkriptase]

Hi Talmo,

Thank you very much for the detailed response. I tried a video with four petries, but since I thought the resolution was not sufficient, I only marked one termite region (thorax) and labeled 6 frames. However, unfortunately, when I trained it, new frames did not open, and the existing labels also changed their positions. I think it would be more reasonable to continue with one. I will research cameras.

So, what is the possibility of behavior detection and real-time monitoring with this software? :) (just wundering)

Thanks again for your help and for this software.

[talmo]

Hey @transkriptase,

I tried a video with four petries, but since I thought the resolution was not sufficient, I only marked one termite region (thorax) and labeled 6 frames.

6 frames is very little (though I appreciate you have many termites per dish) and I'd definitely recommend trying with a ~20-30 to get a good sense for whether it's possible to reliably detect them.

Also, definitely make sure that you're labeling all of the petri dishes since all animals in the frame need to be annotated or the model will mistake the unlabeled instances for background, resulting in poor performance.

the existing labels also changed their positions

Not sure what this means exactly, but it sounds similar to what happens when videos aren't reliably seekable (see this page for more info).

So, what is the possibility of behavior detection and real-time monitoring with this software? :) (just wundering)

We recommend checking out Bonsai-Rx which has a SLEAP integration for realtime tracking! We also have a notebook showcasing how to implement something like this yourself.

Let us know if you have any questions!

Cheers,

Talmo

[transkriptase]

Hello Talmo,

thank you so much for your kind answers. I am running sleap on hpc with one hour video with 50 individuals and get predictions.slp file as output but to convert the file to .h5 file or .mp4 file, I can not open this .slp file because it is too big! I can not also use sleap-convert on hepc because my predictions.slp file and get this error:
"Unable to allocate 19.8 TiB for an array with shape (100001, 13, 2, 1046250) and data type float64"
I do not want to use shorter than one hour video because my research about social behavior of termites in special conditions and I need to track them min 1 hour. Could you please suggest me some way to convert .slp files to .h5 file at least?

Thank you so much in advance

[talmo]

Hi @transkriptase,

Right, so that last dimension is the issue here (1046250). This corresponds to the number of tracks, which clearly is way too many!

You may want to set some track filtering parameters with the sleap-track CLI. Useful here would be:

• -n 200: Set the maximum number of animals detected per frame to 200
• --tracking.post_connect_single_breaks 1 attempt to merge tracks when only a single track fragmentation happens (this may be less useful in your case since you have so many animals)
• --tracking.clean_instance_count 200 keep at most 200 tracks after associating detections
• --tracking.min_new_track_points 3 only create new tracks when at least 3 points are detected on the instance (adjust this to more or fewer depending on the number of nodes on your skeleton)
• --tracking.clean_iou_threshold 0.7 eliminate detections that are highly overlapping (IOU > 0.7)
• --tracking.min_match_points 4 require that at least 4 points are associated with each other when linking detections

There are some more that can tweak the performance, but I think you definitely want to reduce the number of tracks first. Ideally with these settings you'll get down to 200, so you should be able to edit in the GUI and convert to h5 (though I recommend inspecting them in the GUI since you're likely to get a lot of ID switches still).

Another option would be to run inference without tracking (don't set --tracking.tracker), and then use this notebook on post-inference tracking to interactively tweak the tracker options above.

Cheers,

Talmo