Track mice from a bottom-up view camera

[skyshi0]

HI all! Does anyone have experience on tracking mice poses from a bottom-up view camera. I'm currently label[nose tip, left hand, right hand, left leg, right leg, tail base] in a mouse, and only connect paws with each other(nose and tail base are two separate nodes). I got the display of "no instances are predicted". I've tried to adjust the input scale to 1.5 to match the size of the mouse but it still didn't work. I wonder if it's because I didn't connect the nose and tail base with other parts of the body, do you suggest to create a center node to connect everything together, and what way will be the most convenient? Thank you!

[github-actions[bot]]

Thank you for starting a new discussion!

We appreciate your input and will review it soon.

Warning

A friendly reminder that this is a public forum. Please be cautious when clicking links, downloading files, or running scripts posted by others.

• Always verify the credibility of links and code.
• Avoid running scripts or installing files from untrusted sources.
• If you're unsure, ask for clarification before proceeding.

Stay safe and happy SLEAPing!

Best regards,
The Team

[gitttt-1234]

Hi @skyshi0,

Single-instance models doesn't depend on edges (connecting body parts). It only depends on the keypoint locations. So, this won't affect your model accuracy. The receptive field (RF) is very crucial for single instance models. Ideally the little blue box on the image should be almost the size of the animal in the frame (input scale and max stride can be adjusted to get an apt RF). In your case, RF seems to be right!

Could you provide more information about your labelling/ training - how many labeled frames you used for training and what's your image resolution? Was the training loss decreasing over the epochs?

One option to try out is to use top-down model - this model is very efficient in cases where the target is relatively small compared to the rest of the frame. To improve generalization, you could also set the rotation angle to -180/180 instead of -15/15.

Thanks,

Divya

[skyshi0]

Thank you so much for your reply, Here's more information about the project:

1. Currently I labeled 282 frames.
2. Image resolution: 1440x1080 px
3. Training loss over the epoch: it stoped early at the epoch 12, the initial epoch's loss 8.2340e-05 and the epoch 12's loss is 8.2770e-05, this is my 11th training. Two of my previous training returned some predictions so I corrected the labels, but the following training didn't return any instances.
   I'll try the top-down model. Do you suggest me to add a center node?

[gitttt-1234]

Hi @skyshi0,

Thank you, It seems like the model isn't learning (the training loss isn't decreasing much). A couple of options to try out: you can increase the max stride (this will help the model to learn more global context), keeping the scale as 1.0, and increase the rotation augmentation to -180/180. Note: RF should cover the entire animal, i.e., the little blue box on the image should be almost the size of the animal in the frame

Another option is to try out the Top-down model, which is a two-stage model. First, a centroid model is trained to identify the centroids of each animal instance and then, a centered-instance model is trained, which crops each animal instance around the centroids and predicts keypoints from the cropped instance. For top-down mice model, tail-base could be a good centroid node. For centroid model, you can set the input scaling to 0.5 (this model doesn't require high resolution to predict just one node) and adjust the receptive field to make sure the entire mice can fit inside the blue box shown in the GUI.

Let us know if these solutions work for you!

Thanks,

Divya

[skyshi0]

Thank you! I changed the settings to (1)centroid model: scaling 0.5, rotation augmentation -180/180, maximal stripe: 16; (2)centered-instance model: scaling 1; rotation augmentation -180/180, maximal stripe 32. to include the whole body of the mouse including its tail tip. It returns predictions to me now. I'm now refining the model to predict the nodes better(currently can't capture the nose tip well, the predicted nose tip is around the mouth instead even though I corrected them several trainings) I also have a follow-up question about the RF, my other videos may have a slight offset of the mouse size, should I set the RF to include the largest one for all videos? Thank you for your help again!

[roomrys]

Followed-up in #1974.