OKS standard deviation

[antortjim]

Hi SLEAP devs!

I was wondering how was the standard deviation used to compute the OKS chosen

Having a look at the code, I see the ki parameter from https://arxiv.org/pdf/1707.05388 (the Gaussian standard deviation) is implemented in SLEAP here

sleap/sleap/nn/evals.py

Line 134 in 0134ab4

def compute_oks(

as the stddev argument, which by default is 0.025. I see the function expects either a unique value for all body parts (keypoints) or a list of numbers, one for each body part. However, going through the call chain that leads me to compute_oks, I could not find stddev being computed from the data anywhere

sleap/sleap/nn/evals.py

Line 750 in 0134ab4

metrics = evaluate(labels_gt, labels_pr)

i.e. simply the ground truth and predictions are passed, but not the standard deviation nor the scale.
I also found this discussion over here that refers to the same function, and also only passes the ground truth and predicted labels
#1253 (comment)

From the OKS preprint:

The Gaussian’s standard deviation ki is specific to the keypoint type and is scaled by the area of the instance s, measured in pixels, so that the OKS is a perceptually meaningful and easy to interpret similarity measure. For each keypoint type, ki reflects the consistency of human observers clicking on keypoints of type i and is computed from a set of 5000 redundantly annotated images

So I understand the standard deviation should be computed from a set of redundant labels, i.e. labels that have been labeled more than once, as a way to estimate human error. My SLEAP labels do not have any redundancy, so I am guessing I need to use a predefined and constant standard deviation (which probably is the 0.025 value in the code). My question is then how was this stddev value picked?

Thanks again for your great support, really helpful 🙏
Antonio

PS I have a pretty bad Precision-Recall curve OKS @ 0.90 but I suspect it is coming from some challenging body parts that are luckily not essential for my purpose. So I was wondering if I can recompute the metrics plots from https://sleap.ai/notebooks/Model_evaluation.html using only a subset of the skeleton (with the key body parts). In other words, I want to know whether a bad curve at high OKS is coming from all body parts in general or if as I suspect, it is coming from a few very challenging ones.

[talmo]

Hi @antortjim,

We have a discussion about this in the SLEAP paper here.

Basically, we chose a conservative value that is based on the "easiest" keypoint from human pose data. This means that the OKS estimates in SLEAP are likely to be underestimates of the true value because we assume all keypoints

To derive the actual correct values, you'd need to have many samples of the same pose being labeled, which is a huge pain.

Let me know if that helps or if you have any q's!

Talmo

[antortjim]

Thank you Talmo, I forgot that I should not just check the docs but of course also the paper! It makes sense, thanks again!

For completeness, in case someone wants to also do this at some point, to evaluate the model ignoring some body parts:

import sleap.io.dataset
from sleap.nn.evals import evaluate
model_folder="/path/to/models/date_time.model_type.n=n"

ignored_bodyparts=["foo", "bar"]

predictions=sleap.io.dataset.load_file(f"{model_folder}/labels_pr.val.slp")
ground_truth=sleap.io.dataset.load_file(f"{model_folder}/labels_gt.val.slp")

# original metrics
metrics=evaluate(labels_gt=ground_truth, labels_pr=predictions)
print("Error distance (50%):", metrics["dist.p50"])
print("Error distance (90%):", metrics["dist.p90"])
print("Error distance (95%):", metrics["dist.p95"])

ignored_nodes=[node for node in ground_truth.skeleton.nodes if node.name in ignored_bodyparts]
for node in ignored_nodes:
    ground_truth.skeleton.delete_node(node)

# this throws an error because evaluate expects the skeletons to have the same number of nodes
# metrics_simple=evaluate(labels_gt=ground_truth, labels_pr=predictions)
ignored_nodes=[node for node in predictions.skeleton.nodes if node.name in ignored_bodyparts]
# so I remove them also from the predictions
for node in ignored_nodes:
    predictions.skeleton.delete_node(node)

new_metrics=evaluate(labels_gt=ground_truth, labels_pr=predictions)
print("Error distance (50%):", new_metrics["dist.p50"])
print("Error distance (90%):", new_metrics["dist.p90"])
print("Error distance (95%):", new_metrics["dist.p95"])