feature request: recomputing spot detections after users' manual edits

[jiyuuchc]

Background

Trackmate provides a rich set of UIs for users to modify the results from automatic algorithms. For example, it has keyboard shortcuts to either delete or add new spots. However, if the user manually added a spot by specifying a location, it typically is not annotated with as rich set of feature annotations. It would be nice if Trackmate could compute the same features for all the spots including manually added ones.

Feature request

For the detectors that support such a feature, Trackmate should call them to perform an update of spot detection. Trackmate should provide the additional annotations, e.g., the additional spot locations that the user had given. The detector should then fill in the missing information (radius, ROI) as needed.

Potential issues

The timing of the update needs to be discussed. The detector might be slow, so synchronous (blocking) update might be bad user experience. On the other hand, asynchronous update (esp if without UI indicator) can be counter-intuitive and confusing. A potential simple solution might be ask user to click a button to specifically request an update.

Relavance

I am the co-author of trackmate-lacss, a detector using the deep-learning model lacss as its back-end. We are interested in implementing such an "update" feature in our detector. In fact, the core functionality already exists in the back-end, but is not exposed to the user.

In addition, recently there are some research on a type of "segment-anything" models that are specifically designed to do object detection with a user-provided hint, e.g. a location. These models also benefit from this feature, if someone is to implement them in Trackmate.

[tinevez]

Hello @jiyuuchc
I am not sure I get it right.

There is an action in the last panel called 'recompute features' that does what it says. Could it be what you are looking for?

[jiyuuchc]

*** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** User-supplied spots typically only has X-Y values (or XYZ in 3d), which is different from the spots generated from our cell detector, which has the full segmentation, from which TrackMate computes a lot of other features (area, circularity etc). When we worked on analyses of BYTETrack tracking, we had to do a bit of heuristics on user-added spots, because we cannot get correct IOU values on these. For us, a better solution is to ask the detector to re-segment based on the user-supplied hint ( the locations ). This would make the downstream task more consistent. However to do that in Trackmate, we need upstream support to add this capability. Does that make sense?

…

________________________________________ From: Jean-Yves Tinevez ***@***.***> Sent: Thursday, November 7, 2024 3:32 PM To: trackmate-sc/TrackMate Cc: Yu,Ji; Mention Subject: Re: [trackmate-sc/TrackMate] feature request: recomputing spot detections after users' manual edits (Issue #310) *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** Hello @jiyuuchc<https://urldefense.com/v3/__https://github.com/jiyuuchc__;!!Cn_UX_p3!lvM5xA7xaa3NUfoi8Ene9qog6vyrX_PWWAz2Pha7BLJ0NXzyYakQqX1a-osgOcCwo9JaJOZLaAHcLyJnWAg3Gw$> I am not sure I get it right. There is an action in the last panel called 'recompute features' that does what it says. Could it be what you are looking for? — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/trackmate-sc/TrackMate/issues/310*issuecomment-2463158037__;Iw!!Cn_UX_p3!lvM5xA7xaa3NUfoi8Ene9qog6vyrX_PWWAz2Pha7BLJ0NXzyYakQqX1a-osgOcCwo9JaJOZLaAHcLyJRjQy_Sw$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AAKRPNS7AXZXLL4ZCW2HQSDZ7PE6ZAVCNFSM6AAAAABRL3AFLWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRTGE2TQMBTG4__;!!Cn_UX_p3!lvM5xA7xaa3NUfoi8Ene9qog6vyrX_PWWAz2Pha7BLJ0NXzyYakQqX1a-osgOcCwo9JaJOZLaAHcLyKsX5p-JA$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[tinevez]

Yes thank you!
It seems like a gigantic task however:
We need to make it possible for a user to pick any existing algorithm to operate locally in neighborhoods around a set of discrete locations. So a new framework, UI, extension mechanism to do so in TrackMate.

In the forthcoming version 8, it will be possible for users to edit manually segmentation results. Would that work instead?

[jiyuuchc]

Obviously I am bias here, but I don't think it is that difficult to implement.

From the detector-writer point of view, all we need is that the TrackMate make a call at some point -- maybe when user clicked an update button:

if detectorFactory.canUpdate() {
    updatedSpots = detectorFactory.getDetector(interval, frame).updateSpots(currentSpots);
}

This is a bit exaggeration -- there are also globalDetector to worry about, and there are some post-processing to do -- but it is not that far from the truth.

That said, I understand if you think this feature doesn't bring enough value -- after all, currently I don't know any other detector except ours that is actually able to implement this function. And we can always perform the update outside TrackMate using the export file.

[tinevez]

Yes ok. In that case, what about putting such a thing in the 'semi-automatic tracking' feature?
Right now, when the user presses 'shift-A' with a spot selected, TrackMate moves to the next frame, runs detection in a small neighborhood around the spot location, and links it to the most favorable spot found in the neighborhood.
Right now, we just use the LoG detector for detection. But we could make a proper config panel that specifies what detector to use for semi-automatic tracking, with parameters.

[jiyuuchc]

Sorry for the slow response. I am not familiar with the semi-automatic tracking workflow and has to take a bit of time to look up the relevant code in Trackmate to understand what it does.

Unfortunately I don't think the proposal would work for our use case.

One type of data we are currently testing on are bright-field microscopy of proliferating hematopoietic progenitor cells. Our detector can detect/segment most cells correctly, but it still makes some mistakes, e.g., it occasionally detect two contacting cells as one when the cell boundary is blurry in the image. In those cases, we rely on users to manually delete the wrong segmentation and add two new spots indicating there are two cells.

In a semi-automatic tracking setting, the detector would still have no idea that it should segment for two cells instead of one, so it will make the same mistake as it currently does.

As stated before, we fully understand the requested feature may be deemed too niche and not fit into the roadmap of Trackmate development.