Providing micro-sam as napari plugin widgets

[GenevieveBuckley]

I think we need a thread to discuss implementing napari plugin widgets for micro-sam.
This is follow up work to #165

[GenevieveBuckley]

So far I have experimented with a plugin widget for the micro-sam 2D annotator.
Some initial observations:

1. A lot of micro-sam assumes that there will be six layers, named "raw", "auto_segmentation", "committed_objects", "current_object", "prompts", & "box_prompts" . This assumption is hard coded into many of the codebase.
   • How to best to handle users choosing the raw input from layers that are already open? Copying the layer or data seems inefficient. Maybe we can create a view (I can't remember if this caused an error in micro-sam or not). A simple but not-ideal workaround could be to rename the image layer to "raw", but this might make users confused later on.
   • How to handle creating the other empty layers ("prompts", "box_prompts", "committed_objects", ... etc.) There will need to be some refactoring of the existing functions to do this - the current initialize function creates a new napari viewer and then adds those empty layers (we don't want a new viewer), and the current update function assumes the layers already exist (not necessarily true anymore).
2. Type annotations. To have the @magic_factory magicgui thing work in the easiest way, you need to type annotations for the napari layers and viewer. You can't have a Union where the input could be a numpy array, or could be a napari image layer, because then magicgui does not know what kind of widget control to use.
   • We could create a wrapper widget function around the actual function we want. Seems an ok solution, but not as nice as just making sure your functions are type-hinted and then adding a new widget command to the napari.yaml
   • Possibly we could update the existing type hints in micro-sam, but this might cause backwards incompatibility problems we'd like to avoid.
   • Maybe I could think about trying to add support for Union type hints in magicgui (but this might take a long time, even if it is something Talley might be interested in having included)

[GenevieveBuckley]

Also including an error Constantin ran into (I have not seen the same error, but will keep an eye out for it now)
#165 (comment)

When trying to start the 2d annotator plugin I get an error:

RuntimeError: Failed to import command at 'micro_sam.sam_annotator.annotator_2d:annotator_2d_widget': /lib/x86_64-linux-gnu/libtinfo.so.6: version `NCURSES6_TINFO_6.2.20211010' not found (required by /home/pape/software/conda/miniconda3/envs/sam-plugin/lib/python3.10/site-packages/vigra/../../../././libncurses.so.6)

(Just fyi, just importing vigra works).

[constantinpape]

Hi @GenevieveBuckley,
thanks for opening the issue and working on this! Here are my initial thoughts on how to change the architecture to enable the plugins.

1. Computing Image Embeddings and "raw" image layer

We add a new widget for the embedding computation. The user can just select an image layer that is used to precompute the embeddings (and optionally specify a path where the embeddings are cached). This will also solve #161 (because we get a progress bar "for free") and #17 as the name "raw" is not hard-coded anymore.
As far as I understand we can't rely on global variables any more when we switch to the plugin, so we need to store the image embeddings somewhere else than in IMAGE_EMBEDDINGS, e.g. as a property of the viewer or the corresponding image layer. Same is true for the other global state we currently have.

2. Other hard-coded layers

For the other hard-coded layers (prompts, box_prompts, current_object, commited_objects, etc.) it would be best to keep them hard-coded, and if possible to just have the plugin create them when called. From my understanding this should be possible by having type annotated return values for these layers.

3. Type annotations and magicfactory

My current opinion is: we solve this via wrapper functions. We have the "actual" function that corresponds to the plugin and is decorated with @magicfactory and then write wrapper functions around it to keep support for the current functionality l(starting from python and CLI). I think this will be necessary in any case, even if you would solve the Union problem, because some of these wrappers will need specially functionality, e.g. taking the existing viewer and returning the viewer again.

4. Development

I would suggest to start with a PR that creates the plugin for the 2d annotation tool. Let's create the PR onto https://github.com/computational-cell-analytics/micro-sam/tree/dev, so that we can work on all the updates there and only merge into master once the plugins are all there, and the current functionality is adapted to the changes.

Let me know what you think about these points and plan @GenevieveBuckley! (And please let me know if you disagree with any of the technical points or have a better plan!!)

[GenevieveBuckley]

Response:

1. Computing Image Embeddings and "raw" image layer
   • "We add a new widget for the embedding computation" - ok, sure
   • "the name "raw" is not hard-coded anymore" - won't it still be hard coded in all of the sam_annotator python files? It's all the downstream stuff that makes this a little complicated.
   • That's a good point about the global variables. I didn't realise there were so many, that's good to know.
2. Other hard-coded layers
   • I don't quite understand what you mean by "From my understanding this should be possible by having type annotated return values for these layers."
3. Type annotations and magicfactory
   • Agree, wrapper functions for widgets seem like a good approach.
4. Development
   • I don't have access to push to your dev branch. You can either grant me write access, or I can open a PR (we both have permissions to push to that).

[constantinpape]

• "the name "raw" is not hard-coded anymore" - won't it still be hard coded in all of the sam_annotator python files? It's all the downstream stuff that makes this a little complicated.

It's not hard-coded in many cases:

$ grep -Inr "\"raw\"" sam_annotator 
sam_annotator/annotator_3d.py:160:    shape = v.layers["raw"].data.shape
sam_annotator/annotator_2d.py:64:        AMG.initialize(v.layers["raw"].data, image_embeddings=IMAGE_EMBEDDINGS, verbose=True)
sam_annotator/annotator_2d.py:68:    shape = v.layers["raw"].data.shape[:2]
sam_annotator/annotator_2d.py:170:    v.layers["raw"].data = raw
sam_annotator/annotator_tracking.py:226:    shape = v.layers["raw"].data.shape
sam_annotator/annotator_tracking.py:354:    shape = v.layers["raw"].data.shape

I think we can get rid of this without too much effort.

• That's a good point about the global variables. I didn't realise there were so many, that's good to know.

The global state for each annotator is defined in the first few lines of the annotator_... function, e.g. https://github.com/computational-cell-analytics/micro-sam/blob/master/micro_sam/sam_annotator/annotator_2d.py#L220-L221.
It's not that much for annotator_2d and 3d, tracking has a bit more (but that's the most complex one anyways and we should probably tackle it last).
I am not really sure what the best strategy is to represent this state in a plugin.

• I don't quite understand what you mean by "From my understanding this should be possible by having type annotated return values for these layers."

From what I understand from https://napari.org/stable/plugins/guides.html something along these is possible in the function that initializes the plugin:

@magicfactory
def my_plugin() ->  List[LayerTypes]:
   ...
   return [prompt_layeer, box_prompt_layer, ...]

and the viewer will create the corresponding layers, so that we can add these specific layers during plugin creation.
(If this is not clear: I can check this more myself and try to write a minimal example).

• Agree, wrapper functions for widgets seem like a good approach.

👍

• I don't have access to push to your dev branch. You can either grant me write access, or I can open a PR (we both have permissions to push to that).

Sorry, I wasn't quite clear. What I meant: the PRs should be for merging into dev rather than merging into master.
(The tool is used quite a bit now, and some people are using it by installing from current master to have newer features. By merging the PRs for the migration to the plugin in dev rather than master we avoid confusion due to incomplete plugin support etc.)

(And I am happy to give you write access if you want, but I would want to stick with specific PRs onto dev, so we wouldn't really need it).

[GenevieveBuckley]

One other thing I notice is that we want to make sure the path through the code doesn't include napari.run() (or too many v = napari.Viewer()) statements. I think we can manage to balance making things work from scripts as well as interactively.

[constantinpape]

One other thing I notice is that we want to make sure the path through the code doesn't include napari.run()

Good point. I think we can avoid this with the wrapper design (exemplified for the 2d annotation tool).

@magicfactory
def annotator_2d_plugin(...) -> ...:
    """This function implements the core logic and is used to provide the plugin"""

def annotator_2d(image_data, ...):
    """This function can be called in python (or from CLI) to start the annotation tool"""
    v = napari.Viewer()
    v.add_image(image_data)
    layers = annotator_2d_plugin(...)
    for layer_data in layers:
        v.add_layer(layer_data)
    napari.run()

[constantinpape]

A quick summary on what @GenevieveBuckley , @paulhfu and me discussed in a meeting this morning.

• We will break up the work towards the plugin into smaller packages, the next goals are:
  • Refactoring the global state Refactor global state #201
  • Exploring widget containers Explore widget containers #200
  • Using thread workers Thread worker and progress bar for model loading (and other long running computation) #202
  • Implementing an embedding widget Implement embedding widget #203
  • and enabling GUI tests in the CI Enable GUI tests in CI #205
• Based on this we can then
  • Implement the plugin for the 2d annotator, continuing (or replacing) Prototype napari plugin widget for annotator_2d #177
  • And see how to share functionality for the different annotation plugins Best design for plugins? #206 before porting all tools to a plugin

[constantinpape]

This has been addressed by #304. I have opened several follow-up issues to improve the plugin.