Overwriting images and annotations

[mcib3d]

Hi,

When importing images with similar names in OMERO, there will be multiple images with the same name. One possibility is to delete the image with same name before importing.

Maybe an option "overwrite" could be welcome when importing images and adding annotations ?

Thanks.

[ppouchin]

You would like a method that would do something like the following?

import fr.igred.omero.Client
import fr.igred.omero.repository.ImageWrapper;
import fr.igred.omero.repository.DatasetWrapper;
import fr.igred.omero.annotations.TagAnnotationWrapper;
import fr.igred.omero.annotations.FileAnnotationWrapper;
import fr.igred.omero.annotations.TableWrapper;
import fr.igred.omero.roi.ROIWrapper;

Client client = new Client();
client.connect(host, port, username, password.toCharArray(), group);

DatasetWrapper dataset = client.getDataset(datasetId);
List<Long> ids = dataset.importImage(client, path);

Long[] newIds = ids.stream().toArray(Long[]::new);
List<ImageWrapper> newImages = client.getImages(newIds);
for(ImageWrapper image : newImages) {
	List<ImageWrapper> oldImages = dataset.getImages(client, image.getName());
	if(!oldImages.isEmpty()) {
		ImageWrapper oldImage = oldImages.get(0);
		String description = oldImage.getDescription();
		List<TableWrapper> tables = oldImage.getTables(client);
		List<TagAnnotationWrapper> tags = oldImage.getTags(client);
		List<ROIWrapper> rois = oldImage.getROIs(client);
		//List<FileAnnotationWrapper> files = oldImage.getFileAnnotations(client);
		//List<MapAnnotationWrapper> kvPairs = oldImage.getMapAnnotations(client);
		image.setDescription(description);
		image.saveAndUpdate(client);
		for(TableWrapper table : tables) image.addTable(client, table);
		for(TagAnnotationWrapper tag : tags) image.addTag(client, tag);
		for(ROIWrapper roi : rois) image.saveROI(client, roi);
		// for(FileAnnotationWrapper file : files) image.addFileAnnotation(client, file);
		// for(MapAnnotationWrapper kvPair : kvPairs) image.addMapAnnotation(client, kvPair);
		// Missing methods: addFileAnnotation, getMapAnnotations
		client.delete(oldImage);
	}
}

client.disconnect();

Currently, I think 2 methods are missing to do that, but it should be easily fixed.
Even a "copyAnnotations()" method could prove useful, I'll see how this could be done.

However, the code above has one very obvious limitation: how do you handle the case where multiple images share the same name in the dataset? In this snippet, I chose to only replace the first image found with the same name, but should it always be like that?

[mcib3d]

Hi,
Sorry, I will try to be more detailed next time lol . My problem is when I run my automation script, I would like to overwrite the previous version of my image with the new one.
Let say I have an image called MyImage.tif , I run my script and import MyImage-segmented.tif . I then improve my script and import a new version of MyImage-segmented.tif, I would like this new version of the image to overwrite the previous version.
What I do now is check if an image with the same name exists, and delete it before importing the new version.
The same if I want to add a new version of an annotation.
Thanks.

[ppouchin]

Ok, that's what I thought, and the code above should do that.
I have created a new branch to simplify it a bit, but I haven't tested it myself yet (only automated tests).

With this branch, replacing images could be done with something as simple as:

import fr.igred.omero.Client
import fr.igred.omero.repository.ImageWrapper;
import fr.igred.omero.repository.DatasetWrapper;
import fr.igred.omero.roi.ROIWrapper;

Client client = new Client();
client.connect(host, port, username, password.toCharArray(), group);

DatasetWrapper dataset = client.getDataset(datasetId);
List<Long> ids = dataset.importImage(client, path);

Long[] newIds = ids.stream().toArray(Long[]::new);
List<ImageWrapper> newImages = client.getImages(newIds);
for(ImageWrapper image : newImages) {
	List<ImageWrapper> oldImages = dataset.getImages(client, image.getName());
	if(!oldImages.isEmpty()) {
		ImageWrapper oldImage = oldImages.get(0);
		String description = oldImage.getDescription();
		image.setDescription(description);
		image.saveAndUpdate(client);
                image.copyAnnotations(client, oldImage);
		List<ROIWrapper> rois = oldImage.getROIs(client);
		for(ROIWrapper roi : rois) image.saveROI(client, roi);
		client.delete(oldImage);
	}
}

client.disconnect();

Before pulling this to the main branch, I'll have to do some tests, and maybe I'll make a specific method for images to copy description, annotations and ROIs at once.

However, as I said, I'm not sure I can make a method to replace on import as the policy for this could vary from person to person: some people may want to replace all the previous images with the same name and copy all the annotations to the new image before deleting all the older ones, others may prefer to only copy data from the first image with the same name (as the code above does).

If I make a function to reduce this "replace on import" to the following, would it be sufficient?

import fr.igred.omero.Client
import fr.igred.omero.repository.ImageWrapper;
import fr.igred.omero.repository.DatasetWrapper;
import fr.igred.omero.roi.ROIWrapper;

Client client = new Client();
client.connect(host, port, username, password.toCharArray(), group);

DatasetWrapper dataset = client.getDataset(datasetId);
List<Long> ids = dataset.importImage(client, path);

Long[] newIds = ids.stream().toArray(Long[]::new);
List<ImageWrapper> newImages = client.getImages(newIds);
for(ImageWrapper image : newImages) {
	List<ImageWrapper> oldImages = dataset.getImages(client, image.getName());
	if(!oldImages.isEmpty()) {
		ImageWrapper oldImage = oldImages.get(0);
                image.copy(client, oldImage);
		client.delete(oldImage);
	}
}

client.disconnect();

Edit:
Or maybe I missed something with "annotations". You also want to replace annotations?
Like tags?
When you replace an image, do you actually want annotations to be transferred to the new image?

[ppouchin]

Actually, maybe replacing ALL the images (in the dataset) that share the same name would satisfy your needs.
I could imagine a "DatasetWrapper::importAndReplaceImages(client, file)" method that does the following:

1. Import the file
2. For each image imported, search other images with the same name in the dataset
   a. For each image with the same name, copy the annotations and ROIs
   b. Delete all these older images

[mcib3d]

Hi,
Yes you are right, overwriting images means overwriting the pixels information, not necessarily the other information like ROI or attachment. So yes a DatasetWrapper::importAndReplaceImages(client, file) function would be perfect. Thanks !

I guess it should be the same for annotations like attachments.

[ppouchin]

Ok. I thought PR #26 would solve this, but I forgot to include a way to replace file annotations.

Moreover, the importAndReplaceImages does not properly work when images come from a file with multiple images (as all images from the file have to be deleted, but nothing guarantees it is the case).

I think images that are replaced should only be "unlinked", because:

1. They might still be referenced from a well or a different dataset.
2. Other images from the same fileset might still be referenced elsewhere - or may not have been imported/replaced.

Unlinking and then only deleting if every image from the fileset is orphaned should be possible, although I haven't tried yet.

For attachments, unlinking is easy too while deleting poses a similar problem: an attachment could be used by a different object somewhere else. It is a bit harder though: currently, I'm not sure how to get the number of objects linked to an annotation, besides going in HQL through every *AnnotationLink table (which is not pretty). Annotation::annotationLinksCountPerOwner only returns -1, as annotation links are not loaded somehow.

Also, I may rename importAndReplaceImages to just replaceImages, but it may not be as obvious?

[mcib3d]

Yes an importAndReplaceimage method would be perfect, and it is a good idea to just unlink the replaced data, it will be then the user responsibility to delete them. (importAndReplace is less confusing than just replace)

[ppouchin]

The last commit I made included a method only called "replace" which provided a boolean option to specify if the data should be unlinked or deleted.
I've been a bit busy lately, but I'll try to provide an argument to choose between:

1. Unlink
2. Delete
3. Delete if everything is orphaned

And maybe offer a default method with less arguments that only unlinks objects.

[ppouchin]

@mcib3d Ok. I made methods to import and replace images or attachements.
In both cases, there is a default method, which will only unlink old objects and a method with an additional argument to specify if the old objects should be:

• unlinked
• deleted
• deleted if they become orphaned (and other images from the same fileset are orphaned too)

If it's good for you, I can release the 5.9.1 version with these modifications.

[mcib3d]

Hi @ppouchin ,
That seems perfect and quite easy to use. Thanks !

[mcib3d]

And maybe unlinked could be the default policy, just to play safe.

[ppouchin]

And maybe unlinked could be the default policy, just to play safe.

When a method is called without the policy argument, objects are only "unlinked", I guess that's the behavior you're expecting.

One possible limitation that exists though is that annotation links are copied from the old images, but not removed from them, and that includes map annotations (key/value pairs). Therefore editing a value on the new image will be reflected on the old one.

[ppouchin]

Solved with the release of 5.9.1.