Enrich georeferenced items with information from nominatim server

[patrickdalla]

Nominatim maps a geolocation to textual address information. It may be useful to search these items by their city name, for instance. Also, this information can be used to cluster the plotted items by their administrative address information (country, city, etc..).
An URL should be configured, and if so, the processing would lookup addresses for every geolocated item.

[patrickdalla]

Hi nassif,

I've made a stub of this in Nominatim branch. But I have some doubts about the design.

First, the task makes one http request per item with geolocation. This can lead to many http requests. And this kind of request to complement information from other sources can become more common, made from other task also (maybe it already exists). There should be some centralized/normalized subsystem to control these kind of enrichment from http requests?

Second: This external information complement are what some tools call enrichment. Should we make some visual differentiation of enrichment information from original artifacts?

[lfcnassif]

Sorry for my delay.

First, the task makes one http request per item with geolocation. This can lead to many http requests.

You can group items and make requests in batches, overriding the AbstractTask.sendToNextTask() method. Be careful to send all batched items to the next task after they are processed together, otherwise you can miss case items. We have this example:
https://github.com/sepinf-inc/IPED/blob/master/iped-app/resources/scripts/tasks/NSFWNudityDetectTask.py

And this kind of request to complement information from other sources can become more common, made from other task also (maybe it already exists).

We have the --downloadInternetData command line param to enable this behavior. Currently it is used just by WhatsAppParser to download media attachments still present on WhatsApp servers.

Second: This external information complement are what some tools call enrichment. Should we make some visual differentiation of enrichment information from original artifacts?

Yes, WhatsAppParser adds the ExtraProperties.DOWNLOADED_DATA property to flag downloaded medias and it also put a notice about that in the chat HTML report.

[patrickdalla]

"Yes, WhatsAppParser adds the ExtraProperties.DOWNLOADED_DATA property to flag downloaded medias and it also put a
notice about that in the chat HTML report."
Maybe not this flag, as it seems to indicate that the entire item was downloaded. In this case, only the metadata info will contain enriched/downloaded information. We could create some config file with the list of metadata names that are enrichment/downloaded.