Handling duplicates

[gwbischof]

Fixes: #33

[danielballan]

Looks good. Do you think it's worth adding a flag to __init__ ignore_duplicates so that this is configurable and can be run in a "strict" mode for debugging purposes?

[tacaswell]

Do you think it's worth adding a flag to init ignore_duplicates so that this is configurable and can be run in a "strict" mode for debugging purposes?

I agree with this.

Having the check is better than just dropping the document, but I am still very confused by why we are getting multiple inserts and worry that it is indicative of a much more fundamental problem that needs to be run down.

[danielballan]

I am still very confused by why we are getting multiple inserts and worry that it is indicative of a much more fundamental problem that needs to be run down.

AFAIK we have never actually seen multiple inserts. But we know that it's theoretically possible for a Kafka consumer process to fail between successful insertion of a document and confirming to Kafka that the document has been successfully processed and need not be re-sent ("committing"). If we start seeing any log messages from this, we'll investigate, but I would expect to never see this in practice.

When this is used in-process with an RunEngine, it should fail hard if it receives a duplicate, but when this is used downstream of a message bus that is designed to err on the side of retrying, it should tolerate duplicates in this fashion but log them at a high level to prompt a human to investigate.

[gwbischof]

An important note here is that, resource uid index has been made unique. There are one or two beamlines that may have resource uids that are not unique. The plan is to require them to update their profile to only make resources with unique iud.
This change was needed to dedupe resource documents.

[tacaswell]

👍 for paranoia!

[danielballan]

The plan is to require them to update their profile to only make resources with unique iud.

It's important to be clear that this won't work we need to generate unique uids going forward and update their databases to remove all instances of duplicates uids in the past. The first is comparatively easy (and, I think, already taken care of everywhere). The second will require significant effort and great care.