[live stream] external references are not synced

[axelfahy]

Description

I am having trouble syncing external references of a report using live streams.

I tested two cases, one with "perfect synchronization" and one without it:

Without perfect synchronization:
If I create a report, add an external reference, it is correctly sync with the destination.
However, if I remove the external reference in the report afterwards, it is not removed on the destination.
Same thing happends with the label added and then removed.

With perfect synchronization:
If I create a report, add an external reference, the external reference is never added to the destination.
However, the label is added correctly to the report and removed afterwards as well.

Is this expected? How can I sync two opencti instances and have the external references and labels of a report synced when being added and removed as well?

Environment

1. OS (where OpenCTI server runs): MacOS, docker
2. OpenCTI version: 6.3.4
3. OpenCTI client: frontend
4. Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Setup a live stream between two instances of opencti
2. Create a report, add external reference and a label
3. Remove the external reference and the label from the report

Expected Output

I expect to have the external references and the labels added correctly, then removed.

Actual Output

External reference is never removed from the report, or not even added with perfect sync.

Additional information

Screenshots (optional)

Example without perfect sync:

Example with perfect sync:

[SamuelHassine]

@axelfahy You confirm you are in 6.3.4 and you've tester in 6.3.4? Can be related to the bug with have fixed this week-end.

[axelfahy]

Yes, I am in 6.3.4

[SamuelHassine]

Ok, @richard-julien confirmed we seem to have an issue with external references, we will come back to you.

[axelfahy]

To be sure I understand correctly, to have all the removal events applied on the destination, I must set the "perfect synchronization" option? Without this option, no removal will be done on the destination, even if there was no modification on the object to be removed on the destination?

[romain-filigran]

I observed another suspicious behavior with perfect sync.

Steps to reproduce:

1. Create a report with 1 label and 1 external reference --> Report is correctly sync with the external ref and the label
2. I update the report by adding a new label on the report --> New label is correctly added to the report but the external ref is removed.

[richard-julien]

Looks related to an incorrect validation function that prevent the external refs creation.

[SouadHadjiat]

@romain-filigran could you describe the exact steps to reproduce the issue ?