Evaluations being recorded to Kafka multiple times

[zhouzhuojie]

Related issue: openflagr/flagr#463

@fenriskiba

Evaluations sent through the Kafka Recorder are getting duplicated intermittently.

Expected Behavior

Each evaluation should have a single message sent to Kafka.

Current Behavior

Approximately 0.65% (sampled from roughly 6 million Kafka messages) of evaluations are recorded to Kafka 2 or more times.

Steps to Reproduce (for bugs)

1. Use Vegeta to produce about 2000 evaluations per second with randomly generated entity IDs for about 15 minutes
2. Use the Sarama example apps (with some minor formatting adjustments) to consume the generated messages in Kafka and pipe the results into a CSV
3. Identified multiple identical payload messages in the CSV

Context

Creates some confusion/misleading conclusions if someone doesn't know to dedup data when doing analysis on an AB Test.

Your Environment

• Flagr v1.1.12
• Kafka v2.3.1

[zhouzhuojie]

This might be related to kafka's Exactly-once vs At-least-once semantics, and the producer's default setting within Flagr right now is https://github.com/openflagr/flagr/blob/main/pkg/handler/data_recorder_kafka.go#L52

Some more configuration changes might be there to support it. PRs are welcome to enable configurational changes to kafka settings.

https://stackoverflow.com/a/67909822

[github-actions]

Stale issue message