Protect output file updates from potential concurrent access

[atc0005]

The goal is to allow simultaneous submissions from Splunk and other sources, but single batch de-duped writes to an output file. The goal is to prevent race conditions when appending to an existing output file.

[atc0005]

Channels? The Effective Go guide seems to suggest this approach.

[atc0005]

Channels? The Effective Go guide seems to suggest this approach.

(I think) that one workaround would be to use a read/write mutex. A write-only mutex wouldn't work because we'll need to ensure that all "writers" see the same existing data in order to not attempt writing out a new entry to a flat-file if another writer has an update queued up to add the same user account.

Another approach (likely better) is to use a queue for "block user" events and a single goroutine to process that queue. I assume that at least a write mutex would still be needed in case multiple copies of the app run simultaneously (for whatever reason), but needing to lock on read wouldn't apply any more?

[atc0005]

Another approach (likely better) is to use a queue for "block user" events and a single goroutine to process that queue.

This same approach should also work well for concurrent access to a database where a potential high-volume, but low speed requirement would apply. Using a queue approach should help alleviate issues with file-based databases like SQLite where concurrent write access would (at best) prove unreliable.

I assume that at least a write mutex would still be needed in case multiple copies of the app run simultaneously (for whatever reason), but needing to lock on read wouldn't apply any more?

Does a write mutex apply at the OS level, or only the application? If only the application, then there isn't any additional protection applied aside from having a file open at the time another application (or another application instance) wants to write to it.

[atc0005]

Unless things change as development moves further along, I've opted to use a mutex for flat-file access instead of implementing queues via channels. I may still use that approach as I did with the atc0005/bounce project once Teams notifications support is wired in.