exp/lighthorizon: Add thread-safe support for reading account list from FileBackend.

[Shaptic]

What

Three key changes:

• actually read the account list when using a filesystem backend
• using O_APPEND on the file to support concurrent writes
• ensure that the read list is a unique set of accounts

Why

Concurrent writes make FileBackend a viable option for MapReduce-based index building.

Known Limitations

The written list of accounts may contain duplicates due to concurrent writes.

[paulbellamy]

Doesn't this mean the file will only grow, and contain duplicates?

[Shaptic]

That's true, but how else do we coordinate parallel (and partial) flushes? If you decide to flush indices to the filesystem every 512 accounts, say, then there is no way to guarantee a lack of dupes nor that you're done writing.

I added some code to return a unique list in 06df7c1, but I'm not sure how to prevent it from being generated in the first place. We could do something dumb like periodically read/unique-ify/flush it, or maybe store the list in memory at all times? But unsure. Ideas?

[Shaptic]

FlushAccounts() could also do a "set union" instead of a direct append, e.g. Write(Union(ReadAccounts(), CurrentAccounts()). This would slow down flushes but would ensure a unique and correct set of accounts on-disk.

[Shaptic]

Tried it out (and reverted it) in 34b9ba2 if you wanna have a look at that.

[bartekn]

Is O_APPEND thread safe? Ie. given two go routines writing to a file: is the write a blocking call? I can't see any info in godoc claiming it's true.

Namely, if each address is plopped in with a trailing newline, you need a newline at the end of the file, otherwise the next append will squish two addresses together on the same line. But, if there's a newline there, and there are concurrent writes, there may be a bunch of newlines creating weird gaps in the file. So when reading, you'd need to filter out the crap lines. Using the raw XDR binary solves all of this, since you just read chunk by chunk and turn that into xdr.MuxedAccounts (since there may be, and are, M-addresses as participants).

Not sure I understand this. Why can't we just write account + \n together atomically to prevent having empty lines in a file?

[Shaptic]

I'm not sure I understand it either 😆 after revisiting the old code, I don't think I can explain it correctly. It's possible that writes were getting interleaved (or just sloppy coding on my part), but I'm not sure.

Have a look at d821995: it works just fine and the file looks normal (one address per line, clean). This whole PR history is nonsense 🤦‍♂️

This S/O thread suggests that Golang file writes are thread-safe, but we can make further guarantees by ensuring each write happens in a single write() syscall (which is also safe).

[sreuland]

could the intermediate full size array of account strings be avoided by just put'ing to this map on ln 174 instead then iterate the de-dup'd map keys into string slice based on map len for a bit less thrash?

[Shaptic]

Absolutely brilliant point - done in 91468ec.

[sreuland]

nice work vetting this, looks like ideal solution.