handle feed consistency errors more gracefully

[cryptix]

Even though the gossip plugin has strict locking to only fetch each feed once at a time it seems like there is still a problem with this...

What ever the cause may be, if the gossip scheduler throws consistency error: wrong stored message sequence for .. we need to handle that gracefully.

first reported here: sunrise-choir/sunrise-social-android-app#2

[cryptix]

Since messages are appended to the root offset log first and indexes updated after that, it's hard to diagnose with a good reproduction story. In the normal cases the KV shouldn't complain about having to many messages.

One possible idea is that the log-append (and file system write) call goes through, the message(s) get indexed and the KV updated and THEN the app crashes.

I'm not 100% certain but I think if the offset log is written without a filesystem sync, the index COULD be further along since the new messages never made it to disk.

cc @keks

[cryptix]

There is no filesystem check (fsck) code on package sbot and the go-sbot tool using -fsck and `-repair.

The root cause is still diagnosed even though I tried multiple times...

The main protection against fetching feeds multiple times curreently should be this activeFetch map. Which should dictate to skip a feed with a running createHistoryStream but something is still off..

The current test base makes it hard to produce this bug. I feel like I need to make a new one, creating a couple hundred feeds, sync them all to a 2nd bot, truncate it a little to get lots of incomplete feeds and sync again to try to cause this bug.