CB Lite on Android Xamarin fails to replicate when local DB is large

[rainerg2000]

In regard to this Forum thread:

I am using CB Lite 1.3.1 on Android in a Xamarin-based application. The application logs about 1 small data item per second into Couchbase. It also updates a few documents about once per second.
I am using SQLite as a storage engine for CBLite. I am using Sync Gateway to replicate the database to Couchbase Server.

When the local database reaches a size of 100K-200K docs (100-200 MBytes) the push replication fails to start. I am using a non-continuous push replication. Its Status property is 'Active', IsRunning is true, ChangesCount is 0, CompletedChangesCount is 0.

For this replication the Sync Gateway log shows some activity. But I think it's from a pull replication and not from this push, that seems to get stuck.

Attached is the Android Device Monitor log filtered to tag:mono. Around 15:10:10 the push replication begins. Attached is also the log from sync gateway.

CB Lite fails to push when local DB is large ADM log.txt
CB Lite fails to push when local DB is large SG log.txt

[rainerg2000]

here's a more verbose log for this problem.
(using Couchbase.Lite.Util.Log.Domains.Sync.Level = Couchbase.Lite.Util.Log.LogLevel.Verbose; )

Note the state machine error at 17:19:12.728:

CB Lite fails to push when local DB is large ADM log verbose.txt

[borrrden]

That error is not related to the size of the database. What version of Xamarin are you using?

[rainerg2000]

using Xamarin 23.3.0
NuGet offers me 23.4.0.1, do you recommend upgrading ?

[rainerg2000]

I cannot upgrade to 23.4.0.1 for the Xamarin support packages, due to a constraint from Xamarin.Forms. I am using the newest Xamarin.Forms (2.3.2.127) which has dependencies on the Xamarin.Android.Support.* packages. It required exactly version 23.3.0 of the support packages.

[rainerg2000]

I am using the current stable Xamarin VS2015 extension (4.2.0.680) on VS 2015 Pro.
I will upgrade to the current Beta Xamarin 4.2.0.694 and see what happens.

[rainerg2000]

No luck with the current Beta Xamarin (4.2.0.694).

I am targeting a Nexus 5X on Android Nougat. Is N supported by couchbase-lite-net? My app shows popup warnings about libsqlite.so not being accessible.

[rainerg2000]

Interestingly, the push sync of the 200K doc database does work now, but very slowly. It could be that my Xamarin version upgrades improved the situation, or maybe I am giving it more time to do the sync. In any case, the push replication remains at ChangesCount 0 for several hours, and then the sync of a few hundred new docs happens relatively quickly.

[borrrden]

With regards to sqlite, Android N made the decision to disallow native library access to system libs installed on Android (one of which is sqlite). On CI, there is a now a "custom sqlite" storage engine which uses a bundled version of sqlite and going forward on Android it will need to be used. That doesn't likely have anything to do with this issue though. Several hours is not acceptable and something else is going on. Do you think you could run this in a debugger and pause while it is blocked and see how many threads are there and what they are doing? Also keep an eye out for how much garbage collection activity there is.

[rainerg2000]

I'm running the same application on a virtual Android M device. That instance has a much larger database (>800K docs). It does not have the slowness issue. So, you are correct, it is not a size issue.

I will report thread counts and GC activity from Android Device Monitor tomorrow.

[rainerg2000]

@borrrden, I hope this helps:
logcat tag:mono-stdout shows this State Machine Error on a device that is doing the slow push sync (another device does not show this error and is syncing normally)

09-27 08:32:22.592: I/mono-stdout(12082): INFO) DATABASE (Database): [6] 2016-9-27 08:32:22.591-07:00 Database[/data/user/0/com.biotronik.BTLEPerfTestXaPortable/files/.local/share/btlelog1.cblite2] posting change notifications: seq [616595]
09-27 08:32:22.611: I/mono-stdout(12082): INFO) DATABASE (Database): [5] 2016-9-27 08:32:22.610-07:00 Database[/data/user/0/com.biotronik.BTLEPerfTestXaPortable/files/.local/share/btlelog1.cblite2] posting change notifications: seq [616596]
09-27 08:32:22.627: I/mono-stdout(12082): INFO) DATABASE (Database): [4] 2016-9-27 08:32:22.626-07:00 Database[/data/user/0/com.biotronik.BTLEPerfTestXaPortable/files/.local/share/btlelog1.cblite2] posting change notifications: seq [616597]
09-27 08:32:24.077: I/mono-stdout(12082): INFO) DATABASE (Database): [12] 2016-9-27 08:32:24.076-07:00 Database[/data/user/0/com.biotronik.BTLEPerfTestXaPortable/files/.local/share/btlelog1.cblite2] posting change notifications: seq [616598]
09-27 08:32:24.081: I/mono-stdout(12082): INFO) TASK SCHEDULING (SingleTaskThreadpoolScheduler): [4] 2016-9-27 08:32:24.080-07:00 New scheduler created
09-27 08:32:24.246: I/mono-stdout(12082): INFO) TASK SCHEDULING (SingleTaskThreadpoolScheduler): [4] 2016-9-27 08:32:24.246-07:00 New scheduler created
09-27 08:32:24.335: I/mono-stdout(12082): INFO) SYNC (Replication): [4] 2016-9-27 08:32:24.335-07:00 Attempting to start pusher (40d5e994-639c-4de0-8c8d-a6e6357d638d)
09-27 08:32:24.336: I/mono-stdout(12082): INFO) SYNC (Replication): [4] 2016-9-27 08:32:24.336-07:00 Attempting to start puller (aee1498a-8d1f-458e-9c73-d03b35c06e83)
09-27 08:32:24.481: I/mono-stdout(12082): ERROR) SYNC (Replication): [13] 2016-9-27 08:32:24.400-07:00 State machine error:
09-27 08:32:24.481: I/mono-stdout(12082): System.MethodAccessException: Method System.Net.Http.WebRequestHandler:set_ReadWriteTimeout (int)' is inaccessible from methodSystem.Net.Http.HttpClientHandler:EnsureModifiability ()'
09-27 08:32:24.482: I/mono-stdout(12082): at (wrapper managed-to-native) System.Object:__icall_wrapper_mono_throw_method_access (intptr,intptr)
09-27 08:32:24.482: I/mono-stdout(12082): at System.Net.Http.WebRequestHandler.set_ReadWriteTimeout (System.Int32 value) [0x00000] in :0
09-27 08:32:24.482: I/mono-stdout(12082): at Couchbase.Lite.Support.CouchbaseLiteHttpClientFactory.BuildHandlerPipeline (Couchbase.Lite.Util.CookieStore store, Couchbase.Lite.Util.IRetryStrategy retryStrategy) [0x00023] in :0
09-27 08:32:24.482: I/mono-stdout(12082): at Couchbase.Lite.Support.CouchbaseLiteHttpClientFactory.GetHttpClient (Couchbase.Lite.Util.CookieStore cookieStore, Couchbase.Lite.Util.IRetryStrategy retryStrategy) [0x00000] in :0
09-27 08:32:24.482: I/mono-stdout(12082): at Couchbase.Lite.Internal.RemoteSession.Setup (Couchbase.Lite.ReplicationOptions options) [0x00063] in :0
09-27 08:32:24.482: I/mono-stdout(12082): at Couchbase.Lite.Replication.StartInternal () [0x0006d] in :0
09-27 08:32:24.482: I/mono-stdout(12082): at Couchbase.Lite.Replication.m__6 (Stateless.StateMachine2+Transition[TState,TTrigger] transition) [0x00049] in <b8537e1669c6431ba7cf881d30359b75>:0 09-27 08:32:24.482: I/mono-stdout(12082): at Stateless.StateMachine2+StateConfiguration+<>c__DisplayClass19_0[TState,TTrigger].b__0 (Stateless.StateMachine2+Transition[TState,TTrigger] t, System.Object[] args) [0x00000] in <ee2edfa22280454d8821704fcab62b81>:0 09-27 08:32:24.482: I/mono-stdout(12082): at Stateless.StateMachine2+StateRepresentation[TState,TTrigger].ExecuteEntryActions (Stateless.StateMachine2+Transition[TState,TTrigger] transition, System.Object[] entryArgs) [0x00031] in <ee2edfa22280454d8821704fcab62b81>:0 09-27 08:32:24.482: I/mono-stdout(12082): at Stateless.StateMachine2+StateRepresentation[TState,TTrigger].Enter (Stateless.StateMachine2+Transition[TState,TTrigger] transition, System.Object[] entryArgs) [0x00040] in <ee2edfa22280454d8821704fcab62b81>:0 09-27 08:32:24.482: I/mono-stdout(12082): at Stateless.StateMachine2[TState,TTrigger].InternalFire (TTrigger trigger, System.Object[] args) [0x00092] in :0
09-27 08:32:24.482: I/mono-stdout(12082): at Stateless.StateMachine2[TState,TTrigger].Fire (TTrigger trigger) [0x00000] in <ee2edfa22280454d8821704fcab62b81>:0 09-27 08:32:24.482: I/mono-stdout(12082): at Couchbase.Lite.Replication+<FireTrigger>c__AnonStorey0.<>m__0 () [0x00000] in <b8537e1669c6431ba7cf881d30359b75>:0 09-27 08:32:24.485: I/mono-stdout(12082): ERROR) SYNC (Replication): [13] 2016-9-27 08:32:24.482-07:00 State machine error: 09-27 08:32:24.485: I/mono-stdout(12082): System.MethodAccessException: MethodSystem.Net.Http.WebRequestHandler:set_ReadWriteTimeout (int)' is inaccessible from method System.Net.Http.HttpClientHandler:EnsureModifiability ()' 09-27 08:32:24.485: I/mono-stdout(12082): at (wrapper managed-to-native) System.Object:__icall_wrapper_mono_throw_method_access (intptr,intptr) 09-27 08:32:24.485: I/mono-stdout(12082): at System.Net.Http.WebRequestHandler.set_ReadWriteTimeout (System.Int32 value) [0x00000] in <c421b229432947d4b1e8558238a97dfc>:0 09-27 08:32:24.485: I/mono-stdout(12082): at Couchbase.Lite.Support.CouchbaseLiteHttpClientFactory.BuildHandlerPipeline (Couchbase.Lite.Util.CookieStore store, Couchbase.Lite.Util.IRetryStrategy retryStrategy) [0x00023] in <b8537e1669c6431ba7cf881d30359b75>:0 09-27 08:32:24.486: I/mono-stdout(12082): at Couchbase.Lite.Support.CouchbaseLiteHttpClientFactory.GetHttpClient (Couchbase.Lite.Util.CookieStore cookieStore, Couchbase.Lite.Util.IRetryStrategy retryStrategy) [0x00000] in <b8537e1669c6431ba7cf881d30359b75>:0 09-27 08:32:24.486: I/mono-stdout(12082): at Couchbase.Lite.Internal.RemoteSession.Setup (Couchbase.Lite.ReplicationOptions options) [0x00063] in <b8537e1669c6431ba7cf881d30359b75>:0 09-27 08:32:24.486: I/mono-stdout(12082): at Couchbase.Lite.Replication.StartInternal () [0x0006d] in <b8537e1669c6431ba7cf881d30359b75>:0 09-27 08:32:24.486: I/mono-stdout(12082): at Couchbase.Lite.Replication.<InitializeStateMachine>m__6 (Stateless.StateMachine2+Transition[TState,TTrigger] transition) [0x00049] in :0
09-27 08:32:24.486: I/mono-stdout(12082): at Stateless.StateMachine2+StateConfiguration+<>c__DisplayClass19_0[TState,TTrigger].<OnEntry>b__0 (Stateless.StateMachine2+Transition[TState,TTrigger] t, System.Object[] args) [0x00000] in :0
09-27 08:32:24.486: I/mono-stdout(12082): at Stateless.StateMachine2+StateRepresentation[TState,TTrigger].ExecuteEntryActions (Stateless.StateMachine2+Transition[TState,TTrigger] transition, System.Object[] entryArgs) [0x00031] in :0
09-27 08:32:24.486: I/mono-stdout(12082): at Stateless.StateMachine2+StateRepresentation[TState,TTrigger].Enter (Stateless.StateMachine2+Transition[TState,TTrigger] transition, System.Object[] entryArgs) [0x00040] in :0
09-27 08:32:24.486: I/mono-stdout(12082): at Stateless.StateMachine2[TState,TTrigger].InternalFire (TTrigger trigger, System.Object[] args) [0x00092] in <ee2edfa22280454d8821704fcab62b81>:0 09-27 08:32:24.486: I/mono-stdout(12082): at Stateless.StateMachine2[TState,TTrigger].Fire (TTrigger trigger) [0x00000] in :0
09-27 08:32:24.486: I/mono-stdout(12082): at Couchbase.Lite.Replication+c__AnonStorey0.<>m__0 () [0x00000] in :0
09-27 08:32:24.658: I/mono-stdout(12082): INFO) DATABASE (Database): [5] 2016-9-27 08:32:24.658-07:00 Database[/data/user/0/com.biotronik.BTLEPerfTestXaPortable/files/.local/share/btlelog1.cblite2] posting change notifications: seq [616599]
09-27 08:32:24.681: I/mono-stdout(12082): INFO) DATABASE (Database): [6] 2016-9-27 08:32:24.681-07:00 Database[/data/user/0/com.biotronik.BTLEPerfTestXaPortable/files/.local/share/btlelog1.cblite2] posting change notifications: seq [616600]
09-27 08:32:24.697: I/mono-stdout(12082): INFO) DATABASE (Database): [13] 2016-9-27 08:32:24.697-07:00 Database[/data/user/0/com.biotronik.BTLEPerfTestXaPortable/files/.local/share/btlelog1.cblite2] posting change notifications: seq [616601]
09-27 08:32:26.734: I/mono-stdout(12082): INFO) DATABASE (Database): [4] 2016-9-27 08:32:26.733-07:00 Database[/data/user/0/com.biotronik.BTLEPerfTestXaPortable/files/.local/share/btlelog1.cblite2] posting change notifications: seq [616602]
09-27 08:32:26.756: I/mono-stdout(12082): INFO) DATABASE (Database): [6] 2016-9-27 08:32:26.756-07:00 Database[/data/user/0/com.biotronik.BTLEPerfTestXaPortable/files/.local/share/btlelog1.cblite2] posting change notifications: seq [616603]
09-27 08:32:26.777: I/mono-stdout(12082): INFO) DATABASE (Database): [5] 2016-9-27 08:32:26.776-07:00 Database[/data/user/0/com.biotronik.BTLEPerfTestXaPortable/files/.local/share/btlelog1.cblite2] posting change notifications: seq [616604]
09-27 08:32:28.812: I/mono-stdout(12082): INFO) DATABASE (Database): [12] 2016-9-27 08:32:28.811-07:00 Database[/data/user/0/com.biotronik.BTLEPerfTestXaPortable/files/.local/share/btlelog1.cblite2] posting change notifications: seq [616605]
09-27 08:32:28.833: I/mono-stdout(12082): INFO) DATABASE (Database): [6] 2016-9-27 08:32:28.832-07:00 Database[/data/user/0/com.biotronik.BTLEPerfTestXaPortable/files/.local/share/btlelog1.cblite2] posting change notifications: seq [616606]
09-27 08:32:28.845: I/mono-stdout(12082): INFO) DATABASE (Database): [5] 2016-9-27 08:32:28.844-07:00 Database[/data/user/0/com.biotronik.BTLEPerfTestXaPortable/files/.local/share/btlelog1.cblite2] posting change notifications: seq [616607]

[rainerg2000]

@borrrden, some heap and thread stats:
My Nexus 5x, which has slow push sync: 25 threads, 42MB heap, 26MB allocated, 16MB free
My virtual device, where sync is working properly: 30 up to 42 threads, 8MB heap, 5MB allocated, 3MB free

[borrrden]

That error message is something I am seeing from other reports as well and I fear its meaning. It seems to indicate that what I am doing to get socket timeout settings on Xamarin mobile platforms is not valid. It looks like either I'll have to abandon the socket timeout setting or convince Xamarin to ship the WebRequestHandler dll with the mobile profiles.

[borrrden]

Ok, I pushed up some changes to issue/#742 which should address the error message I see here. This unfortunately means losing the SocketTimeout option on replication for mobile platforms :( but the forums keep ignoring my pleas for information on this assembly.

[rainerg2000]

@borrrden, thank you for looking into this issue.

I managed to build issue/#742 and run it on the problematic database where push repl is very slow. I debugged through Batcher.QueueObjects. It tries to add 117000 objects to _inbox which gets progressively slower. The first few thousand objects get added within a few seconds, but after adding 25000 objects it takes more than 20sec to add the next 1000.
While the objects get enqueued over many minutes, I don't see anything getting dequeued from _inbox.

I hope this helps to track down the issue. Let me know how I can help further.

[borrrden]

What about the logs? Do you see any errors in them?

[borrrden]

P.S. This replication is not going to be quick. A database of this size is potentially 500MB+ in size. With your phone's network conditions, how long do you think it will take to upload a 500MB file? That's something to keep in mind. Replicating from scratch will need to send quite a lot of data. I'm currently in the process of generating a database with 1,000,000 entries to test and even generating it takes a few minutes.

[rainerg2000]

I'll look at logs in more detail. For some reason my debug build wasn't
logging to Android Device Monitor. I'll continue tomorrow.

On Sep 28, 2016 6:02 PM, "Jim Borden" [email protected] wrote:

P.S. This replication is not going to be quick. A database of this size is
potentially 500MB+ in size. With your phone's network conditions, how long
do you think it will take to upload a 500MB file? That's something to keep
in mind. Replicating from scratch will need to send quite a lot of data.
I'm currently in the process of generating a database with 1,000,000
entries to test and even generating it takes a few minutes.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#742 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACwxV-gehmQ91IB8obgKoiojoS_7eRp_ks5quw4ogaJpZM4KBBmT
.

[borrrden]

Actually, I've found the source of the problem, I'm pretty sure. I am working on a fix now but Android is always a gigantic pain because the debugger is a pain in the ass that crashes quite a bit and for some reason Xamarin Studio is being even more unreliable than usual...so it might not get done today.

[rainerg2000]

Great news.
I can easily test the fix for you. I will keep the device with the slow database around for this test.
By the way, the logs don't show errors or warnings before and during the Batcher.QueueObjects loop with 117000 docs.
I'd like to suggest a change: in Batcher.cs change "Log.To.NoDomain" => "Log.To.Sync"

[rainerg2000]

I take your point about limited mobile bandwidth. In my estimation a 2G cell connection with 100kBits/sec should be enough on average to push my application data to sync gateway. I use VisualStudio Android emulator to verify this. It can simulate slow connections with packet drops.

[borrrden]

Great, if nothing else comes up I am going to have another look at this today. Xamarin Android 7.x seems to be giving me issues so I've rolled back, and hopefully Xamarin will remain stable while I try to do some more testing. The ultimate issue is deduplication. The string comparison operation gets much slower as the size of the inbox grows since it checks the entries to ensure that it is not already added which causes a linear search of the inbox. I'm trying to coax it to run prematurely to start draining the inbox but that causes the problem of it thinking it is finished too soon. There is one other way I need to try as well.

[borrrden]

Ok at the expense of a slight increase in memory I've made the deduplication O(1) instead of O(n) and the results are immediately evident. I was able to add 1,000,000 items to the inbox in a matter of seconds. The replication itself is still a long process though (it's been going for about 10 minutes and finished ~200k). It's going to be much slower than that on a 100 kilobit / sec connection (kiloBIT? As in maximum 12.5 kiloBYTES per second? At that rate sending a 600 MB file would take no less than 13 1/2 hours)

EDIT I pushed the changes to the issue branch

[rotorgames]

@borrrden Hello. I have same problem. Can you send fix in http://latestbuilds.hq.couchbase.com/couchbase-lite-net/1.3.1/?

[rainerg2000]

@borrrden, Hi Jim,
I verified your fix in issue/#742. Batcher.QueueObjects performs much better. My test device is now able to catch up. As far as I'm concerned you can close this issue.

However, I still have problems, which I'll post in the forum. Then you can decide if they are related to this issue, or not. See https://forums.couchbase.com/t/lastsequence-remains-constant-across-many-push-replications/10218

Thanks.
Rainer.

[borrrden]

@rotorgames I won't be making a patch onto 1.3.1. This fix is currently targeted for 1.5 but I'd like to get it into 1.4 instead since it is a severe performance regression.

[rotorgames]

@borrrden What version I need use that I will not get this error now?

[borrrden]

@rotorgames You will need to build it yourself from the issue/742 branch. If I get this moved to 1.4 then I will let you know because it will go into a CI build.

[rotorgames]

@borrrden Hi. I compiled and tested issue/742 branch and I have new error:

Galaxy s4
Android5.0.1

Number.StringToNumber(System.String str, System.Globalization.NumberStyles options, System.Number+NumberBuffer& number, System.Globalization.NumberFormatInfo info, System.Boolean parseDecimal)
android.runtime.JavaProxyThrowable: System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. ---> System.FormatException: Input string was not in a correct format.

caused.AggregateException(s)
System.Number.StringToNumber(string str, NumberStyles options, ref Number.NumberBuffer number, NumberFormatInfo info, bool parseDecimal)<368820a9888f43ddb85d18e87189adbf>:0
System.Number.ParseInt64(string value, NumberStyles options, NumberFormatInfo numfmt)<368820a9888f43ddb85d18e87189adbf>:0
System.Int64.Parse(string s)<368820a9888f43ddb85d18e87189adbf>:0
Couchbase.Lite.Replication.NotifyChangeListeners(ReplicationStateTransition transition)<3b173af1ebe44071ba8c91dd0a735666>:0
Couchbase.Lite.Replication.SafeAddToCompletedChangesCount(int value)<3b173af1ebe44071ba8c91dd0a735666>:0
Couchbase.Lite.Replicator.Puller.InsertDownloads(IList<T> downloads)<3b173af1ebe44071ba8c91dd0a735666>:0
Couchbase.Lite.Support.Batcher<T>.ProcessNow()<3b173af1ebe44071ba8c91dd0a735666>:0
Couchbase.Lite.Support.Batcher<T>.<ScheduleWithDelay>b__21_0(Task t)<3b173af1ebe44071ba8c91dd0a735666>:0
System.Threading.Tasks.ContinuationResultTaskFromTask<TResult>.InnerInvoke()<368820a9888f43ddb85d18e87189adbf>:0
System.Threading.Tasks.Task.Execute()<368820a9888f43ddb85d18e87189adbf>:0
--- End of inner exception stack trace ---
---> (Inner Exception #0) System.FormatException: Input string was not in a correct format.
System.Number.StringToNumber(string str, NumberStyles options, ref Number.NumberBuffer number, NumberFormatInfo info, bool parseDecimal)<368820a9888f43ddb85d18e87189adbf>:0
System.Number.ParseInt64(string value, NumberStyles options, NumberFormatInfo numfmt)<368820a9888f43ddb85d18e87189adbf>:0
System.Int64.Parse(string s)<368820a9888f43ddb85d18e87189adbf>:0
Couchbase.Lite.Replication.NotifyChangeListeners(ReplicationStateTransition transition)<3b173af1ebe44071ba8c91dd0a735666>:0
Couchbase.Lite.Replication.SafeAddToCompletedChangesCount(int value)<3b173af1ebe44071ba8c91dd0a735666>:0
Couchbase.Lite.Replicator.Puller.InsertDownloads(IList<T> downloads)<3b173af1ebe44071ba8c91dd0a735666>:0
Couchbase.Lite.Support.Batcher<T>.ProcessNow()<3b173af1ebe44071ba8c91dd0a735666>:0
Couchbase.Lite.Support.Batcher<T>.<ScheduleWithDelay>b__21_0(Task t)<3b173af1ebe44071ba8c91dd0a735666>:0
System.Threading.Tasks.ContinuationResultTaskFromTask<TResult>.InnerInvoke()<368820a9888f43ddb85d18e87189adbf>:0
System.Threading.Tasks.Task.Execute()<368820a9888f43ddb85d18e87189adbf>:0

P.S. Error is in Replication.cs:1886. I catch value of LastSequence. It was "477:422". Int64.Parse can't convert it value.

[borrrden]

@rotorgames Filed this as #757

[borrrden]

Alright, issue/742 will be merged into master soon and will be in 1.4

[rotorgames]

@borrrden Good news. Please tell me when you do it.

[rotorgames]

@borrrden Hello. Any news?

[borrrden]

This will be done by the end of this week (and likely tomorrow)

[rotorgames]

@borrrden Ok. Thx

[borrrden]

@rotorgames @rainerg2000 FYI this fix introduced a regression in one of our functional tests. It's a really hard case to hit as it requires a lot of updates from multiple clients to a sync gateway instance on the same set of documents but I have fixed it up while maintaining the performance changes. See the above commit.