fix(datastore) release startStopSemaphore when start returns, not when API sync completes

[richardmcclellan]

• Fixes Unable to acquire orchestrator lock. Transition currently in progress. #1010 ("Unable to acquire orchestrator lock. Transition currently in progress") (also reported in DataStore sync not downloading newly created data from DynamoDB #977 (comment)). Previously, if you call two DataStore operations right after each other, the second one would usually fail. Each operation calls start, which previously required obtaining a semaphore, within a 2 second timeout. Previously, the semaphore was locked until the API sync completed, but with this PR, the lock is released as soon as the storage observer is initialized (which only takes a few milliseconds).

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[jamesonwilliams]

Regarding thread safety and synchronization of concurrent calls:

If I understand correctly, all of it is now handled by the Semaphore? And so we don't need to run the transitions on a Schedulers.single(), and we don't need to throw synchronized on any method signatures?

[richardmcclellan]

Regarding thread safety and synchronization of concurrent calls:
If I understand correctly, all of it is now handled by the Semaphore? And so we don't need to run the transitions on a Schedulers.single(), and we don't need to throw synchronized on any method signatures?

I'm going to take another closer look at this. Before, I think the semaphore handled everything, and the synchronized methods were unnecessary. Now, we may actually be able to get rid of the semaphore and just have synchronized methods. I'm going to update the logic for stopping on errors and retrying first though, in my next PR.

[richardmcclellan]

@jamesonwilliams I think I've addressed all of your comments, so this is ready for review again. I have another PR coming after this one to fix the "crash when going offline" bug, and also to potentially remove the semaphore completely.

[jamesonwilliams]

Nice cleanup!