Service was not ready: transport error

[ilteoood]

In my service I'm trying to define a singleton firebase instance that I will query once I receive some http requests.

Everything seems to be working fine while I run the service normally, but tests break with errors like:

thread 'routes::open::tests::test_ko_handler' panicked at src/firebase.rs:146:23:
called `Result::unwrap()` on an `Err` value: DatabaseError(FirestoreDatabaseError { public: FirestoreErrorPublicGenericDetails { code: "Unknown" }, details: "status: Unknown, message: \"Service was not ready: transport error\", details: [], metadata: MetadataMap { headers: {} } without root cause", retry_possible: false })

Do you have any suggestion how to overcome this? I tried also to execute them serially but without any luck

[abdolence]

Yes, that's limitation coming from underlying Tonic/Tokio (gRPC) implementation, so you are trying to work with different Tokio runtimes.

There was a similar question in the past: #101 if you want to know some details about it. In general I don't recommend to go this path, Firestore instances are clonable and cloning doesn't create multiple connection pool and relatively cheap, so it is better to propagate it and store in structures as a normal field.

[ilteoood]

I think that the 2 scenarios are a bit different.

The main issue here seems to be related to caching.
The original implementation I currently have on production, which relies on manual deserialization of the database content, works perfectly.
Here is an example: https://github.com/ilteoood/sesamo-backend/blob/master/src/firebase.rs

This issue starts to appear when I use the caching strategy, being it in memory or in temp file.
If I still want to create it as a singleton I don't understand how cloning will fix this 🤔

[abdolence]

I think the root cause still the same. You noticed it with caching because cache.load() uses tokio::spawn() after probably which it detects different tokio runtimes. It is mostly guess depends on my previous digging, firestore crate itself doesn't do anything special except reuses Tonic channels, so this requires a bit digging inside Tonic/Tokio to verify more.

[abdolence]

I'll do some additional verification later around cache if there are anything that can be related from this crate, but I doubt it.

[ilteoood]

Thank you so much btw, your help is incredibly appreciated and your work with this lib is awesome.

[ilteoood]

I did a bit of digging, and if I test using the tokio-shared-rt create everything seems to be working fine.
I just had to switch to in memory caching instead of persistent one, in order to avoid the DB already opened error.

called `Result::unwrap()` on an `Err` value: CacheError(FirestoreCacheError { public: FirestoreErrorPublicGenericDetails { code: "RedbDatabaseError" }, message: "Cache error: Database already open. Cannot acquire lock." })

[abdolence]

Are you running some kind of async tests in your application?
Then it is well know behaviour from Tokio, where you can read details here: tokio-rs/tokio#2374

Related to a persistence cache error you probably are reusing the same directory for cache and either :

• have tests running in parallel
• forgetting to shutdown cache which close an opened cache (and a database in cache directory)

[ilteoood]

Yes to both.
I have tests running in parallel and I can't shutdown the cache otherwise I can't re-use the singleton.
Additionally, I can't even shutdown it by myself since the drop trait is not async

[abdolence]

Then this is the reason.
As a singleton it was created was one runtime, and tests are running on separate. This is why enabling shared environment helped you. There is a little I can do to mitigate this, the limitation is coming from Tonic/Tokio.

[abdolence]

I'm closing this for now, let me know if you have additional questions for this.