Firestore checkpointer

[TomDarmon]

Hello !

I think firestore is a great solution for a checkpointer as:

• The setup is easy if you have a GCP account
• It's serverless so for on and off experiments you don't pay if you don't use it (which is great for a small project)

I've implemented a firestore checkpointer in langgraph 0.1.X in #1047 , with the changes to langgraph 0.2.0 checkpointer must be separated in their own official libraries as specified by @vbarda.

So here am I opening a discussion on the subject. I can try to update my connector to the 0.2.0 requirements, I believe new put and aput methods were added.

Additionally, I'm not sure of what would be the requirements and steps to make it a "library and maintain it in an official capacity".

@vbarda Could you list the requirements next steps if me or someone wants to follow up on this ?

[pelguetat]

Hey Tom, I was testing your implementation and it works great, only thing is I'm getting a document size limit error after a few turns in the conversation.
google.api_core.exceptions.InvalidArgument: 400 Document 'projects/bewe-ia/databases/(default)/documents/checkpoints_collection/1542334235' cannot be written because its size (1,070,134 bytes) exceeds the maximum allowed size of 1,048,576 bytes.

Any ideas on what the issue might be?

[TomDarmon]

Interesting to see this, I've used my implementation for a few days with limited number of interactions, so this is an important catch.

After a little research, it looks like it's a limit of firestore. A document cannot be bigger than 1,048,576 bytes.

I think my implementation needs to be optimized further to scale correctly.

I'm curious what did you stored in the state ? It was chat history only that produced this much bytes ?

[pelguetat]

Thanks for the reply. I'm storing the default, plus some config and state variables, but it's all plain text. Usually gives the error after a few turns. Can you try replicating it? I'm curious if Langgraph normally produces this much data and other storage platforms don't enforce any limits or if it's an inefficiency on my side.

[pelguetat]

@TomDarmon

[vbarda]

@TomDarmon for now we're planning to only have official packages that are maintained by LangChain and and our partner providers. since you're interested in creating a library, i would recommend that you create/maintain it in your own repo. would be happy to answer questions / help if you run into any issues! thank you for your interest in contributing!

[NicholasMicali]

@TomDarmon I am trying to write a firestore checkpointer, do you have your implementation saved somewhere? It would be a helpful jumping off point.

[skamalj]

Just published firestore and cosmosdb checkpointers packages - langgraph-checkpoint-firestore and langgraph-checkpoint-cosmosdb. Code is on github - https://github.com/[skamalj/langgraph_checkpoint_firestore](https://github.com/skamalj/langgraph_checkpoint_firestore)