CDC from planetscale

[amcclosky]

Is there already a solution for or plans to implement a feature that would allow change data capture with planetscale? i.e. deliver an event to my application with the changed data when a row is updated.

[johnyerhot]

👋 @amcclosky - so something like a webhook whose payload contains the schema diff that fires when a database branch is merged?

We have been discussing adding webhooks that fire on various events quite a bit internally. I can't make any promises on if or when, though.

[amcclosky]

That would be cool but I was thinking something more along the lines of a web-hook that fires when I insert or update a row in a table.

Alternatively, having mysql binlog access would also work. I haven't attempted to verify myself yet but does planetscale allow a client to connect to a database for replication?

[jgreet]

We don't currently expose MySQL binlogs from PlanetScale databases.

[ethangclark]

binlogs or a hook that fires with created/updated/deleted row information would really set planetscale apart for reactive projects.

Current workaround is polling on updatedAt columns.

[amcclosky]

We don't currently expose MySQL binlogs from PlanetScale databases.

Thanks for the response @jgreet. Completely understandable limitation at this stage of the product. I'd assume exposing binlogs or equivalent is on the roadmap at some point in the future though.

[chinmay-500apps]

binlogs or a hook that fires with created/updated/deleted row information would really set planetscale apart for reactive projects.

Current workaround is polling on updatedAt columns.

Hello team, we could not sync delete operation, whereas create and update operations are getting properly synced up. Any pointers towards getting this will help my team a lot. It will really set planetscale a way ahead, as @ethangclark has rightly point out.

I am just a newbie at planetscale and any help in this regard is greatly appreciated.
I subscribed to the basic plan at planetscale, but not sure where to post my query regarding Airbyte connector for dealing with syncing of delete operations.

[LouisIV]

It would be amazing if planet scale could expose the VStream from the VTGate. This is the difference between me hosting Vitess myself and using planet scale. 😢

A first party example showing how to integrate it with something like debeizum would be 🚀

Let me know if there is anything I can do to move this along

[mscoutermarsh]

Debezium is now available: https://planetscale.com/docs/integrations/debezium

[mscoutermarsh]

PlanetScale Connect is now available: https://planetscale.com/blog/extract-load-and-transform-your-data-with-planetscale-connect

We also have a Debezium connector: https://planetscale.com/docs/integrations/debezium

[streamkap]

Any update on exposing binlogs?

[tiagoefreitas]

This is not an answer, Connect is ETL with Airbyte that is not CDC at all. CDC means realtime updates on every transaction.
Airbyte launches Docker Containers to sync data between source and destination and can run minimum every 5 minutes.

In CDC the destination is not event a database just a message.

[mscoutermarsh]

Update: we now have a debezium connector available for PlanetScale https://planetscale.com/docs/integrations/debezium

[CanRau]

Insert, update, delete webhook(s) would be incredible 🤓
This would be very helpful to build queue (like) data structures directly integrated as part of the main database 😻

[Tang8330]

Any updates on exposing binlogs?

[jgreet]

Nothing to announce regarding binlogs. We'll post an update here if that changes.

[chinmay-500apps]

Anybody, who have used this Airbyte connector and has been able to sync delete operations as well, please help me.

[chinmay-500apps]

Planetscale provides "Full refresh Overwrite" as a refresh option to sync up delete operations. But this can not be accepted as a solution as this will have very bad performance when number of records in source database is huge. Any other suggestions team?

[JohnGemstone]

I think for now we have to use a pub/sub service like Ably or host our own service using something like Soketi. These dont directly connect to pscale but you use them along side, mirroring events and seperating them into their own logical channels clients can subscribe to using webhooks.

[etdej]

Hi there,
Any news on supporting CDC ? Is it prioritised in your roadmap ?
Thanks,

[mscoutermarsh]

We just released a debezium connector for PlanetScale: https://planetscale.com/docs/integrations/debezium

[andrao]

This is not official support, but for those working in TS/JS I've created a package, planetscale-stream-ts, which provides access to the Vitess VStream, a CDC log.

Under the hood it uses the psdbconnect gRPC API, which is used for e.g. the Connect Airbyte adapter. This API is in alpha, so use with caution! This response from PlanetScale Support is encouraging, however:

I received confirmation that the v1alpha1 API will be around for quite a while since it has been adopted for use in a number of places so deprecating it would likely only occur in a very slow / controlled manner at this point.

[etdej]

Thanks ! Does this handle deletes properly ? The Connect Airbyte adapter does not (which limits its usability very much). I am struggling to understand where this limitation comes from.

[andrao]

Thanks ! Does this handle deletes properly ? The Connect Airbyte adapter does not (which limits its usability very much). I am struggling to understand where this limitation comes from.

@etdej it does indeed! See the screencap in the README there.

[etdej]

Yeah 🥳