-
Notifications
You must be signed in to change notification settings - Fork 378
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[EPIC] Add Paimon catalog support for Gravitino #1129
Comments
Sounds like a good idea. We don't have Paimon expert now. Would you like to work on it? @SteNicholas :) |
@JunpingDu, I would like invite other Paimon contributors to support Paimon catalog together. |
@SteNicholas We are very interested it and waiting for the proposal and milestones to dismantle, look forward to achieve paimon catalog together, thx. |
@YxAc Can I ask you to take a little more care with your words. I'm sure no ill intent was intended, but It is often hard to read the tone in messages, and the way that was written could be taken the wrong way. Also, people are volunteers here; sometimes, things may take longer than they first intended. |
@justinmclean Sure, thanks for your remind, I will put it in another way. Actually, we knew each other and talk about Paimon catalog offline, my words above was just a little joke. Thank you for reminding me. |
Another reminder: as we are an open-source project, it is best if all communication is public; that way, all contributors can participate. Please try to have conversations about this feature in public. |
Sure |
@SteNicholas Hi, I did some investigation on Paimon. I found that Paimon does not need HMS to store a We can use Gravitino to manage the Paimon and store the metadata of the database、table. And we may not need a REST catalog like Iceberg. We can just use Gravitino. That makes things more simple. What do you think? |
@coolderli, the implementation of the lock is not designed in Gravitino. A Paimon REST catalog (better have) can facilitate users to use catalog through Rest method, which operation does not have conflict. |
@SteNicholas Yeah, I know what you mean. But Gravitino already has its own Open API. We can use Gravitino Open API to do the same work. Of course, a Paimon REST catalog is meaningful, there is indeed no conflict between the two implementation methods. But using Gravitino Open API is more simple for now. We can finish this work more fast. |
@YxAc, @coolderli, I have updated the proposal of Paimon catalog support. PTAL. |
@SteNicholas Hi, any update about this? Thanks. |
@caican00 can you please leave a message here, so I can assign the epic issue to you. |
@jerryshao sorry for the late. |
Describe the proposal
Gravitino supports Apache Iceberg catalog at present. Apache Paimon is a streaming data lake platform that supports high-speed data ingestion, change data tracking and efficient real-time analytics. We could build Paimon catalog to support managing Paimon metadata.
Paimon exposes
Catalog
pluggable interface and supports several implementation ofCatalog
likeFileSystemCatalog
,HiveCatalog
. It's recommended to build a Gravitino catalog that refers to the implementations of Paimon. Meanwhile, I would propose theRESTCatalog
interface in Paimon community.Task list
The text was updated successfully, but these errors were encountered: