[Subtask] Survey transaction implementation on key-value store and remove RocskDB transactional dependency

[yuqi1129]

Describe the subtask

Currently, our storage layer heavily relies on the transaction mechanism provided by RocksDB to ensure reliability. However some key-value pair databases do not support transaction operations, making it challenging for Gravitino to adapt to other
KV databases such as Redis, Cassandra, Hbase, and so on.

We need to eliminate transactional dependency and come up with alternative solutions.

Parent issue

#603

[yuqi1129]

@jerryshao https://docs.google.com/document/d/1DPvaBcUvmA4-9WOnJ_4cPabZXxlyjK7kF-lBLPiB-CY/edit#heading=h.xe72prsp3h4e
The preliminary documentation is here, and I need to reevaluate today.

[yuqi1129]

@jerryshao Hi, we discussed the document last night and have reached the following points.

• The first step is to determine if we need to support transactions in Gravitino at this time. @xunliu has confirmed that KV databases used in eBay supports transaction and there's no urgency to support transaction in Gravitino.
• Developing and implementing a mature transaction framework is a challenging task that may take months or even years. The solution I proposed in the document is also difficult to implement quickly.

Based on this, please help verify the following :

• Whether we really need a transaction framework in Gravitino.
• Whether the solution outlined in the document is reasonable.

I'm eagerly awaiting your response to this issue.

[jerryshao]

I think you totally misunderstood what I'm saying. I'm not saying that we want to build a transaction layer of our own on top of kv store. I'm saying to get rid of current transactions and use other ways instead.

Thinking of the current operations we support, we are mostly doing put, get, delete, update, list on entity and entities. For most of the operations like put and get, they're operating on a single object, so we don't need transactions.

For update, currently we use transactions, I think we can change to use MVCC (copy-write-commit) mechanism somehow. For bulk delete operations, we can use mark-and-sweep mechanism instead.

Basically what I mean is that: for most of the kv stores, they support atomic operation on a single object, so we should leverage this ability to change our current layout to support single object operation, convert the several-step-operations to a single operation, using atomicity with lock to achieve transactions. If we only depend on the simple actions of kv store, so we can support different stores with no prerequisites.

We don't wanner a transaction layer at all, all we want is a new storage layout that doesn't require transactions to do the operations I mentioned above.

Please think in this way.

[jerryshao]

Please schedule a time slot for the design discussion with us. The current design is too simple to understand the details.

[yuqi1129]

Please schedule a time slot for the design discussion with us. The current design is too simple to understand the details.

OK