Next generation of TiDB BigData Ecosystem Integrations

[sunxiaoguang]

Problem:

The architecture of TiBigData as it currently exists is designed specifically for running in an IDC-based infrastructure, where everything runs on a flat network. However, this is no longer suitable for the growing cloud-native application architecture. Therefore, in order to move TiBigData towards the next generation of application architecture, we plan to retire the existing classic architecture and replace it with a more modern and agile support structure that is capable of operating not only in the cloud, but also keeping up with the fast TiDB release cadence.

Goals

• Improve Reliability and Maintainability: The primary objective of deprecating the TiBigData classic architecture is to enhance the overall reliability and maintainability of the TiBigData product. By streamlining the architecture and removing the bypass TiDB logic, we can take advantage of the mature distributed TiDB service. This strategic move aims to identify and eliminate potential failure points, reducing the occurrence of system crashes and improving the platform's stability. As a result, we will deliver a more reliable and efficient user experience, ensuring that TiBigData performs optimally in demanding production environments.

• Improving User Experience: One of the main goals in deprecating the TiBigData classic architecture is to enhance the overall user experience. We plan to achieve this by simplifying the configuration process and eliminating complicated network topology prerequisites. Our aim is to provide users with a more user-friendly and intuitive experience, allowing them to effortlessly set up and deploy TiBigData just like any other Spark datasources without any additional hassle. By removing unnecessary complexities, we can make it easier for newcomers to quickly get started with TiBigData and promote its widespread adoption.

• Streamline Architecture and Reduce Maintenance Efforts: The deprecation of the classic architecture also aims to streamline the overall architecture of TiBigData. By removing outdated components and simplifying the system design, we can reduce the maintenance efforts required to keep TiBigData up and running. This will free up valuable resources and allow the team to focus on the most important enhancements, ultimately benefiting both the product and the end-users.

• Ensure Seamless Transition: Ensuring a seamless transition from the current generation of TiBigData to the successor product is a crucial objective. We understand the importance of minimizing disruptions and inconveniences for our users during this migration process. By providing clear documentation, proactive support, and backward compatibility where possible, we aim to make the transition as smooth as possible for our existing TiBigData users, ensuring that they can continue leveraging the benefits of the new architecture without major hassle.

By achieving these objectives, we aim to create a more reliable, user-friendly, and maintainable TiBigData product, ultimately enhancing the overall experience for both new and existing users.

Solutions

As part of the architecture evolution, Spark integration is included for a more complete support of popular big data ecosystems. For more information, please refer to the proposal at here.

[sunxiaoguang]

The initial implementation based on the new architecture are available in these repositories.

https://github.com/humengyu2012/TiBigData-jdbc/tree/jdbc
https://github.com/sunxiaoguang/tidb/tree/tispark_ng

[sunxiaoguang]

Patched branch on top of 8.1 LTS: https://github.com/sunxiaoguang/tidb/tree/tispark_ng_on_8.1