Skip to content

Apache Gravitino 0.7.0-incubating

Latest
Latest
Compare
Choose a tag to compare
@jerryshao jerryshao released this 14 Nov 10:22
· 276 commits to main since this release
v0.7.0-incubating
911d23c

Apache Gravitino 0.7.0 is the second major release after entering the ASF. In this release, the community mainly focused on strengthening cloud support to make Gravitino work better in cloud environments.

This release blog will briefly introduce the new features related to cloud support and other significant features and improvements. Please keep reading to learn more about what the community has worked on.

Cloud storage support for Gravitino

As more and more users run their data stacks in the cloud and use cloud object storage, cloud storage support has become an imperative requirement. In this release, the community has mainly focused on adding cloud storage support for Gravitino to make sure Gravitino itself and its connectors/sources can work smoothly with cloud storage.

In this release:

  • Gravitino Iceberg REST catalog server now supports different cloud storage services, including AWS S3, Google GCS, Aliyun OSS. Users can simply configure it to make it work.
  • Gravitino Fileset catalog now supports managing files (objects) stored in S3, GCS, and OSS. Gravitino provides both server-side pluggable framework and client-side Java / Python GVFS (Gravitino Virtual File System) SDK. Users can easily use their existing tools with the Gravitino provided bundled packages to access the data in the cloud. Also, Gravitino provides a pluggable framework for users to implement their own storage support.
  • Gravitino’s Hive, Paimon, and Iceberg catalogs also adds and verifies the support with different cloud storage.
  • Gravitino’s Spark and Trino connectors also verifies cloud storage support.

Overall, with the 0.7.0 release, Gravitino generally supports working with different cloud storage services. To know more, you can check our issue #4396. Also, we’re continuing to add more cloud storage support in future releases. Please stay tuned.

Credential vending support in Gravitino

Besides cloud storage support, credential vending support is also vital for Gravitino, especially to work with cloud storage. The traditional way of using AKSK is not convenient or safe. With credential vending technology, the Gravitino server will help users get temporary tokens for authentication, significantly simplifying the client-side configurations and centralizing authentications.

Gravitino 0.7.0 introduces a framework to support Credential vending and also adds S3 and GCS token support. This framework is integrated into Gravitino's Iceberg REST catalog service. So users can smoothly access Iceberg tables on S3 and GCS with authentication.

This is just the first step of credential vending, and we will add more integrations with Gravitino, like fileset support, connector support, etc, in the next release.

For the details of credential vending, please check the issue #4398 and the design document.

Unified access control improvements

Gravitino 0.6.0 introduced the alpha version of unified access control with Apache Ranger support (here), but this feature still needs improvement. In version 0.7.0, we added many improvements and bug fixes to make this end-to-end access control workable. Now, with the release of 0.7.0, the Gravitino unified access control works well with Spark and Ranger to secure end-to-end table access. To see what we have fixed, please check out our issue #4615. You can also try our playground to try out the unified access control feature.

Centralized audit log support

Thanks to the community, Gravitino now supports centralized audit logs. With this feature enabled, users can get the audit information in a centralized place, whether they’re accessing tables or filesets from various sources.

Gravitino’s audit log framework also supports different plugin formatters and writers, and users can implement their own log format and output destinations.

Please see the issues #4887 and #4021 to know more about Gravitino’s centralized audit log.

New data sources support

As a unified data catalog, the community always aims to add more data sources. In this version, Gravitino adds two new data sources. One is Apache Hudi, the other is OceanBase. You can now use Gravitino to manipulate Hudi and Oceanbase metadata in a unified manner.

Various core features

Apart from the features listed above, this version also improves a lot at its core, here are several important features:

  • Add PostgreSQL support for storage backend #4101. Gravitino already supports using MySQL, H2 as its backend metadata storage. In 0.7.0, the community adds the PostgreSQL support to enlarge its adoption.
  • Unify the catalog and metalake drop behavior #5031. In the previous version, we didn’t enforce the behavior of catalog and metadata drop operations. In this version, we redefine its behavior and make it much safer.
  • Manage columns in Gravitino #4493. In 0.7.0, we introduce the column entity in Gravitino, which Gravitino can manage. With this feature introduced, Gravitino can now support tagging on columns, and in future, it will support column-level operations.
  • Add an event listener for Iceberg REST catalog server #5204 and support pre-event for event listener #5112.

Other notable enhancements

Gravitino core

  • Supporting storing column metadata in Gravitino #4493.
  • Support pre-event for Gravitino #5049.
  • Unify drop metalake and catalog behavior #5031.
  • Add credential vending support in Gravitino #4398.
  • Support audit log in Gravitino #4887.
  • Shrink the package size of Gravitino #4513.

Iceberg REST catalog server

  • Add credential vending for Iceberg REST server. #4993.
  • Add an event listener for Iceberg REST server #5204.
  • Support pre-event for event listener #5112.

Catalog related

  • Add OSS support for fileset catalog #5173.
  • Add GCS support for fileset catalog #5074.
  • Add S3 support for fileset catalog #3379.
  • Add pluggable storage support for fileset catalog #5019.
  • Add S3 support for Paimon catalog #4938.
  • Add catalog support for Hudi #4306.
  • Add catalog support for OceanBase#4848.

API and client

  • Add S3 fileset support for Python GVFS client #5188.
  • Add GCS fileset support for Python GVFS client #5139.
  • Add OSS fileset support for Python GVFS client #5221.
  • Supports unified auditing of Fileset metadata and data operations #4021.
  • Support OAuth2 in Python GVFS #3758.

UI

  • Add UI support for operating fileset #5167.
  • Add UI support for operating schema #5140.

All the resolved issues targeting the 0.7.0 release can be seen at https://github.com/apache/gravitino/issues?q=is%3Aissue+is%3Aclosed+label%3A0.7.0+.

Overall

Apache Gravitino 0.7.0 is the second ASF release. This version adds a bunch of new features. We thank the Gravitino community for their continued support and valuable contributions. Thanks to our users' feedback, we can continue to innovate and build,
so thanks to all those reading this!

To further explore the Gravitino 0.7.0 release, please check the documentation.
Your feedback is invaluable to the community and the project.

Credits

This release acknowledges the hard work and dedication of all contributors who have helped make this release possible.

@FANNG1, @LauraXia123, @LindaSummer, @LiuQhahah, @Naresh-kumar-Thodupunoori, @SeanAverS, @caican00, @coolderli, @diqiu50, @featherchen, @hanwxx, @jerqi, @jerryshao, @jingjia88, @justinmclean, @koonchen, @lsyulong, @lw-yang, @mchades, @noidname01, @puchengy, @shaofengshi, @theoryxu, @xiaozcy, @xloya, @xunliu, @yangyuxia, @yaoderek, @yuanoOo, @yuqi1129

Apache, Apache Fink, Apache Hive, Apache Hudi, Apache Iceberg, Apache Ranger, Apache Spark, Apache Paimon and Apache Gravitino are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

Assets 15
Loading