Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Iceberg: should add caching mechanism for IcebergMetadata #7551

Closed
dixingxing0 opened this issue Apr 11, 2021 · 3 comments
Closed

Iceberg: should add caching mechanism for IcebergMetadata #7551

dixingxing0 opened this issue Apr 11, 2021 · 3 comments
Labels
enhancement New feature or request

Comments

@dixingxing0
Copy link

dixingxing0 commented Apr 11, 2021

I think trino should add caching mechanism for IcebergMetadata for at least two reasons :

  1. to prevent the client from using the same table multiple times in the same query and getting different results, for now iceberg connector accesses hive metastore for multiple times in a single query.
  2. improve query performance, For example,when trino query a small iceberg table, we found a simple count query will cost about 4 seconds, and analyze phase cost more than 2 seconds.
@dixingxing0 dixingxing0 changed the title Iceberg: should use a caching catalog Iceberg: should add caching mechanism for IcebergMetadata Apr 11, 2021
@findepi
Copy link
Member

findepi commented Apr 11, 2021

  • to prevent the client from using the same table multiple times in the same query and getting different results, for now iceberg connector accesses hive metastore for multiple times in a single query.

There should be query- (actually: transaction-) -level "cache".
For reference, in Hive this is done in TransactionalMetadataFactory.
(utilising CachingHiveMetastore under the hood).
Such "cache" should be always used, does not need to be configurable.

Furthermore, there could be additionally an opt-in longer-lived
(not transaction-scoped) metadata cache. For reference, in Hive this is
done in CachingHiveMetastoreModule (also using CachingHiveMetastore,
but with different configuration, this is a cache in a typical sense of the
word).

@findepi findepi added the enhancement New feature or request label Apr 11, 2021
@phd3
Copy link
Member

phd3 commented Apr 16, 2021

this is in progress under #7367. (not long-lived though).

@dixingxing0
Copy link
Author

dixingxing0 commented May 14, 2021

this is in progress under #7367. (not long-lived though).

Thanks @phd3, that's greate, i'm closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Development

No branches or pull requests

3 participants