You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I think trino should add caching mechanism for IcebergMetadata for at least two reasons :
to prevent the client from using the same table multiple times in the same query and getting different results, for now iceberg connector accesses hive metastore for multiple times in a single query.
improve query performance, For example,when trino query a small iceberg table, we found a simple count query will cost about 4 seconds, and analyze phase cost more than 2 seconds.
The text was updated successfully, but these errors were encountered:
dixingxing0
changed the title
Iceberg: should use a caching catalog
Iceberg: should add caching mechanism for IcebergMetadata
Apr 11, 2021
to prevent the client from using the same table multiple times in the same query and getting different results, for now iceberg connector accesses hive metastore for multiple times in a single query.
There should be query- (actually: transaction-) -level "cache".
For reference, in Hive this is done in TransactionalMetadataFactory.
(utilising CachingHiveMetastore under the hood).
Such "cache" should be always used, does not need to be configurable.
Furthermore, there could be additionally an opt-in longer-lived
(not transaction-scoped) metadata cache. For reference, in Hive this is
done in CachingHiveMetastoreModule (also using CachingHiveMetastore,
but with different configuration, this is a cache in a typical sense of the
word).
I think trino should add caching mechanism for IcebergMetadata for at least two reasons :
The text was updated successfully, but these errors were encountered: