Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docs for MemoryFileSystemCacheConfig #23704

Open
mosabua opened this issue Oct 7, 2024 · 6 comments
Open

Add docs for MemoryFileSystemCacheConfig #23704

mosabua opened this issue Oct 7, 2024 · 6 comments
Assignees
Labels

Comments

@mosabua
Copy link
Member

mosabua commented Oct 7, 2024

As mentioned by @raunaqmorarka in #23575

This cache is based on iceberg metadata files which are guaranteed to be immutable by the iceberg spec.
So there is no real need for a refresh, as we never deal with stale metadata.
There are some configurables available at io.trino.filesystem.memory.MemoryFileSystemCacheConfig (undocumented at the moment) which can control TTL, max size and max content length for this cache.

For documenting this we need to figure out where these configs are applicable and either start a separate section, a separate page, or some include that is pulled in at various places in the docs.

@mosabua
Copy link
Member Author

mosabua commented Oct 7, 2024

Need to doc these properties

  • fs.memory-cache.ttl
  • fs.memory-cache.max-size
  • fs.memory-cache.max-content-length

@mosabua mosabua added the docs label Oct 7, 2024
@mosabua
Copy link
Member Author

mosabua commented Oct 7, 2024

If anyone wants to help .. that would be great. I will try to get to this but I am flooded.

@sheajamba sheajamba self-assigned this Oct 9, 2024
@anneyshas
Copy link

Hi @sheajamba how are you going with this issue? Would you mind me helping with documentation as well?

@datasundae
Copy link

From the standpoint of where to add the additional documentation on this matter I would suggest it supplement the following:

https://trino.io/docs/current/connector/iceberg.html

iceberg.metadata-cache.enabled Set to false to disable in-memory caching of metadata files on the coordinator. This cache is not used when fs.cache.enabled is set to true. true

Being new to this I am not sure how much documentation would get added. Clearly based on issue 23575 there's a lot that could be clarified but that seems inconsistent with the way the docs are written.

@datasundae
Copy link

I realize this probably "just isn't done" stylistically, but one could simply add a link to the relevant issue number in the documentation where the issue discussed further explains the topic. In this case, a link to the discussion around this topic would do a lot to add clarity and depth to an otherwise rather opaque statement.

@mosabua
Copy link
Member Author

mosabua commented Nov 5, 2024

Pretty sure those settings apply to all object storage connectors so the iceberg connector alone is not enough. Also .. as you rightly guess .. we dont send readers on a goose chase to find information in old issues and such. The documentation should contain the relevant info.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

4 participants