Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for multi path data directories per server to utilize multiple disks present in a node #14461

Open
chrajeshbabu opened this issue Nov 15, 2024 · 0 comments

Comments

@chrajeshbabu
Copy link
Contributor

chrajeshbabu commented Nov 15, 2024

Currently under pinot.server.instance.dataDir we can only configure single directory path with that we cannot utilize the all the disks present in the system. Because of this limitation we need to add more nodes to launch servers and launch servers per disk to accommodate the data.

By having simple of map of segment vs directory path we can use multiple directories under data directories. It would be better to introduce the new configuration pinot.server.instance.dataDirs and while allocating directory to a segment we can have round robin based or size availability based approaches. During the start up also we can build the map by looking at the segments present in the directories configured and update when ever required.

Working on the design document to support the same and will share in few days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant