Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[3.0] Use fallocate in incremental mode #150

Open
bchess opened this issue Jun 27, 2024 · 1 comment
Open

[3.0] Use fallocate in incremental mode #150

bchess opened this issue Jun 27, 2024 · 1 comment
Assignees
Milestone

Comments

@bchess
Copy link
Contributor

bchess commented Jun 27, 2024

Regarding

        if not incremental:
            write_dependency = self._maybe_fallocate(write_specs)

Eta0 last week
As far as I can tell, this is the only time this flag is used, and it doesn't actually correspond to whether it is an incremental write or not, only whether it is from a call to write_tensor or not, so this flag could be renamed for clarity. That aside, for a large enough single tensor, it's still fine to call fallocate; it can proceed in the background during tensor and metadata preparation, including hash calculation.

@bchess bchess 2 days ago
Yeah, but the logic inside _maybe_fallocate() uses its tensors list to calculate total size. To work in incremental mode it'd need to be updated to account for past writes.

@Eta0
Copy link
Contributor

Eta0 commented Jun 27, 2024

Copying another of my comments from that thread (#127 (comment)):

fallocate extends the file relative to the current file position, so it doesn't need anything special to account for past writes if the file position is reliable; it can just progressively extend the file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants