Constants computed on-the-fly and mmapped Constants #27705
Draft
+296
−21
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ization to IR. Helps to not materialize big computed constants in memory for the entire model. Instead make such Constants one by one during serialization to IR. It consists of two parts: - Serialize pass modification to listen for 'postponed_constant' runtime attribute in a node and on-the-fly replace such a node by a Constant node with constant_fold method. - Python helper `make_postponed_constant` that wrap an arbitrary callback that creates a tensor into a custom Python operation that has runtime attribute `postponed_constant`. This operation will constant folded into a Constant instance with data obtained by calling that callback.
github-actions
bot
added
category: Core
slyalin
commented
Nov 22, 2024
slyalin
commented
Nov 22, 2024
| return PostponedConstant.class_type_info | ||
|
|
||
| def evaluate(self, outputs, _): | ||
| maker().copy_to(outputs[0]) |
There was a problem hiding this comment.
If we allow the assignment of new ov.Tensor instances in outputs, then this copying won't be required. Another (probably better) option is to pass outputs[0] to maker function to build the tensor in place.
… two flavors, reflected in Python API, added move_constant_to_file that dump constant to a file and creates a new Constant with memory mmaped from that file.
github-actions
bot
added
category: tools
slyalin
changed the title
slyalin
changed the title
…ed_memory=True when creating Constant from Tensor adjusting to Python API misalignment with C++ API.
|
This PR will be closed in a week because of 2 weeks of no activity. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A way to avoid the materialization of big constants by computing them on-the-fly during IR serialization. Going to be used in NNCF when compressing big models.