Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Runtime] Allow parameter sharing between modules #3887

Closed
wants to merge 1 commit into from

Conversation

yongsun
Copy link
Contributor

@yongsun yongsun commented Sep 3, 2019

As GraphRuntime does not provide control-flow logics, we have to split
our model to two parts. While we need to share parameters between them
to save memory usage.

Solution:

  1. add "lazy_init_input" in graph's attributes
   "attrs": {
     ... ...
     "lazy_init_input": [
       "list_str",
       [
         "p0"
       ]
     ]
    }
  1. allow un-allocated NDArray entry in SetupStorage
  2. utilize "set_input_zero_copy" function to set parameters

Thanks for contributing to TVM! Please refer to guideline https://docs.tvm.ai/contribute/ for useful information and tips. After the pull request is submitted, please request code reviews from Reviewers.

As GraphRuntime does not provide control-flow logics, we have to split
our model to two parts. While we need to share parameters between them
to save memory usage.

Solution:
1) add "lazy_init_input" in graph's attributes
   "attrs": {
     ... ...
     "lazy_init_input": [
       "list_str",
       [
         "p0"
       ]
     ]
    }
2) allow un-allocated NDArray entry in SetupStorage
3) utilize "set_input_zero_copy" function to set parameters
@icemelon
Copy link
Member

icemelon commented Sep 4, 2019

I realize that there's two issues in this PR.

  1. The input ndarray is a view created from the ndarray in the storage pool. If you use SetInputZeroCopy to set the input ndarray, it only modifies the data ptr in the input ndarray, but not the data ptr in the storage entry. Therefore, if other intermediate tensor shares the same ndarray in the storage pool, it will still have the data ptr as nullpr.
  2. Even if you fix issue 1, there's a follow-up question. If the intermediate tensor requires larger space than the input tensor, it means that using SetInputZeroCopy may not provide enough space for this storage entry.

@yongsun
Copy link
Contributor Author

yongsun commented Sep 4, 2019

Thanks so much for catching the issues! I'll close this PR and find a clean solution.

@yongsun yongsun closed this Sep 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants