Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TensorRT engine context memory sharing #45842

Merged
merged 12 commits into from
Sep 22, 2022

Conversation

yuanlehome
Copy link
Contributor

@yuanlehome yuanlehome commented Sep 7, 2022

PR types

Others

PR changes

Others

Describe

通过createExecutionContextWithoutDeviceMemory()和setDeviceMemory()接口实现TensorRT engine context memory sharing功能。

总结如下:

  1. 增加了config.trt_engine_memory_sharing()函数(内部使用),trt_engine_memory_sharing参数默认为false,在config.EnableTensorRtEngine()函数内开启;
  2. 当TensorRT版本低于7.2时createExecutionContextWithoutDeviceMemory()存在bug,因此当版本不符合要求时关闭此功能;
  3. 由于setDeviceMemory()要求内存对齐(cudaGetDeviceProperties(),依机器不同可能为256字节或512字节),而memory::Alloc接口从显存池中分配的内存并不能满足对齐要求,因此对申请到的内存进行了内存对齐操作。

@paddle-bot
Copy link

paddle-bot bot commented Sep 7, 2022

你的PR提交成功,感谢你对开源项目的贡献!
请关注后续CI自动化测试结果,详情请参考Paddle-CI手册
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@paddle-bot-old paddle-bot-old bot added the contributor External developers label Sep 13, 2022
Copy link
Contributor

@hp03 hp03 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@paddle-bot-old paddle-bot-old bot removed the contributor External developers label Sep 15, 2022
@jiweibo jiweibo merged commit 173b39b into PaddlePaddle:develop Sep 22, 2022
@yuanlehome yuanlehome deleted the engine_context_memory_sharing branch September 22, 2022 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants