-
Notifications
You must be signed in to change notification settings - Fork 250
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
异步合batch对lod的支持。 #1366
Merged
Merged
异步合batch对lod的支持。 #1366
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Thanks for your contribution! |
LGTM |
TeslaZhao
previously approved these changes
Sep 1, 2021
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
TeslaZhao
approved these changes
Sep 2, 2021
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
一、异步框架对输入参数的支持新增:
1、支持输入参数是lod的情况。
2、支持输入参数是nobatch,永远是shape[0] = 1的情况。
二、异步框架对输出参数的支持新增:
1、支持输出参数是lod的情况。
2、支持输出参数是nobatch,永远是shape[0] = 1的情况。
三、新增了一个配置选项,允许用户设置,异步框架是否可以拆req,以达到batch最大利用。
1、默认该选项为开启,即必要时会拆req.
2、修改了原配置参数中的名称歧义。
拆的优势是:
1、能够减少调用预测库的次数,节省时间。
2、即使设置的合并batch的最大数为32,用户输入了64batch,此时也不会一直判断batch不够而卡死,而会分批执行。
3、即使模型最大支持32batch,用户输入了64batch,正常情况下会预测库报错,但此时不会报错,而会拆分执行。
拆的劣势是:
1、由于一个Task被拆分为多个TaskMeta,此时对于输出OutputTensor的初始化,需要加锁,防止竞争。
2、当输出的某个参数是lod的时候,由于一个Task被拆分为多个TaskMeta,必须额外开辟空间记录每个taskmeta的lodfetchvar,最后再合并。此处有内存的额外开销和拷贝开销。(只有某个lod才会有开销)、
具体拆还是不拆好,有待多种情况下的性能测试。
四、修改了HttpClient的部分逻辑,使其与Client完全对齐。
1、目前的示例代码,不需要改动,仅需要替换import Client为import HttpClient即可支持Http模式
2、修改了python/examples/xpu下面的示例和README
五、修复了异步框架bug
1、修复了异步框架中的fetch 为lod的bug
2、修复了异步框架中死锁的问题。