-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Caffe Frontend] introduce caffe frontend for tvm #6206
Conversation
645cee8
to
aab7056
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some initial review comments.
Hi @tqchen @FrozenGene: |
I think we have to install caffe to gpu docker too. |
Aha,what a bad news! Shall we append caffe in gpu docker? I can proposal a new pr. |
Need more advice from you @tqchen |
@fernchen please follow up. |
@fernchen let us skip the tutorials for now and only add the frontend cpu tests |
Hi @tqchen @FrozenGene : |
Hi @tqchen : |
@fernchen please try to send another dummy commit to retrigger the CI, @FrozenGene please followup. I think we can merge once all the tests are resolved |
when testing the caffe frondend.
e9baff5
to
043aff1
Compare
@tqchen Thanks for your suggestions! @FrozenGene I have send a new dummy commit for this pr and now all tests are passed. Please let me konw if you have more suggestions about this pr, thanks. |
target_host = 'llvm' | ||
|
||
ctx = tvm.cpu(0) | ||
with tvm.transform.PassContext(opt_level=2): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why can not be opt level 3?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why can not be opt level 3?
Sorry,this is a historical issue. In previous version, it will throw errors while building LRN layer(Alexnet) with opt_level 3. And this problem has been solved, so i will reset the opt_level to 3. Thanks
m.set_input('data' + str(idx), tvm.nd.array(d.astype(dtype))) | ||
else: | ||
m.set_input('data', tvm.nd.array(data.astype(dtype))) | ||
m.set_input(**params) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't need set parameter now in new module based runtime interface
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't need set parameter now in new module based runtime interface
Thanks! I will delete this line.
weight_filler=dict(type="xavier"), | ||
bias_filler=dict(type="xavier")) | ||
|
||
print("Testing layer Convolution pass!") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove all related print as we have assert
data = np.random.rand(1, 3, 10, 10).astype(np.float32) | ||
_test_batchnorm(data) | ||
_test_batchnorm(data, moving_average_fraction=0.88, eps=1e-4) | ||
print("Testing layer BatchNorm pass!") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto
where is from |
Based on disccussion on #6023 (comment), we will support BVLC Caffe in TVM firstly. So there is no need to compile own caffe.proto in this pr, we use the caffe.proto as follow: from caffe.proto import caffe_pb2 as pb |
Thanks @fernchen It is merged now. |
* [Caffe Frontend] introduce caffe frontend for tvm. * [Caffe Frontend] fix bugs for generating caption in tutorial. * [Caffe Frontend] delete statement for python2 and modify the function name. * [Caffe Frontend] change the directory which will hold the tmp files when testing the caffe frondend. * [Caffe Frontend] delete tutorial about caffe frontend. * [Caffe Frontend] delete some print statements Co-authored-by: fernchen <[email protected]>
* [Caffe Frontend] introduce caffe frontend for tvm. * [Caffe Frontend] fix bugs for generating caption in tutorial. * [Caffe Frontend] delete statement for python2 and modify the function name. * [Caffe Frontend] change the directory which will hold the tmp files when testing the caffe frondend. * [Caffe Frontend] delete tutorial about caffe frontend. * [Caffe Frontend] delete some print statements Co-authored-by: fernchen <[email protected]>
* [Caffe Frontend] introduce caffe frontend for tvm. * [Caffe Frontend] fix bugs for generating caption in tutorial. * [Caffe Frontend] delete statement for python2 and modify the function name. * [Caffe Frontend] change the directory which will hold the tmp files when testing the caffe frondend. * [Caffe Frontend] delete tutorial about caffe frontend. * [Caffe Frontend] delete some print statements Co-authored-by: fernchen <[email protected]>
I mentioned a PR before, see #6000, but because of my mistake, I merged the master branch. In order to correct this mistake, I propose this pr.
Please forgive me for my mistakes and help to review this pr, thanks you very much!
@tqchen @FrozenGene @siju-samuel
Background & Motivation
Caffe is a deep learning framework made with expression, speed, and modularity in mind. Because of its simplicity, good scalability, fast and other characteristics, it is favored by many people. According to riselab who makes statistics of papers collected in arxiv.org, Caffe is ranked in the top four in the deep learning framework, which shows to some extent that Caffe's user base is still large, please refer to blog. In addition, according to our company's research on the market, the demand for Caffe in the production environment is still strong, and many models based on Caffe need to be deployed. For example, existing deployment frameworks, such as MNN, NCNN, MACE, etc., directly support the deployment of Caffe.
TVM only supports caffe2 at present, and the difference between Caffe and caffe2 is quite large. At present, there are two ways to deploy Caffe model in TVM: one is to convert Caffe model to Tensorflow or Pytorch model, the other is to convert Caffe model to onnx and then to relay IR. The two methods are essentially the same. They are all indirectly converted to relay IR through the third-party transformation. However, the problem is that some ops will fail in the process of model transformation, and even the result of transfer out may be different.
Based on the above situation, we decided to open our Caffe frontend codes, hoping to enrich the use scenarios of TVM.
Implementation Approach
The whole process of Caffe front end importing model is divided into:
Finally, we can import the Caffe model as follows:
Work Done
All of the things that we have done are listed as following:
1. List of supported Ops
2. List of supported complete models
3. Caffe frontend test cases
4. Caffe frontend tutorial
TODO
According to the above implementation scheme, based on the front-end framework we built, you can add any new op, you only need to: firstly, add a method in the operatorconverter class, which needs to include your extraction of the layer parameters and the logic of conversion to TVM OP, secondly, register the method to convert_ map.