[InSPyReNet] Swin Transformer Support question #312
Labels
no-issue-activity
OP:LayerNormalization
OP:LayerNormalization
OP:MatMul
OP:MatMul
OP:Softmax
OP:Softmax
Parameter replacement
Use Parameter replacement
third party
Third-party tool issues
Transformer
Transformer
Comments
|
The See this issue. When Onnx Matmul inputs have different dimension #133
Yes. https://github.com/PINTO0309/onnx2tf#validated-models-without-replacementjson
If you share the ONNX files with me, I will look into it. If you can't, I won't look any further. The work cost of generating ONNX wastes my private time. |
PINTO0309
changed the title
|
Thank you for the response! and valuable pointers 😄 I've uploaded the file here: https://github.com/bernii/aut-1-5/releases/download/test/latest.opset17.onnx in case you want to take a look. I'd be happy to debug myself further but I'm not sure what the procedure should be - there's lots of going on here. I just re-run the conversion with and got following result. It seems that there was Out Of Memory error during the verification process as it was trying to allocate ~60GB of RAM. |
I see. This seems to be too large for the model. If time permits, I feel that the path to success is closer if the model is separated and validated in as much detail as possible. Using the https://github.com/PINTO0309/sne4onnx
|
|
Not in a hurry but I'd love to help if possible :) I split a part of the model with and seems that things are going sideways very early as I see Full log |
|
Thank you. It's a Once the CI is all green, I will release the revised package in about an hour.
|
PINTO0309
added
Bug
|
Also, I found one more thing - my onnx model was already run through it it ok if I simplify onnx in my own pipeline or I should not do that when using onnx2tf? Uploaded the onnx no-simplify model in case you need it: https://github.com/bernii/aut-1-5/releases/download/test/latest.opset17.no-simplify.onnx |
Either is fine. However, there is an important technique. For huge onnx models, run onnxsim at least 5 times. |
|
Fixes |
|
🙇 I can see that I made another extraction but getting an exception - not sure if I'm messing model with sne4onnx or that's expected 🤔 |
|
I debugged it using the commands you provided and strangely enough the conversion succeeded without error. Although there is a transposition error somewhere in the OP, it does not appear to be a conversion error. I just copied and pasted your command, so I don't know where the possible problem is, except that the versions of onnxsim may not match. I am surprised that the
|
|
Incidentally,
replace.json{
"format_version": 1,
"operations": [
{
"op_name": "/model/context2/branch1/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context2/branch1/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context2/branch2/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context2/branch2/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/decoder/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/decoder/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context2/branch3/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context2/branch3/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
}
]
}
|
|
I checked what casues the error, it's The rest of my dependencies
|
|
What? It never occurred to me that there could be a problem with onnxsim. Thanks to you I have identified two serious problems today. I will make one more correction and release it. 😸 |
PINTO0309
added
third party
|
I'm continuing the work taking your The input dimensions of transpose previous to it are not the same and yet it's
|
|
The reason why errors are detected in accuracy verification with replace_InSPyReNet.json{
"format_version": 1,
"operations": [
{
"op_name": "/model/context2/branch1/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context2/branch1/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context2/branch2/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context2/branch2/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/decoder/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/decoder/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context2/branch3/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context2/branch3/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context3/branch1/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context3/branch1/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context3/branch2/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context3/branch2/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context3/branch3/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context3/branch3/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context4/branch1/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context4/branch1/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context4/branch2/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context4/branch2/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context4/branch3/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context4/branch3/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context5/branch1/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context5/branch1/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context5/branch2/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context5/branch2/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context5/branch3/Hattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
},
{
"op_name": "/model/context5/branch3/Wattn/Transpose_1",
"param_target": "attributes",
"param_name": "perm",
"values": [0,2,1]
}
]
}The final outputs have been matched. Maybe the fact that the error is flattened by
|
|
thank you for the explanation :) |
|
I would like to implement the logic to determine the correct transposition of Transformer's So far I myself find Transformer conversions to be quite cumbersome. |
|
I do not know enough about most common problems but in case of When it comes to replacement I'm struggling a bit with my new extraction I'm trying to address problems around bigger errors like I'm wondering if there might be a problem with layer normalization though, while I was being able to reduce the error from
|
|
This model is crazy big. 😅 I'll take a look at a few when I have time.
I see. Might be a good idea. Thanks. I'll try various things. |
|
If I see things correctly, the origin of the transpose seems to be related to the onnx2tf/onnx2tf/ops/LayerNormalization.py Lines 101 to 102 in 2916710 and is some cases the value of bias is being transpoed onnx2tf/onnx2tf/ops/LayerNormalization.py Line 90 in 2916710 btw there seems to be a typo there |
|
many places with high errors that I've inspected from this extraction seem to have |
|
[Second step] Further separation of the model to identify problem areas. Temporarily replace sne4onnx \
--input_onnx_file_path latest.opset17.head3.onnx \
--input_op_names /model/backbone/layers.1/downsample/norm/LayerNormalization_output_0 onnx::MatMul_14949 \
--output_op_names /model/backbone/layers.2/blocks.0/Add_output_0 \
--output_onnx_file_path latest.opset17.head3_test.onnxonnx2tf -i latest.opset17.head3_test.onnx
From the inference results, I identified no problems with latest.opset17.head3_test_float32.tflite.zip However, I have noticed that errors occur when local models are extracted and transformed within the following ranges. sne4onnx \
--input_onnx_file_path latest.opset17.head3.onnx \
--input_op_names /model/backbone/layers.2/downsample/norm/LayerNormalization_output_0 onnx::MatMul_15970 \
--output_op_names /model/backbone/layers.3/blocks.0/Add_output_0 \
--output_onnx_file_path latest.opset17.head3_test2.onnxonnx2tf -i latest.opset17.head3_test2.onnx \
-kat /model/backbone/layers.2/downsample/norm/LayerNormalization_output_0Note that the output of
latest.opset17.head3_test2_float32.tflite.zip
{
"format_version": 1,
"operations": [
{
"op_name": "/model/backbone/layers.3/blocks.0/norm1/LayerNormalization",
"param_target": "outputs",
"param_name": "/model/backbone/layers.3/blocks.0/norm1/LayerNormalization_output_0",
"post_process_transpose_perm": [0,2,1]
}
]
}onnx2tf -i latest.opset17.head3_test2.onnx \
-kat /model/backbone/layers.2/downsample/norm/LayerNormalization_output_0 \
-prf replace_latest.opset17.head3.jsonI disabled
|
|
Thanks for checking and the explanation! I took a little detour and tried playing with model after onnx conversion with |
|
Thank you. I have almost identified the problem as being with |
|
I just did take a look into a new extraction and I definitely have couple examples of errors around
|
|
Thanks to your help, I feel I have almost identified some bug in |
|
It will take some time, but the following four steps will help to isolate the problem into smaller pieces and proceed to deal with it.
|
|
I first examine how the operations around the
Netron was used to extract the
The extracted weights were loaded in Numpy and the tensor values were directly compared. This is correct, although the order of tensors is reversed from ONNX due to the tflite specification. The tensor on the tflite side is inverted and then compared to the tensor on the onnx side. Even if the shapes are matched, the tensor values are different. The reason for the Despite the manual calculations, the tflite weights do not match the onnx weights at all. Therefore, let us examine the maximum absolute error.
An incomprehensibly large error was displayed. This is a much larger error than the error detected by the At the moment I have not been able to reproduce the error by hand calculations, so I have not been able to identify the problem area. |
|
Hi @PINTO0309 I'm the author of onnxsim. Sorry for the convenience. I have fixed the bug of onnxsim in the latest version. You might want to have a try and upgrade to the latest version :) |
|
@PINTO0309 anything I can do to help? and errors seem similar to the ones before. |
|
Thank you. I will be very busy until the second week of May and will not have enough time to debug and maintain at the same pace as before. Probably related to this issue as well. Since the accuracy check fails with a simple multiplication, there is most likely a problem with the logic of the accuracy check rather than a problem with the model transformation. |
This was referenced May 3, 2023
|
It takes about 30 minutes for one conversion test. However, the conversion was ultimately successful. When using the https://github.com/PINTO0309/onnx2tf/releases/tag/1.11.0 https://github.com/PINTO0309/onnx2tf/blob/main/json_samples/replace_InSPyReNet.json
https://s3.ap-northeast-2.wasabisys.com/temp-models/onnx2tf_312/latest.opset17_float32.tflite I have removed the |
|
If there is no activity within the next two days, this issue will be closed automatically. |
|
did a quick test today and seems that the converted model gives perfect results, I'll do more tests over the week and report back if there's anything fishy there :) thanks for tremendous work! @PINTO0309 🙇 🙇 |
|
I can confirm - everything works like a charm 👌 ❤️ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue Type
Others
onnx2tf version number
1.9.2
onnx version number
1.13.1
tensorflow version number
2.12.0
Download URL for ONNX
https://github.com/plemeri/InSPyReNet
Parameter Replacement JSON
NoneDescription
This is more of a general question - are there any success stories of using onnx2tf with Swin Transformers (for example https://github.com/plemeri/InSPyReNet is using SwinB)? I tried conversion and it passed without errors to
saved_modelformat but inference gives me output that has nothing to do with the input.Wondering if that's a known issue/untested territory or maybe I do have problem on my side
The text was updated successfully, but these errors were encountered: