Use wget to download models in the onnx model zoo #1433
Conversation
Signed-off-by: Tung D. Le <[email protected]>
Signed-off-by: Tung D. Le <[email protected]>
Signed-off-by: Tung D. Le <[email protected]>
| if (len(excluded_names) != 0): | ||
| excluded_msg = "where " + \ | ||
| str(len(excluded_names)) + \ | ||
| " models are not checked because of old opsets or quantization" |
There was a problem hiding this comment.
Instead of line continuation, it's probably better to use parentheses for multi-line expressions:
excluded_msg = ("where " +
str(len(excluded_names)) +
" models are not checked because of old opsets or quantization")
There was a problem hiding this comment.
Thanks for reminding this! It's better.
Signed-off-by: Tung D. Le <[email protected]>
…nnx-mlir into check-onnx-model-zoo-using-wget
|
@gongsu832 @chentong319 any comments on this? Thanks! |
|
Jenkins Linux ppc64le Build #5020 [push] Use wget to download mod... started at 00:35 |
|
Jenkins Linux amd64 Build #5883 [push] Use wget to download mod... started at 23:33 |
|
Jenkins Linux s390x Build #5898 [push] Use wget to download mod... started at 00:33 |
|
Jenkins Linux s390x Build #5898 [push] Use wget to download mod... passed after 56 min |
|
Jenkins Linux ppc64le Build #5020 [push] Use wget to download mod... passed after 1 hr 9 min |
|
Jenkins Linux amd64 Build #5883 [push] Use wget to download mod... passed after 1 hr 9 min |
The current script
CheckONNXModelZoo.pyusesgit lfsto checkout models in the model zoo, which does not support parallelism.This patch uses
wgetinstead to download models so that we can download and check multiple models at the same time.Some other improvements in
CheckONNXModelZoo.py:On my Z machine, it takes about 17 minutes to check 124 models in the model zoo when using 16 processes, e.g.
python CheckONNXModelZoo.py -njob 16 -compile_args="-O3 --mcpu=z14". FYI, this is the result: #128 (comment)