[Bug] Wrong output shapes of MaxPool

[dkurt]

System information (version)

• OpenVINO Source=> pip install
• OpenVINO Version=> 2023.0
• Operating System / Platform => Linux
• Compiler =>GCC
• Problem classification =>nGraph
• Device use: => any
• Framework => any (reproduced with pure nGraph builder)
• Model name => ❔

Detailed description

Wrong output shape calculation during shape propagation. For a specified parameters (kernel 2, stride 2, pad 1, ceil rounding mode) and input 5x5 expected output is 3x3 (as ceil[ (5 + 1 + 1 - 2) / 2 ] = ceil(2.5) = 3).

Steps to reproduce

using nGraph:

#include <iostream>
#include <vector>
#include <openvino/openvino.hpp>
#include <ngraph/ngraph.hpp>

int main() {
    std::vector<size_t> input_shape{1, 3, 5, 5};
    std::vector<size_t> strides{2, 2};
    std::vector<size_t> kernel{2, 2};
    std::vector<size_t> pads{1, 1};
    auto inp = std::make_shared<ngraph::op::Parameter>(ngraph::element::f32, ngraph::Shape(input_shape));
    auto max_pool = std::make_shared<ngraph::op::v1::MaxPool>(
        inp,
        ngraph::Strides(strides),
        ngraph::Shape(pads),
        ngraph::Shape(pads),
        ngraph::Shape(kernel),
        ngraph::op::RoundingType::CEIL,
        ngraph::op::PadType::EXPLICIT
    );
    std::cout << max_pool->get_shape() << std::endl;

    return 0;
}

using PyTorch:

import numpy as np
import torch
import torch.nn as nn
import openvino
from openvino.runtime import Core

print('PyTorch version', torch.__version__)
print('OV version', openvino.runtime.__version__)

class Model(nn.Module):
    def __init__(self, *args, **kwargs):
        super().__init__()
        self.pool = nn.MaxPool2d(kernel_size=(2, 2), stride=2, padding=1, dilation=1, ceil_mode=True)

    def forward(self, x):
        return self.pool(x)

m = Model()
inp = torch.tensor(np.random.standard_normal((1, 3, 5, 5)))
ref = m(inp)
print('ref shape', ref.shape)

torch.onnx.export(m, inp, "model.onnx")

# Run with OpenVINO

core = Core()
compiled = core.compile_model("model.onnx")
req = compiled.create_infer_request()
out = req.infer(np.array(inp))
out = next(iter(out))
print('out shape', out.shape)

PyTorch version 1.13.1+cpu
OV version 2023.0.0-10926-b4452d56304-releases/2023/0
ref shape torch.Size([1, 3, 3, 3])
out shape [1,3,4,4]

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, Stack Overflow, etc and have not found solution
• There is reproducer code and related data files: images, videos, models, etc.

[praasz]

Thanks for reporting the issue.

According to the pytorch documentation MaxPool2d the output H,W dimension should be calculated as:

ceil((5+2*1-1*(2-1)-1)/2 + 1) = ceil(3.5) -> 4

There is +1 after stride divide. The output shape should be [1,3,4,4]

The ONNX and OpenVINO use same equation for calculation as pytorch.
Please see the exported ONNX model it has H and W dimensions also calculated as 4,4 and same result is from OpenVINO.

It looks like ceil mode in pytorch is not working correctly. When in given example the ceil mode is replaced by floor then output shape is still same.

[dkurt]

Please take a look at the note:

When ceil_mode=True, sliding windows are allowed to go off-bounds if they start within the left padding or the input. Sliding windows that would start in the right padded region are ignored.

So for 5x5 input, kernel 2x2, stride 2x2 and 1x1 paddings there are 3x3 sliding windows:

 0 x0 x1 x2 x3 x4 0
|----|-----|-----|

PyTorch shape compute works correctly and decreases output shape by 1 at the end: https://github.com/pytorch/pytorch/blob/15442915cf450347d077a783a9c3ea20ff220686/aten/src/ATen/native/Pool.h#L41-L56

[praasz]

Please take a look at the note:

When ceil_mode=True, sliding windows are allowed to go off-bounds if they start within the left padding or the input. Sliding windows that would start in the right padded region are ignored.

So for 5x5 input, kernel 2x2, stride 2x2 and 1x1 paddings there are 3x3 sliding windows:

 0 x0 x1 x2 x3 x4 0
|----|-----|-----|

PyTorch shape compute works correctly and decreases output shape by 1 at the end: https://github.com/pytorch/pytorch/blob/15442915cf450347d077a783a9c3ea20ff220686/aten/src/ATen/native/Pool.h#L41-L56

Yes right and this is the difference.
As workaround to exporting PyTorch to ONNX model maybe this will help:

torch.onnx.export(m, inp, "model.onnx", opset_version=9)

Looks like using opset 9 the result is same as PyTorch and from opset 10 conversing is not compatible.

[dkurt]

@praasz, thanks for a workaround, but initially this problem is reproduced with nGraph API as well:

#include <iostream>
#include <vector>
#include <openvino/openvino.hpp>
#include <ngraph/ngraph.hpp>

int main() {
    std::vector<size_t> input_shape{1, 3, 5, 5};
    std::vector<size_t> strides{2, 2};
    std::vector<size_t> kernel{2, 2};
    std::vector<size_t> pads{1, 1};
    auto inp = std::make_shared<ngraph::op::Parameter>(ngraph::element::f32, ngraph::Shape(input_shape));
    auto max_pool = std::make_shared<ngraph::op::v1::MaxPool>(
        inp,
        ngraph::Strides(strides),
        ngraph::Shape(pads),
        ngraph::Shape(pads),
        ngraph::Shape(kernel),
        ngraph::op::RoundingType::CEIL,
        ngraph::op::PadType::EXPLICIT
    );
    std::cout << max_pool->get_shape() << std::endl;

    return 0;
}

prints

[1,3,4,4]

[praasz]

@praasz, thanks for a workaround, but initially this problem is reproduced with nGraph API as well:
...
prints

[1,3,4,4]

This is expected result in OpenVINO and it can be different than PyTorch.
There is PR which improve PyTorch frontend conversion. With change proposed by #18965 converting model from PyTorch to OpenVINO should give same results.

Example to convert model:

import numpy as np
import torch
import torch.nn as nn

from openvino.frontend import FrontEndManager
from openvino.runtime import PartialShape

from openvino.frontend.pytorch.ts_decoder import TorchScriptPythonDecoder

class Model(nn.Module):
    def __init__(self, *args, **kwargs):
        super().__init__()
        self.pool = nn.MaxPool2d(2,stride=2, padding=1, dilation=1, ceil_mode=True)

    def forward(self, x):
        return self.pool(x)


input_shape = (1,3,5,5)
m = Model()
inp = torch.tensor(np.random.standard_normal(input_shape))
ref = m(inp)

fe_manager = FrontEndManager()
fe = fe_manager.load_by_framework('pytorch')
decoder = TorchScriptPythonDecoder(m)
im = fe.load(decoder)
om = fe.convert(im)

om.inputs[0].get_node().set_partial_shape(PartialShape(input_shape))
om.validate_nodes_and_infer_types()
print(f'Input {input_shape} -> results ref: {ref.shape} ov: {om.get_output_partial_shape(0)}')

[dkurt]

As I already mentioned, the problem is not in PyTorch/ONNX conversion but with nGraph itself. There is no data outside the padding and output shape should be 3x3. I have extended my C++ sample above so you can see the garbage at last row/col:

#include <iostream>
#include <vector>
#include <openvino/openvino.hpp>
#include <ngraph/ngraph.hpp>

int main() {
    srand(123);

    std::vector<size_t> input_shape{1, 1, 5, 5};
    std::vector<size_t> strides{2, 2};
    std::vector<size_t> kernel{2, 2};
    std::vector<size_t> pads{1, 1};
    auto inp = std::make_shared<ngraph::op::Parameter>(ngraph::element::f32, ngraph::Shape(input_shape));
    auto max_pool = std::make_shared<ngraph::op::v1::MaxPool>(
        inp,
        ngraph::Strides(strides),
        ngraph::Shape(pads),
        ngraph::Shape(pads),
        ngraph::Shape(kernel),
        ngraph::op::RoundingType::CEIL,
        ngraph::op::PadType::EXPLICIT
    );
    std::cout << "OpenVINO output shape: " << max_pool->get_shape() << std::endl;

    // Create a function
    ngraph::ParameterVector inputs{inp};
    ngraph::ResultVector outs;
    outs.push_back(std::make_shared<ngraph::op::Result>(max_pool));
    auto func = std::make_shared<ngraph::Function>(outs, inputs);

    // Run model
    ov::Core core;
    auto compiled = core.compile_model(func, "CPU");
    auto req = compiled.create_infer_request();

    std::vector<float> inpData(5*5, 0);
    std::vector<float> outData(4*4, 0);
    for (int i = 0; i < inpData.size(); ++i) {
        inpData[i] = static_cast<float>(rand()) / RAND_MAX;
    }

    ov::Tensor inpTensor(ov::element::f32, {1, 1, 5, 5}, inpData.data());
    ov::Tensor outTensor(ov::element::f32, {1, 1, 4, 4}, outData.data());
    req.set_input_tensor(inpTensor);
    req.set_output_tensor(outTensor);
    req.infer();

    std::cout << std::endl << "input:" << std::endl;
    for (int i = 0; i < 5; ++i) {
        for (int j = 0; j < 5; ++j) {
            std::cout << inpData[i * 5 + j] << " ";
        }
        std::cout << std::endl;
    }

    std::cout << std::endl << "output:" << std::endl;
    for (int i = 0; i < 4; ++i) {
        for (int j = 0; j < 4; ++j) {
            std::cout << outData[i * 4 + j] << " ";
        }
        std::cout << std::endl;
    }

    return 0;
}

OpenVINO output shape: [1,1,4,4]

input:
0.0600514 0.788318 0.203068 0.348563 0.361609 
0.134639 0.375968 0.259322 0.0443163 0.879562 
0.630366 0.377145 0.319729 0.827858 0.425105 
0.486632 0.790695 0.95527 0.713757 0.352038 
0.84194 0.36238 0.383021 0.0784863 0.174064 

output:
0.0600514 0.788318 0.361609 -3.40282e+38 
0.630366 0.377145 0.879562 -3.40282e+38 
0.84194 0.95527 0.713757 -3.40282e+38 
-3.40282e+38 -3.40282e+38 -3.40282e+38 -3.40282e+38

[praasz]

This is expected behaviour. To change it new version of operator is required.

Other solution is to use additional sub-graph like in created PR which will remove unwanted data.

[dkurt]

Do you mean #18965? It just have not linked, thanks.

Should I open a separate ticket for nGraph?

[praasz]

Do you mean #18965? It just have not linked, thanks.

Should I open a separate ticket for nGraph?

Yes, #18965.

@ilya-lavrenov Do we need new ticket or this can be used as new feature request for operator?

[dkurt]

@praasz, should we close this issue?

[ilya-lavrenov]

@ilya-lavrenov Do we need new ticket or this can be used as new feature request for operator?

Let's create a new feature request ticket for new operator if required.