The LowerConvsToMatMul function does not seem to support the ordinary im2col transformation of grouped convolutions.

[sansi-zhang]

I got a "reshape" error while using LowerConvsToMatMul to process grouped convolutions.

I set the number of groups to be a factor of the number of input channels, and I've seen the corresponding value being written to the node in onnx, but there's an error when using LowerConvsToMatMul, it seems that it doesn't correctly recognize grouped convolutions with fewer groups than the number of input channels.

Can I fix this by adjusting LowerConvsToMatMul?

If can I hope you can give me some guidance and advice.

Thanks.

[maltanar]

Hi @sansi-zhang - you are correct, the LowerConvsToMatMul currently supports only either fully dense (groups=1) or depthwise (groups=channels) convolutions. To add support for arbitrary grouped convolutions, it would be necessary to create a sparse matrix in a similar way to how the depthwise case is currently handled - here you can see that we create a weight matrix where only the entries linking the same input channel to the same output channel are non-zero for those cases:

                    W_sparse = np.zeros((ofm_ch, ifm_ch, k_h, k_w))  # (OFM, IFM, k_H, k_W)
                    for ch in range(ifm_ch):
                        W_sparse[ch][ch] = W_conv[ch][0]  # W_conv = [OFM, IFM, k_H, k_W]

If this is something you'd like to work on and contribute to qonnx this would make a great addition to the repository.

[sansi-zhang]

Hi @maltanar, thank you very much for your reply.

I've tried to solve it at the software level,

for i in range(OFM):
    OFM_group = OFM // group
    n = i // OFM_group 
    in_start = n * IFM_group
    in_end = (n + 1) * IFM_group
    W_sparse[i, in_start:in_end, :, :] = weights_to_modify[i, :, :, :]

I used the above code to convert the weights of the grouped convolution into the weights of the ordinary convolution, and then I implemented its hardware structure through MVU.

Moreover, this code can also be used as the construction code of the sparse matrix in the LowerConvsToMatMul function. However, I have not thought of a reasonable way to solve the construction of the hardware computing unit.

Hi @sansi-zhang - you are correct, the LowerConvsToMatMul currently supports only either fully dense (groups=1) or depthwise (groups=channels) convolutions. To add support for arbitrary grouped convolutions, it would be necessary to create a sparse matrix in a similar way to how the depthwise case is currently handled - here you can see that we create a weight matrix where only the entries linking the same input channel to the same output channel are non-zero for those cases:

                    W_sparse = np.zeros((ofm_ch, ifm_ch, k_h, k_w))  # (OFM, IFM, k_H, k_W)
                    for ch in range(ifm_ch):
                        W_sparse[ch][ch] = W_conv[ch][0]  # W_conv = [OFM, IFM, k_H, k_W]

If this is something you'd like to work on and contribute to qonnx this would make a great addition to the repository.