From 6ea9a8b35d1f84e576ece0c9de431ff8596f1eb2 Mon Sep 17 00:00:00 2001 From: jdanieck Date: Sun, 31 Jan 2021 19:49:29 +0100 Subject: [PATCH 1/5] DeformableConvolution specification refactoring. --- .../convolution/DeformableConvolution_1.md | 143 +++++++++++++++--- 1 file changed, 119 insertions(+), 24 deletions(-) diff --git a/docs/ops/convolution/DeformableConvolution_1.md b/docs/ops/convolution/DeformableConvolution_1.md index 247c52ea121e7e..986269afe310a6 100644 --- a/docs/ops/convolution/DeformableConvolution_1.md +++ b/docs/ops/convolution/DeformableConvolution_1.md @@ -4,22 +4,24 @@ **Category**: Convolution -**Detailed description**: [Reference](https://arxiv.org/abs/1703.06211) +**Short description**: Computes 1D, 2D or 3D deformable convolution of input and kernel tensors. -**Attributes** +**Detailed description**: *Deformable Convolution* is similar to regular *Convolution* but its receptive field is deformed because of additional spatial offsets used during input sampling. More thorough explanation can be found in [Deformable Convolutions Demystified](https://towardsdatascience.com/deformable-convolutions-demystified-2a77498699e8) and [Deformable Convolutional Networks](https://arxiv.org/abs/1703.06211). + +**Attributes**: * *strides* - * **Description**: *strides* is a distance (in pixels) to slide the filter on the feature map over the (z, y, x) axes for 3D convolutions and (y, x) axes for 2D convolutions. For example, *strides* equal *4,2,1* means sliding the filter 4 pixel at a time over depth dimension, 2 over height dimension and 1 over width dimension. - * **Range of values**: integer values starting from 0 + * **Description**: *strides* is a distance (in pixels) to slide the filter on the feature map over the `(z, y, x)` axes for 3D convolutions and `(y, x)` axes for 2D convolutions. For example, *strides* equal `4,2,1` means sliding the filter 4 pixel at a time over depth dimension, 2 over height dimension and 1 over width dimension. + * **Range of values**: integer values starting from `0` * **Type**: int[] * **Default value**: None * **Required**: *yes* * *pads_begin* - * **Description**: *pads_begin* is a number of pixels to add to the beginning along each axis. For example, *pads_begin* equal *1,2* means adding 1 pixel to the top of the input and 2 to the left of the input. - * **Range of values**: integer values starting from 0 + * **Description**: *pads_begin* is a number of pixels to add to the beginning along each axis. For example, *pads_begin* equal `1,2` means adding 1 pixel to the top of the input and 2 to the left of the input. + * **Range of values**: integer values starting from `0` * **Type**: int[] * **Default value**: None * **Required**: *yes* @@ -27,8 +29,8 @@ * *pads_end* - * **Description**: *pads_end* is a number of pixels to add to the ending along each axis. For example, *pads_end* equal *1,2* means adding 1 pixel to the bottom of the input and 2 to the right of the input. - * **Range of values**: integer values starting from 0 + * **Description**: *pads_end* is a number of pixels to add to the ending along each axis. For example, *pads_end* equal `1,2` means adding 1 pixel to the bottom of the input and 2 to the right of the input. + * **Range of values**: integer values starting from `0` * **Type**: int[] * **Default value**: None * **Required**: *yes* @@ -36,8 +38,8 @@ * *dilations* - * **Description**: *dilations* denotes the distance in width and height between elements (weights) in the filter. For example, *dilation* equal *1,1* means that all the elements in the filter are neighbors, so it is the same as for the usual convolution. *dilation* equal *2,2* means that all the elements in the filter are matched not to adjacent elements in the input matrix, but to those that are adjacent with distance 1. - * **Range of values**: integer value starting from 0 + * **Description**: *dilations* denotes the distance in width and height between elements (weights) in the filter. For example, *dilation* equal `1,1` means that all the elements in the filter are neighbors, so it is the same as for the usual convolution. *dilation* equal `2,2` means that all the elements in the filter are matched not to adjacent elements in the input matrix, but to those that are adjacent with distance 1. + * **Range of values**: integer value starting from `0` * **Type**: int[] * **Default value**: None * **Required**: *yes* @@ -45,44 +47,137 @@ * *auto_pad* * **Description**: *auto_pad* how the padding is calculated. Possible values: - * *explicit*: use explicit padding values from `pads_begin` and `pads_end`. - * *same_upper (same_lower)* the input is padded to match the output size. In case of odd padding value an extra padding is added at the end (at the beginning). + * *explicit* - use explicit padding values from *pads_begin* and *pads_end*. + * *same_upper* - the input is padded to match the output size. In case of odd padding value an extra padding is added at the end. + * *same_lower* - the input is padded to match the output size. In case of odd padding value an extra padding is added at the beginning. * *valid* - do not use padding. * **Type**: string - * **Default value**: None + * **Default value**: explicit * **Required**: *no* * **Note**: *pads_begin* and *pads_end* attributes are ignored when *auto_pad* is specified. + * *group* * **Description**: *group* is the number of groups which *output* and *input* should be split into. For example, *group* equal to 1 means that all filters are applied to the whole input (usual convolution), *group* equal to 2 means that both *input* and *output* channels are separated into two groups and the *i-th output* group is connected to the *i-th input* group channel. *group* equal to a number of output feature maps implies depth-wise separable convolution. - * **Range of values**: integer value starting from 1 + * **Range of values**: integer value starting from `1` * **Type**: int - * **Default value**: 1 + * **Default value**: `1` * **Required**: *no* * *deformable_group* * **Description**: *deformable_group* is the number of groups which deformable values and *output* should be split into along the channel axis. Apply the deformable convolution using the i-th part of the offset part on the i-th out. - * **Range of values**: integer value starting from 1 + * **Range of values**: integer value starting from `1` * **Type**: int - * **Default value**: 1 + * **Default value**: `1` * **Required**: *no* **Inputs**: -* **1**: Input tensor of rank 3 or greater. Required. +* **1**: Input tensor of type *T* and rank 3, 4 or 5. Layout is NCZYX (number of batches, number of channels, spatial axes Z, Y, X). Required. + +* **2**: Deformable values tensor of type *T* and rank 3, 4 or 5. Layout is *TBD*. Required. + +* **3**: Kernel tensor of type *T* and rank 3, 4 or 5. Layout is OIZYX (number of output channels, number of input channels, spatial axes Z, Y, X). Required. +* **Note**: Type of the convolution (1D, 2D or 3D) is derived from the rank of the input tensors and not specified by any attribute: + * 1D convolution (input tensors rank 3) means that there is only one spatial axis X + * 2D convolution (input tensors rank 4) means that there are two spatial axes Y, X + * 3D convolution (input tensors rank 5) means that there are three spatial axes Z, Y, X + -* **2**: Deformable values tensor of rank 3 or higher. Required. +**Outputs**: -* **3**: Convolution kernel tensor. Weights layout is OIYX (OIZYX for 3D convolution), which means that *X* is changing the fastest, then *Y*, then *Input* then *Output*. The size of kernel is derived from the shape of this input and not specified by any attribute. Required. +* **1**: Output tensor of type *T* and rank 3, 4 or 5. Layout is NOZYX (number of batches, number of kernel output channels, spatial axes Z, Y, X). +**Types**: + +* *T*: Any floating point type. + **Example** +1D Convolution +```xml + + + + + 1 + 5 + 128 + + + 16 + 5 + 4 + + + + + 1 + 16 + 63 + + + +``` +2D Convolution +```xml + + + + + 1 + 3 + 224 + 224 + + + 64 + 3 + 5 + 5 + + + + + 1 + 64 + 224 + 224 + + + +``` + +3D Convolution ```xml - - - ... - ... + + + + + 1 + 7 + 320 + 320 + 320 + + + 32 + 7 + 3 + 3 + 3 + + + + + 1 + 32 + 106 + 106 + 106 + + ``` \ No newline at end of file From 2aebe13d2b8c95572e32695d6020906c63c31bab Mon Sep 17 00:00:00 2001 From: jdanieck Date: Sun, 7 Feb 2021 11:45:28 +0100 Subject: [PATCH 2/5] Add ticks to types and layouts. --- .../convolution/DeformableConvolution_1.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/ops/convolution/DeformableConvolution_1.md b/docs/ops/convolution/DeformableConvolution_1.md index 986269afe310a6..8ffd13d0f095c4 100644 --- a/docs/ops/convolution/DeformableConvolution_1.md +++ b/docs/ops/convolution/DeformableConvolution_1.md @@ -14,7 +14,7 @@ * **Description**: *strides* is a distance (in pixels) to slide the filter on the feature map over the `(z, y, x)` axes for 3D convolutions and `(y, x)` axes for 2D convolutions. For example, *strides* equal `4,2,1` means sliding the filter 4 pixel at a time over depth dimension, 2 over height dimension and 1 over width dimension. * **Range of values**: integer values starting from `0` - * **Type**: int[] + * **Type**: `int[]` * **Default value**: None * **Required**: *yes* @@ -22,7 +22,7 @@ * **Description**: *pads_begin* is a number of pixels to add to the beginning along each axis. For example, *pads_begin* equal `1,2` means adding 1 pixel to the top of the input and 2 to the left of the input. * **Range of values**: integer values starting from `0` - * **Type**: int[] + * **Type**: `int[]` * **Default value**: None * **Required**: *yes* * **Note**: the attribute is ignored when *auto_pad* attribute is specified. @@ -31,7 +31,7 @@ * **Description**: *pads_end* is a number of pixels to add to the ending along each axis. For example, *pads_end* equal `1,2` means adding 1 pixel to the bottom of the input and 2 to the right of the input. * **Range of values**: integer values starting from `0` - * **Type**: int[] + * **Type**: `int[]` * **Default value**: None * **Required**: *yes* * **Note**: the attribute is ignored when *auto_pad* attribute is specified. @@ -40,7 +40,7 @@ * **Description**: *dilations* denotes the distance in width and height between elements (weights) in the filter. For example, *dilation* equal `1,1` means that all the elements in the filter are neighbors, so it is the same as for the usual convolution. *dilation* equal `2,2` means that all the elements in the filter are matched not to adjacent elements in the input matrix, but to those that are adjacent with distance 1. * **Range of values**: integer value starting from `0` - * **Type**: int[] + * **Type**: `int[]` * **Default value**: None * **Required**: *yes* @@ -51,7 +51,7 @@ * *same_upper* - the input is padded to match the output size. In case of odd padding value an extra padding is added at the end. * *same_lower* - the input is padded to match the output size. In case of odd padding value an extra padding is added at the beginning. * *valid* - do not use padding. - * **Type**: string + * **Type**: `string` * **Default value**: explicit * **Required**: *no* * **Note**: *pads_begin* and *pads_end* attributes are ignored when *auto_pad* is specified. @@ -61,7 +61,7 @@ * **Description**: *group* is the number of groups which *output* and *input* should be split into. For example, *group* equal to 1 means that all filters are applied to the whole input (usual convolution), *group* equal to 2 means that both *input* and *output* channels are separated into two groups and the *i-th output* group is connected to the *i-th input* group channel. *group* equal to a number of output feature maps implies depth-wise separable convolution. * **Range of values**: integer value starting from `1` - * **Type**: int + * **Type**: `int` * **Default value**: `1` * **Required**: *no* @@ -69,17 +69,17 @@ * **Description**: *deformable_group* is the number of groups which deformable values and *output* should be split into along the channel axis. Apply the deformable convolution using the i-th part of the offset part on the i-th out. * **Range of values**: integer value starting from `1` - * **Type**: int + * **Type**: `int` * **Default value**: `1` * **Required**: *no* **Inputs**: -* **1**: Input tensor of type *T* and rank 3, 4 or 5. Layout is NCZYX (number of batches, number of channels, spatial axes Z, Y, X). Required. +* **1**: Input tensor of type *T* and rank 3, 4 or 5. Layout is `NCZYX` (number of batches, number of channels, spatial axes Z, Y, X). Required. -* **2**: Deformable values tensor of type *T* and rank 3, 4 or 5. Layout is *TBD*. Required. +* **2**: Offsets tensor of type *T* and rank 3, 4 or 5. Layout is `NCZYX*`(number of batches, number of channels, spatial axes Z, Y, X). Required. -* **3**: Kernel tensor of type *T* and rank 3, 4 or 5. Layout is OIZYX (number of output channels, number of input channels, spatial axes Z, Y, X). Required. +* **3**: Kernel tensor of type *T* and rank 3, 4 or 5. Layout is `OIZYX` (number of output channels, number of input channels, spatial axes Z, Y, X). Required. * **Note**: Type of the convolution (1D, 2D or 3D) is derived from the rank of the input tensors and not specified by any attribute: * 1D convolution (input tensors rank 3) means that there is only one spatial axis X * 2D convolution (input tensors rank 4) means that there are two spatial axes Y, X From 486d50f5ff5bc6bd09e957d9b0159e1ebc9019ae Mon Sep 17 00:00:00 2001 From: jdanieck Date: Sun, 7 Feb 2021 11:58:39 +0100 Subject: [PATCH 3/5] Removed 1D and 3D Convolution type. At the current moment CPU plugin and MXnet implemenatations supports only 2D case. --- .../convolution/DeformableConvolution_1.md | 73 ++----------------- 1 file changed, 6 insertions(+), 67 deletions(-) diff --git a/docs/ops/convolution/DeformableConvolution_1.md b/docs/ops/convolution/DeformableConvolution_1.md index 8ffd13d0f095c4..c77fe283071338 100644 --- a/docs/ops/convolution/DeformableConvolution_1.md +++ b/docs/ops/convolution/DeformableConvolution_1.md @@ -4,7 +4,7 @@ **Category**: Convolution -**Short description**: Computes 1D, 2D or 3D deformable convolution of input and kernel tensors. +**Short description**: Computes 2D deformable convolution of input and kernel tensors. **Detailed description**: *Deformable Convolution* is similar to regular *Convolution* but its receptive field is deformed because of additional spatial offsets used during input sampling. More thorough explanation can be found in [Deformable Convolutions Demystified](https://towardsdatascience.com/deformable-convolutions-demystified-2a77498699e8) and [Deformable Convolutional Networks](https://arxiv.org/abs/1703.06211). @@ -12,7 +12,7 @@ * *strides* - * **Description**: *strides* is a distance (in pixels) to slide the filter on the feature map over the `(z, y, x)` axes for 3D convolutions and `(y, x)` axes for 2D convolutions. For example, *strides* equal `4,2,1` means sliding the filter 4 pixel at a time over depth dimension, 2 over height dimension and 1 over width dimension. + * **Description**: *strides* is a distance (in pixels) to slide the filter on the feature map over the `(y,x)` axes. For example, *strides* equal `2,1` means sliding the filter 2 pixel at a time over height dimension and 1 over width dimension. * **Range of values**: integer values starting from `0` * **Type**: `int[]` * **Default value**: None @@ -75,20 +75,16 @@ **Inputs**: -* **1**: Input tensor of type *T* and rank 3, 4 or 5. Layout is `NCZYX` (number of batches, number of channels, spatial axes Z, Y, X). Required. +* **1**: Input tensor of type *T* and rank 4. Layout is `NCYX` (number of batches, number of channels, spatial axes Y and X). Required. -* **2**: Offsets tensor of type *T* and rank 3, 4 or 5. Layout is `NCZYX*`(number of batches, number of channels, spatial axes Z, Y, X). Required. +* **2**: Offsets tensor of type *T* and rank 4. Layout is `NCYX` (number of batches, number of channels, spatial axes Y and X). Required. -* **3**: Kernel tensor of type *T* and rank 3, 4 or 5. Layout is `OIZYX` (number of output channels, number of input channels, spatial axes Z, Y, X). Required. -* **Note**: Type of the convolution (1D, 2D or 3D) is derived from the rank of the input tensors and not specified by any attribute: - * 1D convolution (input tensors rank 3) means that there is only one spatial axis X - * 2D convolution (input tensors rank 4) means that there are two spatial axes Y, X - * 3D convolution (input tensors rank 5) means that there are three spatial axes Z, Y, X +* **3**: Kernel tensor of type *T* and rank 4. Layout is `OIYX` (number of output channels, number of input channels, spatial axes Y and X). Required. **Outputs**: -* **1**: Output tensor of type *T* and rank 3, 4 or 5. Layout is NOZYX (number of batches, number of kernel output channels, spatial axes Z, Y, X). +* **1**: Output tensor of type *T* and rank 4. Layout is `NOYX` (number of batches, number of kernel output channels, spatial axes Y and X). **Types**: @@ -96,31 +92,6 @@ **Example** -1D Convolution -```xml - - - - - 1 - 5 - 128 - - - 16 - 5 - 4 - - - - - 1 - 16 - 63 - - - -``` 2D Convolution ```xml @@ -149,35 +120,3 @@ ``` - -3D Convolution -```xml - - - - - 1 - 7 - 320 - 320 - 320 - - - 32 - 7 - 3 - 3 - 3 - - - - - 1 - 32 - 106 - 106 - 106 - - - -``` \ No newline at end of file From 4e8c9f2ff84ecb44e05437f1ad8177aa9dc87ff3 Mon Sep 17 00:00:00 2001 From: jdanieck Date: Sun, 7 Feb 2021 12:39:46 +0100 Subject: [PATCH 4/5] Fix examples section. --- .../convolution/DeformableConvolution_1.md | 49 +++++++++++++++++-- 1 file changed, 45 insertions(+), 4 deletions(-) diff --git a/docs/ops/convolution/DeformableConvolution_1.md b/docs/ops/convolution/DeformableConvolution_1.md index c77fe283071338..b621e9d2acfe34 100644 --- a/docs/ops/convolution/DeformableConvolution_1.md +++ b/docs/ops/convolution/DeformableConvolution_1.md @@ -92,20 +92,26 @@ **Example** -2D Convolution +2D DeformableConvolution (deformable_group=1) ```xml - + 1 - 3 + 4 224 224 + 1 + 50 + 220 + 220 + + 64 - 3 + 4 5 5 @@ -114,9 +120,44 @@ 1 64 + 220 + 220 + + + +``` + +2D DeformableConvolution (deformable_group=4) +```xml + + + + + 1 + 4 224 224 + + 1 + 200 + 220 + 220 + + + 64 + 4 + 5 + 5 + + + + + 1 + 64 + 220 + 220 + ``` From b57cc07dc1f170aa52fef5be121dc3c2df148228 Mon Sep 17 00:00:00 2001 From: jdanieck Date: Sun, 7 Feb 2021 13:04:23 +0100 Subject: [PATCH 5/5] Fix offests input layout description. --- docs/ops/convolution/DeformableConvolution_1.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ops/convolution/DeformableConvolution_1.md b/docs/ops/convolution/DeformableConvolution_1.md index b621e9d2acfe34..446d6fd07bb8ac 100644 --- a/docs/ops/convolution/DeformableConvolution_1.md +++ b/docs/ops/convolution/DeformableConvolution_1.md @@ -67,7 +67,7 @@ * *deformable_group* - * **Description**: *deformable_group* is the number of groups which deformable values and *output* should be split into along the channel axis. Apply the deformable convolution using the i-th part of the offset part on the i-th out. + * **Description**: *deformable_group* is the number of groups in which *offsets* input and *output* should be split into along the channel axis. Apply the deformable convolution using the i-th part of the offsets part on the i-th out. * **Range of values**: integer value starting from `1` * **Type**: `int` * **Default value**: `1` @@ -77,7 +77,7 @@ * **1**: Input tensor of type *T* and rank 4. Layout is `NCYX` (number of batches, number of channels, spatial axes Y and X). Required. -* **2**: Offsets tensor of type *T* and rank 4. Layout is `NCYX` (number of batches, number of channels, spatial axes Y and X). Required. +* **2**: Offsets tensor of type *T* and rank 4. Layout is `NCYX` (number of batches, *deformable_group* \* kernel_Y \* kernel_X \* 2, spatial axes Y and X). Required. * **3**: Kernel tensor of type *T* and rank 4. Layout is `OIYX` (number of output channels, number of input channels, spatial axes Y and X). Required.