Addition of Reflect Padding Functionality for tf.keras Convolutional Layers

[thesahibnanda]

GitHub Issue Form

Here's why we have that policy:

Keras developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information:

• TensorFlow version (you are using): 2.16.1
• Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state:

We propose the addition of reflect padding functionality to Keras convolutional layers. Reflect padding, also known as symmetric or mirror padding, is a crucial technique in image processing and computer vision tasks. Unlike traditional padding methods, reflect padding extends image borders by mirroring existing pixels, preserving image features, and avoiding edge effects during convolutional operations.

Currently, Keras supports only 'valid' and 'same' padding modes, which may not suffice for certain use cases where maintaining spatial coherence and feature integrity at the boundaries are essential. The absence of reflect padding limits users' ability to address these critical issues effectively.

Will this change the current API? How?

Yes, this feature will introduce new padding modes ('reflect') to the existing set of padding options in Keras convolutional layers. Users will be able to specify 'reflect' as the padding mode for 1D, 2D, and 3D convolutional layers, enabling them to extend image borders using reflect padding.

Who will benefit from this feature?

This feature will benefit researchers, developers, and practitioners in the fields of computer vision, image processing, and deep learning. Specifically, individuals working on tasks such as object detection, image classification, semantic segmentation, and medical imaging will find reflect padding invaluable for preserving image features and ensuring accurate analysis, leading to improved model performance and reliability.

Contributing:

• Do you want to contribute a PR? (Yes/No): Yes
• If yes, please read this page for instructions
• Briefly describe your candidate solution (if contributing): We propose implementing reflect padding as ReflectPadding which extend the image borders using the reflect padding technique for 1D, 2D and 3D convolutional layers. These layers will be seamlessly integrated into Keras models, allowing users to specify reflect padding in convolutional layers. One layer would be applicable on all 1D, 2D and 3D

What is Reflect Padding

Reflect padding, also referred to as symmetric or mirror padding, is a vital technique in image processing and computer vision tasks. Unlike traditional padding methods, reflect padding extends image borders by mirroring existing pixels, preserving image features, and avoiding edge effects during convolutional operations.

The need for reflect padding arises from its ability to maintain image integrity, prevent information loss at boundaries, and ensure accurate analysis in tasks like object detection, image classification, and semantic segmentation.

Real Life Applications

Implementing reflect padding in Keras convolutional layers would enhance the library's functionality and enable users to address these critical issues more effectively. This feature would benefit various real-life applications such as medical imaging, video processing, and image analysis, where maintaining spatial coherence and preserving feature integrity are paramount.

By incorporating reflect padding functionality into Keras, users can expect improved performance and accuracy in their convolutional neural network (CNN) models, facilitating more robust and reliable results in diverse image processing scenarios.

How The Code After Implementation Will Look

Method 1

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense

# Define the model
model = Sequential()

# Add a 2D convolutional layer with reflect padding
model.add(Conv2D(filters=32, kernel_size=(3, 3), padding='reflect', activation='relu', input_shape=(28, 28, 1)))

# Add a max pooling layer
model.add(MaxPooling2D(pool_size=(2, 2)))

# Flatten the output before feeding into the fully connected layers
model.add(Flatten())

# Add fully connected layers
model.add(Dense(units=128, activation='relu'))
model.add(Dense(units=10, activation='softmax'))

# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

# Print model summary
model.summary()

Method 2

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, ReflectionPadding  # Import ReflectionPadding layer

# Define the model
model = Sequential()

# Add a reflection padding layer
model.add(ReflectionPadding(padding=(1, 1), input_shape=(28, 28, 1)))

# Add a convolutional layer without padding
model.add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu'))

# Add a max pooling layer
model.add(MaxPooling2D(pool_size=(2, 2)))

# Add another reflection padding layer
model.add(ReflectionPadding(padding=(1, 1)))

# Add a convolutional layer without padding
model.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))

# Add a max pooling layer
model.add(MaxPooling2D(pool_size=(2, 2)))

# Flatten the output before feeding into the fully connected layers
model.add(Flatten())

# Add fully connected layers
model.add(Dense(units=128, activation='relu'))
model.add(Dense(units=10, activation='softmax'))

# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

# Print model summary
model.summary()

How The Implementation Code Will Look (Just An Overview)

fimport tensorflow as tf

class ReflectPadding(tf.keras.layers.Layer):
    def __init__(self, padding=(1, 1), **kwargs):
        super(ReflectPadding, self).__init__(**kwargs)
        self.padding = padding

    def build(self, input_shape):
        super(ReflectPadding, self).build(input_shape)

    def compute_output_shape(self, input_shape):
        return (input_shape[0], input_shape[1] + 2*self.padding[0], input_shape[2])

    def call(self, inputs):
        pattern = [[0, 0], [self.padding[0], self.padding[0]], [0, 0]]
        padded_inputs = tf.pad(inputs, pattern, mode='REFLECT')
        return padded_inputs

    def get_config(self):
        config = {'padding': self.padding}
        base_config = super(ReflectPadding, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))

[fchollet]

Thanks for the suggestion. We should add it in Keras 3. Rather than a new standalone layer, an alternative option is to add a fill_mode argument in keras.ops.image.PadImages. We use the fill_mode argument in other places, with the following allowed argument values:

           - `"constant"`: `(k k k k | a b c d | k k k k)`
                The input is extended by filling all values beyond
                the edge with the same constant value k specified by
                `fill_value`.
            - `"nearest"`: `(a a a a | a b c d | d d d d)`
                The input is extended by the nearest pixel.
            - `"wrap"`: `(a b c d | a b c d | a b c d)`
                The input is extended by wrapping around to the opposite edge.
            - `"mirror"`: `(c d c b | a b c d | c b a b)`
                The input is extended by mirroring about the edge.
            - `"reflect"`: `(d c b a | a b c d | d c b a)`
                The input is extended by reflecting about the edge of the last
                pixel.

We might also create a corresponding Padding layer (generic) that would simply wrap PadImages.

Are you able to create a PR?

[thesahibnanda]

I'll be able to finish the pull request; your advice is appreciated.