Adding Differential Binarization model from PaddleOCR to Keras3 #1739

gowthamkpr · 2024-08-06T08:24:44Z

This adds the Differntial Binarization model for text detection.

Implemented the architecture based on ResNet50_vd from PaddleOCR and ported the weights.

mattdangerw · 2024-08-06T17:44:02Z

Let's split this up. Start with ResNetVD backbone?

Some notes...

Remove the aliases. One ResNetVDBackbone can handle all of these with different presets.
Conversion scripts as scripts not colabs.
Follow the local style for backbones as closely as possible. See some comments here Add VGG16 and VGG19 backbone #1737
Keep models a flat directory. No backbones/xx etc.
Add some tests.

divyashreepathihalli · 2024-09-25T19:42:33Z

@gowthamkpr is the PR ready for review?

divyashreepathihalli

Thanks for the PR! I have left a reorganization comment.

example for structuring the code - https://github.com/keras-team/keras-hub/tree/master/keras_hub/src/models/sam

divyashreepathihalli · 2024-09-26T19:44:02Z

keras_nlp/src/models/diffbin/diffbin.py

@@ -0,0 +1,243 @@
+# Copyright 2024 The KerasNLP Authors


rename folder to differential_binarization and file to differential_binarization.py

divyashreepathihalli · 2024-09-26T19:46:25Z

keras_nlp/src/models/diffbin/diffbin.py

+        backbone = backbone
+
+        inputs = backbone.input
+        x = backbone.pyramid_outputs


please create a file differential_binarization_backbone.py and move the diffbin_fpn_model and backbone code into that. You can rename the backbone you are using in this file to image_encoder in the differential_binarization_backbone file. The task model should contain the preprocessor, backbone and the task head.

divyashreepathihalli · 2024-09-26T19:47:04Z

keras_nlp/src/models/diffbin/losses.py

+from keras import ops
+
+
+class DiceLoss:


add test coverage for the losses here

divyashreepathihalli · 2024-10-24T19:47:39Z

Hi @gowthamkpr! can you please refactor the code to KerasHub style?

Add a preprocessor flow
subclass image segementer model for the task class
add preset class
add standard test routines

gowthamkpr · 2024-10-29T21:35:06Z

Hi @gowthamkpr! can you please refactor the code to KerasHub style?

I've refactored using SAM as example.

* [ ]  Add a preprocessor flow

I've added DifferentialBinarizationPreprocessor and DifferentialBinarizationImageConverter.

* [ ]  subclass image segementer model for the task class

I've subclassed ImageSegmenter, but I left the custom compile() method, since we need a different loss than the one used in ImageSegmenter's compile().

* [ ]  add preset class

Done. The model is not yet in Kaggle, so I've disabled the presets test for now.

* [ ]  add standard test routines

Done. Not sure if there are additional standard test routines other than the ones used in SAM that should be run.

divyashreepathihalli

Thanks Gowtham! left a few comments!

keras_hub/src/models/differential_binarization/differential_binarization_backbone.py

divyashreepathihalli · 2024-11-06T18:05:54Z

keras_hub/src/models/differential_binarization/differential_binarization_backbone_test.py

+                56,
+                256,
+            ),
+            run_mixed_precision_check=False,


does the mixed precision check pass?

No. I tried adding an explicit dtype argument, but the problem remains that the mixed precision check checks against each sublayer of the model. The ResNet backbone, which is instantiated separately, therefore has the wrong dtype.

keras_hub/src/models/differential_binarization/differential_binarization_test.py

divyashreepathihalli · 2024-11-06T18:29:25Z

keras_hub/src/models/differential_binarization/differential_binarization.py

+            instance.
+        head_kernel_list: list of ints. The number of filters for probability
+            and threshold maps. Defaults to [3, 2, 2].
+        step_function_k: float. `k` parameter used within the differential


I dont think step_function_k is a arg we want to expose.

divyashreepathihalli · 2024-11-06T18:30:49Z

keras_hub/src/models/differential_binarization/differential_binarization.py

+    Args:
+        backbone: A `keras_hub.models.DifferentialBinarizationBackbone`
+            instance.
+        head_kernel_list: list of ints. The number of filters for probability


lets move the head code to backbone.
rename this class to DifferentialBinarizationOCR and just take in preprocessor and backbone.

divyashreepathihalli

Thanks for the PR Gowtham! Left a few comments. Can you please also add a demo colab in the PR description to verify the model is working before merging?

divyashreepathihalli · 2024-11-13T22:30:44Z

keras_hub/src/models/differential_binarization/differential_binarization_backbone.py

+    pyramid network.
+
+    Args:
+        image_encoder: A `keras_hub.models.ResNetBackbone` instance.


add all args in docstring

divyashreepathihalli · 2024-11-13T22:31:33Z

keras_hub/src/models/differential_binarization/differential_binarization_backbone.py

+
+
+def diffbin_fpn_model(inputs, out_channels, dtype=None):
+    in2 = layers.Conv2D(


what is in2 - can we rename this to be more readable?

divyashreepathihalli · 2024-11-13T22:33:23Z

keras_hub/src/models/differential_binarization/differential_binarization_backbone.py

+        )
+
+        outputs = {
+            "probability_maps": probability_maps,


looks like probability_maps and threshold_maps are identical. what is the difference?

divyashreepathihalli · 2024-11-13T22:34:33Z

keras_hub/src/models/differential_binarization/differential_binarization_image_converter.py

+
+@keras_hub_export("keras_hub.layers.DifferentialBinarizationImageConverter")
+class DifferentialBinarizationImageConverter(ImageConverter):
+    backbone_cls = DifferentialBinarizationBackbone


there should be some resizing/rescaling ops here right?

divyashreepathihalli · 2024-11-13T22:35:38Z

keras_hub/src/models/differential_binarization/differential_binarization_ocr.py

+
+
+@keras_hub_export("keras_hub.models.DifferentialBinarizationOCR")
+class DifferentialBinarizationOCR(ImageSegmenter):


we need to add a new base class for ocr, I don't think ImageSegmenter is a good. one. Do you have a specific reason you chose to subclass ImageSegmenter?

mattdangerw changed the base branch from master to keras-hub August 6, 2024 17:36

mattdangerw requested a review from divyashreepathihalli August 6, 2024 20:48

divyashreepathihalli mentioned this pull request Aug 8, 2024

Add OCR model to Keras-nlp/keras hub branch #1727

Open

gowthamkpr mentioned this pull request Aug 9, 2024

Add the ResNet_vd backbone #1766

Merged

mattdangerw force-pushed the keras-hub branch 2 times, most recently from 1826dce to 753047d Compare September 11, 2024 00:01

gowthamkpr force-pushed the diffbin branch from 4dc7f78 to 3d06308 Compare September 13, 2024 13:44

mattdangerw force-pushed the keras-hub branch from 753047d to a5e5d8f Compare September 13, 2024 20:00

gowthamkpr force-pushed the diffbin branch from b9e7a3c to beaf088 Compare September 17, 2024 16:12

divyashreepathihalli requested a review from fchollet September 25, 2024 19:42

divyashreepathihalli reviewed Sep 26, 2024

View reviewed changes

gowthamkpr added 7 commits October 22, 2024 21:28

Add DifferentialBinarization model

49f6bb1

Added tests for DifferentialBinarization losses

5b4e011

Moved DifferentialBinarization to keras_hub

12ab81c

Renamed to differential_binarization.py

e68512c

Refactorings for DifferentialBinarization

0c3235c

More refactorings

6797231

Fix tests

4845b6a

gowthamkpr force-pushed the diffbin branch from beaf088 to 4845b6a Compare October 22, 2024 20:15

gowthamkpr changed the base branch from keras-hub to master October 22, 2024 20:24

gowthamkpr added 7 commits October 29, 2024 20:02

Add preprocessor and image converter

83edf9a

Add presets

f15b7b9

Run formatting script

392dbff

Impl additional tests

db70eb5

Fixed formatting

18fcbfb

Removed copyright statements

898235d

Fix tests, run api_gen.sh

eaec868

Merge branch 'master' into diffbin

21b6312

divyashreepathihalli reviewed Nov 6, 2024

View reviewed changes

gowthamkpr added 3 commits November 11, 2024 20:38

Addressed comments

9fb6e65

Merge with local branch

83b66ed

Fixed torch and jax tests

e4a334d

divyashreepathihalli requested changes Nov 13, 2024

View reviewed changes

Improved code readability

49d6f6d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding Differential Binarization model from PaddleOCR to Keras3 #1739

Adding Differential Binarization model from PaddleOCR to Keras3 #1739

gowthamkpr commented Aug 6, 2024

mattdangerw commented Aug 6, 2024

divyashreepathihalli commented Sep 25, 2024

divyashreepathihalli left a comment

divyashreepathihalli Sep 26, 2024

divyashreepathihalli Sep 26, 2024

divyashreepathihalli Sep 26, 2024

divyashreepathihalli commented Oct 24, 2024

gowthamkpr commented Oct 29, 2024

divyashreepathihalli left a comment

divyashreepathihalli Nov 6, 2024

gowthamkpr Nov 11, 2024

divyashreepathihalli Nov 6, 2024

gowthamkpr Nov 11, 2024

divyashreepathihalli Nov 6, 2024

gowthamkpr Nov 11, 2024

divyashreepathihalli left a comment

divyashreepathihalli Nov 13, 2024

divyashreepathihalli Nov 13, 2024

divyashreepathihalli Nov 13, 2024

divyashreepathihalli Nov 13, 2024

divyashreepathihalli Nov 13, 2024



		def diffbin_fpn_model(inputs, out_channels, dtype=None):
		in2 = layers.Conv2D(



		@keras_hub_export("keras_hub.models.DifferentialBinarizationOCR")
		class DifferentialBinarizationOCR(ImageSegmenter):

Adding Differential Binarization model from PaddleOCR to Keras3 #1739

Are you sure you want to change the base?

Adding Differential Binarization model from PaddleOCR to Keras3 #1739

Conversation

gowthamkpr commented Aug 6, 2024

mattdangerw commented Aug 6, 2024

divyashreepathihalli commented Sep 25, 2024

divyashreepathihalli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

divyashreepathihalli commented Oct 24, 2024

gowthamkpr commented Oct 29, 2024

divyashreepathihalli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

divyashreepathihalli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment