Weird dash lines on ImageProjectiveTransformV2

[Smankusors]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04 (Google Colab)
• TensorFlow installed from (source or binary): pip
• TensorFlow version (use command below): 2.3
• Python version: 3.6.9

Describe the current behavior
I used these transform values

transform = [
             [1, 0.027, -4.905, -0.025, 1.096, 4.518, 0, 0],
             [1.041, 0.01, -10.256, -0.01, 1, -0.67, 0, 0],
             [1, 0, 0, 0, 1.06, -2.536, 0, 0]
]

but the resulting images got.... weird dash lines. To view the images, you can open my notebook from link on the standalone code section.

Describe the expected behavior
It should be seamless without weird lines?

Standalone code to reproduce the issue
https://colab.research.google.com/drive/1z6zDhE6ikQr-aYHluxvlrpOhriztOmB0?usp=sharing

Other question

tensorflow/tensorflow/core/kernels/image_ops.cc

Lines 61 to 83 in 4910e8e

	OP_REQUIRES_OK(ctx, ctx->GetAttr("interpolation", &interpolation_str));
	if (interpolation_str == "NEAREST") {
	interpolation_ = Interpolation::NEAREST;
	} else if (interpolation_str == "BILINEAR") {
	interpolation_ = Interpolation::BILINEAR;
	} else {
	LOG(ERROR) << "Invalid interpolation " << interpolation_str
	<< ". Supported types: NEAREST, BILINEAR";
	}
	string mode_str;
	OP_REQUIRES_OK(ctx, ctx->GetAttr("fill_mode", &mode_str));
	if (mode_str == "REFLECT") {
	fill_mode_ = Mode::FILL_REFLECT;
	} else if (mode_str == "WRAP") {
	fill_mode_ = Mode::FILL_WRAP;
	} else if (mode_str == "CONSTANT") {
	fill_mode_ = Mode::FILL_CONSTANT;
	} else if (mode_str == "NEAREST") {
	fill_mode_ = Mode::FILL_NEAREST;
	} else {
	LOG(ERROR) << "Invalid mode " << mode_str
	<< ". Supported types: REFLECT, WRAP, CONSTANT, NEAREST";
	}

Also is this is only logging? It's not stopping me entering random string into fill_mode and interpolation parameters?

Speculation
My speculation is, it seems like the code responsible for map the coordinate miss by 1 pixel? I tried to understand image_ops code but I don't get which one it is.

[ravikyram]

I have tried in colab with TF version 2.3, nightly version(2.4.0-dev20200802) and was able to reproduce the issue.Please, find the gist here.Thanks!

[bhack]

@Smankusors Have you tried to check coordinates calcs in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/image_ops.h?

[Smankusors]

@Smankusors Have you tried to check coordinates calcs in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/image_ops.h?

yea but unfortunately I can't quite understand it. It's the MapCoordinate structs that's responsible right? Maybe I will try tinkering/fiddling it to understand it futher.

[bhack]

Ok let me know if you will find something in the meantine.
As It Is c++ code I will try to build a fresh version but It Is a very time consuming job (tensorflow/build#5)

[bhack]

@Smankusors As you have already some example code can you try to add a small PR to extend or add a test in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/image_ops_test.py to let to cover you error?
Thanks

[Smankusors]

@Smankusors As you have already some example code can you try to add a small PR to extend or add a test in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/image_ops_test.py to let to cover you error?
Thanks

huh... sure.... but currently I don't know what RGB values should be filled on that dash lines 😕

[bhack]

/cc @WindQAQ @tanzhenyu Are we using any reference impl to compare these image ops outputs (e.g. PIL etc..)?

[bhack]

@Smankusors Can I ask you how you have formed the transformation matrix in the example?

[Smankusors]

@Smankusors Can I ask you how you have formed the transformation matrix in the example?

it's random combination of

• rotation (2 degrees)
• zoom (x=(0.95, 1.05), y=(0.9, 1.1)); and
• translation (x=(0, 0), y=(-10, 10))

where each of them have 50% chance

[Smankusors]

alright I just get this compiled after encountering this issue (#42066).

it seems the fix is simple, only put -1 somewhere in the code. And the dash lines is gone. 😲

Although........ I don't fully understand the code yet...

[Smankusors]

Fixed by 86276ae

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[WindQAQ]

@Smankusors Please let me know if there is any further problem. Thanks 😃