MobileScannerState.size is not equal to BarcodeCapture.size on iOS

[jsroest]

Hi,

I would like to implement a picklist mode.

This is a scanning mode known from manufacturers like Zebra and Honeywell, where the user is able to mark and scan precisely a single barcode from a barcode sheet (a paper sheet containing a large number of barcodes). With these hardware devices, a laser crosshair is projected by the barcodescanner and only the barcode that intersects with that crosshair is scanned.

To have something similar, I project a red crosshair above the mobile_scanner live-camera feed. Touching the screen temporarily ignores any detected barcodes, to give the user the opportunity to move to the right barcode to scan. The crosshair is drawn in a green color, to give the user feedback that the scanning is temporarily disabled.

The idea is that once the user touches a single barcode with the (active) crosshair, that single barcode is scanned.

For this to work, I need the following:

1. The corners of the detected barcodes
2. The size of the image to have a reference for the corners

The corners for each barcode are passed in the BarcodeCapture.barcode.corners, which is great.

I do have issues finding the size of the image.

I preferably do not want to receive the image in the BarcodeCapture, because I expect unnecessary performance issues with passing the image to the dart side only for having the size as a reference.

Here some samples where I place a barcode as much as possible to the lower-right corner of the camera feed.

Android 9 on a Galaxy S8
MobileScannerState.size = Size(480.0, 640.0)
BarcodeCapture.size = Size(480.0, 640.0)
BarcodeCapture.barcodes[0].corners =
[Offset(272.0, 570.0), Offset(479.0, 566.0), Offset(486.0, 631.0), Offset(280.0, 638.0)]

iOS 18, on a iPhone 15
MobileScannerState.size = Size(3024.0, 4032.0)
BarcodeCapture.size = Size(1178.0, 1572.0)
BarcodeCapture.barcodes[0].corners =
[Offset(731.0, 1409.0), Offset(1110.0, 1407.0), Offset(1115.0, 1540.0), Offset(736.0, 1547.0)]

My question boils down to the following:

1. Is it a bug that MobileScannerState.size is not equal to BarcodeCapture.size on iOS?
2. If it's not a bug, how can I get the size of the image where the barcode.corners are placed on, without setting returnImage to true in the BarcodeController.

[jsroest]

I've added my implementation of the 'picklist mode' to the example app, which you can find here.

jsroest@19eedfd

[navaronbracke]

@jsroest Thanks for the picklist example! Would you mind adding this to the official samples? It would be a nice showcase of the capabilities of the plugin.

Is it a bug that MobileScannerState.size is not equal to BarcodeCapture.size on iOS?

That definitely sounds like a bug.

If it's not a bug, how can I get the size of the image where the barcode.corners are placed on, without setting returnImage to true in the BarcodeController.

We should indeed always provide the output size, but have only the image data behind the flag. Currently that is not the case. Should be easy enough to add, though.

[jsroest]

@jsroest Thanks for the picklist example! Would you mind adding this to the official samples? It would be a nice showcase of the capabilities of the plugin.

Sure. I will make a PR for that.

Is it a bug that MobileScannerState.size is not equal to BarcodeCapture.size on iOS?

That definitely sounds like a bug.

If it's not a bug, how can I get the size of the image where the barcode.corners are placed on, without setting returnImage to true in the BarcodeController.

We should indeed always provide the output size, but have only the image data behind the flag. Currently that is not the case. Should be easy enough to add, though.

The size is now calculated on the dart side. Fixing it by sending it from the native side will require a platform specific implementation on all supported platforms, and it may be a breaking change for some package users.

[navaronbracke]

The size is now calculated on the dart side. Fixing it by sending it from the native side will require a platform specific implementation on all supported platforms, and it may be a breaking change for some package users.

That isn't entirely correct? We get the output size of the MobileScannerState from the return value of the start() method implementation. Similarly, we get the BarcodeCapture.size from the barcode event.

See

mobile_scanner/lib/src/method_channel/mobile_scanner_method_channel.dart

Lines 244 to 249 in 58dfefe

	if (startResult['size']
	case {'width': final double width, 'height': final double height}) {
	size = Size(width, height);
	} else {
	size = Size.zero;
	}

https://github.com/juliansteenbakker/mobile_scanner/blob/master/ios/Classes/MobileScannerPlugin.swift#L147

and

https://github.com/juliansteenbakker/mobile_scanner/blob/master/ios/Classes/MobileScanner.swift#L281-L285

I don't think that fixing this is a breaking change, though, as the API does not change.

[jsroest]

Yes, you are right, the size is passed through the MethodChannel, I misread the code. With that being true, it is also not a breaking change.
Thanks @navaronbracke!

[Dileriuml]

The size is now calculated on the dart side. Fixing it by sending it from the native side will require a platform specific implementation on all supported platforms, and it may be a breaking change for some package users.

That isn't entirely correct? We get the output size of the MobileScannerState from the return value of the start() method implementation. Similarly, we get the BarcodeCapture.size from the barcode event.

See

mobile_scanner/lib/src/method_channel/mobile_scanner_method_channel.dart

Lines 244 to 249 in 58dfefe

	if (startResult['size']
	case {'width': final double width, 'height': final double height}) {
	size = Size(width, height);
	} else {
	size = Size.zero;
	}

https://github.com/juliansteenbakker/mobile_scanner/blob/master/ios/Classes/MobileScannerPlugin.swift#L147

and

https://github.com/juliansteenbakker/mobile_scanner/blob/master/ios/Classes/MobileScanner.swift#L281-L285

I don't think that fixing this is a breaking change, though, as the API does not change.

Hey there!

I targeted your commit specifically, and if returnImage is set to false size on the value is still 4k /3k, while to apply corners drawing (crosshair) I would need proper reference size (output size). @jsroest mentioned performance. Or is it a performance problem to get image on every reading?

[navaronbracke]

@Dileriuml The initial goal is to provide the raw output size (the actual size of the camera output, which is indeed relatively big for cameras with a high resolution) to Dart. Because of the performance implication of sending an image that is for example 4k x 3k in pixels, we want to provide only the size (which is what this bug is referring to).

However, the output size of the camera is not the same as the output size in the widget tree, as the Texture widget is resized to fit the widget constraints in the Flutter app. We will have to provide a way to deal with that as well. The same problem exists for the size of a barcode and its corner points (those are in the coordinate space of the raw camera output)

We do have something similar for the scan window here: https://github.com/juliansteenbakker/mobile_scanner/blob/master/lib/src/scan_window_calculation.dart

Maybe we can leverage that idea to fix the coordinate space.

So to sum it up, we will have to do the following:

1. provide the camera output size to Dart, regardless of returnImage
2. document that both the BarcodeCapture.size and Barcode.corners are in the coordinate space of the BarcodeCapture.size
3. Provide a way to scale BarcodeCapture.size and Barcode.corners to the coordinate space of the Texture widget
4. If we figure out 3, ideally we modify an existing example to show how to do this (or create an entirely new sample)
   -> the PR for the picklist example might be a good fit here

[navaronbracke]

@jsroest I added a fix for point 1 just now. This will be available in the next release.

I still need to look into point 3, but maybe you can work around this by taking the BarcodeCapture.size and the BoxConstraints from the overlay builder?

I plan on adding a public conversion method to the plugin that allows for scaling the Barcode.size into the coordinate space of the BoxConstraints in which the camera preview is built in the widget tree (that is, the actual layout size of the camera preview in your app, not the size of the actual camera output, as those are in different coordinate spaces)

Once that is done, I'll also fix the docs for point 2.

[jsroest]

Thanks @navaronbracke!

Fixing point 1 is all I need for my use case. With the BarcodeCapture.size and the Barcode.corners, I'm able to detect if a barcode touches the center of the image. As long as my crosshair is also at the center of the image, it will work, regardless of the coordinate space of the widgets.

I was already able to do this, but on iOS I found that the MobileScannerState.size was reported incorrectly and then tried to use BarcodeCapture.size, which I could only access after setting returnImage to true.

This bug might still be present on iOS:

Is it a bug that MobileScannerState.size is not equal to BarcodeCapture.size on iOS?

[navaronbracke]

Is it a bug that MobileScannerState.size is not equal to BarcodeCapture.size on iOS?

Yeah, that is something that I still need to investigate. That should be the same.

[jsroest]

Hi,

I did some tests with and Android and iOS device.

On iOS everything looks ok and works as expected.

iOS
BarcodeCapture.Size = (1178.0, 1572.0)

Barcode.Corners:           X/Y
TL: Offset( 856.0, 1460.0) Top Left
TR: Offset(1147.0, 1456.0) Top Right
BR: Offset(1169.0, 1573.0) Bottom Right
BL: Offset( 876.0, 1567.0) Bottom Left

But on Android there are some quirks.

Android
BarcodeCapture.Size = (640.0, 480.0)

Barcode.Corners:         X/Y
TL: Offset(368.0, 632.0) Bottom Left
TR: Offset(370.0, 577.0) Top Left
BR: Offset(470.0, 580.0) Top Right
BL: Offset(470.0, 636.0) Bottom Right

On both Android and iOS i've moved a barcode slowly from bottom-right into the view until the first barcode is recognised. On both devices the barcode was still at the far bottom-right when the first barcode was detected.

1. On Android the BarcodeCapture.Size X and Y seems swapped, as the image was shown in portrait mode on the device.
2. On Android the contents of Barcode.Corners does not seem to be in the right order (clockwise starting at top left).

I've updated the picklist sample, with a workaround for Android.

[navaronbracke]

Both of those points seem like as if MLKit is:

1. Not correcting the size for the orientation of the preview
2. Returning the corners in the wrong order (starts with bottom left, not top left)

We just pass the data from the MLKit barcode through this getter:
https://github.com/juliansteenbakker/mobile_scanner/blob/master/android/src/main/kotlin/dev/steenbakker/mobile_scanner/MobileScannerUtilities.kt#L29-L47

Since you said that everything works as expected on iOS, is the original issue, "BarcodeCapture.size != MobileScannerState.size" still a problem?

[jsroest]

Since you said that everything works as expected on iOS, is the original issue, "BarcodeCapture.size != MobileScannerState.size" still a problem?

That issue still persists.

MobileScannerState.Size = (3024.0, 4032.0).
BarcodeCapture.Size = (1178.0, 1572.0)

[jsroest]

I have trouble to reproduce the issue, but I'm on to something and it might be related to the orientation the camera thinks it is versus the orientation the device has.

Pls take a look at this piece of code:
https://github.com/juliansteenbakker/mobile_scanner/blob/master/android/src/main/kotlin/dev/steenbakker/mobile_scanner/MobileScanner.kt#L122-L150

What is suspicious here is that the width and height originate from a different source depending on wether returnImage is set to true or false. And if returnImage is true, the image may, based on a number of conditions, be rotated.

Related commit:
0dbce20

[navaronbracke]

I see.

Should we always take the width/height from the media image and not the rotated bitmap? (then we do not give guarantees on the size of the bitmap)

[jsroest]

I'm leaning to the width/height of the rotated bitmap, but that's only an educated guess.

All I know is that I want the Barcode.corners to be in the same coordinate space as the size.