Add ML Kit support

[EddyVerbruggen]

Yesterday at Google I/O a new feature was announced for Firebase: ML Kit. Let's see how neatly we can integrate its features into this plugin.

Feature checklist

If you're curious about the implementation: I'm continuously pushing my changes to this branch.

The checkboxes indicate my progress (on both platforms). Some items may move to different issues/versions.

Generic

• Bump the native Firebase SDK's to the versions that support ML Kit
• Fix the Android build to support different sub-SDK versions (as that's a new thing for Firebase)
• Fix the iOS build to correctly load the TensorFlowLite SDK
• Fix the iOS build to correctly load the required bundles for offline ML support (likely a {N} issue)
• Allow configuration of frame-processing-throttling (less strain on the device)
• Add ML Kit installation prompts to the installer (we don't want to add it if folks don't want it)
• Demo's
• Documentation
• Regression-test all non-ML features (because of the SDK bumps)

Text Recognition

• Recognize text from an image on the device
• Recognize text from a camera stream (🤙realtime OCR, anyone?)
• Expose position information for each detected text block
• Recognize text from an image in the cloud, for better accuracy (but less speed)

Barcode Scanning

• Recognize barcodes from an image on the device (there's no cloud option)
• Not only return the encoded value, but also return the type of barcode to the client
• (Continuously) recognize barcodes from a camera stream
• Limit barcode formats to certain types (speeds up recognition)
• Create an embedded barcodescanner (😱be very afraid, barcodescanner plugin!)

Face Detection 👩‍👧‍👦

• (Continuously) detect faces from a camera stream (only create a photo when everyone in the frame has their eyes open and smiles 😀?)
• Detect faces from an image on the device (there's no cloud option) - can't get this to work on iOS currently, will mark it as a known issue for now

Image Labeling 🏷 🏷

• Get labels from an image on the device
• (Continuously) label images from a camera stream
• Get labels from an image in the cloud, for better accuracy (but less speed)

Landmark Recognition 🗼🗽

• Recognize well-known landmarks in an image in the cloud (there's no device option)

Custom Models (moved to 6.1.0, see #702)

Use a custom TensorFlow Lite model, on the device
Use a custom TensorFlow Lite model, stored in Firebase (so users always have the latest model)

[jlooper]

Yasssss!!!! So excited by this!

[davidpanic]

Is there some sort of beta on npm yet? I'd really like to use this in my app, or at least try it out. If there's no beta, can you reference some build instructions please, I wasnt able to find any 😝
EDIT: nevermind I found the pack.sh script under publish.

[EddyVerbruggen]

@psrcek Thanks for asking! As you found out you can create a local package of this branch, but you're best off waiting a day or two because I'm currently hammering down on this feature like a madman. Expect it to be more stable very soon!

[davidpanic]

So what you're saying is that you will make a release after a day or two with mlkit included?

[EddyVerbruggen]

My comment was about trying this branch.

I'll make a release once I'm happy with all parts of the release and there's no regression. That's not gonna happen in two days, but I hope to find enough time to release it next week.

[ZachRoberts25]

@EddyVerbruggen Hey dude, you are the man. Thanks for the great work! Following :)

[davidpanic]

@EddyVerbruggen I've successfuly used the new features! Seems awesome, the only thing i'm bummed about is that i can't get the elements and their location in the text recognition like I can in pure java on android:

By looking at the code it doesn't seem that hard to add and it is IMO a must have feature!

[EddyVerbruggen]

Ha, @psrcek very cool that you've already tried it! Indeed, that feature is not exposed through the plugin yet but it can easily be added. I'll add it to the featurelist above.

To anyone interested, I just published a beta release: tns plugin add nativescript-plugin-firebase@next. It contains most features I planned in the original post above. Note that there's no documentation yet, but there's a very elaborate demo app that should help you get started.

@psrcek In your case you seem to know how to install the plugin from a cloned Git repo so don't bother installing the beta.

[EddyVerbruggen]

@psrcek The ML Kit branch has been updated to give you more details when recognizing text locally on a device.

If you're using a still image then it's good to know the function name has changed from recognizeText to recognizeTextOnDevice because there's now also recognizeTextCloud (and the input and output params of those functions is different). The Cloud bits of the FIrebase SDK are in 'preview' mode and will probably change. Text recognition is probably more accurate but because of the unstable API in their SDK I simply return the raw text back to the client (for now).

[EddyVerbruggen]

UPDATE:

• I'll release the plugin (version 6.0.0) tomorrow, including the ML Kit feature. The plugin is ready to go, I just need to test for regression issues.
• The docs are ready. Please check them out and let me know if anything needs to be changed: https://github.com/EddyVerbruggen/nativescript-plugin-firebase/blob/issue699-mlkit-support/docs/ML_KIT.md
• Custom models will not be part of 6.0.0. Simply because it feels like a lower priority and I don't want to block the release. Probably part of 6.1.0. Tracked by Add custom (TensorFlow Lite) models support to the ML Kit feature #702.
• Known issue: detecting faces from still images doesn't currently work on iOS. On Android it works fine, and detecting faces from a camera stream is perfectly fine as well. Tracked by Fix face detection on iOS from still images #703.

[jlooper]

Over-the-moon excited to try this, just in time for My Epic Next Demo 💃 #ThanksEddy

[davidpanic]

@EddyVerbruggen I used the new text recognition and I have a suggestion: your interfaces should look like this IMO to reduce confusion, somewhat match the java implementation and make lines accessible (they aren't right now):

export interface MLKitRecognizeTextResultBounds {
	x: number;
	y: number;
	width: number;
	height: number;
}

export interface MLKitRecognizeTextResultElement {
	text: string;
	bounds: MLKitRecognizeTextResultBounds;
}

export interface MLKitRecognizeTextResultLine {
	text: string;
	bounds: MLKitRecognizeTextResultBounds;
	elements: MLKitRecognizeTextResultElement[];
}

export interface MLKitRecognizeTextResultBlock {
	text: string;
	bounds: MLKitRecognizeTextResultBounds;
	lines: MLKitRecognizeTextResultLine[];
}

export interface MLKitRecognizeTextOnDeviceResult extends MLKitResult {
	blocks: MLKitRecognizeTextResultBlock[];
}

[davidpanic]

I made a pull request with changes to the android code, I didn't change the iOS code though as I don't have a mac so I haven't even tried making anything with iOS and I don't know how to code for iOS. 🤣

[EddyVerbruggen]

Hey @psrcek I'll check it out.. I was actually implementing your changes myself and have the iOS part ready so should be easy :) Thanks!

UPDATE: yep, we aligned quite perfectly. The only change I'm making is the type of MLKitRecognizeTextResultBounds because of alignment with iOS (and IMO it's clearer).

export interface MLKitRecognizeTextResultBounds {
    x: number;
    y: number;
    width: number;
    height: number;
}

Becomes

export interface MLKitRecognizeTextResultBounds {
  origin: {
    x: number;
    y: number;
  },
  size: {
    width: number;
    height: number;
  }
}

[EddyVerbruggen]

6.0.1 has been RELEASED!

• iOS: Please run pod repo update from the command line.
• Android: Please update you Play services (Android Studio > Tools > SDK manager).
• Both platforms: Make sure to remove your 'platforms' folder so the old native SDK's are removed.

To configure and play with ML Kit, follow the instructions in the doc.

In case of issues: please post them here, on GitHub. Not on Slack/Twitter/Email. Thanks!

[davidpanic]

@EddyVerbruggen I made my friend lend me his mac and tried to compile my project on iOS, but I keep getting this message spammed in console whenever i try to use any live preview components:

An empty result returned from from GMVDetector for VisionTextDetector.

The preview is displayed just fine, but does not detect anything.
(keep in mind that my project works fine on android)

EDIT: I found this on StackOverflow, it seems like it's a bug.

[EddyVerbruggen]

@psrcek Is that with the demo app mentioned at the top of the ML Kit readme in this repo (linked in my previous post)? If not, please try that first. The steps are in the readme. It's working fine on my iPhone X...

Otherwise please share a repo reproducing this issue. For instance, do you have all these set to true?

The reason I ask is because the message "An empty result returned from from GMVDetector for VisionTextDetector." is a symptom of the GoogleMVTextDetectorResources.bundle not being part of your project.

Btw please open a new issue as this one is closed and shipped ;)

[davidpanic]

Yeah, oops i only had "ml_kit" set to true, you must have added those at some point in between me trying this 😛
And i'm sorry about not opening another issue, i'm a derp.

[EddyVerbruggen]

That's a very good point, I did add those (I don't want to ship stuff with your app you don't actually need). 👍

[coderaven]

I just want to leave a comment here that you all sirs are heroes to us! Thanks for making this plugin happen! 😄