Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not able to use smart-reply feature in react-native-firebase/@react-native-firebase/ml-natural-language #2478

Closed
1 of 8 tasks
gitathrun opened this issue Aug 12, 2019 · 17 comments
Assignees
Labels
Keep Open avoids the stale bot platform: android type: enhancement Implements a new Feature
Milestone

Comments

@gitathrun
Copy link

gitathrun commented Aug 12, 2019

Issue

Describe your issue here
I am trying to use the smartreply feature according to https://twitter.com/mikediarmid/status/1128838915878805508
#2117
I try to use the code in react-native-firebase/tests/app.smartreply.js
But the error shows on emulator as:
image

smartReplyConversation = firebase.naturalLanguage().newSmartReplyConversation();

my package.json shows:

"dependencies": {
    "@react-native-firebase/app": "^0.1.5",
    "@react-native-firebase/ml-natural-language": "^0.1.5",
    "react": "16.8.6",
    "react-native": "0.60.4",
    "react-native-firebase": "^5.5.6",
    "react-native-gifted-chat": "^0.9.11"
  },

Then I tried with another approach following this instruction: https://invertase.io/oss/react-native-firebase/v6/ml-natural-language/reference/module

buttonHandleFunctionSuggestReplies = async () => {
        console.log("Button is pressed");
        // console.log("current state is:" + this.state.input_text);
        // for smart reply suggestion.
        const replies = await firebase.naturalLanguage().suggestReplies([
            // { text: "Hey, long time no speak!", },
            { text: 'I know right, it has been a while..', userId: 'xxxx', isLocalUser: false },
            // { text: 'We should catchup some time!', },
            // { text: 'Definitely, how about we go for lunch this week?', userId: 'xxxx', isLocalUser: false },
          ]);
        console.log("suggested replies: " + replies);
    }

But it still shows error:
image

I setup my firebase.json
wit following content

{
    "react-native": {
      "ml_natural_language_language_id_model": true,
      "ml_natural_language_smart_reply_model": true,
    },
}

Can anyone update a proper documentation to achieve the example on the twitter?


Project Files

iOS

Click To Expand

ios/Podfile:

  • I'm not using Pods
  • I'm using Pods and my Podfile looks like:
# N/A

AppDelegate.m:

// N/A


Android

Click To Expand

Have you converted to AndroidX?

  • [ x] my application is an AndroidX application?
  • I am using android/gradle.settings jetifier=true for Android compatibility?
  • I am using the NPM package jetifier for react-native compatibility?

android/build.gradle:

// Top-level build file where you can add configuration options common to all sub-projects/modules.

buildscript {
    ext {
        buildToolsVersion = "28.0.3"
        minSdkVersion = 22
        compileSdkVersion = 28
        targetSdkVersion = 28
        supportLibVersion = "28.0.0"
    }
    repositories {
        google()
        jcenter()
    }
    dependencies {
        classpath("com.android.tools.build:gradle:3.4.1")

        // NOTE: Do not place your application dependencies here; they belong
        // in the individual module build.gradle files
        classpath 'com.google.gms:google-services:4.2.0'
    }
}

allprojects {
    repositories {
        mavenLocal()
        maven {
            // All of React Native (JS, Obj-C sources, Android binaries) is installed from npm
            url("$rootDir/../node_modules/react-native/android")
        }
        maven {
            // Android JSC is installed from npm
            url("$rootDir/../node_modules/jsc-android/dist")
        }

        google()
        jcenter()
    }
}

android/app/build.gradle:

dependencies {
    implementation project(':@react-native-firebase_app')
    implementation project(':@react-native-firebase_ml-natural-language')
    implementation project(':react-native-firebase')
    implementation fileTree(dir: "libs", include: ["*.jar"])
    implementation "com.facebook.react:react-native:+"  // From node_modules
    implementation 'com.google.firebase:firebase-core:17.0.0'

    // implementation 'com.google.firebase:firebase-ml-natural-language-smart-reply-model:20.0.1'

    if (enableHermes) {
      def hermesPath = "../../node_modules/hermesvm/android/";
      debugImplementation files(hermesPath + "hermes-debug.aar")
      releaseImplementation files(hermesPath + "hermes-release.aar")
    } else {
      implementation jscFlavor
    }
}

android/settings.gradle:

rootProject.name = 'GiftedChatDemo'
include ':@react-native-firebase_app'
project(':@react-native-firebase_app').projectDir = new File(rootProject.projectDir, '../node_modules/@react-native-firebase/app/android')
include ':@react-native-firebase_ml-natural-language'
project(':@react-native-firebase_ml-natural-language').projectDir = new File(rootProject.projectDir, '../node_modules/@react-native-firebase/ml-natural-language/android')
include ':react-native-firebase'
project(':react-native-firebase').projectDir = new File(rootProject.projectDir, '../node_modules/react-native-firebase/android')
include ':react-native-uuid-generator'
project(':react-native-uuid-generator').projectDir = new File(rootProject.projectDir, '../node_modules/react-native-uuid-generator/android')
apply from: file("../node_modules/@react-native-community/cli-platform-android/native_modules.gradle"); applyNativeModulesSettingsGradle(settings)
include ':app'

MainApplication.java:

package com.giftedchatdemo;

import android.app.Application;
import android.util.Log;

import com.facebook.react.PackageList;
import com.facebook.hermes.reactexecutor.HermesExecutorFactory;
import com.facebook.react.bridge.JavaScriptExecutorFactory;
import com.facebook.react.ReactApplication;
import io.invertase.firebase.app.ReactNativeFirebaseAppPackage;
import io.invertase.firebase.ml.naturallanguage.ReactNativeFirebaseMLNaturalLanguagePackage;
import io.invertase.firebase.RNFirebasePackage;
import com.facebook.react.ReactNativeHost;
import com.facebook.react.ReactPackage;
import com.facebook.soloader.SoLoader;

import java.util.List;

public class MainApplication extends Application implements ReactApplication {

  private final ReactNativeHost mReactNativeHost = new ReactNativeHost(this) {
    @Override
    public boolean getUseDeveloperSupport() {
      return BuildConfig.DEBUG;
    }

    @Override
    protected List<ReactPackage> getPackages() {
      @SuppressWarnings("UnnecessaryLocalVariable")
      List<ReactPackage> packages = new PackageList(this).getPackages();
      // Packages that cannot be autolinked yet can be added manually here, for example:
      // packages.add(new MyReactNativePackage());
      return packages;
    }

    @Override
    protected String getJSMainModuleName() {
      return "index";
    }
  };

  @Override
  public ReactNativeHost getReactNativeHost() {
    return mReactNativeHost;
  }

  @Override
  public void onCreate() {
    super.onCreate();
    SoLoader.init(this, /* native exopackage */ false);
  }
}

AndroidManifest.xml:

<!-- N/A -->


Environment

Click To Expand

react-native info output:


first -----------

TypeError: _app.default.naturalLanguage().newSmartReplyConversation is not a function. (In '_app.default.naturalLanguage().newSmartReplyConversation()', '_app.default.naturalLanguage().newSmartReplyConversation' is undefined)
------------

second ------------
Possible Unhandled Promise Rejection (id: 0):
                             │ Error: [naturalLanguage/unknown] com.google.firebase.ml.common.FirebaseMLException: Error loading SmartReply model
                             │ NativeFirebaseError: [naturalLanguage/unknown] com.google.firebase.ml.common.FirebaseMLException: Error loading SmartReply model
                             │ [email protected]:99275:111
                             │ [email protected]:23447:23
                             │ [email protected]:23622:32
                             │ [email protected]:23447:23
                             │ [email protected]:23523:30
                             │ index.android.bundle:23553:19
                             │ [email protected]:24758:9
                             │ [email protected]:24922:25
                             │ [email protected]:24781:14
                             │ [email protected]:23552:29
                             └ [email protected]:23557:157
-------------
  • Platform that you're experiencing the issue on:
    • iOS
    • [x ] Android
    • iOS but have not tested behavior on Android
    • Android but have not tested behavior on iOS
    • Both
  • react-native-firebase version you're using that has this issue:
    • e.g. 5.4.3
  • Firebase module(s) you're using that has the issue:
    • e.g. Instance ID
  • Are you using TypeScript?
    • Y/N


Think react-native-firebase is great? Please consider supporting all of the project maintainers and contributors by donating via our Open Collective where all contributors can submit expenses. [Learn More]

@Ehesp
Copy link
Member

Ehesp commented Aug 12, 2019

Please check out: https://invertase.io/oss/react-native-firebase/v6/ml-natural-language/reference/module#suggestReplies

Since that tweet, we have changed the API around and removed the builder classes. It won't change again :)

@Ehesp Ehesp closed this as completed Aug 12, 2019
@gitathrun
Copy link
Author

gitathrun commented Aug 13, 2019

@Ehesp Hi, Ehesp, Thanks for the heads up, but I do mentioned in the issue that when I use the same code to invoke the smartreply feature as shown in the documentation with the link you posted,

buttonHandleFunctionSuggestReplies = async () => {
        console.log("Button is pressed");
        // console.log("current state is:" + this.state.input_text);
        // for smart reply suggestion.
        const replies = await firebase.naturalLanguage().suggestReplies([
            // { text: "Hey, long time no speak!", },
            { text: 'I know right, it has been a while..', userId: 'xxxx', isLocalUser: false },
            // { text: 'We should catchup some time!', },
            // { text: 'Definitely, how about we go for lunch this week?', userId: 'xxxx', isLocalUser: false },
          ]);
        console.log("suggested replies: " + replies);
    }

but it shows the following error, it does show as yellow box instead of red screen on device, but if you check the react-native log, it shows following error.

image

I have tried all @react-native-firebase/ml-natural-language package version above 0.1.0, shows the same result, is there any step I missed? or any configuration about dependencies is wrong?

Would you mind to reopen this issue and have a look on it?

@Ehesp
Copy link
Member

Ehesp commented Aug 13, 2019

Walk through these steps:

  • Set version to 0.1.5 - (remove node_modules & yarn install)
  • Make sure the following is in your firebase.json: "ml_natural_language_smart_reply_model": true".
  • Rebuild Android or pod install for iOS.

If it still errors let me know and I'll re-open.

@gitathrun
Copy link
Author

Hello, @Ehesp,

I have tried those solution, even with latest version 0.2.3 (0.1.3 ~ 0.1.5 as well), still not work.

  "dependencies": {
    "@react-native-firebase/app": "^0.2.3",
    "@react-native-firebase/ml-natural-language": "^0.2.3",
    "react": "16.8.6",
    "react-native": "0.60.5"
  },

I have attached the information in the following.

  1. code

MySmartReply.js

import React, { Component } from 'react';
import { View, Text, Button } from "react-native";
import { firebase } from '@react-native-firebase/ml-natural-language';
// import console = require("console");

export default class MySmartReply extends Component {
    
    constructor(props){
        super(props);
    }

    buttonHandleFunctionSuggestReplies = async () => {
        console.log("Button is pressed");
        // for smart reply suggestion.
        const replies = await firebase.naturalLanguage().suggestReplies([
            { text: "Hey, long time no speak!", },
            { text: 'I know right, it has been a while..', userId: 'xxxx', isLocalUser: false },
            { text: 'We should catchup some time!', },
            { text: 'Definitely, how about we go for lunch this week?', userId: 'xxxx', isLocalUser: false },
          ]);
        console.log("suggested replies: " + replies);
    }

    componentDidMount() {
        console.log("MySmartReply component is mounted.");
    }

    
    render () {
        return (
            <View>
                <Text>
                    This is the MySmartReply component.
                </Text>

                <Button 
                    // onPress={this.buttonHandleFunctionLanguageIdentification}
                    onPress={this.buttonHandleFunctionSuggestReplies}
                    title="Inference"
                    color="#841584"
                />
            </View>
        );
    }
}

main App.js

import React, { Component } from "react";
import MySmartReply from "./MySmartReply";

export default class App extends Component {
    render() {
        
        return <MySmartReply />;
        
    }
}
  1. firebase.json
    I used the codein GitHub repo firebase.json.
{
    "react-native": {
      "ml_natural_language_language_id_model" : true,
      "ml_natural_language_smart_reply_model" : true
    }
}
  1. after rebuild Android project, it still gave me following error in Yellow box.
    image

PS 1: I have tried auto-link and manual-link the packages, but none of them worked, shows the same error.
PS 2: I have tried with both real device and android emulator, both shows the same error.
PS 3: I have tried the same setting, but with languageIdentification model. This model works fine, and able to return correct result (predicted "Hello World" as "en").

Would you mind re-open this issue and check is there anything I did wrong?

Many thanks

@Ehesp Ehesp reopened this Aug 15, 2019
@gitathrun
Copy link
Author

Hi, @Ehesp , I think this might be the solution to this problem, but it still requires your test to verify this if you can.

My solution is:

Add following content in the app-level build.gradle file, rebuild Android project and run again.
In app/build.gradle, within defaultConfig {}, add

defaultConfig{
......
 //add following config 
// for react-native-smart-reply
    aaptOptions{ 
        noCompress "tflite","model" 
    }

.......
 }

The reason behind this is the react-native successfully build and link the native Android SmartReply library, and the native module try to load the target .tflite file in the assets directory. But this file is actually compressed in a .aar file (which is the native Java smartreply library), which causes “file not found” problem:
image

By setting the non-compression configuration, the .tflite file stores as a standalone file in the assets, no compression is applied, so the framework/library could load the file properly.
image

If you test this and consider this as a solution, it would be better to update your documentation on Android Setup page.

Many thanks.

@mikehardy
Copy link
Collaborator

Those aaptOptions are in the v6 build.gradle for the tests app, seems on target

@Ehesp
Copy link
Member

Ehesp commented Aug 16, 2019

@Salakar any idea on this?

@Salakar
Copy link
Member

Salakar commented Aug 18, 2019

@Ehesp ye, the noCompress option needs adding to the docs as we cant automate this part as far as I know.

@mikehardy
Copy link
Collaborator

The right way to do this (I think, with social proof via a bunch of modules) is to have a gradle file you ask people to include:

https://github.com/oblador/react-native-vector-icons#android

@mikehardy
Copy link
Collaborator

(then you can make changes etc but people only have to fiddle with it once)

@stale
Copy link

stale bot commented Sep 15, 2019

Hello 👋, to help manage issues we automatically close stale issues.
This issue has been automatically marked as stale because it has not had activity for quite some time. Has this issue been fixed, or does it still require the community's attention?

This issue will be closed in 15 days if no further activity occurs.
Thank you for your contributions.

@stale stale bot added the Type: Stale Issue has become stale - automatically added by Stale bot label Sep 15, 2019
@Salakar
Copy link
Member

Salakar commented Sep 16, 2019

I like that idea @mikehardy - I think we should apply this fix that way then instead of documentation, can put the gradle file in app and do conditional checks if projects exist on gradle (projects as in specific firebase modules) and apply changes based on that. Crashlytics has a similar need for properly enabling NDK reporting

@stale stale bot removed the Type: Stale Issue has become stale - automatically added by Stale bot label Sep 16, 2019
@Salakar Salakar added this to the v6.0.0 milestone Sep 16, 2019
@Salakar Salakar added Keep Open avoids the stale bot platform: android Type: Docs type: enhancement Implements a new Feature labels Sep 16, 2019
@mikehardy
Copy link
Collaborator

react-native-background-geolocation (and it's licensed/paid release version react-native-background-geolocation-android) just converted to this "import a gradle file" style as well. And now that I think about it, react-native and react-native-cli do it as well. Seems much more robust

https://github.com/transistorsoft/react-native-background-geolocation/blob/master/help/INSTALL-ANDROID-AUTO.md#open_file_folder-androidappbuildgradle

@Salakar
Copy link
Member

Salakar commented Nov 24, 2019

I've added it to the template to be there by default, could do with some documentation for non-template users also.

We can look at including an import gradle file in a later build once RN CLI auto-linking supports it (react-native-community/cli@master...bartolkaruza:feature/custom-gradle-script-option - we hacked on this together in France a couple weeks back)

@mikehardy
Copy link
Collaborator

We can look at including an import gradle file in a later build once RN CLI auto-linking supports it (react-native-community/[email protected]:feature/custom-gradle-script-option - we hacked on this together in France a couple weeks back)

Oooo! I know at least a couple modules (vector-icons and background-geolocation) that could use this feature, nice!

@Salakar
Copy link
Member

Salakar commented Nov 24, 2019

Oooo! I know at least a couple modules (vector-icons and background-geolocation) that could use this feature, nice!

Just need to find some time to get https://github.com/bartolkaruza/cli/tree/feature/custom-gradle-script-option up as a PR, unless someone beats me or @bartolkaruza to it 😅

@Salakar Salakar modified the milestones: v6.1.0, v6.2.0 Nov 25, 2019
@Salakar Salakar modified the milestones: v6.2.0, v6.3.0 Jan 20, 2020
@Salakar Salakar modified the milestones: v6.3.0, v6.4.0, v7.0.0 Feb 4, 2020
@dackers86
Copy link
Member

Closing due to inactivity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Keep Open avoids the stale bot platform: android type: enhancement Implements a new Feature
Projects
None yet
Development

No branches or pull requests

5 participants