tf.loadLayersModel is 24x slower with bundled model

[alexstanbury]

TensorFlow.js version

"@tensorflow/tfjs": "^1.7.3",
"@tensorflow/tfjs-react-native": "^0.2.3",

Describe the problem or feature request

I'm seeing extremely slow model loading times when testing model loading on an iPhone 7.
I have built a test project which I can share (but not with the model file I am using)
The model is approx 40MB in size.

In dev mode testing on the simulator (which loads from the metro server), the model load takes ~1000ms consistently.

However, when testing on the device (which loads the file from local file system), in debug mode it takes ~10,000ms consistently and in release mode ~24,000ms consistently!

From digging into the code it looks like the usage of react-native-fs may be the issue, see here for an idea of the poor performance that library gives, it appears to be specific to iOS. In the same thread someone mentions rn-fetch-blob improves read speeds massively.

I am happy to test with rn-fetch-blob and eventually create a PR when I have time (not much at all at the moment) but would like clarification that it would be welcome before doing so.

[alexstanbury]

Further to this, I am able to get the model to load in ~5000ms by avoiding calling loadLocalAsset and using loadViaHttp for all model loads instead of only those that have a uri beginning with 'http'. Is there any downside to doing it this way?

My main problem with all this is the UI becomes unresponsive during the load, leaving a bad UX for my users. At ~5000ms it's almost manageable. I may look into possibly doing the file read and conversion to UInt8Array with a native module, but presumably I'll run into problems when passing that back over the bridge. Perhaps I could batch that so it doesn't eat up all the bandwidth. I am not so fussed with the time it takes to load as I am how it affects the interface - my use case is loading the model in the background upon app start whilst my user navigates the app, then when they get to the point they want to do a prediction the model should be ready to go.

[rthadur]

@alexstanbury the first detection is slow since the backend are caching the shader programs, the prediction after the first one should much faster and stable, for example after the model is loaded, you can run a warming up step, to feed the model prediction call with all zero tensor. tf.zeros([tensorShape]).

[tafsiri]

@alexstanbury Thanks for the detailed report! A PR on this would be very welcome! If rn-fetch-blob works well cross platform this would be great.

To your second question, i'm not sure loadViaHttp will work in release mode across iOS and Android. The webserver used during dev mode is not present in release afaik (I remember testing this on android) .

@rthadur Here Alex is referring to model loading time only, not inference/detection time.

[alexstanbury]

@tafsiri I did some brief testing with rn-fetch-blob and didn't see any improvement, in fact doing a streamed and chunked load with that plugin resulted in a much longer load time and also made the ui unresponsive. As I say it was very brief though so I haven't had a chance to really drill into the problem unfortunately.

I also noticed that the bottleneck I originally experienced may not actually be to do with react-native-fs but in fact with the transformation into the array buffer, this was purely based on logging after each step though. Again I'd like to have more time to test this (if my kids will let me!)

The http fetch load definitely works on iOS in release mode, I stopped the metro server to ensure this. The examples that show loading an image for inference on the library announcement on the tensorflow blog use fetch to load an image that's bundled with the app so I presume this approach is valid.

[rthadur]

@alexstanbury do you still need help on this or will you be able to contribute a PR for the fix ?

[alexstanbury]

@rthadur I haven't had time to look into further as I cannot get this library working at all now since we have upgraded to RN 0.62.2.

Are you able to confirm that using fetch to load bundled models in release mode is a valid approach as mentioned above?

[tafsiri]

@alexstanbury I'm doubtful that loadViaHttp would work in release mode reliably (seems to work in iOS but not Android—which I believe was the motivation to add that code path).

To your issue with RN 0.62.2 it looks like expo-gl does not yet support that version of RN expo/expo#7575

[aelich]

A quick question please, @tafsiri is there any news or plan to fix, regarding this issue I have same problem as @alexstanbury

[aelich]

@tafsiri is it possible that it's slow because of Metro (bundleResourceIO)? As than Metro needs to transfer a big quantity of data for the model.

If we get it from URL it's much faster as it doesn't go though the Metro server?

[rthadur]

Closing this due to lack of activity, feel to reopen. Thank you

[Caundy]

Having experienced slow load times with bundled model files (60s when loaded using filesystem vs 17s when served from a server and loaded using automl.loadImageClassification), have there been any workarounds found or plans to fix this issue? 🙂