feat: ✨ V3 ✨

[mrousavy]

See #1376

Breaking Changes

• Frame Processors are now synchronous. Previously they ran on a separate Thread. If you want to run something on a separate Thread now, use runAsync inside a Frame Processor
• Frame Processor Plugins are no longer in the global object with the __ prefix, but rather stored directly in the FrameProcessorPlugins object exported by react-native-vision-camera. (e.g. replace __scanQRCodes(frame) with FrameProcessorPlugins.scanQRCodes(frame))
• frameProcessorFps no longer exists. Use runAtTargetFps inside a Frame Processor to throttle some calls.
• onFrameProcessorPerformanceSuggestionAvailable no longer exists. Use the FPS display (enableFpsGraph={true}) to see how your Frame Processor performs over time. This is more in-line with how React Native works (Dev Tools / Perf Monitor)
• VisionCamera V3 will not work on RN 0.70 or below. You need to use RN 0.71. This is because the build script got way simpler and smaller, making it faster to build and way less error prone. Backwards compatibility is just too complex here.
• Reanimated is no longer used as a Worklet Runtime. Instead, VisionCamera now uses react-native-worklets-core.

Progress

You can test the latest V3 release by creating a new RN project with RN 0.71 and installing VisionCamera + RNWorklets:

yarn add [email protected]
yarn add react-native-worklets-core
yarn add @shopify/react-native-skia

Things to test:

• TensorFlow Lite plugin to load any .tflite model!! ✨ (see this PR for more info, will be a separate library soon)
• Drawing onto a Frame using Skia!! 🎉
• Using frame.toArrayBuffer() to get the Frame's byte content in JS
• New Android build script. This should drastically speed up the build time! 💨
• New Worklet library. This replaces Reanimated Worklets. Should be faster and more stable :)
• New synchronous Frame Processors. Should be faster :)
• runAtTargetFps and runAsync in Frame Processors
• Using HostObjects or HostFunctions (like models from PyTorch) inside a Frame Processor. This will probably require a few native bindings on PyTorch's end to make the integration work (cc @raedle)

Overall V3 is close to completion. I have a few things to do the coming days so not sure how much work I can put into this. If anyone wants to support the development of v3, I'd appreciate donations / sponsors: https://github.com/sponsors/mrousavy ❤️ :)

Related issues

features

• resolves ‼️‼️‼️‼️ ✨ VisionCamera V3 ‼️‼️‼️‼️‼️ #1376
• fixes 🐛 Android: incorrect resolution/size photo output. #281
• resolves ✨ Android Multi-Camera #211
• resolves 💨 Make image flipping faster #130
• resolves ✨ Get rid of preset #117
• fixes ✨ Android: Correctly implement Format (Resolution, FPS) #76
• resolves ✨ Android: Support takePhoto(...) options #75
• resolves ✨ Pausing and resuming record #562
• resolves ✨ Sound when taking picture #565
• fixes 🐛 App crashes on view unmount #570
• fixes 🐛 EXIF orientation #287
• resolves ✨ How do I enable/disable the shutter sound #311
• fixes 🐛 device?.supportsParallelVideoProcessing is true but library fails to init frameProcessor #315
• resolves 🐛 Preset prop has no effect on Android #323
• fixes 🐛 [Android] - Does not build within monorepo #340
• fixes 🐛 Android: Changing device front to back mid video record stops recording #354
• resolves ✨ Mention Frame Processor getting started steps more explicitly #420
• fixes 🐛 Camera preview does not appear and hangs on the capture process #434
• fixes 🐛 Recorded video is stretched #452
• fixes 🐛 iOS capture photo throws an error when enableAutoDistortionCorrection is enabled #496
• fixes ✨ Add support for generating videos in mp4? #497
• resolves ✨ Support the v8 version of react-native-reanimated #499
• fixes 🐛 Android: NFC not working after Camera is unmounted #516
• fixes ❓ Requiring module "node_modules/vision-camera-qrcode-scanner/src/index.ts", which threw an exception: TypeError: undefined is not a function, js engine: hermes  #527
• fixes Android: Screen transitions broken in Example App🐛  #542
• fixes 🐛 Changing position when record result in frames stretch in wrong orientation  #548
• fixes 🐛 iOS optimization level bug #561
• fixes 🐛 CameraProvider failed to init: No static method getInstanceAsync(...) #740
• fixes ❓ Android resolution and FPS setting #770

...and then pretty much every Android issue lol

• fixes 🐛 Recorded video appears green with only audio #1675 (maybe, please test @PrernaBudhraja)
• fixes "app:checkDebugDuplicateClasses" error for Android. #1671

.. maybe also (not tested):

• fixes 🐛rn 0.72.3 Property '_setGlobalConsole' doesn't exist #1698
• fixes [Android] Flash light turn off when its used with frameprocessor🐛  #1687
• fixes [Android] camera session is bound and idle on unmount 🐛  #1685
• fixes V3.0.0-rc.4 RN 0.72.3 android Execution failed for task ':react-native-vision-camera:buildCMakeDebug[arm64-v8a]'.🐛🐛  #1681
• fixes 🐛 Trying to get face detector to work but frame processing is unavailable and are not enabled #1650
• fixes 🐛 On android device unable to get reference of camera, on iOS it is working fine. #1646
• fixes Crash with SIGSEGV on Android #1635
• fixes 🐛Vision Camera Android example App keeps crashing after installation #1631
• fixes 🐛 node_modules/react-native-vision-camera/android/CMakeLists.txt debug|x86 : No compatible library found [//shopify_react-native-skia/rnskia] Affected Modules: react-native-vision-camera  #1621
• fixes 🐛 Error: EPERM: operation not permitted, lstat '...\node_modules\react-native-vision-camera\android\build\intermediates\cxx\Debug\3j3y4s34\obj\arm64-v8a\libVisionCamera.so.tmpfc89ebc'  #1615
• fixes 🐛 scanBarcodes returns undefined upon scanning a QR code #1612
• fixes when I press startRecording button an error occurs after adding 'frameProcessor' props.🐛  #1605
• fixes Camera section goes black on some android devices #1599
• fixes 🐛 takePhoto() not always resolving Android #1585
• fixes 🐛 library "libjscexecutor.so" not found AND frame-processor/unavailable #1581
• fixes In Android not able to get all devices, whereas in iOS its working perfectly fine.🐛  #1569
• fixes Not building for RN 0.64.2🐛  #1568
• fixes Runtime JAR files in the classpath should have the same version. 🐛  #1565
• fixes 🐛 [Android] Prop Camera orientation='landscapeLeft' results in picture oriented landscape right #1561
• fixes 🐛 StopRecording doesn't emit any event (finish or error), on specific android devices #1558
• fixes 🐛 "Camera View" is not Fabric compatible yet #1554
• fixes 🐛 Third-party Android NDK suffering from vulnerabilities to multiple CVE exploits #1551
• fixes 🐛 RN 0.71.4 run-android ==> Task :react-native-vision-camera:compileDebugKotlin FAILED #1547
• fixes Property 🐛 __scanQRCodes doesn't exist! #1543
• fixes 🐛 cannot find symbol new CameraPackage() in PackageList.java [com.mrousavy.camera.example.CameraPackage] #1538
• fixes Invalid argument type ImageProxy when trying to use native module and passing the frame as argument🐛  #1536
• fixes Task :app:installDebug FAILED Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. 🐛  #1534
• fixes Worklet Frame Processor Crash Android 🐛  #1528
• fixes Example app fails to start in Android Emulator🐛  #1520
• fixes Failed to retrieve default CameraXConfig.Provider from meta-data🐛  #1498
• fixes 🐛 No able to build Android - react-native-vision-camera:downloadBoost FAILED #1489
• fixes The Android Gradle plugin supports only kotlin-android-extensions Gradle plugin version 1.6.20 and higher. #1477
• fixes EPERM: operation not permitted, lstat, errno: -4048  #1474
• fixes 🐛 No implementation found for void CameraView.frameProcessorCallback #1463
• fixes Camera.getAvailableCameraDevices() returns null on Android, physical device🐛  #1462
• fixes 🐛 The Android Gradle plugin supports only kotlin-android-extensions Gradle plugin version 1.6.20 and higher. #1449
• fixes 🐛 Execution failed for task ':react-native-vision-camera:extractAARHeaders'. #1443
• fixes 🐛 An unknown camera error occured. #1437
• fixes 🐛 The Android Gradle plugin supports only kotlin-android-extensions Gradle plugin version 1.6.20 and higher #1431
• fixes 🐛 Android EXIF data returning with "FocalLenIn35mmFilm" as 0  #1429
• fixes "Expected directory 'node_modules/react-native/android' to contain exactly one file, however, it contains no files"🐛  #1427
• fixes 🐛[android] Camera.requestCameraPermission() does not resolve after giving camera permissions. #1423
• fixes Camera is not loading on certain devices. #1416
• fixes 🐛 Unable to use external USB camera #1407
• fixes [android] White screen before camera on 🐛  #1403
• fixes 🐛 Android app crashes with java.util.ConcurrentModificationException  #1402
• fixes 🐛 Android build error in monorepo #1398
• fixes 🐛 camera crashes when implementing tap to focus #1396
• fixes [Crash] Android - AssertionException: Expected to run on UI thread!🐛  #1395
• fixes 🐛 Error in build type resolution #1379
• fixes Execution failed for task ':react-native-vision-camera:compileDebugKotlin'.🐛  #1377
• fixes Execution failed for task ':react-native-vision-camera:extractJNIFiles'. #1374
• fixes 🐛 package FrameProcessorRuntimeManager does not exist #1373
• fixes 🐛 Can not save video > 1minute on android #1365
• fixes 🐛 When I turn on frameProcessor prop on, app crashes with SIGABRT #1356
• fixes 🐛 Android build error when using frame processor: 'folly/dynamic.h' file not found #1353
• fixes 🐛 Support for kotlin-android-extensions Gradle plugin version 1.6.20 and higher. #1352
• fixes java.util.concurrent.RejectedExecutionException: Task androidx.camera.core.impl.utils.executor.SequentialExecutor  #1351
• fixes 🐛 React Native: Recording crash on few android mobiles due to react-native-video used  #1343
• fixes 🐛 Android phones videoWidth/videoHeight are null #1340
• fixes 🐛 build fail #1334
• fixes 🐛 Mute phone call sound on Android  #1330
• fixes 🐛 AndroidRuntime FATAL EXCEPTION when trying to run Android app - No virtual method runOnJSQueueThread #1322
• fixes 🐛 Cannot swap between cameras while recording on Android #1296
• fixes 🐛 Execution failed for task ':react-native-vision-camera:buildCMakeDebug[arm64-v8a]' #1283
• fixes Execution failed for task ':react-native-vision-camera:extractJNIFiles'. > Cannot expand ZIP 🐛  #1260
• fixes react-native-vision-camera:downloadDoubleConversion FAILED🐛  #1253
• fixes 🐛 Android Black Screen #1251
• fixes 🐛 Autofocus returning errors on newest Android devices #1245
• fixes onHandlerStateChange is not triggered at all🐛  #1238
• fixes 🐛 Camera scale in samsung j7 #1227
• fixes 🐛 In react-native-vision-camera the contours values are always showing as [Array] #1226
• fixes Android: Torch stops working after going back from another screen 🐛  #1225
• fixes 🐛 Can't build assembleRelease - Hermes #1222
• fixes 🐛 Crash on Android using FrameProsessor from vision-camera-code-scanner #1211
• fixes 🐛 Android 7.1.2 Call Camera and show the white screen #1208
• fixes 🐛 Bulid failed: Configure project :react-native-gradle-plugin #1193
• fixes Black screen in the cameraView🐛  #1191
• fixes 🐛 Expected directory '.../node_modules/react-native/android' to contain exactly one file, however, it contains more than one file. #1184
• fixes 🐛 react-native 0.69.3 android build crash #1164
• fixes 🐛 The App close itself without any error when camera isActive = true #1143
• fixes 🐛 Android frame processor not works  #1128
• fixes 🐛 Example app don't working on real device #1122
• fixes 🐛 react-native-reanimated build issue #1120
• fixes 🐛 Execution failed for task ':react-native-vision-camera:compileDebugKotlin'. #1110
• fixes 🐛 App crashes on Android: UnsatisfiedLinkError (vision-camera-face-detector) #1097
• fixes The front camera returns null metadata🐛  #1081
• fixes When the camera starts, the screen stays black ِAnd Flash not work #1080
• fixes 🐛 Camera formats are not working consistently on Android #1064
• fixes 🐛 onInitialized seems to be called too early on Android #1053
• fixes 🐛 Metro crashing on every build with EPERM error #1047
• fixes ❓ shutter sound not playing on Android #1044
• fixes 🐛 Unmounting the Camera component does not turn off the torch #1032
• fixes 🐛 Application crashes on start recording #1026
• fixes 🐛 Video output #1023
• fixes 🐛 Caused by: groovy.lang.MissingPropertyException: Could not get unknown property 'android' for task ':react-native-vision-camera:jitsiAndroidSourcesJar' of type org.gradle.api.tasks.bundling.Jar. #1015
• fixes 🐛 Task :react-native-vision-camera:compileDebugKotlin FAILED #1012
• fixes 🐛 With Hermes on, If the component is updated too many times using the frame processor, the application will crash. #997
• fixes 🐛 Frame is sideways on android portrait orientation #960
• fixes 🐛 [capture/inactive-source] The recording failed because the source becomes inactive and stops sending frames. One case is that if camera is closed due to lifecycle stopped, the active recording will be finalized with this error, and the output will be generated, containing the frames produced before camera closing. Attempting to start a new recording will be finalized immediately if the source remains inactive and no output will be generated. #959
• fixes App crashes in release mode🐛  #954
• fixes 🐛 java.lang.NullPointerException in FrameProcessorRuntimeManager.<init> #946
• fixes 🐛 google pixel 4a, has night sight, though lib returns supportsLowLightBoost: false #945
• fixes 🐛 VisionCamera component displayed previous screen's view #922
• fixes Error when building for Android 🐛  #908
• fixes 🐛 App crash for LEGACY camera device with video set to true #907
• fixes 🐛 Wrong format on ANDROID. #868
• fixes Camera is showing white screen instead of showing camera in android🐛  #855
• fixes 🐛 Example app broken when switching devices #834
• fixes 🐛 Orientation is always 0 #793
• fixes 🐛 Camera: Native Module 'CameraView' was null! Did you run pod install? #779
• fixes 🐛 Android- frame processor alway return wrong width/height of frame #746
• fixes 🐛 CameraProvider failed to init: No static method getInstanceAsync(...) #740
• fixes 🐛 The given Camera device could not be found for use-case binding! #727
• fixes 🐛 Release build System err on Samsung device when initialising the camera #671
• fixes Constantly getting random crashes every now and then🐛  #613
• fixes 🐛 Android phones not reporting all existing cameras #595
• fixes CMake Error: The following variables are used in this project, but they are set to NOTFOUND.🐛  #588
• fixes 🐛 App crashes on view unmount #570
• fixes 🐛 Android - Taking picture with automatic flashlight #569
• fixes Android: Screen transitions broken in Example App🐛  #542
• fixes 🐛 Android: NFC not working after Camera is unmounted #516
• fixes 🐛 Android build failure: "Multiple task action failures occurred"  #515
• fixes 🐛 Camera preview does not appear and hangs on the capture process #434
• fixes 🐛 Android: Changing device front to back mid video record stops recording #354
• fixes 🐛 Preset prop has no effect on Android #323
• fixes 🐛 device?.supportsParallelVideoProcessing is true but library fails to init frameProcessor #315
• fixes 🐛 Android: incorrect resolution/size photo output. #281
• fixes ✨ Android Multi-Camera #211
• fixes ✨ Android: Correctly implement Format (Resolution, FPS) #76

[github-actions]

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

[mrousavy]

Notice you spelt it workets in the package name, not worklets.

Ahh, thank you!

I think I just assumed you weren't supporting Paper due to this #1376 (comment)

Edited the comment :)

I am getting build errors on Android.

Yep I mainly focused on iOS for now, am currently looking more into Android. thanks for the logs, will investigate!

[mrousavy]

Update 21.2.2023:

I just released VisionCamera v3.0.0-rc.2, which includes the following features:

• Drawing onto the Frame using Skia!! 🎉 (iOS only for now, I will probably need to rewrite the entire Android part from CameraX to Camera2 (one of Android's worst APIs) for this)
• Frame now has .toByteArray() to read the bytes straight from JS!
• Frame now also has .orientation, .isMirrored and .timestamp
• Add fpsGraph to show the Camera's preview FPS. Good for debugging performance
• Fixed the regex thing build error
• Makes runAsync truly async.
• Uses latest Skia and RN Worklets for more compatibility

Try it:

yarn add react-native-vision-camera@rc
yarn add react-native-worklets@https://github.com/chrfalch/react-native-worklets#d62d76c
yarn add @shopify/[email protected]

Things to test: all of the features above

[mrousavy]

Okay I had a mistake in android/src/main/AndroidManifest.xml, test on iOS for now (or hotpatch this commit if you want to test Android) :)