Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Reanimated v3 support in RNVisionCamera v2 #1759

Merged
merged 51 commits into from
Sep 18, 2023
Merged
Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
3d5dc03
Update Podfile.lock
mrousavy Jun 14, 2023
87fa369
Install REA v3
mrousavy Jun 14, 2023
61d2be6
Upgrade project to RN 0.71
mrousavy Jun 14, 2023
0f82985
Update build.gradle
mrousavy Jun 14, 2023
d791ee3
Update packages
mrousavy Jun 14, 2023
a2e0e1d
Add VisionCamera pod again
mrousavy Jun 14, 2023
efb91f8
Update dev deps
mrousavy Jun 14, 2023
a5f4711
Extend TSConfig from RN
mrousavy Jun 14, 2023
b3cfca9
Fix Linting setup
mrousavy Jun 14, 2023
1f93ccc
fix tsconfig
mrousavy Jun 14, 2023
b254017
Update project.pbxproj
mrousavy Jun 14, 2023
7e93460
Update Info.plist
mrousavy Jun 14, 2023
6681be2
Use `JSRuntimeHelper` (temporary workaround)
mrousavy Jun 14, 2023
f336643
Wrap in GestureHandlerRootView
mrousavy Jun 22, 2023
79e7692
Use new console.log
mrousavy Jun 22, 2023
294301a
fix: Update CameraRoll
mrousavy Jun 22, 2023
be7a5ef
It works!
tomekzaw Jul 6, 2023
d85e02a
Print _WORKLET in frameProcessor
tomekzaw Jul 10, 2023
df20ac2
Enforce shareable type
tomekzaw Jul 10, 2023
fc5a63d
It works again!
tomekzaw Jul 11, 2023
73055bb
console.log works
tomekzaw Jul 11, 2023
91d2aa8
Setup console in Reanimated
tomekzaw Jul 11, 2023
4b2ff25
Adapt to WorkletRuntime
tomekzaw Aug 2, 2023
6d043ef
Merge branch 'mrousavy:main' into @tomekzaw/rea-v3
tomekzaw Aug 22, 2023
673d73b
Update Podfile.lock
tomekzaw Aug 22, 2023
afdaa54
Update CameraPage.tsx
tomekzaw Aug 22, 2023
3a4140f
Android works!
tomekzaw Aug 26, 2023
56a9716
Fix iOS
tomekzaw Aug 26, 2023
e174560
Remove paths to Reanimated headers from node_modules from `target_inc…
tomekzaw Aug 29, 2023
a39dbd3
Bump `com.android.tools.build:gradle` from 7.3.1 to 7.4.2 fix issue G…
tomekzaw Aug 29, 2023
e659768
Use `@tomekzaw/prefab` branch instead of `link:`
tomekzaw Aug 30, 2023
a71ff07
Remove unnecessary header
tomekzaw Aug 30, 2023
4b2f2be
Consume Reanimated from commit
tomekzaw Aug 30, 2023
d2341ce
Remove dependency on Reanimated Scheduler on iOS
tomekzaw Aug 30, 2023
d519c3e
Go back
tomekzaw Aug 30, 2023
c48f7a7
Remove dependency on Reanimated Scheduler on Android
tomekzaw Aug 30, 2023
e569e68
Pass job as runnable
tomekzaw Aug 30, 2023
32ad37b
Remove `makeJSIRuntime`
tomekzaw Aug 30, 2023
f287052
Use nightly
tomekzaw Aug 31, 2023
8d805a0
Remove react-native-reanimated from `watchFolders` in metro.config.js
tomekzaw Aug 31, 2023
f0404e7
Fix crash on reload
tomekzaw Aug 31, 2023
bba0ee6
Fix import
tomekzaw Aug 31, 2023
f124b54
Barcode detection works
tomekzaw Sep 1, 2023
db4ff0b
Use latest commit
tomekzaw Sep 1, 2023
6b2cb36
Use `runGuarded` instead of `callWithThis`
tomekzaw Sep 1, 2023
e7b10d6
Remove private field
tomekzaw Sep 4, 2023
021417a
Remove unused type
tomekzaw Sep 4, 2023
401ea32
fix: Optionally require Reanimated
mrousavy Sep 7, 2023
366872b
Update .gitignore
mrousavy Sep 7, 2023
a7a4af1
chore: Revert example app changes
mrousavy Sep 7, 2023
e323a61
chore: Remove unused packages from example app
mrousavy Sep 7, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .eslintrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ module.exports = {
ecmaVersion: 2018,
sourceType: 'module',
},
ignorePatterns: ['scripts', 'lib', 'docs', 'example', 'app.plugin.js'],
ignorePatterns: ['scripts', 'lib', 'docs', 'example', 'app.plugin.js', '.eslintrc.js'],
plugins: ['@typescript-eslint'],
extends: ['plugin:@typescript-eslint/recommended', '@react-native-community'],
extends: ['plugin:@typescript-eslint/recommended', '@react-native'],
rules: {
// eslint
semi: 'off',
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/validate-cpp.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
with:
github_token: ${{ secrets.github_token }}
reporter: github-pr-review
flags: --linelength=230 --exclude "android/src/main/cpp/reanimated-headers"
flags: --linelength=230
targets: --recursive cpp android/src/main/cpp
filter: "-legal/copyright\
,-readability/todo\
Expand Down
10 changes: 10 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ build/
*.perspectivev3
!default.perspectivev3
xcuserdata
ios/.xcode.env.local

*.xccheckout
*.moved-aside
DerivedData
Expand All @@ -39,6 +41,10 @@ android.iml
# Cocoapods
#
example/ios/Pods
/vendor/bundle/

# Temporary files created by Metro to check the health of the file watcher
.metro-health-check*

# node.js
#
Expand Down Expand Up @@ -68,3 +74,7 @@ docs/docs/api
# External native build folder generated in Android Studio 2.2 and later
.externalNativeBuild
.cxx/

.cxx/
34 *.keystore
35 !debug.keystore
mrousavy marked this conversation as resolved.
Show resolved Hide resolved
38 changes: 2 additions & 36 deletions android/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -60,15 +60,6 @@ if(${REACT_NATIVE_VERSION} GREATER_EQUAL 71)
"${NODE_MODULES_DIR}/react-native/ReactCommon/react/renderer/graphics/platform/cxx"
"${NODE_MODULES_DIR}/react-native/ReactCommon/runtimeexecutor"
"${NODE_MODULES_DIR}/react-native/ReactCommon/yoga"
# --- Reanimated ---
# New
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/AnimatedSensor"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/Tools"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/SpecTools"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/SharedItems"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/Registries"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/LayoutAnimations"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/hidden_headers"
"src/main/cpp"
)
else()
Expand All @@ -95,22 +86,6 @@ else()
"${NODE_MODULES_DIR}/hermes-engine/android/include/"
${INCLUDE_JSI_CPP} # only on older RN versions
${INCLUDE_JSIDYNAMIC_CPP} # only on older RN versions
# --- Reanimated ---
# Old
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/headers/AnimatedSensor"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/headers/Tools"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/headers/SpecTools"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/headers/SharedItems"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/headers/Registries"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/headers/LayoutAnimations"
# New
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/AnimatedSensor"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/Tools"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/SpecTools"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/SharedItems"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/Registries"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/LayoutAnimations"
"${NODE_MODULES_DIR}/react-native-reanimated/Common/cpp/hidden_headers"
"src/main/cpp"
)
endif()
Expand Down Expand Up @@ -154,7 +129,6 @@ if(${FOR_HERMES})
"${BUILD_DIR}/third-party-ndk/hermes/jni/${ANDROID_ABI}/libhermes.so"
)
endif()
file (GLOB LIBREANIMATED_DIR "${BUILD_DIR}/react-native-reanimated-*-hermes.aar/jni/${ANDROID_ABI}")
else()
file (GLOB LIBJSC_DIR "${BUILD_DIR}/android-jsc*.aar/jni/${ANDROID_ABI}")

Expand All @@ -173,9 +147,6 @@ else()
${PACKAGE_NAME}
${JS_ENGINE_LIB}
)

# Use Reanimated JSC
file (GLOB LIBREANIMATED_DIR "${BUILD_DIR}/react-native-reanimated-*-jsc.aar/jni/${ANDROID_ABI}")
endif()

if(${REACT_NATIVE_VERSION} LESS 71)
Expand Down Expand Up @@ -234,12 +205,7 @@ else()
)
endif()

find_library(
REANIMATED_LIB
reanimated
PATHS ${LIBREANIMATED_DIR}
NO_CMAKE_FIND_ROOT_PATH
)
find_package(react-native-reanimated REQUIRED CONFIG)
mrousavy marked this conversation as resolved.
Show resolved Hide resolved

find_library(
LOG_LIB
Expand All @@ -252,7 +218,7 @@ target_link_libraries(
${PACKAGE_NAME}
${LOG_LIB}
${JSI_LIB}
${REANIMATED_LIB}
react-native-reanimated::reanimated
${REACT_NATIVE_JNI_LIB}
${FBJNI_LIB}
${FOLLY_LIB}
Expand Down
11 changes: 4 additions & 7 deletions android/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -111,9 +111,10 @@ def resolveBuildType() {
}

// plugin.js file only exists since REA v2.
def hasReanimated2 = file("${nodeModules}/react-native-reanimated/plugin.js").exists()
def disableFrameProcessors = rootProject.ext.has("disableFrameProcessors") ? rootProject.ext.get("disableFrameProcessors").asBoolean() : false
def ENABLE_FRAME_PROCESSORS = hasReanimated2 && !disableFrameProcessors
// def hasReanimated2 = file("${nodeModules}/react-native-reanimated/plugin.js").exists()
// def disableFrameProcessors = rootProject.ext.has("disableFrameProcessors") ? rootProject.ext.get("disableFrameProcessors").asBoolean() : false
// def ENABLE_FRAME_PROCESSORS = hasReanimated2 && !disableFrameProcessors
def ENABLE_FRAME_PROCESSORS = true
mrousavy marked this conversation as resolved.
Show resolved Hide resolved

if (ENABLE_FRAME_PROCESSORS) {
logger.warn("VisionCamera: Frame Processors are enabled! Building C++ part...")
Expand Down Expand Up @@ -303,10 +304,6 @@ dependencies {
def jscAAR = fileTree("${nodeModules}/jsc-android/dist/org/webkit/android-jsc").matching({ it.include "**/**/*.aar" }).singleFile
extractJNI(files(rnAAR, jscAAR))
}

def jsEngine = FOR_HERMES ? "hermes" : "jsc"
def reaAAR = "${nodeModules}/react-native-reanimated/android/react-native-reanimated-${REACT_NATIVE_VERSION}-${jsEngine}.aar"
extractJNI(files(reaAAR))
}

implementation "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
Expand Down
2 changes: 1 addition & 1 deletion android/gradle.properties
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
#Fri Feb 19 20:46:14 CET 2021
VisionCamera_buildToolsVersion=30.0.0
VisionCamera_compileSdkVersion=31
VisionCamera_kotlinVersion=1.5.30
VisionCamera_kotlinVersion=1.6.20
VisionCamera_targetSdkVersion=31
VisionCamera_ndkVersion=21.4.7075529
android.enableJetifier=true
Expand Down
2 changes: 1 addition & 1 deletion android/gradle/wrapper/gradle-wrapper.properties
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-6.9-all.zip
distributionUrl=https\://services.gradle.org/distributions/gradle-7.5.1-all.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
68 changes: 28 additions & 40 deletions android/src/main/cpp/FrameProcessorRuntimeManager.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,6 @@
#include <utility>
#include <string>

#include "RuntimeDecorator.h"
#include "RuntimeManager.h"
#include "reanimated-headers/AndroidScheduler.h"
#include "reanimated-headers/AndroidErrorHandler.h"

#include "MakeJSIRuntime.h"
#include "CameraView.h"
#include "FrameHostObject.h"
#include "JSIJNIConversion.h"
Expand Down Expand Up @@ -55,7 +49,6 @@ TSelf vision::FrameProcessorRuntimeManager::initHybrid(
auto runtime = reinterpret_cast<jsi::Runtime*>(jsRuntimePointer);
auto jsCallInvoker = jsCallInvokerHolder->cthis()->getCallInvoker();
auto scheduler = std::shared_ptr<VisionCameraScheduler>(androidScheduler->cthis());
scheduler->setJSCallInvoker(jsCallInvoker);

return makeCxxInstance(jThis, runtime, jsCallInvoker, scheduler);
}
Expand All @@ -65,16 +58,8 @@ void vision::FrameProcessorRuntimeManager::initializeRuntime() {
"Initializing Vision JS-Runtime...");

// create JSI runtime and decorate it
auto runtime = makeJSIRuntime();
reanimated::RuntimeDecorator::decorateRuntime(*runtime, "FRAME_PROCESSOR");
runtime->global().setProperty(*runtime, "_FRAME_PROCESSOR",
jsi::Value(true));

// create REA runtime manager
auto errorHandler = std::make_shared<reanimated::AndroidErrorHandler>(scheduler_);
_runtimeManager = std::make_unique<reanimated::RuntimeManager>(std::move(runtime),
errorHandler,
scheduler_);

__android_log_write(ANDROID_LOG_INFO, TAG,
"Initialized Vision JS-Runtime!");
Expand All @@ -86,6 +71,11 @@ global_ref<CameraView::javaobject> FrameProcessorRuntimeManager::findCameraViewB
return make_global(weakCameraView);
}

void FrameProcessorRuntimeManager::registerPlugins() {
static const auto registerPluginsMethod = javaPart_->getClass()->getMethod<void()>("registerPlugins");
registerPluginsMethod(javaPart_.get());
}

void FrameProcessorRuntimeManager::logErrorToJS(const std::string& message) {
if (!this->jsCallInvoker_) {
return;
Expand All @@ -105,16 +95,18 @@ void FrameProcessorRuntimeManager::logErrorToJS(const std::string& message) {
});
}

void FrameProcessorRuntimeManager::setFrameProcessor(jsi::Runtime& runtime,
void FrameProcessorRuntimeManager::setFrameProcessor(jsi::Runtime& rnRuntime,
int viewTag,
const jsi::Value& frameProcessor) {
const jsi::Value& frameProcessor,
const jsi::Value& workletRuntimeValue) {
__android_log_write(ANDROID_LOG_INFO, TAG,
"Setting new Frame Processor...");

if (!_runtimeManager || !_runtimeManager->runtime) {
throw jsi::JSError(runtime,
"setFrameProcessor(..): VisionCamera's RuntimeManager is not yet initialized!");
}
workletRuntime_ = reanimated::extractWorkletRuntime(rnRuntime, workletRuntimeValue);
mrousavy marked this conversation as resolved.
Show resolved Hide resolved
jsi::Runtime &visionRuntime = workletRuntime_->getJSIRuntime();
visionRuntime.global().setProperty(visionRuntime, "_FRAME_PROCESSOR", jsi::Value(true));

registerPlugins();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously, we would register plugins before calling setFrameProcessor since we already had _runtimeManager initialized. After the changes, when registerPlugin is called, workletRuntime_ is not set yet. We need to find a better way to synchronize this.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can use the same approach as in V3 where I have one proxy object, then make the plugins lazy? Does this ever go out of sync? I think it should be fine


// find camera view
auto cameraView = findCameraViewById(viewTag);
Expand All @@ -123,27 +115,18 @@ void FrameProcessorRuntimeManager::setFrameProcessor(jsi::Runtime& runtime,
// convert jsi::Function to a ShareableValue (can be shared across runtimes)
__android_log_write(ANDROID_LOG_INFO, TAG,
"Adapting Shareable value from function (conversion to worklet)...");
auto worklet = reanimated::ShareableValue::adapt(runtime,
frameProcessor,
_runtimeManager.get());
auto shareableWorklet = reanimated::extractShareableOrThrow<reanimated::ShareableWorklet>(rnRuntime, frameProcessor);
__android_log_write(ANDROID_LOG_INFO, TAG, "Successfully created worklet!");

scheduler_->scheduleOnUI([=]() {
// cast worklet to a jsi::Function for the new runtime
auto& rt = *_runtimeManager->runtime;
auto function = std::make_shared<jsi::Function>(worklet->getValue(rt).asObject(rt).asFunction(rt));

// assign lambda to frame processor
cameraView->cthis()->setFrameProcessor([this, &rt, function](jni::alias_ref<JImageProxy::javaobject> frame) {
try {
// create HostObject which holds the Frame (JImageProxy)
auto hostObject = std::make_shared<FrameHostObject>(frame);
function->callWithThis(rt, *function, jsi::Object::createFromHostObject(rt, hostObject));
} catch (jsi::JSError& jsError) {
auto message = "Frame Processor threw an error: " + jsError.getMessage();
__android_log_write(ANDROID_LOG_ERROR, TAG, message.c_str());
this->logErrorToJS(message);
}
cameraView->cthis()->setFrameProcessor([=](jni::alias_ref<JImageProxy::javaobject> frame) {
// create HostObject which holds the Frame (JImageProxy)
auto frameHostObject = std::make_shared<FrameHostObject>(frame);
jsi::Runtime &runtime = workletRuntime_->getJSIRuntime();
auto hostObject = jsi::Object::createFromHostObject(runtime, frameHostObject);
workletRuntime_->runGuarded(shareableWorklet, hostObject);
});

__android_log_write(ANDROID_LOG_INFO, TAG, "Frame Processor set!");
Expand Down Expand Up @@ -189,10 +172,15 @@ void FrameProcessorRuntimeManager::installJSIBindings() {
throw jsi::JSError(runtime,
"Camera::setFrameProcessor: Second argument ('frameProcessor') must be a function!");
}
if (!arguments[2].isObject()) {
throw jsi::JSError(runtime,
"Camera::setFrameProcessor: Third argument ('workletRuntime') must be an object!");
}

double viewTag = arguments[0].asNumber();
const jsi::Value& frameProcessor = arguments[1];
this->setFrameProcessor(runtime, static_cast<int>(viewTag), frameProcessor);
const jsi::Value& workletRuntimeValue = arguments[2];
this->setFrameProcessor(runtime, static_cast<int>(viewTag), frameProcessor, workletRuntimeValue);

return jsi::Value::undefined();
};
Expand Down Expand Up @@ -235,11 +223,11 @@ void FrameProcessorRuntimeManager::installJSIBindings() {

void FrameProcessorRuntimeManager::registerPlugin(alias_ref<JFrameProcessorPlugin::javaobject> plugin) {
// _runtimeManager might never be null, but we can never be too sure.
if (!_runtimeManager || !_runtimeManager->runtime) {
if (!workletRuntime_) {
throw std::runtime_error("Tried to register plugin before initializing JS runtime! Call `initializeRuntime()` first.");
}

auto& runtime = *_runtimeManager->runtime;
auto& runtime = workletRuntime_->getJSIRuntime();

// we need a strong reference on the plugin, make_global does that.
auto pluginGlobal = make_global(plugin);
Expand Down
9 changes: 5 additions & 4 deletions android/src/main/cpp/FrameProcessorRuntimeManager.h
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,7 @@
#include <memory>
#include <string>

#include "RuntimeManager.h"
#include "reanimated-headers/AndroidScheduler.h"
#include "WorkletRuntime.h"

#include "CameraView.h"
#include "VisionCameraScheduler.h"
Expand Down Expand Up @@ -46,18 +45,20 @@ class FrameProcessorRuntimeManager : public jni::HybridClass<FrameProcessorRunti
jni::global_ref<FrameProcessorRuntimeManager::javaobject> javaPart_;
jsi::Runtime* runtime_;
std::shared_ptr<facebook::react::CallInvoker> jsCallInvoker_;
std::shared_ptr<reanimated::RuntimeManager> _runtimeManager;
std::shared_ptr<reanimated::WorkletRuntime> workletRuntime_;
std::shared_ptr<vision::VisionCameraScheduler> scheduler_;

jni::global_ref<CameraView::javaobject> findCameraViewById(int viewId);
void registerPlugins();
void initializeRuntime();
void installJSIBindings();
void registerPlugin(alias_ref<JFrameProcessorPlugin::javaobject> plugin);
void logErrorToJS(const std::string& message);

void setFrameProcessor(jsi::Runtime& runtime, // NOLINT(runtime/references)
int viewTag,
const jsi::Value& frameProcessor);
const jsi::Value& frameProcessor,
const jsi::Value& workletRuntimeValue);
void unsetFrameProcessor(int viewTag);
};

Expand Down
30 changes: 0 additions & 30 deletions android/src/main/cpp/MakeJSIRuntime.h

This file was deleted.

Loading
Loading