Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update NativeCamera to support MediaStreamTrack API #1158

Merged
merged 30 commits into from
Nov 17, 2022

Conversation

CoPrez
Copy link
Contributor

@CoPrez CoPrez commented Oct 26, 2022

The main goal of this PR is to update the NativeCamera implementation to fulfill more of the MediaStreamTrack WebAPI and in particular the capabilities and constraints portion of the spec. The capabilities and constraints API allows the client (Babylon Native in this case) to declare which capabilities the camera device supports (torch, zoom, focus mode, etc.). At the same time the consumer can pass in constraints, both at instantiation and during playback, to update the configuration of the camera.

This is a snippet of how the consumer would use this code:

const mediaStream = await navigator.mediaDevices.getUserMedia({
  video:{
    torch: true // Ideally pick a device that supports torch (flashlight)
    width: { ideal: 1280, max: 1920 } // Pick a stream that is closest to 1280 pixels wide, but no more than 1920
    height: { ideal: 720, max: 1080 } // Pick a stream that is closest to 720 pixels heigh, but no more than 1080
  }
}

// Save the MediaStreamTrack for later
const mediaTrack = mediaStream.getVideoTracks()[0];

// Crate the VideoTextureto use in the scene
const videoTexture = await BABYLON.VideoTexture.CreateFromStreamAsync(scene, mediaStream, {});

...

// After some action, check if the device supports the torch capability and toggle it
if (mediaTrack.getCapabilities().torch)
{
  // Get the current set of applied constraints
  let currentConstraints = mediaTrack.getConstraints();
  // Toggle the torch constraint
  currentConstraints.torch = !mediaTrack.getSettings().torch;
  // Then apply the modified constraints
  await mediaTrack.applyConstraints(currentConstraints);  
}

The major changes I've made to achieve the above flow are:

  • Introduced a native polyfill for the MediaStream and MediaStreamTrack objects which are constructed when navigator.mediaDevices.getUserMedia() is called.
    • Calling getUserMedia() is what now opens the camera stream and prompts for camera permission (which matches the web API). Previously we were opening the camera stream when constructing the native video element which is why VideoTexture.CreateFromStreamAsync takes in a constraints object that was only used by NativeEngine.
    • The Video element native polyfill no longer declares the isNative=true property which causes Babylon.JS to assign the result of navigator.mediaDevices.getUserMedia() to the Video elements .src property.
    • The native polyfill of the MediaStream object is now the one to maintain the lifecycle of the actual camera stream (which matches the web API as well)
  • Introduced a CameraCapabilities object which fulfills the capabilities and constraints spec
    • The platform specific NativeCameraImpl classes now iterate over available camera devices on the platform and translate them into platform agnostic CameraDevice objects with registered capabilities
    • Choosing which camera device to open a stream for is now handled in the platform agnostic code as the logic to choose the device is a lot more complicated now to match the capabilities and constraints spec of the web api
  • The NativeCameraImpl classes now all share a common header file so the interface is ensured to be consistent across all platform implementations

@CoPrez CoPrez requested review from Alex-MSFT and ryantrem October 26, 2022 23:19
@CoPrez CoPrez marked this pull request as draft October 26, 2022 23:19
@CoPrez CoPrez marked this pull request as ready for review October 31, 2022 17:14
Plugins/NativeCamera/Source/NativeCameraImpl.h Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/NativeCameraImpl.h Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/Android/NativeCameraImpl.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/NativeVideo.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/NativeVideo.h Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/UWP/NativeCameraImpl.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/Unix/NativeCameraImpl.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/Win32/NativeCameraImpl.cpp Outdated Show resolved Hide resolved
Copy link
Contributor

@bghgary bghgary left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lots of little things and stuff I don't understand, but overall, it looks fine. I probably need a second pass.

Plugins/NativeCamera/Source/NativeVideo.cpp Show resolved Hide resolved
Plugins/NativeCamera/Source/NativeVideo.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/UWP/NativeCameraImpl.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/UWP/NativeCameraImpl.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/NativeCamera.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/Android/NativeCameraImpl.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/CameraCapability.h Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/CameraCapability.h Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/CameraCapability.h Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/MediaStream.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/NativeCameraImpl.h Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/NativeCameraImpl.h Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/Android/NativeCameraImpl.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/UWP/NativeCameraImpl.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/CameraCapability.h Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/MediaStream.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/Android/NativeCameraImpl.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/MediaStream.cpp Outdated Show resolved Hide resolved
Copy link
Contributor

@bghgary bghgary left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bunch of minor stuff, but overall looks good.

Plugins/NativeCamera/Source/Apple/CameraDevice.mm Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/Android/CameraDevice.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/Android/CameraDevice.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/Android/CameraDevice.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/Android/CameraDevice.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/MediaStream.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/MediaStream.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/MediaStream.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/MediaStream.cpp Outdated Show resolved Hide resolved
Plugins/NativeCamera/Source/NativeCamera.cpp Outdated Show resolved Hide resolved
@CoPrez CoPrez merged commit 44ceb84 into BabylonJS:master Nov 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants