Add the option to Cast the Phone Camera instead of the hole phone screen

[obed-vazquez]

A lot of us are looking for apps like:

• IPWebcam
• DroidCam
• ManyCam
• Kinoni
• WO Webcam

When you have an already working app that Casts all the phone in a better resolution and with much less lag! I can just open the camera app in my phone but you can imagine the problems with it.

please add this functionality (Y)

[luigifcruz]

I'm working on a Pull Request to add this feature to the project. I'm running into a problem and I was wondering if someone could help me. When I try to open the camera inside the server I get the permission error bellow. I'm not sure how can I register permissions since this isn't a complete app. Any ideas of how can I do it?

Caller "android" (PID 2000, UID 6860) cannot open camera "0" without camera permission

[rom1v]

Any ideas of how can I do it?

You can't 😞

There are several reasons to add support for installing a real APK. The main one is forwarding audio using Android 10 API. For now, I can't make it work: https://github.com/rom1v/qaudio
But if it works, definitely scrcpy should support it.

There is also the need for an IME for entering non-ASCII chars. And now there is the camera capture…

[mjg]

Meanwhile, I have found Open Camera which supports hiding all UI. Share that with scrcpy ...

[Hocuri]

As lots people are doing videochats currently (for obvious reasons), it would be even greater to have this feature. And then we would obviously need to find a way to simulate a camera input device on the computer so that we can use the smartphone as a webcam (of course we could just do screensharing if I am asking for too much here).

This way, scrcpy would have the same functionality as DroidCam and so on but greater as you just have to enable ADB once, not start an app every time. And of course free and open source.

It is such a pity - I have this camera on my phone that is more than enough for videochats but I would need to install a closed-source driver.

scrcpy is good already but this feature would make it brilliant.

[orogenic]

I also have come here after using a couple non-free Android-as-webcam applications. Personally I've used DroidCam and Iriun Webcam. scrcpy + v4l2loopback + OpenCamera works quite well for me on Linux. I keep Iriun Webcam installed in case I am using Windows.

One interesting thing I noticed about Open Camera is the Video frame rate and Video bitrate debug options appear to affect the preview, so I was able to get quite smooth result by setting the preview framerate to 60FPS, as well I can lower the preview bitrate from its default on my Google Pixel (sailfish) of around 33Mbps to something closer to the 6Mbps I set when using scrcpy + v4l2loopback, with the hope that this would lower the power consumption of the phone.

I can just open the camera app in my phone but you can imagine the problems with it.

I have been imaging what the problems might be with it, and have grown just slightly frustrated that I don't actually understand more concretely what any of them are! I'm just going to list a couple vague impressions I've been carrying with me.

• I wish I could somehow get accurate and precise power consumption information either on the USB supply side or on the phone, under various resolutions/framerates/bitrates. It seems when using scrcpy + v4l2loopback + Open Camera (screen off), my phone's battery does drain when plugged into USB 2.0, and can charge very slowly when plugged into USB 3.0

• It seems likely sub-optimal for the camera image to be rendered within the Open Camera app, then the screen image rendered by the scrcpy server? I am curious how much more efficient can it become in terms of latency and power consumption for a scrcpy server with camera access permission to open the camera, create a stream (likely H.264?) of the camera on the device, and send this directly to scrcpy/v4l2loopback?

• I gathered an impression that webcams tend to implement some non-standard interface(s) to be discovered by common Windows applications...? I have not looked much into this. I am curious is it possible for an android app on an unrooted Android device to pose as a USB webcam for some driverless Windows webcam framework, if one exists, or for scrcpy to pose for common Windows applications? If not, how about a more privileged piece of software on a rooted Android device? (I think I saw discussion about Windows on other issues on this repo, maybe)

• I am unsure and have been curious if my particular device Google Pixel (sailfish) and more generally many Android devices have accelerated H.264 encoding APIs in the SDK and/or NDK, to create the video stream of the camera with even lower power consumption? I'd like to believe this is fairly standard... I cannot help but notice how hot my phone tends to get with the camera app left open, and I've been curious about this, but thus far found no clear, obvious information suggesting if camera apps are commonly using accelerated encoding or not...

edit: H.264 encoding API, not decoding API

[miguetronic]

Please make it real!

[rom1v]

Refs #3049 (comment)

[yume-chan]

Update to #3049 (comment), I'm trying to add camera mirror support to Scrcpy.

--video-source=camera, --camera=<id> and --list-cameras options are added.

Example output of --list-cameras:

./scrcpy --list-cameras
scrcpy 2.1.1 <https://github.com/Genymobile/scrcpy>
INFO: ADB device found:
INFO:     --> (tcpip)         emulator-5554            device  Android_SDK_built_for_x86_64
D:\dev\yume-chan\scrcpy-devcontainer\scrcpy\dist\scrcpy-wi...file pushed, 0 skipped. 106.2 MB/s (60911 bytes in 0.001s)
[server] INFO: List of cameras:
    --video-source=camera --camera=0    (Back, 1280x720)
    --video-source=camera --camera=1    (Front, 1920x1080)

Version compatibility:

• Android 10 or before: Shell doesn't have CAMERA permission.
• Android 11: Similar to audio forward, the calling process must be at foreground to call openCamera and createCaptureSession, except this time it's much harder to implement the hack.
• Android 12 and later: Just works.

Future works:

1. Workaround Android 11
2. Add --camera-facing=front/back/external to pick default camera by facing/position, if both --camera and --camera-facing are specified, it will throw an error if the two are not matching.
3. Add a keyboard shortcut to cycle through cameras (will be affected by --camera-facing)
4. Some front facing cameras are rotated by 90° (because they are physically installed in that direction), and the crop option doesn't work. Investigate if it's possible to rotate and crop the camera view without manually processing the image.

EDIT:

In regard to #994 on XiaoMi devices caused by using getSystemContext, I found a hack to disable the extra printing:

https://github.com/yume-chan/scrcpy/blob/0aa22364e401b15b2600192a68d66e8a622f4929/server/src/main/java/com/genymobile/scrcpy/Workarounds.java#L164-L170

Sadly MIUI added another one when accessing camera:

./scrcpy --video-source=camera --camera=0 --no-audio
scrcpy 2.1.1 <https://github.com/Genymobile/scrcpy>
INFO: --video-source camera disables control
INFO: ADB device found:
INFO:     -->   (usb)              73294bba            device  M2011K2C
D:\dev\yume-chan\scrcpy-devcontainer\scrcpy\dist\scrcpy-wi... file pushed, 0 skipped. 70.6 MB/s (60911 bytes in 0.001s)
[server] INFO: Device: [Xiaomi] Xiaomi M2011K2C (Android 13)
INFO: Renderer: direct3d
java.lang.ClassNotFoundException: miui.tipclose.impl.TipHelpersManager
        at java.lang.Class.classForName(Native Method)
        at java.lang.Class.forName(Class.java:454)
        at java.lang.Class.forName(Class.java:379)
        at miui.tipclose.TipHelperProxy.<init>(TipHelperProxy.java:28)
        at miui.tipclose.TipHelperProxy.<clinit>(TipHelperProxy.java:20)
        at miui.tipclose.TipHelperProxy.getInstance(TipHelperProxy.java:52)
        at android.hardware.CameraInjector.showTipViewForCamera(CameraInjector.java:806)
        at android.hardware.camera2.impl.CameraExtImplXiaoMi.cameraOpened(CameraExtImplXiaoMi.java:125)
        at android.hardware.camera2.impl.CameraExtStub.cameraOpened(CameraExtStub.java:198)
        at android.hardware.camera2.CameraManager$CameraManagerGlobal.onCameraOpenedLocked(CameraManager.java:2611)
        at android.hardware.camera2.CameraManager$CameraManagerGlobal.onCameraOpened(CameraManager.java:2580)
        at android.hardware.ICameraServiceListener$Stub.onTransact(ICameraServiceListener.java:192)
        at android.os.Binder.execTransactInternal(Binder.java:1290)
        at android.os.Binder.execTransact(Binder.java:1249)
Caused by: java.lang.ClassNotFoundException: miui.tipclose.impl.TipHelpersManager
        ... 14 more
INFO: Texture: 5920x2664
WARN: Killing the server...

(It still works)

[rom1v]

I was considering investigating camera for scrcpy in a few months, but that's great, you did all the work 😄 Thank you.

I have not tested yet (I'm on my phone, I will not have access to my computer until the end of the month).

Add a keyboard shortcut to cycle through cameras

I think we don't need this, the camera (like the display) is selected for the whole scrcpy instance. This would add complexity and cause issues (for example with V4L2).

If I remember correctly, each camera supports a set of predefined resolutions (which can be listed). Contrary to screen capture, the aspect ratio is not fixed, so --max-size (-m) is not sufficient to select a specific size. I don't know what we should do. Maybe expose an option to select the aspect ratio in addition (default to 16:9), or an option to select the exact resolution. (A first version could allow only a specific aspect ratio.)

By the way, IIRC cameras expose different resolution for portrait and landscape (1080×1920 and 1920×1080), so this can be used to solve the 90° rotation. IMO we should reset the camera capture on device rotation (so that it works the same way as screen capture), unless --lock-video-orientation is set (but maybe that there are only 2 positions, portrait and landscape, whereas there are 4 for screen capture, for example --lock-video-orientation=3).

Similar to audio forward, the calling process must be at foreground to call openCamera and createCaptureSession, except this time it's much harder to implement the hack.

Why? The same hack does not work? Or is it just more complicated due to callbacks?

both --camera and --camera-facing are specified

Related (just an idea): the camera selection could print the results the same way as the adb device selection (with a --> in front of matching cameras).

Thank you again for your work.

[yume-chan]

I think we don't need this, the camera (like the display) is selected for the whole scrcpy instance.

I listed some features camera and video chat/conference apps commonly support:

1. Preview (what I'm adding)
2. Take photos (again Added Save Screenshot #2040)
3. Record videos (Scrcpy already supports --record)
4. Switch among cameras, IMO it's different from switching among displays:
   1. Many devices only have one display, but most phones have at least two cameras.
   2. I personally use this feature pretty frequently in video chat or video conference apps to switch between showing myself and the hardware we are developing sitting in front of my phone.
   3. For native apps, the OS-provided API usually requires user consents before each screen capture session, so it's technologically impossible to achieve the one-button switching behavior like switching cameras do.
   4. That said, Scrcpy can capture displays silently, so it may also be interesting to have this shortcut switching between displays when --video-source=display, for example, professional presentations usually use OS's multi-desktop feature to quickly switching between the slide show and other applications.
   5. The shortcut can be disabled when V4L2 is enabled
5. Stylize filters (don't think it fits Scrcpy)
6. Manual camera control (focus, exposure, zoom, etc): these are valuable bonus features, but difficult to implement without a UI widget system.

Contrary to screen capture, the aspect ratio is not fixed, so --max-size (-m) is not sufficient to select a specific size.
A first version could allow only a specific aspect ratio

I can't think of any use case that requires a specific aspect ratio. I will only choose 16:9 aspect ratio in first version, and wait for feedbacks for use cases of other aspect ratios.

IMO we should reset the camera capture on device rotation (so that it works the same way as screen capture)

This should require the use of accelerometer (if our Android Shell app had this permission). Camera apps can rotate its UI when screen rotation is locked, and show the "photo direction" indicator when the device is nearly parallel to the ground, so they should be already using the accelerometer. I will check some open-sourced camera apps for inspirations. But this is definitely a hard task.

It it's not possible, the users can always manually rotate the displayed texture using keyboard shortcuts, except that it won't be recorded.

If it's possible, another implementation is sending the device's exact orientation to the client, and let the client rotate the displayed texture. I mean it can rotate freely, not in 90 degrees increment. So for camera mirroring, the displayed image always has the correct rotation. It can also be useful for mirroring racing games that use accelerometer to steer, and render the scene in reverse to the device rotation.

Like this

Similar to audio forward, the calling process must be at foreground to call openCamera and createCaptureSession, except this time it's much harder to implement the hack.

Why? The same hack does not work? Or is it just more complicated due to callbacks?

I have converted callbacks into synchronous calls using a worker thread. The difficulty is at coordinating with audio starting. My plan is starting the workaround in Server.java, adding a started notifications to each AsyncProcessor, and stopping the workaround after all AsyncProcessors are started.

Related (just an idea): the camera selection could print the results the same way as the adb device selection (with a --> in front of matching cameras).

So it would always print camera list? Or only when some option combinations?

[yume-chan]

Update:

1. Android 11 works now. The device must be unlocked before starting, same as audio capturing.
2. --camera-position added
3. --max-size chooses a resolution that has the native aspect ratio of the sensor
4. Switching cameras is extra complex. Now I disabled control when --video-source=camera, but both switching cameras and syncing orientation require the control socket.

Here is a Windows x64 build for testing:

scrcpy-win64-v2.1.1-14-g51d29fa5.zip

Usage:

1. Works on Android 11 and later
2. Use --list-cameras to see camera IDs, their facing/position, and sensor resolution
3. Use --video-source=camera to capture a camera, chooses first camera by default
4. Use --video-source=camera --camera-position=front/back/external to choose the first camera with specified facing/position
5. Use --video-source=camera --camera=<id> to choose a camera by id
6. Use --video-source=camera --audio-source=mic to capture microphone instead of internal audio along with camera

[rom1v]

@yume-chan Thank you for you work 👍

I just tested your branch, with just the following diff:

diff --git a/app/src/server.c b/app/src/server.c
index c8f7c31df..bb78850f0 100644
--- a/app/src/server.c
+++ b/app/src/server.c
@@ -247,7 +247,7 @@ execute_server(struct sc_server *server,
             sc_server_get_codec_name(params->audio_codec));
     }
     if (params->video_source != SC_VIDEO_SOURCE_DISPLAY){
-        assert(params->audio_source == SC_VIDEO_SOURCE_CAMERA);
+        assert(params->video_source == SC_VIDEO_SOURCE_CAMERA);
         ADD_PARAM("video_source=camera");
     }
     if (params->audio_source != SC_AUDIO_SOURCE_OUTPUT) {

Here are some bug reports (as a simple user, I haven't investigated at all).

I run with:

./run d --video-source=camera --no-audio -m1024

I get one frame, but it stays frozen.

Also, on my phone, scrcpy --list-cameras gives:

[server] ERROR: Exception on thread Thread[main,5,main]
java.lang.RuntimeException: Can't access camera
	at com.genymobile.scrcpy.LogUtils.buildCameraListMessage(LogUtils.java:104)
	at com.genymobile.scrcpy.Server.main(Server.java:202)
	at com.android.internal.os.RuntimeInit.nativeFinishInit(Native Method)
	at com.android.internal.os.RuntimeInit.main(RuntimeInit.java:438)
Caused by: java.lang.NullPointerException: Attempt to invoke virtual method 'boolean java.lang.String.equals(java.lang.Object)' on a null object reference
	at android.hardware.camera2.CameraManager.getCameraCharacteristics(CameraManager.java:469)
	at com.genymobile.scrcpy.LogUtils.buildCameraListMessage(LogUtils.java:79)
	... 3 more

That's probably minor things 😉

Could you please rebase your branch on current dev (and without merge commits) and open a PR please (when it's ready)? (with just the commits related to the camera, that might simplify reviewing, for example I think that Hide warnings on XiaoMi devices is unrelated, there might be others)

IMO we should reset the camera capture on device rotation (so that it works the same way as screen capture)

This should require the use of accelerometer (if our Android Shell app had this permission).

Ah yes, we could not reuse the rotation listener. Maybe the shortcut MOD+r can be re-used for camera capture rotation then :)

Why? The same hack does not work? Or is it just more complicated due to callbacks?

The difficulty is at coordinating with audio starting.

👍

So it would always print camera list?

I don't know. Why not.

[rom1v]

Implemented in scrcpy v2.2.