Add --scale, --position-x-offset, --position-y-offset and --rotation-offset

[willykc]

This addresses #4135 and #4345 by enabling additional transformations to frame the image within the window boundaries. It also applies the transformations on input control signals. Recording is not supported.

[rom1v]

Thank you for your contribution 👍

This feature is indeed often requested for the Meta Quest 3.

---

I had some compilation issues, that I fixed by replacing float32_t by float and by adding the math library as a dependency:

sed -i 's/float32_t/float/g' app/src/screen.c

diff --git a/app/meson.build b/app/meson.build
index 88e2df9aa..d6f403dbe 100644
--- a/app/meson.build
+++ b/app/meson.build
@@ -99,6 +99,7 @@ endif
 cc = meson.get_compiler('c')
 
 dependencies = [
+    cc.find_library('m'),
     dependency('libavformat', version: '>= 57.33'),
     dependency('libavcodec', version: '>= 57.37'),
     dependency('libavutil'),

---

Congrats, the input location seems to work correctly 🎉

I did not really review yet (it may take time, I have other changes to review/implement beforehand).

But I have some general remarks. A priori, I am a bit worried by the (intrinsic) complexity added for supporting any rotation/scale for the rendering. This means that every future change will have to deal with this, and may not assume a landcape/portrait view anymore.

I think that in practice, the Meta Quest 3 is the only use case. Also, this has to be used in conjonction with --crop, and since Android 14 capture cropping is broken (#4011). But anyway.

Semantically, maybe --rotation-offset could be merged with --rotation. One problem is that the window size currently does not depend on the rotation offset. (Should it expand to view the whole content? I don't know.)

[rom1v]

As an alternative, I think it should be possible to filter the video directly on the device: IIRC we should be able to capture the screen and filter it via an OpenGL ES shader, and provide the result as input for the encoder (see SurfaceTexture).

There would be several benefits:

• it would impact recording
• it would provide an alternative for cropping (broken since Android 14)
• it would be less invasive

[willykc]

I did look into a server-side implementation, but the client offered the path of least resistance. The company I work for was interested in casting the Quest 3 for public events, and this implementation got us there without too much effort. In the case of the Quest 3, It turned out to be useful to keep window size independent from rotation and scale so the image could be framed without black corners.

Regarding --crop, it can technically be used without it by scaling up and moving the image around with position offsets, but it is always a waste of bandwidth. If possible, a server-side solution would be better in the long run.

[vuisme]

I did look into a server-side implementation, but the client offered the path of least resistance. The company I work for was interested in casting the Quest 3 for public events, and this implementation got us there without too much effort. In the case of the Quest 3, It turned out to be useful to keep window size independent from rotation and scale so the image could be framed without black corners.

Regarding --crop, it can technically be used without it by scaling up and moving the image around with position offsets, but it is always a waste of bandwidth. If possible, a server-side solution would be better in the long run.

I built it successfully and it works well. But how to rotate and then crop instead of cropping and then rotate like now?

[willykc]

I built it successfully and it works well. But how to rotate and then crop instead of cropping and then rotate like now?

Unfortunately, at the moment, cropping will always happen first since it is done on the device. The only alternative is to be more generous with the initial cropping and then frame the image by playing with --scale, --position-x-offset and --position-y-offset.

[rolandwellinger]

When is this coming to the mainbranch and release?

[vuisme]

When is this coming to the mainbranch and release?
Because the applicability is low, I think it won't make it to the main branch. If scrcpy has a plugin mechanism like yt-dlp it would be better

@willykc : Do you have a solution that can control the Quest screen? I can only control via display ID (main Quest windows such as horizon feed, settings, windows store...) and cannot control via the main screen or in the app

[willykc]

Unfortunately no, I haven't looked at controlling the Quest externally.

[ZaherAlali001]

I did look into a server-side implementation, but the client offered the path of least resistance. The company I work for was interested in casting the Quest 3 for public events, and this implementation got us there without too much effort. In the case of the Quest 3, It turned out to be useful to keep window size independent from rotation and scale so the image could be framed without black corners.

Regarding --crop, it can technically be used without it by scaling up and moving the image around with position offsets, but it is always a waste of bandwidth. If possible, a server-side solution would be better in the long run.

Can you post the last scrcpy command for casting to the pc with the rotatiom and cropping...etc, that you use for the quest 3?

[willykc]

Can you post the last scrcpy command for casting to the pc with the rotatiom and cropping...etc, that you use for the quest 3?

This worked well for me on a 16:9 display in fullscreen mode:
scrcpy --crop=2744:1544:20:350 --rotation-offset=22 --scale=159 --position-x-offset=-520 --position-y-offset=-490 --video-bit-rate=16M -f
I think cropping could be more aggressive though. It can still be optimized.

[Varsett]

This worked well for me on a 16:9 display in fullscreen mode:

Hi! I'm develope special software for Meta Quest's headsets and scrcpy included to it. Where i can find this modified version scrcpy, with support rotation for Quest 3?

[rom1v]

As an alternative, I think it should be possible to filter the video directly on the device

Refs #4011 (comment) for information about implementing a server-side filter.

[vuisme]

This worked well for me on a 16:9 display in fullscreen mode:

Hi! I'm develope special software for Meta Quest's headsets and scrcpy included to it. Where i can find this modified version scrcpy, with support rotation for Quest 3?

Just build from @willykc source. This is my build if you need https://github.com/vuisme/escrcpy/tree/main/electron/resources/extra/win/scrcpy

[Varsett]

This is my build if you need

Thank you so much!

[rolandwellinger]

When is this coming to the mainbranch and release?
Because the applicability is low, I think it won't make it to the main branch. If scrcpy has a plugin mechanism like yt-dlp it would be better

What? This must come to the mainbranch. i really wonder what sidequest ist doing without that. sorry but quest 3 is sold out!

[rolandwellinger]

This worked well for me on a 16:9 display in fullscreen mode:

Hi! I'm develope special software for Meta Quest's headsets and scrcpy included to it. Where i can find this modified version scrcpy, with support rotation for Quest 3?

Just build from @willykc source. This is my build if you need https://github.com/vuisme/escrcpy/tree/main/electron/resources/extra/win/scrcpy

Absolute Hero. Thanks für the Build. Will test it right away.

[hiroyamochi]

This worked well for me on a 16:9 display in fullscreen mode:

Hi! I'm develope special software for Meta Quest's headsets and scrcpy included to it. Where i can find this modified version scrcpy, with support rotation for Quest 3?

Just build from @willykc source. This is my build if you need https://github.com/vuisme/escrcpy/tree/main/electron/resources/extra/win/scrcpy

Great work! I will use your modified scrcpy to include it in my app which specially made for Meta Quest mirroring.

[qianlishun]

This worked well for me on a 16:9 display in fullscreen mode:

Hi! I'm develope special software for Meta Quest's headsets and scrcpy included to it. Where i can find this modified version scrcpy, with support rotation for Quest 3?

Just build from @willykc source. This is my build if you need https://github.com/vuisme/escrcpy/tree/main/electron/resources/extra/win/scrcpy

Good work. However, when using -- record, there may be some issues where the recorded video is not rotated or cropped.
Is there any way?

[willykc]

Unfortunately recording is not supported by this approach. Implementing the rotation on the device as discussed previously would address that limitation though.

[ReadOnlyMAIN]

Can you post the last scrcpy command for casting to the pc with the rotatiom and cropping...etc, that you use for the quest 3?

This worked well for me on a 16:9 display in fullscreen mode: scrcpy --crop=2744:1544:20:350 --rotation-offset=22 --scale=159 --position-x-offset=-520 --position-y-offset=-490 --video-bit-rate=16M -f I think cropping could be more aggressive though. It can still be optimized.

Hello, using this command as a basis, I found out that this crop and scale were absolutly perfect for quest 3.
scrcpy --crop=2064:2208:2064:100 --window-title='Quest3Cast' --rotation-offset=-22 --scale=195 --position-x-offset=-520 --position-y-offset=-490 --video-bit-rate=16M -f

[Xericore]

Hello, using this command as a basis, I found out that this crop and scale were absolutly perfect for quest 3. scrcpy --crop=2064:2208:2064:100 --window-title='Quest3Cast' --rotation-offset=-22 --scale=195 --position-x-offset=-520 --position-y-offset=-490 --video-bit-rate=16M -f

This command works well, thank you. The rotation-offset feature is great for the Quest 3 and Quest Pro. Any reason this is not being merged into the main branch?

[rom1v]

Any reason this is not being merged into the main branch

Because I would like the transform to be performed on the server side (before encoding): #4658 (comment)

[qianlishun]

I tried to modify a window size of Quest3, it feels good for me.

--crop=1920:1440:20:350 --rotation-offset=20 --scale=159 --position-x-offset=-800 --position-y-offset=-500

[beaugunderson]

I applied the chunks individually against master and this does still work as advertised.

[Montana420]

Sorry to pester any updates on this one? Works fine via the alternative method/branch but my god would it be nice to just use SideQuest..

[demencia89]

Is there any way to run the modded build that supports rotation on linux? Thx!

[luixo]

@rom1v Can this be merged with a TODO for server-side transform?
That will improve quality of life for a lot of Quest users.

[rom1v]

I just published a prototype to apply a video filter on the device: #5455

You can edit the code to hardcode the AffineTransform you need for the Quest 3 (typically, a "reframe" followed by a rotation).

[rom1v]

Please test #5455 and the new --angle option.

[rom1v]

Superseded by #5455.