Screencast support

[emersion]

KDE code: https://github.com/KDE/xdg-desktop-portal-kde/blob/master/src/screencast.cpp

DMA-BUF in PipeWire: https://github.com/PipeWire/pipewire/issues/173

[Algram]

@emersion Just to get a grasp of the problem: Am I correct that once this issue is resolved, gnome-network-displays (https://github.com/benzea/gnome-network-displays) would just work? Are there any more steps after that to get support for this into wlroots and then sway?

Thanks!

[emersion]

Yes, it should work, they use org.freedesktop.portal.ScreenCast.

[danshick]

I've been working on this recently, but ran into a snag. The example provided in PipeWire/pipewire#173 which uses PW_STREAM_FLAG_ALLOC_BUFFERS to allow zero-copy DMA-BUF fds from wlroots to pipewire is only available in the work branch of pipewire, which currently builds libpipewire-0.3. It doesn't look like xdg-desktop-portal will build against anything newer than libpipewire-0.2.6 yet, so until those are released, this is stuck.

I'll create a PR once pipewire ships 0.3.

[Algram]

Thanks for working on this. Fyi there is already a PR open for updating to 0.3: flatpak/xdg-desktop-portal#253

[emersion]

Thanks. Feel free to create a PR even if PipeWire hasn't released the changes yet.

[danshick]

I might, but I can no longer test with D-Bus because invoking 'xdg-desktop-portal -r' fails with 0.3 manually compiled due to some breaking change (I symlinked the shared library, no luck). I guess I could directly call the screenshare function from main.c and still see if pipewire streams everything correctly. I still haven't married the exported dmabuf with pipewire, so there's value in trying that.

edit: I didn't read your comment right @Algram, I thought you said issue, not PR. Looks like a minor change too. I'll build this and see if I can proceed. Thanks!

[Algram]

No Problem. If you need any help testing feel free to let me know!

[danshick]

Not sure what commit https://github.com/PipeWire/pipewire/issues/158#issuecomment-535855558 is in reference to, but I think this broke flatpak/xdg-desktop-portal#253 again. I'm not sure it's worth it to chase this down just yet, or just go back to ignoring xdg-desktop-portal for now.

[danshick]

I've gone ahead and pulled out the dbus activation and the dependency on xdg-desktop-portal. Just wanted to see if I could get a stream of my desktop to play through gstreamer.

At this point, the master and work branches of pipewire that support zero copy don't seem to be able to run the video-src-alloc example. In fact, I can't get the gstreamer plugin to play anything from pipewire. I've tried raising some issues and reaching out on IRC, but no luck.

I think we'll have to wait until pipewire 0.3 reaches a more stable state to proceed, unless we want to lean on the CPU. Looked into the KDE repo more, and that's exactly what they're doing:

https://github.com/KDE/xdg-desktop-portal-kde/blob/master/src/screencaststream.cpp#L413

[danshick]

Oh wow, pipewire disabled github issues and are only leaving PRs open. I don't blame them. Tons of volume.

[mvdan]

It looks like they simply want people to use the gitlab issue tracker?

Thanks for looking into this, by the way. I'm one of many who depends on screen casting for the day job.

[danshick]

Good catch. I'll open an issue over there. If it is legit broken, I'll be happy to see it fixed, but if my issue turns out to be PEBKAC, I'm PR'ing some good getting started documentation for that project.

[emersion]

They also have an IRC channel (#pipewire on Freenode) if you want to ask questions.

[danshick]

I tried them a few times on IRC. Got the impression that "I can't get your GStreamer plugin to work" was noise in an otherwise productive channel. I was kind of just hoping for a "yeah, we don't really expect that to work at the moment", or a "you're doing something wrong" but I just got dead air. On the issue I created, @wtay fixed a portion where pipewire was hanging when killed with an interrupt, but the core part of the issue was untouched.

Last time I tried building master, I couldn't even get something like the following to work.

gst-launch-1.0 videotestsrc ! pipewiresink

gst-launch-1.0 pipewiresrc ! videoconvert ! xvimagesink

I couldn't get the pipewire cli to show any nodes being created. Enabling debugging output would show new clients connecting on both sides, but no state changes.

My issue is duplicated here if you're interested:

https://gitlab.freedesktop.org/pipewire/pipewire/issues/193

[danshick]

My plan for now is to keep toying with master branch builds to see if I can get gst-launch to pass a test video through pipewire. If that starts working (or if I figure out what I'm doing wrong), I'll actually start fiddling with my fork again. Seems like I can trigger events in pipewire to queue and dequeue buffers when I get frame ready from wlr. I don't even think I'll need any timers. Overall, the only thing that looks confusing are the video format details, but I can't start to make sense of that until I can get something working.

[danshick]

Okay, I was able to create a new issue that was recently fixed. Big thanks to @wtay. I'm sure he had more important issues than a broken gst plugin. The zero-copy example, video-src-alloc.c can now be played back with gstreamer, so I should be able to start testing this again.

https://gitlab.freedesktop.org/pipewire/pipewire/issues/198

[danshick]

Okay, I have some video output coming through to gstreamer, but it appears smeared. I'm kinda stuck at this point.

For anyone willing to help me debug this a little, I've created a separate repo for testing just the pipewire stuff (without worrying about xdg-desktop-portal/dbus/etc.) over here:

https://github.com/danshick/wlr_zerocopy_pipewire

You'll need a build of pipewire. At the time of writing, the master branch works fine. Make sure you stop any installed instance of pipewire and start the one you built. Also, use the included pw_uninstalled.sh script in the pipewire repo for each shell you're working from so that you're using the right libs and binaries when building and running.

You'll also need gstreamer and gstreamer-plugins-good installed if you want to test with the little test_me script included. Once again, I'm using the pw_uninstalled.sh in any shells where I'm running gstreamer.

For the moment, there are some defines for the hardware device, width, height, framerate (25 seems to work fine), and alignment towards the top of the screencast.c file. Apologies, this is messy at the moment. I'd be happy to try to answer any questions that people have when experimenting.

EDIT:
I should add, I am only passing one frame object (plane?) at the moment for testing purposes. I'm not exactly clear what each plane represents.

[danshick]

Okay, so I think my issue has to do with DRM format modifiers. If I log the mod_high and mod_low params, I get a modifier that corresponds to I915_FORMAT_MOD_Y_TILED_CCS and from reading the description, it looks like the "smeared" corruption I'm seeing has to do with this.

Pipewire has a video format option (SPA_FORMAT_VIDEO_MODIFIER), but setting it seems to do nothing. I'm going to see if I can compensate in my gstreamer pipeline for testing purposes, but I think this might be another pipewire dealbreaker for downstream consumers.

If anyone can confirm my suspicions, I'd love to know.

EDIT:
I couldn't see a way to create a gstreamer pipeline (with gst-launch at least) that could handle tiled ccs format. I found this issue which had an interesting workaround (kernel option i915.enable_fbc=0). Sadly, that didn't seem to work either.

[fooishbar]

There's no support in GStreamer pipewiresrc for handling modifiers, so it just drops them on the floor. GStreamer then later tries to work with no modifier, but it can't.

I guess you sort of discovered this, because you only ever pass the first dmabuf. CCS is a two-plane modifier, so requires two dmabufs. If CCS was being used properly, the import would fail because you only had one dmabuf.

[danshick]

Yeah, after I wrote this up, I went digging for this function...https://gitlab.freedesktop.org/pipewire/pipewire/blob/master/src/gst/gstpipewireformat.c#L820. No mention of modifiers. That's interesting about CCS, that wasn't clear from what little documentation I found. I almost thought from its small size that it might hold a cursor image or something. Pipewire offers the option to create multiple buffers, multiple data elements per buffer, and multiple chunks per data element, so not sure where additional planes are best put. Also curious if SPA_FORMAT_VIDEO_MODIFIER is meant to hold DRM fourcc modifiers or if it needs conversion. It doesn't seem documented.

Gstreamer documents this format flag:
GST_VIDEO_FORMAT_FLAG_TILED (256) – The format is tiled, there is tiling information in the last plane.
I wonder if getting that flag into a caps string and passing both planes would change anything. If that's all there was to it, it would be easy enough to create a PR for the gstreamer plugin.

More importantly though, I'm only using gstreamer for testing. If every consumer needs to handle these modifiers independently, and pipewire, currently, is ignoring them for its own gst plugin, what is the likelihood this will work correctly in most consuming applications?

[danshick]

Oh, @fooishbar, bumped into you over here while trying to answer my own gstreamer question.

Did these patches ever get included? I don't see any format types related to DRM in the docs.

[fooishbar]

Veering off-topic, but CCS is canonically described in drm_fourcc.h. The first buffer is a 'normal' colour buffer with Y-tiling (rather than row-linear) layout; the second buffer is kind of a lookaside buffer for compression. Crudely, for every 16x8 (?) tile in the primary surface, the compression buffer has one byte containing information for that tile; if looking up that address in the compression buffer yields a particular value, then the entire tile is assumed to have a solid fill of the first pixel value seen in the primary buffer. The GPU takes advantage of this to skip reading and writing the entire tile. So for solid patterns in particular you'll see a repeating 'snow' dot pattern where the correct value for every tile is in the top-left corner, and then the rest of the tile is random.

Anyway.

Also curious if SPA_FORMAT_VIDEO_MODIFIER is meant to hold DRM fourcc modifiers or if it needs conversion. It doesn't seem documented.

I'm almost certain it does.

Gstreamer documents this format flag:
GST_VIDEO_FORMAT_FLAG_TILED (256) – The format is tiled, there is tiling information in the last plane.
I wonder if getting that flag into a caps string and passing both planes would change anything. If that's all there was to it, it would be easy enough to create a PR for the gstreamer plugin.

Not quite. From what I understand, the plan for modifiers in GStreamer is passing the modifier set as an appendix to the buffer type, negotiated via caps separately to the format. TILED is a hack which can't actually express enough useful information about the buffer layout.

Oh, @fooishbar, bumped into you over here while trying to answer my own gstreamer question.

Did these patches ever get included? I don't see any format types related to DRM in the docs.

@ndufresne Can you please let us know what the current status is and how we can properly use modifiers from GStreamer, particularly pipewiresrc?

[ndufresne]

Also curious if SPA_FORMAT_VIDEO_MODIFIER is meant to hold DRM fourcc modifiers or if it needs conversion. It doesn't seem documented.

I'm almost certain it does.

This is a bit of a place holder that Wim added while we discussed that in Lyon. So it's not fully defined yet, or implemented, but I'm pretty sure it's one-to-one, best to ask Wim to confirm. I think the type here is to avoid deps on external headers.

Gstreamer documents this format flag:
GST_VIDEO_FORMAT_FLAG_TILED (256) – The format is tiled, there is tiling information in the last plane.
I wonder if getting that flag into a caps string and passing both planes would change anything. If that's all there was to it, it would be easy enough to create a PR for the gstreamer plugin.

Not quite. From what I understand, the plan for modifiers in GStreamer is passing the modifier set as an appendix to the buffer type, negotiated via caps separately to the format. TILED is a hack which can't actually express enough useful information about the buffer layout.

TILED formats aren't a hack, but they are indeed limited to fixed size tiles, with a non-dynamic number of tiles. That is a massive limitation. There is too many of these formats and having them in an enum in the API does not scale. The only benefit of having a tiled formats implemented in libgstvideo is that it gives a (slow path) color converter, which can be handy for debugging. But for these HW tiling, most of them are not documented, they are very complex. And they are not just about tiling, but also compression, which adds to the complexity.

Oh, @fooishbar, bumped into you over here while trying to answer my own gstreamer question.
Did these patches ever get included? I don't see any format types related to DRM in the docs.

@ndufresne Can you please let us know what the current status is and how we can properly use modifiers from GStreamer, particularly pipewiresrc?

So, this is all experimental in my branch. In general, we'll only use this with caps feature memory:DMABuf, as most system memory interface in the GPU/GFX don't have modifiers support. The idea is to pass the modifier in the caps format string, in the form <format>:<modifier_hex>. This will always The <format>, should not have much meaning. In the current PoC though, I'm parsing caps with the modifier part removed, just so I can extract other information from the caps. I think we need something cleaner.

[danshick]

Thanks, that's excellent information. It sounds like with your experimental branch plus some updates to the pipewiresrc gst plugin, I could possibly build a very alpha end-to-end wlroots->pipewire->gstreamer zerocopy example.

@emersion I think the bigger question is, is this worth while yet? I think the ecosystem stability isn't there to support this approach at the moment. I think there are three options.

1. Don't do anything for now. Wait for pipewire to finalize support for format modifiers, and wait for gstreamer and other downstream libraries (libyuv [1]) to support them as well (libavutil already has, as the dmabuf capture example works on my system). Then pick this back up when we can safely pass these buffers to most downstream programs that will consume them. Problem is that until there is a need, support may not land.

2. Build out zero copy in master and release it as soon as pipewire 0.3 and a compatible xdg-desktop-portal land. This may not be useful, but creates a use case to support format modifiers in downstream libraries and applications (browsers [libyuv] [2] [3], screen recorders[?], gnome-network-displays [gstreamer] [4], etc.)

3. Go for an approach like other screen recording utilities have and do the heavy lifting on CPU. Get this approach working in master, and move zero copy to another branch to await better support.

I think the best direction to take is best informed by how long it might take pipewire 0.3 to land. If the horizon is long enough, it might be best to choose option 3.

[1] https://chromium.googlesource.com/libyuv/libyuv/+/HEAD/docs/formats.md

[2] https://dxr.mozilla.org/mozilla-central/rev/3a51a5d0794209a3cf261dc23a8fa19ca2c02247/media/webrtc/trunk/webrtc/modules/desktop_capture/linux/base_capturer_pipewire.cc#346

[3] https://dxr.mozilla.org/mozilla-central/rev/3a51a5d0794209a3cf261dc23a8fa19ca2c02247/dom/media/systemservices/video_engine/desktop_capture_impl.cc#492

[4] https://github.com/benzea/gnome-network-displays/blob/a8014cc54c6ff9d184c3bc3b93b2374f1ecfac7d/src/nd-window.c#L120

[emersion]

Thanks for taking the time to investigate!

I think having a slow, CPU, fallback path wouldn't hurt, as in my experience zero-copy tends to be broken depending on the hardware and driver support. Both the CPU and zero-copy paths can be worked on independently.

[danshick]

Okay, since we have to do this with the CPU anyway, I've gone back and written a demo using pipewire 0.2. This will work with out of the box packages today. Needs a lot of refactoring and some minor features (output selection, proper width and height support, need to make decisions about handling framerate, etc.) and then this has to be added back into my fork that includes the d-bus code and it'll work.

https://github.com/danshick/wlr_cpu_pipewire

I think we should create a separate issue for zerocopy and another to update this repo when pipewire 0.3 officially lands. Spoke with @wtay today on IRC and in addition to helping me figure out what I was doing wrong, he gave me the following links to consider for the pipewire 0.3 API changes:

PR for support for 0.3 in xdg-desktop-portal
Mutter's PR for 0.3 screencasts