Blur/modify background in video call

[jancborchardt]

This is not something for soon as it’s a bunch of work and there’ other things to be worked on first. But just for reference.

---

From https://news.ycombinator.com/item?id=22670896

I can’t begin to tell you how many times we have had people that need extra time to prepare for a meeting because of the background. As a manager, I have had lots of personal conversations with employees who legitimately stress out about getting the computer into a place with a good background.

• Zoom has custom backgrounds: https://support.zoom.us/hc/en-us/articles/210707503-Virtual-Background
• Microsoft Teams has background blur: https://support.office.com/en-us/article/blur-your-background-in-a-teams-meeting-f77a2381-443a-499d-825e-509a140f4780
• Skype has background blur: https://support.skype.com/en/faq/FA34896/what-is-background-blur-in-skype
• Jitsi has "Blur my background" in beta now as well: https://github.com/jitsi/jitsi-meet/blob/master/react/features/stream-effects/blur/JitsiStreamBlurEffect.js

[PVince81]

Research technologies

The goal is mostly to find one that performs best and where the license is compatible.

• tensorflow/body-pix

  • license ok?
  • prototype: Background blur effect demo #5918 by @ipasanec

• mediapipe

  • license ok? (Apache 2)
  • prototype
  • potential challenges with updates, because the library only comes in minified form and need some local changes

• tflite ? (as pointed at by @LukasReschke internally)

  • license ok?
  • prototype ?

• how big is the package to download, size of model ? should we load it on-demand after turning it on the first time ?

• is it also possible to replace the blur with an image or video ?

Integration

• feature flag (experimental, needs JS code copy-pasted to console to enable it, OR debug mode in config.php so that every developer sees it) @PVince81

• implement plugin endpoint to allow registering video filter plugins to avoid the need to touch any of the internals (see Background blur effect demo #5918 for locations that need touching)

• function to toggle blur background on and off: does this need stream swapping gymnastics to be able to get back to the original stream + performance ?

• add button to toggle blur background on and off, possibly in the three dots menu with a keyboard shortcut "B"

• remember last blur setting per conversation/token like we do for video/audio state

• to test:

  • make sure that disabling/enabling video still keeps the blur
  • make sure the effect can survive disconnections/reconnections and the new "null stream" approach when idle
  • effect should not apply to screen shares, only videos
  • must continue working after video constraints are applied automatically
  • must work without HPB
  • must work with HPB
  • must work in all supported browsers

@nickvergessen FYI

[nickvergessen]

CSP places that might need adjusting:

• https://github.com/nextcloud/spreed/blob/master/lib/Controller/PageController.php#L262-L275
• https://github.com/nextcloud/spreed/blob/master/lib/Controller/PageController.php#L323-L335
• https://github.com/nextcloud/spreed/blob/master/lib/Listener/CSPListener.php#L49-L60

[PVince81]

Next steps:

• fixing linting Background blur effect demo #5918 @ipasanec
• a second PR with mediapipe (see CSP hints above to solve webassembly loading issues) @ipasanec
• third PR that provides an action to enable/disable blur + event, but not wired up yet. Only show the action if debug mode is enabled @PVince81

[PVince81]

new PR here #6036

[PVince81]

as discussed, a better wrapper around mediapipe ready to use: https://github.com/jpodwys/virtual-background

[nickvergessen]

Done with #6322
Minor follow ups in #6345