-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide a means to select only part of a screen to capture #105
Comments
I don't see how this would be compatible with user selection of content. |
@martinthomson The feature request would be compatible and consistent with user selection of content. The "Take a Screenshot" feature of Firefox Developer Tools provides a basic template of how to implement the feature, by way of selection of the content that should be captured, which could be translated into the appropriate corresponding
One example use case is
The Media Capture Screen Share API should provide a means to achieve such a requirement; by either fine-tuning the constraints for such selection, or allowing the user to select the content to be captured in similar fashion as the Firefox "Take a Screenshot" Developer Tool. What is the reason such functionality should not be provided by this API? |
@martinthomson BTW |
@martinthomson The above workarounds would not be necessary if an HTML |
@martinthomson Tested the code at Mozilla browser at
which sets the |
It seems there are multiple requirements here. One is for a browser to allow a user selecting a part of a screen. I do not see benefits in adding a constraint for that so this seems like an implementation decision, not a spec one. The second requirement is to capture a browser tab w/o title, location bar, slider... This seems somehow more closely related to cursor. Are you only interested in the latter? |
@youennf Am only interested in concatenating multiple media files into a single media file. While attempting to achieve that requirement have tried several approaches where summaries of some of the approaches tried can be found at each branch of the above-linked repository.
The live The selected screen can be translated to The benefits for the end-user should be self-evident: exacting capture of a given screen or application, without unnecessary portions of the screen or application. Such an specification extension of
The constraint Created a workaround for not recording the local and title bars at Chromium 73 after a day trying different approaches. The original branch of using |
What I mean is that crop constraints at getDisplayMedia call time does not make a lot of sense. Allowing a web page to efficiently manipulate video tracks (be they screen, camera or peer connection originating) with operations like cropping makes sense. |
@youennf Crop constraints at If the Edit Taking it a step further, an API which exposed the respective browsers' MediaDecoder/MediaEncoder and webm writer code. And/or human-readable form of EBML for the ability to write audio and video as |
@youennf Firefox includes |
@youennf FWIW after testing various approaches which included Firefox freezing the operating system on several occasions requiring hard re-boot, setting |
So what happens when you capture part of the screen, and the user moves the window? |
@alvestrand What do you mean by "moves the window"? |
These are non-spec and should be removed. They originate from a browser-tab sharing experiment Firefox had behind a pref years ago, and worked solely with In short, they weren't general purpose pixel croppers, which I agree with @youennf belongs elsewhere.
Track constraints are source-specific. If the specification of a source does not mention a constraint, then it is not supported for tracks from that source. w3c/mediacapture-main#578 is hoping to clarify this. |
@jan-ivar Am not certain what the issue, hesitancy and/or reluctance is with adding a constraint which provides a means to capture only part of a screen that will be shared? That is, if am gathering the hesitancy to add this constraint correctly from the responses so far in this issue. This feature request is consistent with screen capture (still image and live stream) programs, and in fact, consistent with the concept of setting It is reasonable to have the ability to select only part of a screen both by a selection tool to physically outline the portions of the screen to be captured and by setting Why should such functionality not be specified? |
@jan-ivar FWIW dove in to |
A rudimentary POC https://plnkr.co/edit/UmrSwN?p=preview to select only part of a screen for a screen shot (the movement and resizing of the selection element can be fine-tuned to have similar functionality as, for example, https://codepen.io/zz85/pen/gbOoVP)
Ideally, specific for a screen shot (i.e., #107)
where the permission is not for a screen shot, but for a continuous media stream to be captured
|
I would use this in my implementation of WebRTC. Getting exact dimensions is useful, and it is currently not allowed (eg. https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia section about TypeError (no exact or min spec's allowed)). Is this question still open for discussion? |
@hoolahoop How does not allowing setting The issue has not been closed, yet. |
This seems to be possible to do with current APIs, so no compelling reason seen for more browser surface. |
I think the spec allows browsers to have a way to crop if they wanted to, but I don't think the application should have control which part of the screen the user can pick |
@alvestrand @henbos Have you tried "Take a Screenshot" at Firefox Devloper Tools? That is what am asking to be specified as part of mediacapture-screen-share. Is there a compelling reason such functionality should not be specified? |
This functionality is allowed by the spec. "Take a Screenshot" at Firefox Devloper Tools does not seem to require any specific spec. This is really up to browser implementors to decide whether to add support for that functionality or not. |
I still don't see mediacapture-screen-share as a screenshot API, which makes me reluctant to add any language advising the browser to that it must or even should have this type of a picker. I'm not saying there aren't any use cases for capturing part of a screen, but I'm not compelled that we should attempt to mandate this, and as-is browsers are allowed to implement such a picker. If this use case was more compelling for getDisplayMedia then I do think we should advise implementations so that you don't end up having to do cropping as part of the application depending on whether or not the browser supported cropping, but again this is not the intent of getDisplayMedia, and I think it is within the browser's decision about what UI to support. For what you're asking I would like to see a different API than getDisplayMedia. |
See the current language in the specification, emphasis added
We could ignore that language and write a specification from scratch which included the same language to achieve double-redundancy. It that is what you believe to be necessary, where to post the specification? WICG discourse? Am not a "member" of W3C, and not really interested in becoming beholden to an organization, particularly one which cannot write the words "patent and copyright" when that is what they are asking about. The specification should be very simple. In fact, since Mozilla has already written the and and implemented the code, the only question would be is will Chrome implement the specification? Using In any event, how do you suggest to proceed? |
@henbos There is an existing "Screenshot API" topic at WICG discourse https://discourse.wicg.io/t/proposal-screenshot-api/2412. There are comments citing privacy and security concerns. Well,
From perspective here Can you explain the reasoning for your statement concerning the intent of How does the reading of the specification infer what was not the intent of the specification when the actual language includes the term
|
@MonsieurWave The issue was closed at #105 (comment). No current movement. |
@MonsieurWave One workaround is to use |
But this only works on a ScreenCapture within the browser, does it? |
Can you clarify "within the browser"? At Chromium it should be possible to select a "Tab" ( Images from the From perspective here one simpler procedure could be to provide a means to select only part of a screen to capture using constraints, thus this issue. |
@MonsieurWave For example, this code scales the entire screen to the dimensions passed as constraints, neither
8b295485-3452-4d90-a3a3-ef38e8e5b0a3.webm.zip One approach to capture only the previews of capture shown in prompt which that was intended to be fixed in the specification by #114
|
@MonsieurWave Entering fullscreen can reduce the captured screen to only |
The spec allows a user agent to implement capture of a part of a screen. |
My specific use case is for recording data (ie taking a screen shot) of data/image displayed by another application. Taking the whole window of the application would reveal compromising meta-data also displayed by the application, which the user would not like to show. |
getDisplayMedia seems overkill for this use-case. |
However, the user would thus need to use an external application to take the screenshot, thus requiring a lot more user input. |
If you try A similar flow could be implemented for screenshots. |
Provide a means to select only part of a screen to capture is a self-evident use case: The user only wants to capture certain portions of the tab, window, monitor surface. Is your question essentially why a user would want to capture only parts of a display surface, instead of use Desktop browsers other than MacOS based do not support
This can be achieved usinf constraints to
Why would this not be specified? |
@youennf This issue was filed initially due to, at the time, Chromium browser crashing when attempting to render variable width and height (pixel dimensions) video at Do wait around for specification authors, maintainers or implementers to do stuff, here. If there is a bug observed, generally immediately commence creating multiple workarounds, proof-of-concepts, and testing. This issue just happens to strike a chord with users that obviously have other use cases, particularly taking screen shots, where again, exclusion of tabs, other windows, and non-essential parts of the capture should be excluded, or rather, the ability to capture precisely what is intended to be captured the first time, and nothing else. |
@youennf I think I have one such use case completely away from the initial topic of screenshots. My (and others) default behaviour when sharing an entire screen used to be to share a screen with nothing on it - take the scenario of a laptop and an external monitor; I used to use the laptop screen as a "blank canvas" I could put content on - applications of all nature. When you're sharing content in a meeting or such, its a completely fluid thing - one minute I might be sharing code, another I could be looking at some docs in a browser. I don't want to keep switching which application is shared using getDisplayMedia so I just share an entire window. The trouble is I used to use my laptop monitor because it wasn't 4k (like my external monitor) - it meant people could actually read what was being shared etc etc. Long story short, I've just changed from having one external 4k monitor to 2 and I no longer have my laptop open - so what option do I get given when wanting to do exactly what I did before? I'll show you. This is the output of Chrome's getDisplayMedia GUM - you can see I have 3 monitors.... an ipad pro using sidecar (my temporary hack for this scenario), a vertical 4k monitor and a horizontal 4k monitor. Some others have their own solutions like changing the resolution on their monitor to lesser so that they can share their screen and make it easily readable but that's not going to work here. I have a vertical monitor that will never share well. Should I rotate it every time I want to do a screen share in a meeting and take the resolution down? No. Should I change the resolution on the primary monitor? No - because then everything else on my display would get shown to the people on the other end of the call - thats another partial reason for always using the "blank canvas" of the laptop monitor before - when you work with multiple clients - you don't want your clients seeing content from your other clients. So now, I'm lucky enough to be in this position. I've got two lovely 4k monitors infront of me and an ipad pro enabling my "hacky" solution for the moment. But for a moment don't think about the specifics, and go back to something else I just said.... I need a way to be able to share multiple applications all within one stream without showing off the content on the rest of my display - that's a perfect use case for this and if its being done to enable privacy reasons - then it should be down to the browser to do it and not be up to the application to action with cropping (ouch... cropping of real time video within javascript userland....) - for privacy reasons the browser should be the one giving over a stream only containing what the user selected when they say.... dragged an area they're OK with sharing. The way I see this working is very similar to how Quicktime does screen recording - you drag an area, everything else goes slightly darker and you're able to see exactly what has been shared with the application. So what do you think? This is definitely more than wanting to create screenshots. This is about offering flexibility. But while we're at it - with the likes of project fugu around.... don't we want web applications and the like to be just as good as say Zoom? Zoom lets me share a portion of my screen.... why shouldn't the browser? |
The initial use case was actually to capture variable width and height video displayed on a screen using Screenshot use cases and issues were mentioned as also applicable after that initial use case, which at the time was not simple to explain what was trying to do (also using Picture in Picture window for the same use case). |
@danjenkins tl;dr https://github.com/guest271314/MediaFragmentRecorder/tree/chromium_crashes, https://bugs.chromium.org/p/chromium/issues/detail?id=992235 and links included therein for the origin of this issue. See also https://bugs.chromium.org/p/chromium/issues/detail?id=1100746 for "Tab" and "Application" capture issue. See w3c/picture-in-picture#163 for Picture-In-Picture issue. In lieu of specified fix, a general solution piping original |
One approach is to utilize |
@danjenkins Another issue is that even though #114 fixed #108 in the specification, Chromium still captures UI, which means that given an HTML element that is effective a green square, simulating the window on your 4k monitor, the first frame can still include the UI - which is not expected to be exposed in the capture
In the case of capturing the |
Ultimately this shouldn't be a userland addition. This should be a browser feature. Lets say I only want to share the top half of my vertical monitor. I have something private and confidential on the bottom half that I haven't moved off it - there's no way I'm giving say Google Meet access to my entire screen, would I trust that application to not be sending that data somewhere? What you're describing above feels like a hack around a solution - and I'm not even certain if its really fulfils the solution to the problem - what if I don't have the browser foregrounded at all. I'm showing off VsCode and another browser different to the one running the meeting application I'm sharing my screen with.... In the web world we stop HTTPS pages being able to talk to HTTP APIs for a reason.... how is this any different? You want the user to know the web application truly only has access to the portion of the screen I allowed it access to. Also, the above solutions only work if developers implement the ability to do it - shouldn't it be a browser GUM "thing" so that any web application taking advantage of getDisplayMedia is able to use it? |
@danjenkins If |
Do you really trust browser implementations to that degree? I do not. It took 2 years just to convey the reality that Chromium was consistently crashing when variable pixel dimension frames were decoded. Re the arbitrary restriction on Picture-In-Picture window implemented by Chromium, see this response w3c/picture-in-picture#186 (comment)
Process the frames for yourself to ensure you are only exposing what you expect. Again, the PR that fixes exposure of prompt (potentially including windows, applications not intended to be captured) at first frame is not fixed in Chromium source code.
Well, that is all we have right now. Feel free to create a repository specifically for what your goal is and will help where able.
Would explore using "Application" capture at Chromium.
There are various way to get around that limitation, e.g., https://gist.github.com/guest271314/04a539c00926e15905b86d05138c113c.
Agree that the specification could solve the problem, though the issue was closed. Thus, here, Fix WontFix on own, without waiting for anyone else. |
@danjenkins As an example of what could be fixed by simply specifying capture of monitor devices MUST be the same as capturing only microphone devices, created an entire repository with multiple solutions, where that same subject matter is applicable to this specification #140, where audio capture is described in an ambiguous manner https://github.com/guest271314/captureSystemAudio. |
@guest271314 We need the browser to be able to do it so that all webrtc applications, no matter whether the developer wants to, can offer this. How can we get this re-opened? |
@danjenkins if by "it" you mean the browser should allow to capture a subset of a given screen, I believe nothing in the spec prevents it, and it's not clear that there is anything the spec can say to require it since ultimately the user-selection mechanism of what gets shared is left to browsers. Has any filed bugs with browsers to get this new display selection option implemented? Until it is clear that the spec needs to change, it's not obvious there would be much point in reopening the issue. |
@dontcallmedom thanks for this - yes I meant the browser - I still wasn't 100% on whether the spec allowed for this or not. Thanks for the clarification! I don't know of any bugs filed yet but I must admit I'm not the best at searching crbug and co. I'll go make a crbug and mozilla bug entry and go from there I guess :) thanks! |
@danjenkins Do not recollect filing Chromium and Mozilla bugs for this. For Mozilla bug would suggest incorporating the already existing screenshot code #105 (comment) into the feature request. That is, similar to the overlay Chromium uses for |
|
Is there a way to record a region of interest today using get user media api and media recorder? |
If the specification states that |
Firefox provide a "Take a Screenshot" feature which allows the user to select only a portion of a screen. That option should be provided for
getDisplayMedia()
in the form of constraints where specific coordinates can be passed; e.g., using.getBoundingClientRect()
in the form of{topLeft:<pixelCoordinate>, topRight:<pixelCoordinate>, bottomLeft:<pixelCoordinate>, bottomRight:<pixelCoordinate>}
, or at the selection UI, similar to how Firefox implements "Take a Screenshot" feature. Use case: The user only wants to share a specific element, e.g. a<video>
.The text was updated successfully, but these errors were encountered: