Detail magic window support #237

toji · 2017-06-01T23:03:28Z

Described how magic window mode works with the outputCanvas proposal.
Also, because they’re natural variants of the same mechanism, described
mirroring and tracking only behavior. Follows the proposal in #224.

Also changed VRPresentationFrame -> VRFrame because if we moved forward
with the tracking only portion of this there would be scenarios in
which the VRPresentationFrames had no Presentation component.

toji · 2017-06-07T22:19:40Z

Updated this pull request after some conversations with @NellWaliczek to drop the concept of a "Tracking only" session for the moment. While I'd like to discuss that further in the future it's the least important part of this proposal and I don't care for it to complicate things.

Very keen to get feedback, since I consider this to be one of the last big issues to work out prior to TAG review and starting on the spec proper.

NellWaliczek

The core concepts seem pretty solid at first glance,
There's a few things I'd like to look into a bit more, but I think we're trending in a good direction.

NellWaliczek · 2017-06-13T04:09:00Z

explainer.md

  // VRDevice.requestSession must be called within a user gesture event
  // like click or touch when requesting exclusive access.
-  vrDevice.requestSession({ exclusive: isExclusive })


It seems a bit odd to pass the exclusive entry explicitly in the supportsSession() sample code and then skip it here. Was there a specific reason for that?

Agreed that it's a bit confusing, reverting for now. I also think that we need to push a change to supportsSession soon, as you've been suggesting, and perhaps at that point it'll be easier to demonstrate effectively.

NellWaliczek · 2017-06-13T04:09:47Z

explainer.md

@@ -123,7 +123,7 @@ function BeginVRSession(isExclusive) {
 ```
 Once the session has started, some setup must be done to prepare for rendering.
 - A `VRFrameOfReference` must be created to define the coordinate system in which the `VRDevicePose` objects will be defined. See the Advanced Functionality section for more details about frames of reference.
- The depth range of the session should be set to something appropriate for the application. This range will be used in the construction of the projection matricies provided by `VRPresentationFrame`.
+- The depth range of the session should be set to something appropriate for the application. This range will be used in the construction of the projection matricies provided by `VRFrame`.


This naming change perhaps isn't as relevant with the removal of the non-visual sessions?

Fair point. I'll revert these for now. (I think Input might push them away from the Presentation verbiage again, though.)

NellWaliczek · 2017-06-13T04:14:56Z

explainer.md

@@ -336,6 +325,62 @@ if(frameOfRef.bounds) {
 }
 ```

+### Mirroring


Should this go in the advanced functionality section? On mobile devices, this won't be particularly useful and on Windows Mixed Reality headsets, there is automatic mirroring in the Mixed Reality Portal.

This is in the advanced functionality section, actually. (Not always easy to tell with the sparse diffs)

NellWaliczek · 2017-06-13T04:40:47Z

explainer.md

+
+When mirroring only one eye's content will be shown, and it should be shown undistorted. The UA may choose to crop the image shown, display it at a lower resolution than originally rendered, and the mirror may be multiple frames behind the image shown in the headset. The mirror may include or exclude elements added by the underlying VR system (such as visualizations of room boundaries) at the UA's discretion. Pages should not rely on a particular timing or presentation of mirrored content, it's really just for the benefit of bystanders or demo operators.
+
+On devices without an external display to mirror to, such as mobile or all-in-one systems, the `outputCanvas` will be ignored.


Are you operating under the assumption that external display == need to mirror? What if a user agent/OS is already doing its own mirroring (such as the Windows Mixed Reality Portal)? It seems like a better end user experience might come from not mirroring a second time. It might be a bit heavy handed to say that a UA can only ignore the outputCanvas if it doesn't have an external display.

Of course with the current design the page has no way of distinguishing this situation. We'd be likely to see pages always supplying an outputCanvas and then there might be a big gaping hole in the middle of the page in some UA. Perhaps it's worth indicating the need for mirroring (as a attribute off an object returned by calls to supportsSession?) There is still the risk that developers will ignore it though. Hrmmmm....

I could probably find a way to highlight this better, but the intent is not so much "external display == need to mirror" but "no external display == pointless to mirror". UA should still have the option to ignore a requested mirror when it's appropriate, even if the system indicates that it has an external display.

NellWaliczek · 2017-06-13T04:50:14Z

explainer.md

+
+These scenarios can make use of non-exclusive sessions to render tracked content to the page. Similar to mirroring, to make use of this mode an `outputCanvas` is provided at session creation time, as well as the `exclusive: false` flag. Just like with mirroring the `outputCanvas` must have a `ImageBitmapRenderingContext`. At that point content rendered to the `VRSession.baseLayer` will be rendered to the `outputCanvas`. In the future, if multiple `VRLayers` are used their composited result will be what is displayed in the `outputCanvas`. Given that the session is tied to a specific output canvas it's permissible to have multiple non-exclusive sessions active at once to handle multiple output surfaces.
+
+Exclusive and non-exclusive sessions use the same render loop code, but there are some differences in behavior to be aware of. The sessions may run their render loops at at different rates. During exclusive sessions the UA runs the rendering loop at the `VRDevice`'s native refresh rate. During non-exclusive sessions the UA runs the rendering loop at the refresh rate of page (aligned with `window.requestAnimationFrame`.) Also, when rendering with an exclusive session the projection matrices provided each frame are based on the device optics, while non-exclusive sessions provide projection matrices based on the output canvas dimensions.


That last sentence might be accidentally a bit too restrictive. The "punch-through" scenarios we've discussed also take into account the headpose the same as I imagine zSpace would.

Expanded the language in upcoming commit.

NellWaliczek · 2017-06-13T04:50:54Z

explainer.md

+
+Exclusive and non-exclusive sessions use the same render loop code, but there are some differences in behavior to be aware of. The sessions may run their render loops at at different rates. During exclusive sessions the UA runs the rendering loop at the `VRDevice`'s native refresh rate. During non-exclusive sessions the UA runs the rendering loop at the refresh rate of page (aligned with `window.requestAnimationFrame`.) Also, when rendering with an exclusive session the projection matrices provided each frame are based on the device optics, while non-exclusive sessions provide projection matrices based on the output canvas dimensions.
+
+Most instances of non-exclusive sessions will only provide a single `VRView` to be rendered, but UA may request multiple views be rendered if, for example, it's detected that that output medium of the page supports stereo rendering. As a result pages should always draw every `VRView` provided by the `VRFrame` regardless of what type of session has been requested.


An example here might help to clarify a lot :)

I guess I could, but it would literally be a copy/paste of a previous render loop example with a comment saying: "This is literally a copy/paste of the previous render loop example." ;)

NellWaliczek · 2017-06-13T04:52:59Z

explainer.md

+
+UAs may have different restrictions on non-exclusive contexts that don't apply to exclusive contexts. For instance, a different set of `VRFrameOfReference` types may be available with a non-exclusive session versus an exclusive session.
+
+The UA may reject requests for a non-exclusive sessions for a variety of reasons, such as the inability of the underlying hardware to provide tracking data without actively rendering to the device. Pages should be designed to robustly handle the inability to aquire non-exclusive sessions.


Did I miss it, or are we calling out somewhere that no user initiated action is required for non-exclusive sessions?

It's called out in the "Detecting and advertising VR mode" section, but didn't show up in this diff because it was made as part of #236, which I (accidentally) made the upstream for this patch. Hm... We'll have to sort that out.

NellWaliczek · 2017-06-13T04:56:21Z

explainer.md

@@ -578,10 +623,12 @@ interface VRDevice : EventTarget {

 dictionary VRSessionCreateParametersInit {
  boolean exclusive = true;
+  HTMLCanvasElement outputCanvas = null;


Out of curiosity, why HTMLCanvasElement rather than the ImageBitmapRenderingContext given the hard requirement listed above?

Primarily because the intent is to create a link to a specific canvas on the page, but I'm flexible on this point. Probably want to address @RafaelCintron's comments about the ImageBitmapRenderingContext before we make a decision here, though.

RafaelCintron · 2017-06-13T18:08:53Z

Here's what ImageBitmapRenderingContext looks like today:

4.12.5.2.2 The ImageBitmapRenderingContext interface
interface ImageBitmapRenderingContext {
readonly attribute HTMLCanvasElement canvas;
void transferFromImageBitmap(ImageBitmap? bitmap);
};

context . transferFromImageBitmap(imageBitmap)
Transfers the underlying bitmap data from imageBitmap to context, and the bitmap becomes the contents of the canvas element to which context is bound.

Calling transferFromImageBitmap replaces contents of the canvas with the passed in ImageBitmap. It’s as if the internal canvas texture is switched out with the backing store of the ImageBitmap. The corresponding transferToImageBitmap has been explained to me as something you “tear off” the corresponding offscreen canvas.

The problem with using ImageBitmapRenderingContext for mirroring is that its contents belong to the developer. The rendering context we want is one that belongs to the user agent. What happens when transferFromImageBitmap is called while the UA is busy mirroring? Will mirroring be disabled? Is the developer telling the UA “I want the ImageBitmap to now be this other thing, please stop messing with it”? Or will the user see the contents of their image bitmap for a fleeting moment until the user agent stomps on it again?

If we introduce spec language that says “transferFromImageBitmap is disabled while we’re mirroring” then I think we’re better off just making our own “mirroring” rendering context type that doesn’t have this function. At that point, we can introduce getImageData functions to this rendering context type so that developers can get the contents of what the UA has rendered for testing purposes.

toji · 2017-06-13T19:19:00Z

Made several updates based on @NellWaliczek's feedback, haven't yet taken into account @RafaelCintron's comments because I need some more time to consider them.

This change removes any concepts that assume a device may only have a single active session at a time. This allows multiple non-exclusive VRSessions to be active at once, which will be important for enabling various interesting magic window scenarios in the future. With this change starting an exclusive session suspends the non-exclusive ones, but doesn’t end them. IDL Changes: - Remove VRDevice.activeSession - VRDevice.onsessionchange -> VRSession.onended

Wanted to explicitly state the differences between the two modes and why you’d want to use one or the other.

Some security discussion has been deferred to a follow up PR for the sake of readability.

Following discussion on implementor call, for the moment we want to just remove elements of the explainer that would explicitly block multiple non-exclusive sessions at once (like the activeSession attribute) while not explicitly stating that multiple are support just yet. (Might be a 2.1 feature? We’ll see.)

Described how magic window mode works with the `outputCanvas` proposal. Also, because they’re natural variants of the same mechanism, described mirroring and tracking only behavior. Follows the proposal in #224. Also changed VRPresentationFrame -> VRFrame because if we moved forward with the tracking only portion of this there would be scenarios in which the VRPresentationFrames had no Presentation component.

It’s the least important and yet most contentious part of this proposal AFAICT, so I don’t mind dropping it to help move things along.

toji mentioned this pull request Jun 3, 2017

Allow multiple active sessions at once #236

Merged

toji mentioned this pull request Jun 12, 2017

F2F Followup: Magic Window (and friends) Proposal #179

Closed

NellWaliczek reviewed Jun 13, 2017

View reviewed changes

toji force-pushed the multi_session branch from 91f7a39 to f113613 Compare June 15, 2017 21:16

toji force-pushed the magic_window branch from 855eae2 to aa11513 Compare June 15, 2017 21:24

toji mentioned this pull request Jun 22, 2017

Clarify behavior of browser window in VR #244

Closed

ddorwin mentioned this pull request Jun 27, 2017

Normatively mitigate privacy concerns related to poses outside presenting (e.g., magic window) #250

Closed

toji force-pushed the multi_session branch from 565c439 to 2456728 Compare June 28, 2017 04:04

toji added 9 commits June 27, 2017 21:53

Clarify the difference between Exclusive and Non-Exclusive modes

956d5a2

Wanted to explicitly state the differences between the two modes and why you’d want to use one or the other.

Updated in response to multiple suggestions from @ddorwin

58861ef

Some security discussion has been deferred to a follow up PR for the sake of readability.

More changes from more @ddorwin feedback.

b699b2d

Incorporating some more feedback from @NellWaliczek

a6a2f24

Dropped "Tracking Only" session for now

5e0b6b5

It’s the least important and yet most contentious part of this proposal AFAICT, so I don’t mind dropping it to help move things along.

Several updates based on @NellWaliczek's feedback.

6332b4d

toji force-pushed the magic_window branch from aa11513 to 6332b4d Compare June 29, 2017 18:50

toji closed this Jun 29, 2017

toji mentioned this pull request Jul 7, 2017

Detail magic window support #255

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detail magic window support #237

Detail magic window support #237

toji commented Jun 1, 2017

toji commented Jun 7, 2017

NellWaliczek left a comment

NellWaliczek Jun 13, 2017

toji Jun 13, 2017

NellWaliczek Jun 13, 2017

toji Jun 13, 2017

NellWaliczek Jun 13, 2017

toji Jun 13, 2017

NellWaliczek Jun 13, 2017

toji Jun 13, 2017

NellWaliczek Jun 13, 2017

toji Jun 13, 2017

NellWaliczek Jun 13, 2017

toji Jun 13, 2017

NellWaliczek Jun 13, 2017

toji Jun 13, 2017

NellWaliczek Jun 13, 2017

toji Jun 13, 2017

RafaelCintron commented Jun 13, 2017

toji commented Jun 13, 2017


		When mirroring only one eye's content will be shown, and it should be shown undistorted. The UA may choose to crop the image shown, display it at a lower resolution than originally rendered, and the mirror may be multiple frames behind the image shown in the headset. The mirror may include or exclude elements added by the underlying VR system (such as visualizations of room boundaries) at the UA's discretion. Pages should not rely on a particular timing or presentation of mirrored content, it's really just for the benefit of bystanders or demo operators.

		On devices without an external display to mirror to, such as mobile or all-in-one systems, the `outputCanvas` will be ignored.


		These scenarios can make use of non-exclusive sessions to render tracked content to the page. Similar to mirroring, to make use of this mode an `outputCanvas` is provided at session creation time, as well as the `exclusive: false` flag. Just like with mirroring the `outputCanvas` must have a `ImageBitmapRenderingContext`. At that point content rendered to the `VRSession.baseLayer` will be rendered to the `outputCanvas`. In the future, if multiple `VRLayers` are used their composited result will be what is displayed in the `outputCanvas`. Given that the session is tied to a specific output canvas it's permissible to have multiple non-exclusive sessions active at once to handle multiple output surfaces.

		Exclusive and non-exclusive sessions use the same render loop code, but there are some differences in behavior to be aware of. The sessions may run their render loops at at different rates. During exclusive sessions the UA runs the rendering loop at the `VRDevice`'s native refresh rate. During non-exclusive sessions the UA runs the rendering loop at the refresh rate of page (aligned with `window.requestAnimationFrame`.) Also, when rendering with an exclusive session the projection matrices provided each frame are based on the device optics, while non-exclusive sessions provide projection matrices based on the output canvas dimensions.


		Exclusive and non-exclusive sessions use the same render loop code, but there are some differences in behavior to be aware of. The sessions may run their render loops at at different rates. During exclusive sessions the UA runs the rendering loop at the `VRDevice`'s native refresh rate. During non-exclusive sessions the UA runs the rendering loop at the refresh rate of page (aligned with `window.requestAnimationFrame`.) Also, when rendering with an exclusive session the projection matrices provided each frame are based on the device optics, while non-exclusive sessions provide projection matrices based on the output canvas dimensions.

		Most instances of non-exclusive sessions will only provide a single `VRView` to be rendered, but UA may request multiple views be rendered if, for example, it's detected that that output medium of the page supports stereo rendering. As a result pages should always draw every `VRView` provided by the `VRFrame` regardless of what type of session has been requested.


		UAs may have different restrictions on non-exclusive contexts that don't apply to exclusive contexts. For instance, a different set of `VRFrameOfReference` types may be available with a non-exclusive session versus an exclusive session.

		The UA may reject requests for a non-exclusive sessions for a variety of reasons, such as the inability of the underlying hardware to provide tracking data without actively rendering to the device. Pages should be designed to robustly handle the inability to aquire non-exclusive sessions.

Detail magic window support #237

Detail magic window support #237

Conversation

toji commented Jun 1, 2017

toji commented Jun 7, 2017

NellWaliczek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RafaelCintron commented Jun 13, 2017

toji commented Jun 13, 2017