Launch immersive experiences directly

[felipeerias]

This PR implements support for opening immersive experiences directly as soon as the application is launched.

We do this by adding a built-in extension which will activate a particular element in the page.

We need to set the preference dom.vr.require-gesture to false so this script can launch immersive WebXR experiences without user input. Media autoplay also need to be allowed in this mode.

The information required to identify the element to activate is passed as parameters to the Intent:

• launch_immersive
• launch_immersive_parent_xpath
• launch_immersive_element_xpath
• a target URL to open

One additional concern is that we should only change the configuration prefs when that is needed for launching Wolvic directly in immersive mode.

The problem is that we only write that prefs file the first time that we create the browser engine, which happens several times at launch time before we have parsed the incoming Intent and we can know in which mode we are launching.

The solution is that WidgetManagerDelegate.isOpenInImmersive() internally will check the current Intent, to cover the case where it is being called too early.

We then pass VRBrowserActivity as the context to the objects that may initialize the engine, so eventually SessionUtils.prepareConfigurationPath() is able to decide whether specific prefs should be added to the configuration file.

For launching Wolvic directly in immersive mode, these prefs are:

  dom.vr.require-gesture: false
  media.autoplay.default: 0

[felipeerias]

This PR can be tested from the command line with:

(deprecated)

[felipeerias]

I have updated the PR so that it uses the XPath of the parent element, which is a more flexible solution.

[svillar]

I think I told @felipeerias about this in private, but I'll add it here for everybody interested to know. The idea of using an extension is nice, but the problem is that the Chromium backend won't have extensions support initially as it's a lot of work. So provided that we don't have a solution at the moment I'd prioritize one that does not involve extensions.

That said I agree that having an extension that inspects the DOM is likely the best we could do, so I indeed have mixed feelings.

[felipeerias]

This implementation might be the best that we can do, but it still has several issues.

First of all, the command to launch Wolvic in immersive mode needs to know the exact element in the page which will open the WebXR experience. If the page changes, it will be necessary to update that command.

Secondly, the approach of using a Web extension to trigger specific elements in the page will not be possible on Chromium, because we don't support extensions there just yet.

Third, in some cases our extension is not able to reach the immersive button because the WebXR experience is inside of an iframe from a different origin, and security rules prevent us from accessing a cross-origin object from the embedding page. It might be possible to work around this issue by passing messages between the content scripts running on the page and on the iframe.

This last problem is particularly painful because it affects HeyVR, a large repository of WebXR games: the pages are in https://heyvr.io but their iframes point towards https://games.heyvr.io.

Finally, some pages might have specific quirks that we have to consider. For example, Moon Rider requires two clicks to open the immersive experience, and Magical Reflections requires the user to accept cookies before anything else.

With all that in mind, the implementation in this PR does work reasonably well. Check out the following video:

ScreenRecording_2024.07.10-16.57.13.mp4

Command to launch A-Painter:

(deprecated)

Command to launch Moon Rider:

(deprecated)

[svillar]

Why two?

[felipeerias]

There was a case where one click() didn't work. I think that I will remove this and test again different experiences, to check if it is still needed.

[svillar]

So, we still have the two clicks... does it mean it's still required?, I still don't get why

[svillar]

I think this should be private

[felipeerias]

Yes, of course.

[felipeerias]

@svillar I have updated the PR following your comments. I also had to allow autoplay when we start in this mode to remove an error with Moon Rider, although for some reason it is still not working properly (it was working fine before).

Reference: https://wiki.mozilla.org/Media/block-autoplay

[felipeerias]

Access Mars:

(deprecated)

(Note that depending on your setup you might have to add .dev to the package name)

[svillar]

private?

[felipeerias]

The other Intent parameters in that file are public.

I think it's a good idea for documentation purposes, as a way to make clear that these strings are supposed to be used outside of this class.

[svillar]

So, we still have the two clicks... does it mean it's still required?, I still don't get why

[svillar]

I know this is kind of arbitrary but a maximum 20" retry seems to much to me. Maybe good for poor connections though...? In that case the user would have some other problems. My point is that users won't get any idea of what's going on for a lot of time. The feeling after 4-5" is that nothing works, nobody will wait for 20" for an app to launch.

[felipeerias]

Yes, but that's a problem in general with this feature. Some experiences just take a long time to load from the network.

In this initial implementation, we show the browser window before launching the immersive experience. Eventually my plan is to go directly from the loading screen to the immersive experience.

[svillar]

I feel a bit uneasy having this. Would it be possible not to add this here and manually add it to the runtime settings builder in EngineProvider.kt somehow? We don't need to spend much time on that, but if it's possible to add it just whenever is needed it'd be much better.

[felipeerias]

One possibility could be to do it in SessionUtils.prepareConfigurationPath(), which copies the configuration file from the assets to the device's filesystem. Basically, the idea would be to append an extra line to the file if needed:

final String REQUIRE_GESTURE_CONFIG = "dom.vr.require-gesture: false";
WidgetManagerDelegate widgetManager = (WidgetManagerDelegate) aContext;

if (widgetManager.isOpenInImmersive()) {
        outputStream.write(REQUIRE_GESTURE_CONFIG.getBytes());
}

Another approach could be to have two files in the assets, and copy one or the other depending on the current configuration. Maybe that would be better?

[svillar]

I think it's enough with the first approach you mentioned. Adding a line to the stream should make it

[felipeerias]

I've been working on this but it is a bit more complex than I expected because prepareConfigurationPath() is called very early, when we have not parsed the incoming Intent yet and so we don't know that we will be in the automatically immersive mode.

Specifically, as a result of VRBrowserActivity.onCreate() calling

((VRBrowserApplication) getApplication()).onActivityCreate(this)

[felipeerias]

I finally found a way to change the configuration prefs only when needed for launching in immersive mode.

The problem is that we only write the prefs file the first time that we create the browser engine, which happens several times at launch time before we have parsed the incoming Intent and we can know in which mode we are launching.

The solution is that WidgetManagerDelegate.isOpenInImmersive() internally will check the current Intent, to cover the case where it is being called too early.

We then pass VRBrowserActivity as the context to the objects that may initialize the engine, so eventually SessionUtils.prepareConfigurationPath() is able to decide whether specific prefs should be added to the configuration file.

For launching Wolvic directly in immersive mode, these prefs are:

  dom.vr.require-gesture: false
  media.autoplay.default: 0

[felipeerias]

I have thoroughly updated this PR.

Unfortunately, it is hard to find experiences which will work well with it:

• some will show an ad before the game, like heyvr.io
• some don't have stable XPath identifiers for the button that launches the experience, like magische-spiegelungen.de
• some work from the JS console via remote debugging but not from an extension (perhaps a timing issue?), like moonrider.xyz

These are a couple of experiences which work consistently.

A-Painter:

adb shell am start -n "com.igalia.wolvic.dev/com.igalia.wolvic.VRBrowserActivity" \
-a android.intent.action.VIEW                                                     \
-c android.intent.category.LAUNCHER                                               \
-d "https://aframe.io/a-painter/"                                                 \
--ez launch_immersive true                                                        \
--ez hide_webxr_interstitial true                                                 \
-e launch_immersive_element_xpath '\/html\/body\/a-scene\/div[2]\/button'

Access Mars:

adb shell am start -n "com.igalia.wolvic.dev/com.igalia.wolvic.VRBrowserActivity" \
-a android.intent.action.VIEW                                                     \
-c android.intent.category.LAUNCHER                                               \
-d "https://accessmars.withgoogle.com/"                                           \
--ez launch_immersive true                                                        \
--ez hide_webxr_interstitial true                                                 \
-e launch_immersive_element_xpath '\/html\/body\/div\[1\]\/header\/div\/button'

(remove the .dev from the package name if necessary)

[svillar]

Looking good, we're superclose to land this. I have some comments

[felipeerias]

I had to modify AndroidManifest.xml so we could launch Wolvic with parameters without needing to specify the specific Activity.

Now that VersionCheckActivity is the entry point of our application and the only Activity that we export, it needs to use the intent filters that were previously associated to VRBrowserActivity.

These are the new comands to test this feature:

adb shell am start -p com.igalia.wolvic.dev \
-a android.intent.action.VIEW               \
-d "https://aframe.io/a-painter/"           \
--ez launch_immersive true                  \
--ez hide_webxr_interstitial true           \
-e launch_immersive_element_xpath '\/html\/body\/a-scene\/div[2]\/button'

adb shell am start -p com.igalia.wolvic.dev \
-a android.intent.action.VIEW               \
-d "https://accessmars.withgoogle.com/"     \
--ez launch_immersive true                  \
--ez hide_webxr_interstitial true           \
-e launch_immersive_element_xpath '\/html\/body\/div\[1\]\/header\/div\/button'

[felipeerias]

The change which assigned the intent filters to VersionCheckActivity has been already integrated by #1575

I have rebased this PR and it is once again ready for review.

[svillar]

OK let's land this