xrshader

Experimenting with fragment shaders for WebXR - work in progress, but feedback always welcome.

Try it: https://matthewarcus.github.io/xrshader

Main repository: https://github.com/matthewarcus/xrshader

Introduction

WebXR is an emerging W3C standard that allows use of Virtual and Augmented Reality technology from within web browsers.

This project is an experiment in using Shadertoy-style fragment shaders to generate WebXR images.

The code is modelled on the examples to be found at the Immersive Web website, so if those examples don't work for you, it's likely that the code here won't work either. Also, I can't pretend to understand everything in the standard, so the code may contain misconceptions or errors - I'm very happy to have any issues pointed out.

Running the code

Clone the repo to somewhere that is visible to a webserver. I use nginx and the top level directory is /var/www/html/xrshader. To work, WebXR must be served with https - otherwise you will see "WebXR not supported" messages, so for testing with nginx, enable https and use the self signed certs generated by the ssl-cert package (don't use them in a production server!).

Tested with Chrome on Samsung A5 phone (Android 8.0, Chrome 87.0), and with Firefox and Chrome with WebXR API emulator. Find the WebXR emulator for either Firefox or Chrome in the Add-on store - it's the same for both.

If you need it, the source is at https://github.com/MozillaReality/, but the Store version is up to date (as of Jan 2021), including for example AR support.

Get developer tools up with F12 and set a suitable emulated device - note that you can either use an AR device (eg. Samsung Galaxy S8+ (AR)) or a VR device (currently all the other devices), no device supports both simultaneously.

How it works

• Projection & view matrix. In clipspace, the viewer is at (0,0,1,0), ie. infinitely far away in the positive z direction. To get a real viewer position, apply the inverse of the projection matrix combined with the view matrix (which defines how the user is positioned in the reference frame).

  mat4.mul(transformMatrix,projectionMatrix,viewMatrix);
  mat4.invert(transformMatrix,transformMatrix);
  gl.uniformMatrix4fv(gl.getUniformLocation(program, "transformMatrix"), false, transformMatrix);

• Selective AA. Use distance from centre of image to determine level of antialiasing:

  vec3 drdx = dFdx(r);
  vec3 drdy = dFdy(r);
  float k = dot(r,rcentre);
  float aa = float(k > 0.96 ? 3 : k > 0.9 ? 2 : 1);
  for (float i = 0.0; i < aa; i++) {
    for (float j = 0.0; j < aa; j++) {
      if (scene(p,normalize(r+i/aa*drdx+j/aa*drdy),col1)) {
        color += col1;
      }
    }
  }

• Centre ray. In clipspace, the centre of the the view is at (0,0,-1.0), ie. infinitely far away in the negative z direction, to map that to a viewer relative direction, apply the inverse of the view matrix (so the result is still a projective point at infinity):

  vec3 rcentre = (inverse(iView)*vec4(0,0,-1,0)).xyz;

• Alpha blending. In AR mode, the framework uses alpha blending to combine the generated image with the real world image.
• Selection events. Selection events are detected by the javascript framework and are passed to the shader as the w component of the iMouse uniform.
• Errors. Generally, the code raises exceptions on errors, better reporting would be good. Debugging is easier with the emulator & making use of the Javascript console.

To be continued...