WIP: ARCore-based position tracking

[knuxify]

Partially fixes #93, fixes #311.

Enable HMD position tracking based on ARCore's motion tracking capabilities. Note that this only works on devices that support ARCore; see the official list for more information. (All devices on this list are supported, regardless of depth API support/camera resolution. Tracking accuracy may vary.)

Performance looks to be pretty good, though I am testing this on a Samsung Galaxy S23 which is pretty much a best-case scenario. The device does heat up after a while, but from what I've heard that's pretty standard for ARCore. However, as of writing, it's not stable enough yet to be actually used, which is why this PR is marked as a draft.

I'm opening this PR to get some feedback and perhaps some help with development. Here's a general TODO list:

• Add user-side toggle for enabling/disabling ARCore-based tracking (right now it's enabled by default)
• Add user-side settings for ARCore (whether to use orientation data from Cardboard or ARCore directly (not recommended unless I can work out the delay, see later comment), camera FPS limit, filtering to camera configs without depth support/with lower resolution for processing power/accuracy tradeoff, etc.)
• Figure out orientation drift. The orientation (rotation) that we get from the Cardboard API drifts away from the ARCore tracking orientation over time, so e.g. walking forwards starts moving sideways instead. Using the orientation data from ARCore itself solves this, but also introduces a ~0.5s delay which is very noticeable. I don't think there's a way to reduce this delay - it's just a side effect of how ARCore works, and is probably going to be worse on weaker devices. The solution would likely be to somehow adjust the orientation to match the one that ARCore reports...?
  • Occasionally when ARCore loses tracking it will "teleport", i.e. the position will be shifted; I think this could be related to the repositioning behavior described in the Pose docs. This completely breaks orientation when it occurs, so fixing the above would probably fix this too (in fact, I'm willing to believe that it's the same issue).
• Figure out pose scaling. Currently the actual speed of movement is a little out of wack, we will probably need to scale the values from ARCore. I've done some more testing and I think it's actually fine, but someone with more VR experience would need to test.
• Figure out altitude. Right now it seems to be wrong (maybe it just needs scaling? or maybe it's picked up from the first plane it identifies), so you'll usually end up clipping through the floor or just generally being very low. The ARCore geospatial API has the ability to get altitude data, but it must use Google's location APIs which is privacy-intrusive and has a ratelimit. I was also considering using the phone's barometer sensor and a short calibration process before each play session to determine the floor position and what scaling to use for anything higher than it, but I'm not sure if that's actually possible + wouldn't work on phones that don't have a barometer/pressure sensor.
  • This is done! The solution I went with is to find the lowest-placed plane and place an anchor on it; this way we can track the Y position of the floor, then subtract it from the headset position. While working on this, I also found out that ARCore's world space is based on the position of the device when the session is set up, so placing the device on the floor while turning on PhoneVR works best (but this also works very well once it gains tracking, so the UX is at least somewhat nice).
  • I did have a WIP patch for barometer-based tracking, but the barometer values were very jittery, and to smooth them out accurately and maybe include data from other sensors for extra accuracy, I'd probably need something like a Kalman filter. Unfortunately I do not know enough maths to actually implement one, so that's left as an exercise to someone more knowledgeable than me.
  • We could probably add a setting to force a specific altitude.

I saw somebody say that iVRy supports ARCore-based tracking (personally I didn't know that, I'd have to check) - if that's true, it'd be interesting to see how they tackle some of these problems.

[knuxify]

note to self: we should probably tie this to the same version we pull in app/build.gradle.

[knuxify]

this pattern kinda sucks; ideally we'd set CTX.arcoreEnabled to false if arSession fails to initialize

[ShootingKing-AM]

Also getPose is called many times perframe, so you might want to deInit/init arcore contexts and other checks outside it.

[ShootingKing-AM]

Nice initiative, I didnt check the code yet, also i dont have a arCore supported device :c
I was actually thinking of removing the hard coded device fingerprint gating in official arcoreApp to see if i can run this on my phone. I actually dont see the that some devices can use arcore and some cannot other than google marketing effort.

[knuxify]

I was actually thinking of removing the hard coded device fingerprint gating in official arcoreApp to see if i can run this on my phone.

IIRC this isn't as simple as removing a check, each device has calibration data specific to it (and of course there's no public documentation on how to add that calibration data yourself). Could work to just copy the calibration data from another device and live with the inaccuracy (quick googling shows that people have spoofed the device information to report the Pixel 2 way back in 2019 and got it working, but that was a long time ago...)

[ShootingKing-AM]

True that too, but whats the use of development or including arcore if only some set of devices can support it ? We need to have something for the others who dont have arcore too, i only think only 10% (assumptions) of devices in market support arcore

[knuxify]

True that too, but whats the use of development or including arcore if only some set of devices can support it ?

The use is that the devices that support it can make use of it :D

On a more serious note though - I can certainly see that this is a bit unfair, and would probably cause added burden to you as the maintainer (since you don't have a device to test this). However, I think having at least some devices able to use this is still better than not having this feature at all. I also hope that supporting one method for getting position can help with adding support for more methods in the future).

(The only other currently-available method I know for tracking headset position is using an external method like PSMoveServiceEX + a LED bulb mounted to the HMD, but that's more involved than just running an app, and is difficult to do homebrew from my own experience.)

Figuring out alternatives to ARCore is likely out-of-scope for this PR, though.

[ShootingKing-AM]

I mean i dont have an issue adding this, but i dont have a way to test it (as of now) as long as you are okay with future pings for issues related to ARCore :p

Also i dont think its that hard to add 6DOF by directly getting accelerometer reading, thought you have some knowhow regarding that.

[knuxify]

oops, leftover from trying to fix orientation

[ShootingKing-AM]

/ok-to-test sha=6d74a41

[ShootingKing-AM]

/ok-to-test sha=b1b640d

[github-actions]

📷 Screenshots of tests:

📱gvr: emulator-5554 - 11

🔧 ALVRActivityTest_saveDeviceScreenBitmap

🔧 InitActivityTest_saveDeviceScreenBitmap

🔧 MainActivityTest_saveDeviceScreenBitmap

🔧 SettingsActivityTest_saveDeviceScreenBitmap

🔧 view-op-error-1

🔧 view-op-error-2

📱noGvr: emulator-5554 - 11

🔧 InitActivityTest_saveDeviceScreenBitmap

🔧 view-op-error-1

For commit b1b640d

[ShootingKing-AM]

I think you need to make another cpp files because arcore itself seems to have a lot more glue code to use it.

[ShootingKing-AM]

Ok I got hold of a ArCore supported android device. I was expecting some moving in SteamVR home (translation) but thats seems not to be the case. How should one test this ? Also you can join #dev channel on discord for more discussion, if you are already on discord am not able to find your ID knuxify