experiment(native): implement missing Blob/URL/fetch APIs

[CodyJasonBennett]

Fixes #1972, #2577.

Adds a fast path for data URIs in JSI, fixes for drawables in Android APK, and implements missing APIs as per the title.

Below asset is from https://github.com/KhronosGroup/glTF-Sample-Models/blob/master/2.0/Duck/glTF-Binary/Duck.glb, rendered on iOS emulator.

---

To recap, this issue comes from usage of URL.createObjectUrl(new Blob([arrayBuffer], { type: 'image/png' })) which is used by GLTFLoader to create a valid image url from inlined buffer data. react-native does not implement the Blob constructor from ArrayBuffer parts, nor does it URL.createObjectURL in the networking stack. Furthermore, only recently in 0.72 react-native implements FileReader.readAsArrayBuffer with facebook/react-native#36332 which is possible to polyfill for older versions. This previously necessitated patches for loaders since THREE.FileLoader moved to fetch -- facebook/react-native#30769 (comment).

I've been able to implement the missing Blob and URL.createObjectURL methods as a JS polyfill in lieu of mrousavy/react-native-blob-jsi-helper (we can't ship native code), but run into the same issue as #1972 (comment) when this is requested anywhere -- NSURLErrorDomain Code=-1002 "unsupported URL".

Show code:

import { NativeModules } from 'react-native'
import BlobManager from 'react-native/Libraries/Blob/BlobManager.js'
import { fromByteArray } from 'base64-js'

let BLOB_URL_PREFIX = null

const { BlobModule } = NativeModules

if (BlobModule && typeof BlobModule.BLOB_URI_SCHEME === 'string') {
  BLOB_URL_PREFIX = BlobModule.BLOB_URI_SCHEME + ':'
  if (typeof BlobModule.BLOB_URI_HOST === 'string') {
    BLOB_URL_PREFIX += `//${BlobModule.BLOB_URI_HOST}/`
  }
}

function uuidv4() {
  return 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, (c) => {
    const r = (Math.random() * 16) | 0,
      v = c == 'x' ? r : (r & 0x3) | 0x8
    return v.toString(16)
  })
}

BlobManager.createFromParts = function createFromParts(parts, options) {
  const blobId = uuidv4()

  const items = parts.map((part) => {
    if (part instanceof ArrayBuffer || (global.ArrayBufferView && part instanceof global.ArrayBufferView)) {
      const data = fromByteArray(new Uint8Array(part))
      return {
        data,
        type: 'string',
      }
    } else if (part instanceof Blob) {
      return {
        data: part.data,
        type: 'blob',
      }
    } else {
      return {
        data: String(part),
        type: 'string',
      }
    }
  })
  const size = items.reduce((acc, curr) => {
    if (curr.type === 'string') {
      return acc + global.unescape(encodeURI(curr.data)).length
    } else {
      return acc + curr.data.size
    }
  }, 0)

  BlobModule.createFromParts(items, blobId)

  return BlobManager.createFromOptions({
    blobId,
    offset: 0,
    size,
    type: options ? options.type : '',
    lastModified: options ? options.lastModified : Date.now(),
  })
}

URL.createObjectURL = function createObjectURL(blob) {
  if (BLOB_URL_PREFIX === null) {
    throw new Error('Cannot create URL for blob!')
  }
  return `${BLOB_URL_PREFIX}${blob.data.blobId}?offset=${blob.data.offset}&size=${blob.size}`
}

I notice that react-native adds special Blob handling to XMLHttpRequest (source) with similar mechanisms in FileReader (facebook/react-native#22681 (comment)), but this will need to be implemented in a polyfill here since user-land code will assume functional behavior. This notably won't work for code that uses fetch or other means like expo-asset. All of these problems stem from facebook/react-native#22681, which was unfortunately closed as stale.

Combining XMLHttpRequest with FileReader lets me get a handle on the blob data after it's converted into a URI, with EXGL giving special treatment to { localUri } for image data. This correctly implements Blob up to this point, and Image.getSize reports the right dimensions. I'm getting OOM crashes from EXGL when passing large data urls through JSI, so they're offloaded to the filesystem as a uri. Performance still needs to be measured and tweaked, but this finally solves this issue with a path to upstream fixes. Native support should remove the overhead of base64 encoding back and forth.

Show code:

// from glTF loader, passed to TextureLoader
const uri = URL.createObjectURL(new Blob([...], { type: '...' }))

const blob = await new Promise((res, rej) => {
  const xhr = new XMLHttpRequest()
  xhr.open('GET', uri)
  xhr.responseType = 'blob'
  xhr.onload = () => res(xhr.response)
  xhr.onerror = rej
  xhr.send()
})

const data = await new Promise((res, rej) => {
  const reader = new FileReader()
  reader.onload = () => res(reader.result)
  reader.onerror = rej
  reader.readAsText(blob)
})

const localUri = `data:${blob.type};base64,${data}`

const texture = new THREE.Texture()

const { width, height } = await new Promise((res, rej) =>
  Image.getSize(asset.localUri, (width, height) => res({ width, height }), rej),
)

texture.image = {
  data: { localUri },
  width,
  height,
}
texture.flipY = false
texture.unpackAlignment = 1
texture.needsUpdate = true

// @ts-ignore
texture.isDataTexture = true

[codesandbox-ci]

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

Latest deployment of this branch, based on commit 450774f:

Sandbox	Source
example	Configuration

[XantreDev]

Looks great. Are you tested it on Android release mode.

[XantreDev]

Are you have plans to make some kind of prerelease npm tag with this fixes. Or please say when it will be released?)

[wangzuo]

It seems patching Blob will break existing fetch behaviour.

[CodyJasonBennett]

Can you detail here or create an issue? I've been upstreaming that in particular to react-native and didn't harden fetch since it does not interact with the native BlobManager -- only XMLHttpRequest does. That is something else I want to address.

[Off2Race]

Below asset is from https://github.com/KhronosGroup/glTF-Sample-Models/blob/master/2.0/Duck/glTF-Binary/Duck.glb, rendered on iOS emulator.

Hi, @CodyJasonBennett – My apologies for a slightly off-topic question but how are you rendering this on an iOS Simulator? I thought it was necessary to use actual iOS and Android devices for testing. In fact, when I try to use the simulator, all I get is a blank screen.

[CodyJasonBennett]

I noticed that simulator occasionally pushes frames so I reloaded the app until one was presented.

[Off2Race]

I noticed that simulator occasionally pushes frames so I reloaded the app until one was presented.

Thanks! I'll try that myself

[corysimmons]

Whoa, is this in? Can we do something like this now?

import { RNGltf, RNCanvas } from '@react-three/fiber'
import { View } from 'react-native'

<View>
  <RNCanvas>
    <RNGltf src="/foo.gltf" />
  </RNCanvas>
</View>

Also, does this work in both iOS and Android? Is it performant?

Sorry, I'm dying to use 3D in React Native. 😅

[CodyJasonBennett]

See https://docs.pmnd.rs/react-three-fiber/getting-started/installation#react-native. Same as web, although /native targets give you correct IntelliSense.