Skip to content

Commit

Permalink
Merge pull request KhronosGroup#1 from omigroup/khr-audio-feedback-ad…
Browse files Browse the repository at this point in the history
…justments

factors in KHR PR feedback and fixes readme
  • Loading branch information
antpb authored Oct 19, 2023
2 parents 55084f3 + 9ca359e commit 3bed830
Show file tree
Hide file tree
Showing 8 changed files with 360 additions and 347 deletions.
193 changes: 110 additions & 83 deletions extensions/2.0/Khronos/KHR_audio/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,13 @@

## Contributors

* Robert Long, Element Inc.
* Anthony Burchell, Individual Contributor
* K. S. Ernest (iFire) Lee, Individual Contributor
* Michael Nisbet, Individual Contributor
* humbletim, Individual Contributor
* Norbert Nopper, UX3D [@UX3DGpuSoftware](https://twitter.com/UX3DGpuSoftware)
- Robert Long, Element Inc.
- Anthony Burchell, Individual Contributor
- K. S. Ernest (iFire) Lee, Individual Contributor
- Michael Nisbet, Individual Contributor
- humbletim, Individual Contributor
- Norbert Nopper, UX3D [@UX3DGpuSoftware](https://twitter.com/UX3DGpuSoftware)
- Aaron Franke, The Mirror Megaverse Inc.

## Status

Expand All @@ -27,73 +28,79 @@ Audio emitter objects may be added to 3D nodes for positional audio or to the sc

```json
{
"extensions": {
"KHR_audio": {
"emitters": [
{
"name": "Positional Emitter",
"type": "positional",
"gain": 0.8,
"sources": [0, 1],
"positional": {
"coneInnerAngle": 6.283185307179586,
"coneOuterAngle": 6.283185307179586,
"coneOuterGain": 0.0,
"distanceModel": "inverse",
"maxDistance": 10.0,
"refDistance": 1.0,
"rolloffFactor": 0.8
}
"extensions": {
"KHR_audio": {
"emitters": [
{
"name": "Positional Emitter",
"type": "positional",
"gain": 0.8,
"sources": [0, 1],
"positional": {
"coneInnerAngle": 6.283185307179586,
"coneOuterAngle": 6.283185307179586,
"coneOuterGain": 0.0,
"distanceModel": "inverse",
"maxDistance": 10.0,
"refDistance": 1.0,
"rolloffFactor": 0.8
}
},
{
"name": "Global Emitter",
"type": "global",
"gain": 0.8,
"sources": [1]
}
],
"sources": [
{
"name": "Clip 1",
"gain": 0.6,
"autoPlay": true,
"loop": true,
"audio": 0
},
{
"name": "Clip 2",
"gain": 0.6,
"autoPlay": true,
"loop": true,
"audio": 1
}
],
"audio": [
{
"uri": "audio1.mp3"
},
{
"bufferView": 0,
"mimeType": "audio/mpeg"
}
]
}
],
"sources": [
{
"name": "Clip 1",
"gain": 0.6,
"autoPlay": true,
"loop": true,
"audio": 0
},
},
"scenes": [
{
"name": "Clip 2",
"gain": 0.6,
"autoPlay": true,
"loop": true,
"audio": 1
"name": "Default Scene",
"extensions": {
"KHR_audio": {
"emitters": [1]
}
}
}
],
"audio": [
{
"uri": "audio1.mp3",
},
],
"nodes": [
{
"bufferView": 0,
"mimeType": "audio/mpeg"
}
]
}
},
"scenes": [
{
"name": "Default Scene",
"extensions": {
"KHR_audio": {
"emitters": [0]
}
}
}
],
"nodes": [
{
"name": "Duck",
"translation": [1.0, 2.0, 3.0],
"extensions": {
"KHR_audio": {
"emitter": 1
"name": "Duck",
"translation": [1.0, 2.0, 3.0],
"extensions": {
"KHR_audio": {
"emitter": 0
}
}
}
}
}
]
]
}
```

Expand All @@ -109,8 +116,8 @@ The extension must be added to the file's `extensionsUsed` array and because it
{
"asset": {
"version": "2.0"
}
"extensionsUsed" : [
},
"extensionsUsed": [
"KHR_audio"
],
"scenes": [...],
Expand All @@ -129,7 +136,7 @@ The extension must be added to the file's `extensionsUsed` array and because it

Audio data objects define where audio data is located. Data is either accessed via a bufferView or uri.

When storing audio data in a buffer view, the `mimeType` field must be specified. Currently the only supported mime type is `audio/mpeg` for use with MP3 files. MP3 was chosen due to its wide support across browsers and 3D engines as well as its lossy compression with variable bitrate. Other supported audio formats may be added via another extension.
When storing audio data in a buffer view, the `mimeType` field must be specified. The base specification supports `audio/mpeg` and `audio/wav` MIME types. These were chosen with consideration for the wide support for these types acrosss 3D engines and common use cases. Other supported audio formats may be added via extensions.

Note that in tools that process glTF files, but do not implement the KHR_audio extension, external files referenced via the `uri` field may not be properly copied to their final destination or baked into the final binary glTF file. In these cases, using the `bufferView` property may be a better choice assuming the referenced `bufferView` index is not changed by the tool. The `uri` field might be a better choice when you want to be able to quickly change the referenced audio asset.

Expand All @@ -151,7 +158,7 @@ Audio sources define the playing state for a given audio data. They connect one

#### `gain`

Unitless multiplier against original audio file volume for determining audio source loudness.
Unitless linear multiplier against original audio file volume used for determining audio source loudness.

#### `loop`

Expand All @@ -173,7 +180,7 @@ Positional or global sinks for playing back audio sources.

Specifies the audio emitter type.

- `positional` Positional audio emitters. Using sound cones, the orientation is `+Z` having the same front side for a [glTF asset](https://www.khronos.org/registry/glTF/specs/2.0/glTF-2.0.html#coordinate-system-and-units).
- `positional` Positional audio emitters. Using sound cones, the orientation is `-Z` having the same emission direction as [`KHR_lights_punctual`](https://github.com/KhronosGroup/glTF/tree/main/extensions/2.0/Khronos/KHR_lights_punctual) and [glTF cameras](https://github.com/KhronosGroup/glTF-Tutorials/blob/master/gltfTutorial/gltfTutorial_016_Cameras.md).
- `global` Global audio emitters are not affected by the position of audio listeners. `coneInnerAngle`, `coneOuterAngle`, `coneOuterGain`, `distanceModel`, `maxDistance`, `refDistance`, and `rolloffFactor` should all be ignored when set.

#### `gain`
Expand Down Expand Up @@ -214,7 +221,7 @@ The gain of the audio emitter set when outside the cone defined by the `coneOute

Specifies the distance model for the audio emitter.

- `linear` A linear distance model calculating the gain induced by the distance according to:
- `linear` A linear distance model calculating the gain induced by the distance according to:
`1.0 - rolloffFactor * (distance - refDistance) / (maxDistance - refDistance)`
- `inverse` (default) An inverse distance model calculating the gain induced by the distance according to:
`refDistance / (refDistance + rolloffFactor * (Math.max(distance, refDistance) - refDistance))`
Expand Down Expand Up @@ -243,7 +250,23 @@ Audio emitters of type `global` may be added to scenes using the following synta
{
"extensions": {
"KHR_audio": {
"emitters": [0, 1]
"emitters": [1, 3]
}
}
}
]
}
```

Audio emitters of type `global` may be added to nodes using the following syntax:

```json
{
"nodes": [
{
"extensions": {
"KHR_audio": {
"emitter": 1
}
}
}
Expand All @@ -259,7 +282,7 @@ Audio emitters of type `positional` may be added to nodes using the following sy
{
"extensions": {
"KHR_audio": {
"emitter": 2
"emitter": 0
}
}
}
Expand All @@ -270,6 +293,7 @@ Audio emitters of type `positional` may be added to nodes using the following sy
Note that multiple global audio emitters are allowed on the scene, but only a single audio emitter may be added to a node.

### Audio Rolloff Formula

The Audio Rolloff range is `(0.0, +∞)`. The default is `1.0`.

The rolloff formula is dependant on the distance model defined. The available distance models are `linear`, `inverse`, and `exponential`.
Expand All @@ -279,10 +303,13 @@ The rolloff formula is dependant on the distance model defined. The available di
- exponential formula: `pow((Math.max(distance, refDistance) / refDistance, -rolloffFactor))`

### Audio Gain Units

The gain unit range is `(0.0, +∞)`. The default is `1.0`.

- gain formula: `originalVolume * gain`

### Audio Cone Vizualized

<img alt="Audio cone showing how cone parameters impact volume based on relative distance to the source." src="./figures/cone-diagram.svg" width="500px" />

Figure 1. A modified graphic based on the <a href="https://webaudio.github.io/web-audio-api/#Spatialization-sound-cones" target="_blank">W3C Web Audio API Audio cone Figure</a>
Expand All @@ -291,7 +318,7 @@ The cone properties relate to the `PannerNode` interface and determine the amoun

The gain relative to cone properties is determined in a similar way as described in the web audio api with the difference that this audio emitter extension uses radians in place of degrees. [Cone Gain Algorithm Example](https://webaudio.github.io/web-audio-api/#Spatialization-sound-cones)

### Units for Rotations
### Units for Rotations

Radians are used for rotations matching glTF2.

Expand All @@ -301,14 +328,14 @@ Radians are used for rotations matching glTF2.

## Known Implementations

* Third Room - https://github.com/thirdroom/thirdroom
* Three Object Viewer (WordPress Plugin) - https://wordpress.org/plugins/three-object-viewer/
* UX3D Experimental C++ implementation - https://github.com/ux3d/OMI
- Third Room - https://github.com/thirdroom/thirdroom
- Three Object Viewer (WordPress Plugin) - https://wordpress.org/plugins/three-object-viewer/
- UX3D Experimental C++ implementation - https://github.com/ux3d/OMI

## Resources

Prior Art:
* [W3C Web Audio API](https://www.w3.org/TR/webaudio/)
* [MSFT_audio_emitter](https://github.com/KhronosGroup/glTF/pull/1400)
* [MOZ_hubs_components Audio](https://github.com/MozillaReality/hubs-blender-exporter/blob/04fc1d1/default-config.json#L298-L324)

- [W3C Web Audio API](https://www.w3.org/TR/webaudio/)
- [MSFT_audio_emitter](https://github.com/KhronosGroup/glTF/pull/1400)
- [MOZ_hubs_components Audio](https://github.com/MozillaReality/hubs-blender-exporter/blob/04fc1d1/default-config.json#L298-L324)
65 changes: 31 additions & 34 deletions extensions/2.0/Khronos/KHR_audio/schema/audio.schema.json
Original file line number Diff line number Diff line change
@@ -1,42 +1,39 @@
{
"$schema": "http://json-schema.org/draft-04/schema",
"title": "Audio Source",
"type": "object",
"description": "A source for audio data referenced in Audio Emitters Audio can be referenced by URI or `bufferView` index. `mimeType` is required when `bufferView` is used.",
"allOf": [ { "$ref": "glTFProperty.schema.json" } ],
"properties": {
"uri": {
"type": "string",
"description": "The uri of the audio file.",
"format": "uriref",
"gltf_detailedDescription": "The uri of the audio file. Relative paths are relative to the .gltf file.",
"gltf_uriType": "audio"
},
"mimeType": {
"type": "string",
"description": "The audio's MIME type. Required if `bufferView` is defined. Unless specified by another extension, the only supported mimeType is `audio/mpeg`.",
"anyOf": [
"$schema": "http://json-schema.org/draft-04/schema",
"title": "KHR_audio Audio Data",
"type": "object",
"description": "A source for audio data referenced in Audio Emitters Audio can be referenced by URI or `bufferView` index. `mimeType` is required when `bufferView` is used.",
"allOf": [{ "$ref": "glTFProperty.schema.json" }],
"properties": {
"uri": {
"type": "string",
"description": "The uri of the audio file.",
"format": "uriref",
"gltf_detailedDescription": "The uri of the audio file. Relative paths are relative to the .gltf file.",
"gltf_uriType": "audio"
},
"mimeType": {
"type": "string",
"description": "The audio's MIME type. Required if `bufferView` is defined. Unless specified by another extension, the only supported mimeType is `audio/mpeg`.",
"anyOf": [
{
"enum": [ "audio/mpeg" ]
"enum": ["audio/mpeg"]
},
{
"type": "string"
}
]
},
"bufferView": {
"allOf": [ { "$ref": "glTFid.schema.json" } ],
"description": "The index of the bufferView that contains the audio data. Use this instead of the audio source's uri property."
},
"name": { },
"extensions": { },
"extras": { }
},
"dependencies": {
"bufferView": [ "mimeType" ]
},
"oneOf": [
{ "required": [ "uri" ] },
{ "required": [ "bufferView" ] }
]
},
"bufferView": {
"allOf": [{ "$ref": "glTFid.schema.json" }],
"description": "The index of the bufferView that contains the audio data. Use this instead of the audio source's uri property."
},
"name": {},
"extensions": {},
"extras": {}
},
"dependencies": {
"bufferView": ["mimeType"]
},
"oneOf": [{ "required": ["uri"] }, { "required": ["bufferView"] }]
}
Loading

0 comments on commit 3bed830

Please sign in to comment.