New Entity Framework

[netpro2k]

This PR introduces the foundations for the new framework for writing core "client engine" code in Hubs.

To prove things out this PR ports the media frame and the camera tool functionality. These are now built entirely without using any AFRAME entities/components/systems and using a new networking system. This also required rewriting the interactions system as it needed to be able to inter op with both new and legacy entities.

This is just the beginning steps and things will likely be changing quite a bit in the near term while we continue to flesh things out. More documentation and examples will happen after things settle down into a more steady state.

Motivations

• The current Hubs client code is quite tricky to reason about. One of the biggest contributing factors is that AFRAME sits between us and ThreeJS adding a bunch of incidental complexity. We have largely outgrown the usefulness of many of the things AFRAME trades off this complexity for (namely having a "HTML like" API and being "easy to get started without writing any code"). AFRAME continues to be a useful tool, but no longer the right one for our usecase.
• Components having behavior and "lifecycles" coupled with the custom element browser API makes it especially hard to reason about the execution order of things. This is made worse by the "events" pattern that is quite common in AFRAME components. This leads to a whole mess of race conditions and defensive code to handle the various order things may happen to execute in.
• AFRAME also introduces a significant number of superfluous Object3Ds into the scene graph, since every AFRAME entity is also a THREE.Group node. Updating matrices remains one of the more expensive things in our main loop, so cutting down on the number of matrices that need to be updated is important. We should be able to do better if we are more opinionated and restrictive about what a "component" can be.
• Our networking system is tied to AFRAME and so inherits a lot of these problems. It also means we need a new one if we want to move off of AFRAME.

Goals

• Make hubs client code easier to reason about. Particularly with regards to execution order.
• Minimize unnecessary scene graph nodes
• Minimizing dynamic allocations and garbage collection
• Prefabs should be easy to edit and version control
• Networking should be mostly "dumb" on the server side and be amicable to binary encoding over data channels
• Transition must be able to be done alongside existing code

Non-Goals

• Making it "easy" for people with little or no game development experience to "get started". We are optimizing for simplicity and a lower total ceiling on learning curve, not a shallower learning curve. Our target audience are people like ourselves and people doing deeper customization to the client. While we don't intend to make things intentionally difficult, we are optimizing for long term development, not one off experiments.
• Abstracting away ThreeJS. We rely heavily on ThreeJS and intend to use it for the forseeable future, so we mostly embrace that rather than trying to abstract it. This may change somewhat as we move to try and multithread things in the future.

High Level Plan

• All AFRAME entities are now also BitECS entities. Several of the simpler AFRAME components are also mirrored as BitECS components.
• New code should be written in terms of BitECS entities and components whenever possible, only reaching into AFRAME entities for interop with legacy code. (Using ThreeJS is fine and expected)
• For networking, all AFRAME entities continue to use NAF, while all non-AFRAME entities will use our new networking system
• Gradually phase out use of AFRAME over time by replacing AFRAME components & systems with BitECS
• Goal is not 1:1 replacing each existing component with an equivalent it BitECS. Functionality and especially implementation will be rethought with new constraints. Many things just go away as they were solving for incidental complexity.
• This is likely done one "chunk" at a time for each network template in hub.html, porting the required functionality for each as we go.
• It is ok for some functionality to live in "both worlds" for now. For example we are still using AFRAME slice9 and text components but also have new versions for non AFRAME entities. We will phase out the AFRAME counterparts when they are no longer being used.
• Even once we have ported all of our existing stuff we will likely want to keep AFRAME around as an option in some minimal way for quite some time to give custom client developers a gradual migration path.

Some things worth noting

• "entities" are just ids, "components" are just data, and "systems" are just regular functions that happen to be (explicitly) called in some order every frame. See the BitECS Intro for a rundown of the (thankfully quite minimal) API.
• all component properties are just numbers (no strings or object references). If more complex data types are required you can use entity ids to associate other data with them. (ex: see world.eid2obj map)
• all of the component data for all entities you will ever create* during the entire lifetime of the app is allocated at startup (as a bunch of TypedArrays). This leads to a significantly larger upfront memory footprint, but much faster creation, removal, querying, and updating of component data - without any need for garbage collection. We will need to see how this goes. Depending on how big the footprint becomes we will likely want to have different storage strategies for different components depending on usage, but for now we are keeping it simple. (* We actually allocate for 10,000 entities. Going above that will create more data for 20,000, 40,000, and so on as needed.)
• The new networking system uses JSX to define "prefabs". This is roughly equivalent to the "templates" we have today in hub.html but more expressive and importantly not doing any text parsing or interfacing with any custom element lifecycles.
• Though we are using JSX this is not using React, and there should be no expectation of "re-rendering". It is simply a way to describe a desired scene graph of components and entities. They should be though of as semantically equivalent to GLTF files with hubs components, but easier to edit by hand and version control. The object3d property on <entity/> tags is an escape hatch that potentially breaks these semantics, and likely will be phased out as we build out the core library of components.
• The new networking system has roughly the same semantics as NAF with regards to ownership, though "creation" of networked entities is more explicit as is the concept of "scene owned objects". It is currently still using quite noisy JSON serialization over the same socket as NAF, but the plan is to transition it to a purely binary protocol.

Known TODOs

There are many inline TODOs in the new code introduced in this PR. None of them are worth holding back merging these changes but a few are worth noting more directly:

Camera Tool

• camera tool rendering used to run in tock (camera tool is now likely a frame behind)
  • The camera tool renders at a reduced FPS anyway so its quite unlikely this will be noticeable..
  • To fix we should re-order where render happens in our main loop and specifically run some systems before/after it
• basic camera state networking (right now only the position is networked, not the snapping state, countdown, etc).
• no way to rotate camera on mobile or non 6-dof VR (currently hard coded to grab and right mouse button only)
  • We don't want to port object menus since we know they are the wrong solution, we should rrethink these basic interactions and use camera tool as the testbed for them.
• The camera tool UI isn't particularly nice looking. I did not want to port the sprite system yet, so all the buttons are just using text labels. It's unlikely what we want resembles either what we had or what this PR has, so not much time was spent on it.

Netcode

• string identifiers are never cleaned up.
  • We may not want to be using these long term anyway
• The transport layer is piggybacking on NAF. Use a custom phoenix channel, ideally with binary encoding instead.
• NetworkTransorms (used for the camera tool) are not interpolated, so movement of other people's cameras feels "laggy"
  • This is probably best done with a rethinking or at least cleanup of the physics system, as network transforms should be physics-aware
• reconnect and blocking support
• These only apply to the camera tool since it is the only network spawned object currently using this system. Not having this yet feels fairly low impact. This should likely be done as part of the move off of NAF transport.

Media Frames

• Some ugly hacks were done around initializing media frames with the physics system. It likely doesn't matter in practice and fixing it likely requires some pretty deep physics fixes, but this should be kept in mind when we get to fixing physics in general.
• pinned objects don't snap into media frames on load (this is not a regression, but still something we want to support)
• When grabbing an object you do not own that is in a media frame it will snap out of the media frame even if you don't move it out of the bounds. This is due to the physics system briefly not reporting any collisions for that object. This is also not a regression, but something we likely want to fix by fixing the physics system.

Pre Merge TODOS

• merge aframe changes
• merge networked-aframe changes
• update package.json and package-lock.json with the updated commits and restore branches
• revert changes in webpack config. They are needed to turn on physics debugging but break image loading in the media browser locally.

[netpro2k]

This was introduced to simplify camera tool code, but is also something we want for other reasons in the future (namely being able to isolate UI from tonemapping after we add post effects)

[netpro2k]

We technically had this exposed in gltf-component-mappings but it is not used by Spoke or the Blender Exporter so it is highly unlikely anyone would have used it for anything. Simplifying the schema to more directly map to the counterpart in BitECS land.

[netpro2k]

We were previously leaking this geometry. I noticed it while making sure to clean up the geometry in the new system.

[netpro2k]

This changes the default for object rotation to unsapped. This was useful for the camera tool, and is something we have said we should do for a long time, so I decided not to make it conditional.

[netpro2k]

Ultimately GLTF loading will eventually go entirely through the new system of inflators, but for now just exposing media frames here is useful, even if slightly awkward.

[netpro2k]

This will not be a "supported" feature and will likely go away in the future but is a useful way of testing the new networking system.

[netpro2k]

We were previously storing away messages that were not u or um (like the pen drawings) but acting as if they were update messages... This likely caused some of the odd behavior we saw with drawings. The new networking system also runs over one of these custom message types and currently just ignores freeze state

[netpro2k]

Note the concept of what "my camera" is was intentionally simplified. Previously if someone grabbed a camera you spawned we (for some purposes) considered it to be "their" camera (though it would still get deleted if you left the room. Now you can only spawn 1 camera, and that one is always "yours". To remove it you click the button (or press c) again. The camera has no object menu.

[netpro2k]

Doing all the work in 1 spot eliminates the need for something like our "frame scheduler" we have. Instead you just do only exactly the work you want to do each frame.

[netpro2k]

We were previously leaking these. Noticed while working on object cleanup for the new system

[netpro2k]

This is the function JSX transforms to:

<entity foo={{bar: 1}}>
  <entity biz />
</entiti>

translates to:

createElementEntity("entity", {foo: {bar: 1}}, [
  createElementEntity("entity", {biz: true}}, null)
])

For the name, I was going for something like React.createElement but I think this name is confusing. Open to suggestions.

[netpro2k]

This is about as noisy as our current JSON encoding in NAF... It should be quite easy to make this much better when we move to a binary socket.

[netpro2k]

I actually don't have a good idea of why this fix was needed (and if it was even a regression). Without it I noticed that sometimes objects would stick to the cursor and never be dropped)

[netpro2k]

This got suspiciously simple... I am still not 100% convinced some functionality was not lost here... But everything seems to be working correctly.