3D Tiles - Request scheduling improvements

[lilleyse]

• Time slice tiles that are processing
• Explicitly cancelling requests that go out of view
• Cull received tiles that are now out of view
• Tweaks for better http2 performance. Some notes in Tile Request Prioritization #5317
• Stagger renders to allow more request throughput: https://groups.google.com/forum/#!topic/cesium-dev/WQ4mgLcoBqI
• Debug visualizations to see load order.
  • Colorize by load order
  • Secondary camera see loaded tiles from a better vantage points
• Statistics / tracking improvements
  • Statistics for average load time
  • Statistics for number of cancelled requests
• Evaluate performance for smaller or larger tile sizes
• Prioritize based on center of screen: Request performance improvements #6243 (comment)
• How to mix and match different prioritization modes. Additive tiles use distance, replacement tiles use SSE, terrain uses its own priority. Is there any way to normalize all values to a 0-1 range?
• Fix 3D Tiles starving terrain/imagery: RequestScheduler starves imagery requests for 3D Tiles #5954
• Implement Request pool to avoid allocating many Request objects.

[loshjawrence]

Time slicing and/or sorting of recieved tiles ended up not being something worth even investigating since there wasn't really anything to time slice or sort. If you monitor the processing queue it never really fills up since it drains very quickly. On a rare occasion it would get up to 10 things to process but again those would drain quickly. I threw some sorting and time slicing in there anyway and there was not benefit.
The cancelling of recieved tiles ended up begin another path no worth exploring since there wasn't anything to cancel to begin with(only single digits even for long complicated flight paths), since request canceling was doing its job.

Request canceling (whether going out of view or artificially raising the maxSSE while cam is moving so they don't even get request to begin with) was weird because while the cancel numbers are good (for some flights you could cancel 1/3 of the requests) The total load times are pretty much the same, maybe 10% depending on the scenario.
Like Patrick said, total load time isn't the only metric to judge whether something is a win, these are tiles the clients don't have to load and pay for. In more realistic use cases,in master you can see that the scene settled but the finished timer doesn't go off because some off screen stuff is still loading.
It should be said that the current request throttling mimics request culling in a way by making sure the request queue isn't bloated at any one point. Like if your flying over stuff the request queue only ever gets filled with cheap-to-load LODs that can totally be loaded on the fly anyway. There are several thousands of requests that get canceled before making onto the active queue because of the current request throttler. Master also has an advantage of prefetching some of the final dest cheap LODs because theres no maxSSE bump(like in the branc) and the cam frustum will touch those before the camera stops.

[ ] Going of the observation, perhaps we can extend niche use case of prefetching the flyTo spot to more useful general case by prefetching tiles where the cam will be in N Frames or just the final camera drift stopping point. This will be slightly nuanced since it will interfere with offscreen canceling so those tiles may need a pretched tag and this tag should be removed if they are no longer on the flight path or if the user stops the camera suddenly while it was drifting. Or revamp the frustum culling so that its a swept volume instead of just the frustum for this frame. If doing swept volume, will need another priority for tiles in the near sweep vs far sweep.

Didn't see much advantage for http2 (maybe this will come into play more when I experiment with smaller tile sizes).

"Stagger renders for better request throughput" seems like it's missing the big picture, it optimizes for total load time at the expense of perceived perf, it will have even worse pop-in and longer delays between visual updates, while it maybe faster it will feel slower. Tabling this for now.

Priority colorization is displayed as a heat map(black loads first, white loads last). This is done with with monitoring min and max values of your priority so that you can get a tight bounds on the exposure mapping that is used to bring it down to 0-1 range. I didn't go with filmic or something that can linearize it a little better because we actually do want smearing of lower values(higher priorities) so that they can be better distinguished (bigger relative jumps between values when they get mapped down into 0-1) (and it's totally ok if there "z-fighting" on higher values (much lower priorities, stuff in the background) when the get mapped down because its absolutely a trade-off worth making). Freeze frame is sufficient to get a grasp of whats going on and having a dynamic secondary camera would be to much effort for a value that is already realized with freeze frame. Tabling for now.

I wanted prioritization to look like a heat diffusion from the camera, so this was done by adding the view space distances of distance to near plane and distance to screen center plane. Because space is distorted stuff near to the camera but on also near edge of the screen might have similar priority to something further back but in the center of the screen. for each z slice priority should fall off to the side

For mixing matching priorities I'm currently trying to map it like a number system. say you had 3 priorities that you wanted to distinguish and there were clear breaks between each. since the request scheduler sees low values as higher priorities, the highest priority category would be you 100's digit, next highest 10's digit and lowest would be 1's digit. Then you can tone map the value knowing the range of values are between 0-1000 and then do 1-tonemapped result. If there aren't clear breaks between priorities then they'll need to bleed into each other a bit with a more careful weighting scheme.

Starvation was done by separating out the queues into each category then swiping across the heads of the queues and pushing them the active request queue. Unifiying the priorities also helped for different sources that fall into the same category (ex: nyc photogram, nyc 3d buildings both fall into the 3d tiles bucket (apparently no good way to distinguish between sources inside requests scheduler))

Currently looking into percieved perf stuff at the moment, have a tileset for investigating smaller tiles sizes but was hitting the 'nomralized result is not a number' error. Tabling average load time stats until those are investigated.

[loshjawrence]

Priority colorization from point of view of camera (black/blue highest priority)

[loshjawrence]

Priority colorization from top down view of camera frustum

[loshjawrence]

Before and after of load time colorization heat map (again, it's reversed: black/blue=first load, red/white=last load) colorized by time since new scene load.

[lilleyse]

@loshjawrence just a reminder to also post the gif compare.

[loshjawrence]

Compare gif before -> after. Scene takes about 20 sec on a 20Mbps bandwidth. Takes about 7sec for the right to resolve its foreground(blue part), master takes whole time.

[loshjawrence]

Just adding a note about google earth. protocol is "http/2+quic/43" ours is "http2 spdy".Typical request size is in the 10's of KB (Ours are in the 100's). Most of it is x-protobuf, available for use by others as well. Doesn't seem like they do any fancy feature detection for placing their vertices in the most important spots to reduce mesh size (An example of what I'm talking about would be something like the paper
Feature Preserving Mesh Generation from 3D Point Clouds Nader Salman, Mariette Yvinec, Quentin Mérigot but you would not put all those vertices in the low freq spots (see their images)).

[loshjawrence]

Prefetching before->after
FoveatedSSE: off
Flight: linear, 6 sec
Bandwidth: 30Mbps

[loshjawrence]

Foveated SSE Max Interp val: 128

[loshjawrence]

non-skip lod alpha blending, instant zoom

[loshjawrence]

non-skip lod alpha blending, instant zoo

m

[loshjawrence]

non-skipLOD alpha blending, manual zoom