Terrain and imagery roadmap

[kring]

Cesium already has excellent support for streaming terrain and imagery, but there are lots of ways we can improve it. This issue catalogs the many ideas and possibilities. Have ideas or want to get involved? Introduce yourself on the development mailing list.

Ideas and Plans

Items with a ⭐ are higher priority and likely to happen in the not-too-distant future.

Rendering

• ⭐ Do per-pixel imagery LOD selection, blending between adjacent LODs. This should drastically reduce hard lines between LODs when the imagery provider has a very different appearance at different LODs.
• ⭐ Terrain tile selection in Columbus View does not closely match 3D. For example, it's hard to get the most detailed version of Mt. Everest in Columbus View.
• Sort tiles using the natural ordering of a quadtree rather than explicitly sorting tiles by distance.
• Use multiple passes to successfully render tiles even if the number of textures exceeds the number of texture units supported by the WebGL stack (minimum: 8). May also need to render in multiple passes if we use too many uniforms.
  • Use blending in the render state to support layer alpha.
• For better performance, we can minimize the complexity of fragment shading so only the most complex shading is done once. Ignoring alpha, a layer can be seen as a diffuse map (which may need to go through a brightness filter given how dark some tiles are). Instead of doing full lighting and atmosphere when rendering each tile, we can simply render the diffuse component, and then in a final pass, we can perform lighting with specular map, bump map, etc.
  • This is missing a lot of detail.
  • Of course, we could also lay down z first, but the CPU overhead could be too high.
• Shade surface with ambient occlusion, either precomputed and stored with terrain data or computed in screen space on the fly.
• Interesting ways to morph terrain during transitions between 2D, 3D, and Columbus View?
• Show terrain in 2D by bump mapping.
• Minimize popping artifacts by morphing between terrain LODs.
• Does laying down Z first improve performance? Horizon views only? Is walking the tree in front-to-back order enough?
• Use a separate shader, without any water code, for tiles that do not contain water.
• Optimize the computation of screen-space error. In theory, we can compute the switching distance once per tile and just compare that, rather than doing a bunch of math per tile per frame.

Culling

• ⭐ Don't compute 2D bounding spheres every frame.
• ⭐ Improve computation of the "occludee point" used for horizon culling. It currently uses a sphere based on the ellipsoid's minimum radius, which is conservative, but using the actual ellipsoid will allow more tiles to be culled. Also, it currently ignores terrain. These two wrongs seem to cancel each other out well enough that I've never seen artifacts from it, but the horizon culling is probably not optimal.
• ⭐ Frustum cull more accurately than with a tile bounding sphere. Bounding spheres don't fit tiles very tightly. Some ideas here: http://outerra.blogspot.com/2012/11/view-frustum-culling-of-sphere-mapped.html
• OBB improvements from Oriented bounding box & frustum culling optimizations #2782
  • Hook up to heightmapped terrain (and flat terrain, if that's separate); not just quantized meshes
  • Hook up to 2D and Columbus view rendering modes

Interaction with other parts of Cesium

• Vector Data on Terrain Vector Data on Terrain #2172
  • ⭐ Render polygons on terrain - probably using shadow volumes
  • ⭐ Render polylines on terrain - perhaps see if we can use our SIGGRAPH work without a geometry shader.
  • ⭐ How should billboards interact with terrain? We generally expect them to be get hidden behind hills, but when their bottoms get clipped off because they're below terrain it looks like a bug.
  • Notifications when LOD under a point switches, so the point can be kept on terrain?
• ⭐ Allow imagery and terrain providers to specify the credit as a simple string, so they don't all have to draw strings to a canvas as they currently do.
• ⭐ Add support for programmatic height queries.
  • Pick on terrain and get exact position.
  • Query height at point. Get back exact height or height according to currently rendered LOD?
• ⭐ Don't allow the camera to go below the terrain.
• Layer primitives on the ground - at least polylines and polygons - seamlessly with imagery (Vector Data on Terrain #2172). Many people think in terms of vector layers, KML layers, etc. Everything is a layer. We should support this, but shouldn't lose sight of our focus on air and space where this paradigm breaks down.
• Avoid duplicate credits/logos when two layers from the same provider are shown, or if two providers have identical credits.
• Show/hide layers based on viewer height(careful in 2D and Columbus view) or time? This may be done in Dynamic Scene, not Scene, but we need to think about it. More general display conditions could also be useful.
• Ability to move - and possibility rotate - a texture on the globe so it better lines up with where the user expects it, e.g., with building models. To start, we need to decouple the extent the texture thinks it has with the extent that it is rendered in.

Loading

• ⭐ Support skipping intermediate tiles in the tile pyramid. For example, if the camera starts zoomed into, we should be able to load the high-res tiles immediately rather than refining down through the tree first and waiting at each level for the previous level to complete.
• ⭐ Combine multiple images from the same layer into a single texture at load time. This should substantially improve performance.
• ⭐ Add a callback or some other way to learn that level 0 terrain/imagery is done loading.
• Consider baking multiple layers into a single texture per geometry tile. The downside to this is that it will be a performance hit when adjusting layer order, alpha, gamma, etc.
• Handle other projections? UTM?
• Many APIs can restrict the zoom level. I assume it's useful - it's certainty trivial to expose in our code - but I'm not sure of the use cases. Performance?
• Fix blurriness in 2D and Columbus View when using Mercator imagery and a Mercator projection. We'll probably do this by keeping the original Mercator images around for the first couple of levels. Currently we do a round-trip from Mercator->Geographic->Mercator.
• If a texture is completely occluded by opaque layers higher in the z-order, it does not need to be rendered. Actually, it doesn't need to be requested from the server. Hmm - could a proxy server do some compositing for us?
• Upsample TerrainMesh instead of TerrainData? This could yield better performance, but at a memory cost because the mesh is bigger than the raw data, at least for heightmap terrain.
• Allow upsampling across more than one level at a time. This (may?) be necessary to skip levels when refining.
• More efficiently encode terrain data for transmission:
  • Use the Google Body techniques, encoding the mesh as a UTF-8 string: http://code.google.com/p/webgl-loader/wiki/BenCompressionStats
  • Encode each tile as a delta from its parent:
    • Parent tile indicates which of its vertices are in which quadrant with index ranges.
    • Child has new vertices, index list linking parent and child vertices into triangles, and new index ranges describing the division of its vertices among its child quadrants.
• When we've exceeded the actual terrain detail available, generate it procedurally using, for example, fractional Brownian motion: http://design.osu.edu/carlson/history/PDFs/p371-fournier.pdf
• When upsampling terrain by subsetting it, can we avoid copying heights to a separate TypedArray and instead pass a range to the HeightmapTessellator? The Web Worker will copy the whole thing anyway, so this may or may not be worthwhile.
• Combine children to minimize the number of separate requests.
• Avoiding "blinking" blue tiles when removing one base layer and adding another. Similarly, when a layer is added on top of the base, we should avoid showing what's under it temporarily before the layer loads. Both of these artifacts happen because tiles continue to be renderable after layer changes even though the tile's layers aren't fully loaded.

Data Sources

• ⭐ Many terrain sources, possibly including VR-TheWorld, provide heights relative to mean sea level (MSL) instead of relative to the WGS84 ellipsoid. Cesium should apply the correction on the client.
• ⭐ Consider using this high-res, vector coastline dataset to create the water mask: http://shoreline.noaa.gov/data/datasheets/wvs.html
• New imagery providers
  • NASA GIBS
  • Web Map Tile Service (WMTS)
  • ArcGIS ImageServer
  • VR-TheWorld
• New terrain providers
  • ArcGIS MapServer
• Improvements to existing providers
  • ArcGisMapServerImageryProvider
    • Read the extent and maybe tiling scheme from non-tiled services, instead of assuming global extent and geographic projection.
    • Support non-default, non-worldwide tiling schemes for tiled services. http://webhelp.esri.com/arcgisserver/9.3/java/index.htm#what_is_map_caching.htm

Additional types of layers and effects

• Ability to draw a layer at a fixed altitude above the ellipsoid.
• ⭐ Separate water mask from terrain so the two can come from different sources. This will let us have fancy water even when terrain providers other than CesiumTerrainProvider are in use.
• Do we need layers for night lights? Why not layer a day image over night? In general, we need to revisit the architecture for night, bump, specular, clouds, etc. Clouds may happen sooner rather than later.
• Image processing to fix images, e.g.,
  • Bing Maps imagery is really dark, so brighten it. (done)
  • Landsat imagery is usually 11 or 12 bits, and loses its dynamic range when dropped to 8-bits.
  • For Landsat multispectral imagery, we might want to color one band.
  • Histogram, gamma corrections (done), etc.
  • We should do this in a fragment shader so the user can adjust it with sliders.
  • There can also be a fast path for final adjustment where the processed textures are saved; however, to start adjusting again, we will need the source data.
• Show/hide layers based on viewer height(careful in 2D and Columbus view) or time? This may be done in Dynamic Scene, not Scene, but we need to think about it. More general display conditions could also be useful.
• Blend maps for layers, e.g., specular, dirt, or destruction maps. I don't have any use cases, but these are certainty popular in games; however, the shading for each layer is different, i.e., they are not just diffuse components.
• Add support for tiled normal maps.
• Procedural shading by height, slope, etc.
• Shadows. Check out: http://organicvectory.com/index.php?option=com_content&view=article&id=54:shadow-mapping-on-large-terrain&catid=1:articles&Itemid=61

Resources

Other APIs with Layers

• Insight3D - Imagery Overlay and Web Imagery Overlays.
• OpenWebGlobe - Tutorial and reference.
• WebGL Earth - See WebGLEarth.Map in the API.
• ReadyMap - Layers demo.
• Glob3 - Layers example.
• OpenLayers - doc; reference doc; and lots of examples such as layer opacity.
• Leaflet - doc and examples.
• ArcGIS - See Layer in the doc and samples.

Sources of data

• Terrain
  • National Elevation Dataset - High quality terrain data for the conterminous United States, Alaska, Hawaii, and territorial islands. Resolution as high as 3 meters for parts of the U.S.
  • ASTER Global Digital Elevation Map - 30m resolution data for most of the world.
  • Shuttle Radar Topography Mission (SRTM) - 90m resolution data for most of the world. There's also a nicely-processed version of available from CGIAR-CSI, but special permission is required to use this processed version commercially.

Papers

In addition to the work mentioned in our book, these recent papers could be worth a read:

• Interactive Editing of GigaSample Terrain Fields. 2012. Uses GPU ray casting. Compression part could be interesting. Didn't look into what they are using CUDA for, but we don't want to rely on WebCL.
• On-the-Fly Decompression and Rendering of Multiresolution Terrain. 2010. Uses a geometry shader but still potentially useful.
• Terrain Rendering with the Combination of Mesh Simplification and Displacement Mapping. 2010. Sounds like combining low resolution mesh with higher resolution displacement maps, but I only read the abstract.
• GPU-Aware Hybrid Terrain Rendering. 2010. Interesting work, but I'm not sure how to efficiently map GPU ray casting to an ellipsoid.
• Rendering Large Terrains in Real-Time. 2007.
• C-BDAM - Compressed Batched Dynamic Adaptive Meshes for Terrain Rendering. 2006.
• Out-of-core terrain rendering with reparameterized textures. 2004. Could be interesting for improving texture quality on steep cliffs.

[mramato]

First, this is an awesome roadmap and I like the idea of using a Github issue (with the new checkboxes) over the wiki. Thanks for putting it together.

Second, what am I missing that you have VR-TheWorld imagery provider as unimplemented under the Data Sources section? We can already use VR-TheWorld imagery via WebMapServiceImageryProvider

[kring]

Thanks, Matt. I think if we implemented an actual VRTheWorldImageryProvider, we'd get better performance because we could access the tiles directly. Using the WMS provider involves some amount of server-side processing.

[mramato]

That makes sense, I always forget that WMS involves extra processing rather than just being a tiling scheme.

[kring]

One downside to using issues for the roadmap: when someone edits the issue, there's no notification or tracking of history. It's a little unfortunate that GitHub has such disjoint feature sets for issues and wiki pages.

[mramato]

That's true. But I like having the neon purple roadmap tag on the issues page. Maybe we should do something like have an issue for each major roadmap item that just links to the wiki page? That way we can have public, archived discussion (the create issue) but history and notification via the wiki (for the actual roadmap). Thoughts? I'm thinking of writing up a roadmap for widgets and it would be nice if we all followed the same patterns.

[pjcozzi]

Maybe we should do something like have an issue for each major roadmap item that just links to the wiki page?

Yeap, organically over time as existing ones need attention.

We still need the dev list to at least funnel discussion to the issue.

[Carl4]

Not sure if this goes here or elsewhere, but it appears to me that the units aren't captured correctly. By this, I mean that I put an object (say an ellipsoid) where I believe the peak of Mount Everest should be, turn on terrain, and find that my ellipsoid is a sizable distance above the top of Mount Everest. I'm fairly confident that my lla to ecf conversion is right. Maybe I'm missing something in the docs?

[kring]

@Carl4, you need to specify the height as meters above the WGS84 ellipsoid, not meters above mean sea level. If that's not the problem, can you tell me a little more about where you're getting the height data and how you're converting it to ECF?

[Carl4]

@kring Seemed like we were going down a rabbit hole, so I turned it into a stackoverflow question.

[kring]

Thanks @Carl4, I responded over there. In the future, our forum (http://cesium.agi.com/forum.html) might be a good choice for questions like this as well.

[kainino0x]

Under "OBB improvements from #2782":

• Hook up to heightmapped terrain (and flat terrain, if that's separate); not just quantized meshes

This can be checked off; it was added to the PR.

[pjcozzi]

• Performance idea: to save GPU load (and maybe some CPU load in the browser/driver, but I doubt it), if a tile is surrounded by tiles of the same LOD, use a different count in drawElements to not draw the skirts (assuming they are appended to the end of the index buffer). Are the meshes watertight?

[kring]

Are the meshes watertight?

They weren't when I did it. Alex may have changed that. But in general I don't think quantized-mesh guarantees it.

[pjcozzi]

If this optimization is worth it (I dunno, it could be significant for horizon views at 4K where the skirts make the depth complexity really high) then we can add it to a 1.0.1 spec.

[bagnell]

• Quantize more/less and pack terrain meshes at different levels of the tree.

[pjcozzi]

• Texture compression (for imagery) - perhaps still send jpg over the wire and immediately upload it to a texture to reduce latency, but then DXT compress it in a web worker and replace the texture after compression. Plan B - just send compressed texture over the wire.

[wallw-teal]

Many APIs can restrict the zoom level. I assume it's useful - it's certainty trivial to expose in our code - but I'm not sure of the use cases

This allows clients to set up multiple layers as the base map imagery. Something like NASA's Blue Marble is only useful at zoom levels 0-5, where you might then use World Street Map from 5-15, and then World Imagery from 15 on. In addition, our client allows the users to modify the min/max zoom level to meet their needs, which lets them potentially "mix" layers by dropping the opacity on one so that the other shows through.

[rahwang]

Forum interest in support for rendering imagery layers above the ellipsoid surface.

https://groups.google.com/d/msg/cesium-dev/mW0VxrU3yhA/jcqoRnu3BAAJ

[florentdu]

Hello,

I'm very insterested in providing altitudes in MSL reference instead of WGS84 reference.

Could you confirm the item "Many terrain sources, possibly including VR-TheWorld, provide heights relative to mean sea level (MSL) instead of relative to the WGS84 ellipsoid. Cesium should apply the correction on the client." will address this issue ?

Have you got any idea when this functionnality will be implemented ?

Thank you very much for you work !

[hpinkos]

Hello @florentdu, Cesium has support for both the heightmap and quantized mesh terrain formats. Your terrain must be processed into one of these open formats. AGI (the company that employs most of the Cesium team) offers a commercial product called the STK Terrain Server to process your raw terrain data into the quantized mesh format. This feature will also be available as part of the upcoming cesium.com application. If you have any follow up questions, please ask them on our forum. Thanks!

[florentdu]

Hello @hpinkos and thank you for your answer. I'm sorry if I was not clear but my question was not related to terrain. I have flight data with mean sea level (MSL) altitudes whereas cesium use altitude above ellipsoid. My question was : do you plan to implement MSL altitude <-> ellipsoid altitudes conversion in a near future.

Some guys use GeographicLib to do the job. I succeeded to do it with this small library geoidheight but I am not sure I got the best performance.

Thank you

[kring]

Hi, We have a class in TerriaJS to compute geoid heights. Feel free to use it. https://github.com/TerriaJS/terriajs/blob/master/lib/Map/EarthGravityModel1996.js Kevin

…

On Jul 11, 2017 12:01 AM, "florentdu" ***@***.***> wrote: Hello @hpinkos <https://github.com/hpinkos> and thank you for your answer. I'm sorry if I was not clear but my question was not related to terrain. I have flight data with mean sea level (MSL) altitudes whereas cesium use altitude above ellipsoid. My question was : do you plan to implement MSL altitude <-> ellipsoid altitudes conversion in a near future. Some guys <https://cesiumjs.org/2015/10/26/CloudAhoy-Migration-to-Cesium/> use GeographicLib <https://sourceforge.net/projects/geographiclib/> to do the job. I succeeded to do it with this small library geoidheight <https://github.com/vandry/geoidheight/blob/master/geoid.js> but I am not sure I got the best performance. Thank you — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#526 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4a1qqS3M9RD6fVx92zCMaPEfySuB0uks5sMi6NgaJpZM4AcIsW> .

[thw0rted]

I noticed that "How should billboards interact with terrain" is checked off, but I believe the default behavior is still pretty ugly. Is this still being worked? I saw a few other billboard/terrain issues but I'm not sure what's being treated as a priority.

[pjcozzi]

@thw0rted this is not currently being worked; however, contributions are very welcome if you have the bandwidth. Otherwise, please add any info you have in the specific issues or create new issues.

[runaas]

I see from your own qmesh terrain tiles that you generate "geometricerror", "leaf" and "surfacearea" metadata fields.
Like for example: {"geometricerror":0.6018822311607437,"leaf":true,"surfacearea":1478369.140916789}

In the source code it seems like those fields are not used (I can only find the "available" field in use). However, I would believe individual tile-specific geometric error could improve efficiency and wonder if you intend to use this in the client?

[kvandreev]

Hello. I don't quite understand what elevation data is used for terrain generation. If it is SRTM, then these heights are plotted on the EGM 96 geoid, which corresponds to sea level. Or are you converting from a geoid to an ellipsoid?