3D Tiles - Support tile expiration #4136

lilleyse · 2016-07-19T19:03:06Z

Summary:

When a tile is created it looks at expire.duration or expire.date to create an expire date. When the tile's content is downloaded, it will update its expire data if expireDuration is defined.

In selectTiles it checks whether a tile is past it's expire date. If so, the content is marked as EXPIRED and new content is requested. There are two situations now: the request comes back successfully with new content, or the request fails and the tile is unloaded. If the request succeeds, the old resources will be destroyed in each content's initialize stage right before the new resources are created. Likewise if the content is an external tileset, the old subtree will be unloaded immediately before the new subtree is created in loadTileset.

If the request fails, the tile is unloaded. There are three cases for unloading: the tile is a leaf tile, the tile is a non-leaf tile with content, or the tile content is an external tileset. In the first case the tile is unloaded and removed from the tree. In the second case, the content is unloaded and the tile becomes an empty tile - since this tile may still have children it can't be removed. In the last case, the whole subtree is unloaded and removed from the tree.

With the current design there may be a few frames of emptiness between when the new content is downloaded and when it is ready to render. I don't think this will be a problem, but I'll look out for it as I'm testing. The alternative is a messier to handle because then you have to manage two sets of resources - the old ones that are still rendering and the new ones that are loading. Not to mention handling two subtrees when dealing with external tilesets.

To do:

Create a test server for generating new tilesets when old ones expire
Update expire.date when fetching new content: Tile expiration 3d-tiles#99 (comment)
Unit tests
Example tileset for Sandcastle example

Questions:

Should the tileset itself have an expiration? Putting an expiration on the root tile is not the same, since this will only reload the content and not the whole tree.
@mramato brought up caching issues when requesting new content with the same url. Maybe we can append a timestamp or guid to the url to prevent caching.

pjcozzi · 2016-07-20T01:39:14Z

@mramato brought up caching issues when requesting new content with the same url. Maybe we can append a timestamp or guid to the url to prevent caching.

What is standard with KML network links? And is it still a good idea today?

pjcozzi · 2016-07-20T01:43:37Z

Source/Scene/Batched3DModel3DTileContent.js

@@ -154,6 +154,9 @@ define([
     * Part of the {@link Cesium3DTileContent} interface.
     */
    Batched3DModel3DTileContent.prototype.initialize = function(arrayBuffer, byteOffset) {
+        // Destroy expired resources if they exist.
+        destroyResources(this);


Throughout the content providers, this is a bit awkward. Is it the cleanest solution? Would it be better to call destroyResources as part of unloadContentAndMakeEmpty?

I kind of agree. The main reason I did that is the i3dm and b3dm contents resolves the contentReadyToProcessPromise after they have created their resources, so there was no good sync point to destroy the resources from the outside. But if I move those promises to the start of initialize it would be easier to handle.

So it might be a little more complicated than just that...

Anyway one benefit of the approach here is that it's not tied to the expire or unloading systems. You can just call tile.requestContent and it will automatically update the content. This is how it works right now; all the more complicated unloadeExpired**** functions are only called when the request fails.

We can talk more tomorrow or after Siggraph.

Let's discuss briefly today when you are free.

pjcozzi · 2016-07-20T01:44:29Z

Added above:

Unit tests

Example tileset for Sandcastle example

pjcozzi · 2016-07-20T01:46:48Z

Source/Scene/Cesium3DTileset.js

+    function recheckRefinement(tile) {
+        var ancestor = getAncestorWithContent(tile);
+        if (defined(ancestor) && (ancestor.refine === Cesium3DTileRefine.REPLACE)) {
+            prepareRefiningTiles([ancestor]);


Does this happen a lot? Is this array allocation OK here?

It doesn't happen much, only when a tile expires for good.

pjcozzi · 2016-07-20T02:00:21Z

With the current design there may be a few frames of emptiness between when the new content is downloaded and when it is ready to render. I don't think this will be a problem, but I'll look out for it as I'm testing...

This does not sound ideal to me, but I agree the complexity could be painful. Let's see how this looks in practice and we can write the spec so that the client has some flexibility as to when to stop displaying the old content and start displaying the new content, e.g., the most trivial client might just unload the content before making the request, and the most complex client might alpha blend the old/new contents.

pjcozzi · 2016-07-20T02:02:00Z

I need to take a more careful look at this after SIGGRAPH, but I think there is enough here for a demo in our tutorial.

mramato · 2016-07-20T14:59:23Z

What is standard with KML network links? And is it still a good idea today?

KML has multiple triggers for refreshing <Link> tags which is specified by the refreshMode enum. The one that matches this feature the closest is the onExpire property and it probable the only one worth implementing (though check that link for yourself):

onExpire - refresh the file when the expiration time is reached. If a fetched file has a NetworkLinkControl, the time takes precedence over expiration times specified in HTTP headers. If no time is specified, the HTTP max-age header is used (if present). If max-age is not present, the Expires HTTP header is used (if present). (See Section RFC261b of the Hypertext Transfer Protocol - HTTP 1.1 for details on HTTP header fields.)

Since caching is a property of the server/browser working in concert, caching isn't an issue in Google Earth because they can ignore headers and refresh the link without doing anything special. However, we can't control how content gets cached client side because the browser handles that. As far as I know, this leaves us only two options:

They traditional way to work around this issue without depending on server configuration is to add a random value as a query parameter to force a refresh in the cache. a GUID or timestamp would work best since they would never be repeated.
If we wanted to avoid adding the query parameters, then we would have to stipulate that the server must make sure the cache headers for the expiring tile are properly set so that the cache expires before the tileset triggers an expire.

The first option puts the burden on the client but is standard and is easy to implement, the second option puts burden on the server and would be easy to get wrong. So I would strongly recommend 1 unless there is a third option I'm not aware of.

This is only a problem the browser-based implementations of the spec, since thick clients would be free to ignore headers, just like Google Earth.

colek42 · 2016-08-01T15:38:53Z

It would be nice if the request included the current cesium time (timeline). Might be out of the scope of this PR, but tiles should also 'expire' if you move backwards in time before the tile should be valid.

Since streaming tiles looks to be far out the above changes should allow us to create 3D tiles dynamically on the server side.

lilleyse · 2017-04-21T18:19:31Z

This is built on top of the no-tile-extensions #5206 now, and really is much cleaner because of it.

To see it working point to http://localhost:8003/tilesets/TilesetWithExpiration with 3d-tiles-samples on this branch: CesiumGS/3d-tiles-samples#13

With the current design there may be a few frames of emptiness between when the new content is downloaded and when it is ready to render. I don't think this will be a problem, but I'll look out for it as I'm testing. The alternative is a messier to handle because then you have to manage two sets of resources - the old ones that are still rendering and the new ones that are loading. Not to mention handling two subtrees when dealing with external tilesets.

This comment still applies. I'll check how noticeable this is now that we have mixed-lod rendering.

Another difference from above is tiles are no longer made empty when the request fails, instead they stay in the failed state which has the same effect. I may rethink this, but its pretty clean this way.

pjcozzi · 2017-04-23T13:24:26Z

This comment still applies. I'll check how noticeable this is now that we have mixed-lod rendering.

How did this go?

I don't think this is going to be acceptable behavior for most users - do you think folks would use CZML if content could disappear between time intervals?

Let me know if you want to discuss in person about how we could come up with a clean implementation. If it is not feasible in the near-term, then let's push tile expiration post 1.0 since it is a niche feature and I do not want to distract us from the core.

lilleyse · 2017-04-23T18:09:11Z

How did this go?

I haven't tried yet, but I'll report back soon.

I don't think this is going to be acceptable behavior for most users - do you think folks would use CZML if content could disappear between time intervals?

If we can force the processing stage to be synchronous then I can get around the flicker. Alternatively a tile could store the expired and new content during the frames in which the new content is processing. I think either approach is doable.

I am more worried about expired external tilesets since a newly fetched subtree needs to coexist with the old subtree until it is ready to swap in its place (whenever that would be...). A brief period of emptiness might be more acceptable in this case.

lilleyse · 2017-04-24T14:11:19Z

Alternatively a tile could store the expired and new content during the frames in which the new content is processing.

I went with this approach which fixed the empty frames.

pjcozzi · 2017-04-25T12:51:07Z

Alternatively a tile could store the expired and new content during the frames in which the new content is processing.

I went with this approach which fixed the empty frames.

Sounds good. What is the latest with external tilesets? We could also limit this in the spec to control the scope, but I think this will be a common use case...we need to balance this with the implementation complexity.

lilleyse · 2017-04-25T15:02:49Z

External tilesets will destroy the subtree immediately as it expires, and then load the new one. So there will be empty frames as it reloads. Getting a seamless experience for external tilesets is much harder.

pjcozzi · 2017-04-25T15:06:52Z

OK, please write the spec accordingly to allow for this.

Is this ready for review?

lilleyse · 2017-04-25T15:10:08Z

The code is basically ready, but I want to write the tests first to be sure. Should be ready soon.

lilleyse · 2017-04-26T21:46:37Z

Updated with tests.

lilleyse · 2017-04-26T21:47:39Z

Source/Scene/Cesium3DTile.js

            get : function() {
-                return this._requestServer;
+                return this.contentReady || (defined(this._expiredContent) && this._contentState !== Cesium3DTileContentState.FAILED);


This could be tweaked depending on the behavior we want. If a request fails do we want the tile to not be rendered, or do we want to render the expired content? This is doing the former.

The former is OK for now. The spec might need some flexibility though, e.g., would some apps want to show the previous content but mark that it is expired? I suspect so. At some point we might need to provide app-level hooks to support this.

lilleyse · 2017-04-26T21:49:38Z

Source/Scene/Cesium3DTile.js

@@ -118,8 +120,6 @@ define([
         */
        this.computedTransform = computedTransform;

-        this._transformDirty = true;


Removed this since it was only used in point clouds and it was easy to get around.

lilleyse · 2017-04-26T21:50:18Z

Source/Scene/Cesium3DTile.js

-        // Restore properties set per frame to their defaults
-        this.distanceToCamera = 0;
-        this.visibilityPlaneMask = 0;
-        this.selected = false;


These aren't needed to be reset anymore since they are always updated right away in the traversal.

pjcozzi · 2017-04-27T12:03:58Z

@lilleyse can you update the tasklist at the top of this PR?

pjcozzi · 2017-04-27T12:11:58Z

At quick glance, the code looks good to me. Definitely needs a Sandcastle example.

@bagnell could you please review this? It would be good for you to become more familiar with the core 3D Tiles engine.

bagnell · 2017-04-28T19:00:54Z

The code looks good to me. There is a failing test and +1 for a Sandcastle example.

lilleyse · 2017-05-01T20:06:09Z

For the Sandcastle demo would it be acceptable to modify server.js similar to CesiumGS/3d-tiles-samples#13? This feels bad to do, but I don't have any other ideas.

lilleyse · 2017-05-01T20:07:18Z

It would make sense to hold off on merging this until the traversal refactor (#5254) is in.

mramato · 2017-05-01T20:48:27Z

For the Sandcastle demo would it be acceptable to modify server.js similar to CesiumGS/3d-tiles-samples#13? This feels bad to do, but I don't have any other ideas.

Not really, because the Sandcastle example wouldn't work with other servers, such as the one we use to deploy on cesiumjs.org. We don't have any Sandcastle examples that rely on server functionality (outside of the proxy, but that's different).

lilleyse · 2017-05-03T17:15:05Z

Not really, because the Sandcastle example wouldn't work with other servers, such as the one we use to deploy on cesiumjs.org. We don't have any Sandcastle examples that rely on server functionality (outside of the proxy, but that's different).

Ok that's a pretty clear deal breaker. I'm not sure I can include a Sandcastle example for this PR. But at least we have one in https://github.com/AnalyticalGraphicsInc/3d-tiles-samples

lilleyse · 2017-05-09T18:11:48Z

This is ready to go.

pjcozzi · 2017-05-10T00:29:08Z

Thanks @lilleyse!

lilleyse force-pushed the 3d-tiles-expire branch from 9733e3b to 3f0479c Compare July 19, 2016 19:03

pjcozzi mentioned this pull request Jul 20, 2016

Tile expiration CesiumGS/3d-tiles#99

Closed

pjcozzi reviewed Jul 20, 2016
View reviewed changes

pjcozzi added the priority - high label Jan 11, 2017

lilleyse force-pushed the 3d-tiles-expire branch 6 times, most recently from 1d68cd8 to d3aeef7 Compare March 16, 2017 21:46

lilleyse mentioned this pull request Mar 16, 2017

Tile expiration in the server CesiumGS/3d-tiles-samples#13

Merged

Expire temp

6aaff1b

lilleyse force-pushed the 3d-tiles-expire branch from d3aeef7 to 6aaff1b Compare April 21, 2017 18:07

lilleyse changed the base branch from 3d-tiles to no-tile-extension April 21, 2017 18:09

Keep expired content while new content loads

5fb5f91

lilleyse force-pushed the 3d-tiles-expire branch 2 times, most recently from 808f4e5 to df3b31f Compare April 24, 2017 17:58

lilleyse force-pushed the 3d-tiles-expire branch 2 times, most recently from f0ea129 to 39a8c6c Compare April 24, 2017 20:44

Handle expiring subtrees

63b899d

lilleyse force-pushed the 3d-tiles-expire branch from 39a8c6c to 63b899d Compare April 24, 2017 20:59

Clean up model matrix code

18e641b

lilleyse changed the base branch from no-tile-extension to 3d-tiles April 26, 2017 19:31

Added tests + styling fix

a46ca4c

lilleyse commented Apr 26, 2017

View reviewed changes

Fix test

d6a1463

lilleyse mentioned this pull request May 1, 2017

Tile expiration CesiumGS/3d-tiles#217

Closed

lilleyse added 2 commits May 8, 2017 11:17

Merge branch '3d-tiles' into 3d-tiles-expire

11bede6

Fix tests

b5c719d

pjcozzi merged commit 68e2aa5 into 3d-tiles May 10, 2017

pjcozzi deleted the 3d-tiles-expire branch May 10, 2017 00:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

3D Tiles - Support tile expiration #4136

3D Tiles - Support tile expiration #4136

lilleyse commented Jul 19, 2016 •

edited

Loading

pjcozzi commented Jul 20, 2016

pjcozzi Jul 20, 2016

lilleyse Jul 20, 2016

lilleyse Jul 20, 2016

pjcozzi Jul 20, 2016

pjcozzi commented Jul 20, 2016

pjcozzi Jul 20, 2016

lilleyse Jul 20, 2016

pjcozzi commented Jul 20, 2016

pjcozzi commented Jul 20, 2016

mramato commented Jul 20, 2016

colek42 commented Aug 1, 2016 •

edited

Loading

lilleyse commented Apr 21, 2017

pjcozzi commented Apr 23, 2017

lilleyse commented Apr 23, 2017

lilleyse commented Apr 24, 2017

pjcozzi commented Apr 25, 2017

lilleyse commented Apr 25, 2017

pjcozzi commented Apr 25, 2017

lilleyse commented Apr 25, 2017

lilleyse commented Apr 26, 2017

lilleyse Apr 26, 2017

pjcozzi Apr 27, 2017

lilleyse Apr 26, 2017

lilleyse Apr 26, 2017

pjcozzi commented Apr 27, 2017

pjcozzi commented Apr 27, 2017

bagnell commented Apr 28, 2017

lilleyse commented May 1, 2017

lilleyse commented May 1, 2017

mramato commented May 1, 2017

lilleyse commented May 3, 2017

lilleyse commented May 9, 2017

pjcozzi commented May 10, 2017

3D Tiles - Support tile expiration #4136

3D Tiles - Support tile expiration #4136

Conversation

lilleyse commented Jul 19, 2016 • edited Loading

pjcozzi commented Jul 20, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pjcozzi commented Jul 20, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pjcozzi commented Jul 20, 2016

pjcozzi commented Jul 20, 2016

mramato commented Jul 20, 2016

colek42 commented Aug 1, 2016 • edited Loading

lilleyse commented Apr 21, 2017

pjcozzi commented Apr 23, 2017

lilleyse commented Apr 23, 2017

lilleyse commented Apr 24, 2017

pjcozzi commented Apr 25, 2017

lilleyse commented Apr 25, 2017

pjcozzi commented Apr 25, 2017

lilleyse commented Apr 25, 2017

lilleyse commented Apr 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pjcozzi commented Apr 27, 2017

pjcozzi commented Apr 27, 2017

bagnell commented Apr 28, 2017

lilleyse commented May 1, 2017

lilleyse commented May 1, 2017

mramato commented May 1, 2017

lilleyse commented May 3, 2017

lilleyse commented May 9, 2017

pjcozzi commented May 10, 2017

lilleyse commented Jul 19, 2016 •

edited

Loading

colek42 commented Aug 1, 2016 •

edited

Loading