Accessing frame timestamp | index from last updateTexImage

[VincentBilodeauFP]

Hi,

We need to have precise information about which frame is currently being rendered in order to trigger frame index based events such as a change in configuration, effects, image resizing, etc.

To get the current frame, we initially used getCurrentPosition and converted to frame based on what's the frame rate of the video, and we truncate with a small epsilon to cover floating number imprecision. Unfortunately, this is not precise. First, the getCurrentPosition will increment regardless of if a new frame is available. Second, there doesn't seem to be a callback triggered fast enough to know that we must update the frame index (i.e., the delay between when the player provides a frame and when a callback might happen to update the time is non deterministic and may be rather big).

I tried getting the surface timestamps, but they seem to be offset by System.nanoTime() at some key points that are unknown to me (initialization? decoder creation? play? seek? it even seems to change randomly while playing).

I would like to know if there's a way to get my current frame, and if so, how? If not, can you implement something about this?

Using the MediaPlayer, I can achieve a relatively precise frame index using getAnchorSytemNanoTime and getAnchorMediaTimeUs, but to my understanding, it should be the same as using System.nanoTime and getCurrentPosition. For now, my conclusion is either that getCurrentPosition is imprecise in this implementation and/or the surface timestamp are constantly offset by something dynamic, which makes them unusable unless that offset is exposed somewhere.

[marcbaechinger]

I think there is no trivial way to trigger frame-accurate behaviours. The codec itself is rendering on the surface and I think the information you are already referring to above is what you can get.

If you want to have to eg. display an overlay or similar at exactly a given frame you may need to render the frame off-screen, composite and then copy the image onto a surface manually. When doing this you have frame-accurate information as you have to do it yourself. That's obviously not trivial to do and also does not work for protected content, but I think there is no other way I'm afraid.

[marcbaechinger]

@andrewlewis do you have some ideas how this could be achieved?

[tonihei]

You can also try to add a VideoFrameMetadataListener to SimpleExoPlayer which provides you with a callback for each video frame including the presentation and release timestamps. I think that should be closest to what you'd like to have. Note however, that this callback is called on the playback thread and you shouldn't do any blocking operation in there.

[VincentBilodeauFP]

Thanks tonihei, I'll need to update the exoplayer for that, I'll be testing this out after and come back with results

[andrewlewis]

Just to elaborate a bit on how to use this: for the frame last rendered via updateTexImage, you can access the frame release time that ExoPlayer passed in the corresponding earlier call to releaseOutputBuffer using SurfaceTexture.getTimestamp. So you can store a mapping from frame release time (in nanoseconds) onto media presentation time (in microseconds) in VideoFrameMetadataListener.onVideoFrameAboutToBeRendered, then map the frame release time returned by SurfaceTexture.getTimestamp back from release time to presentation time. This lets you do frame-exact processing based on presentation time.

See also the spherical SceneRenderer for an example.

[VincentBilodeauFP]

So presentation time is what I need in term of precision, I'm just missing when I need to update my current time to the presentation time. I'll test the TimedValueQueue, it's probably gonna do the trick. Thanks for being helpful :D

[VincentBilodeauFP]

Tested with the the TimedValueQueue, I'll just add that pollFloor worked better in this case for me :) (sorry it took some time for me to close this, I had to properly update the exoplayer from a rather old version, and had some trouble doing it)