Add support for preview images AdaptionSet/Representation in DASH

[pi-andrei]

To implement seek thumbnails, usually custom solutions of loading images is used. But there is a standard of defining seek thumbnail images in dash manifest. Please see this document: http://dashif.org/wp-content/uploads/2017/09/DASH-IF-IOP-v4.1-clean.pdf in chapter "6.2.6. Tiles of thumbnail images". Can ExoPlayer dash manifest parser add support for such AdaptionSet/Representation and provide method in player for getting preview image? Thanks.

[pi-andrei]

Hello. Any update on this?
Can you maybe guide me, which classes I need to edit/create to make it work in some way?

[islor]

I'd like to know if there's any way to do this now too (showing preview while seeking), please help. Thanks.

[LightSun]

I'd want to know how to seek and preview real-time. Thanks.

[ojw28]

@drjunja86 - There is no update on this currently, sorry.

[ppodgorski]

@ojw28 Is there any new information on this issue?

[giladna]

hi @ppodgorski
you can find our solution for the media preview here:
https://github.com/kaltura/kaltura-player-android-samples/tree/master/AdvancedSamples/MediaPreviewSample

[YongtaoS]

Looking forward to the support of DASH tiles thumbnail seeking in ExoPlayer as the native android-player.

[jarhoax]

Any update/plans on implementing this? Thanks.

[skyswitzerland]

Also interested to get a status update and to evaluate how we can make it happen together as a community.
Maybe @marcbaechinger or @ojw28 would have a suggestion too.

[porcelijn]

For reference (and interop testing), here are some sample streams that work with dash js:

• OnDeman profile:
  https://demo.unified-streaming.com/k8s/features/stable/no-handler-origin/tears-of-steel/tears-of-steel-tiled-thumbnails-static.mpd
• Dash live with $number$ segment templates:
  https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel-tiled-thumbnails-numbered.ism/.mpd
• Dash live with $time$ based segment templates:
  https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel-tiled-thumbnails-timeline.ism/.mpd

[skyswitzerland]

Hi Olly @ojw28,

It seems there is an industry wide interest to make this feature happening, we would be ready to assign Java developer if we have a chance to make as a core exoplayer feature. What would be your guidelines to succeed ?
Thanks for your guidance.
Nicolas STEFANI

[souissihaythem]

Hello
this feature is very interesting, I will be happy (as an android developer) to help here.

But first, is Exoplayer team motivated to do such a feature or to help do it by providing all necessary support?

Can any developer describe the process to validate a merge request in ExoPlayer project?
are there any rules to respect?
Do we need to implement unit tests?

thanks

[ojw28]

We would be ready to assign Java developer if we have a chance to make as a core exoplayer feature. What would be your guidelines to succeed?

We would welcome a high quality contribution in this area. That said, we see this feature as a pretty difficult one. It's not immediately clear to us (the core dev team) what the best way of implementing it is, and so someone would need to spend some time coming up with a detailed proposal (or proposals) for review. In terms of maximizing the probability that a contribution would be useful, I'd therefore suggest that a contributor should start by writing a fairly thorough design proposal before anything else, that can be reviewed. To write such a proposal it may of course be necessary to hack around in code a little, to learn about what's already there and figure out what's possible, but I'd expect the design proposal to precede actual pull requests. You can see some past examples of design proposals here to get an idea of the kind of thing we'd be looking for. Once we're able to agree on a design, implementation is (probably!) the easier part :).

[suraneau]

Hello @ojw28, hello community,

thank you for providing the information on how to contribute via design proposal.

We would like to share this proposal regarding "Add support for thumbnail/preview images from AdaptationSets in DASH" with you and the community (Google Docs document).
Our proposed implementation will follow the approach of how the DASH-IF reference player dash.js is handling the display of thumbnails. An API will be implemented which provides a list of metadata objects. This approach allows the App/UI logic to fetch the thumbnail image itself and handle the caching of images. For this purpose we use the analogous data flow (URL and metadata) as used for the subtitles but will provide an API for requesting these metadata only.

We're looking forward to feedback, discussion and the okay for the start of implementation.

Regards
Sinh

[tonihei]

Thanks for putting together this proposal! Looks like you are addressing multiple missing parts that are needed to allow thumbnail previews / trick play. It's probably worth focusing on one specific part of the problem first to ease the design discussion and increase the chances we can merge the changes.

Note that there is a considerable overlap with #474, which tracks trick play / thumbnails for HLS and we should try to keep the common functionality generic enough to work with both. 

I haven't spent a huge amount of time thinking about the details, but I believe that full support will eventually touch many parts of the library: 

1. Parse the trick play track information from the manifest/playlist and add it as a Format (or multiple Formats) in the video TrackGroup. This work is HLS/DASH specific. For HLS, support has been added by Support for HLS EXT-X-IFRAME-ONLY playlists #6270 and DASH support is still missing. Please note the discussions in HLS support for trick play with EXT-X-I-FRAMES-ONLY and EXT-X-I-FRAME-STREAM-INF #474 (comment) about how the trick play tracks are ideally part of the video track group and how they should be marked with the role flag ROLE_FLAG_TRICK_PLAY. This is different from your document where you seem to propose a separate track group that can be loaded and played in parallel.
2. Extract the actual sample data through the Extractor instances set up by the HLS/DASH module that will write the extracted samples into the sample queue. For HLS, this step is already covered for video container based thumbnails (=I-frame only videos), but not for image formats. Similarly, we'd need support for parsing the image formats in the DASH image tracks. Note that we already have a JpegExtractor, but its main purpose is to support motion photos at the moment. This step should leave the image data compressed (i.e. as the raw jpeg bytes in the simplest case).
3. Provide a way to render the images. This could be a callback for the app similar to how text track are handled (see onCues), or a dedicated image Renderer class in ExoPlayer.
4. Handle trick-play selection in DefaultTrackSelector. Turning on trick-play requires a track override that selects the right track while disabling all other video, audio and text tracks. In addition, we may even think about including trick play track in the normal adaptive video selection so it can be used for very high playback speeds. A further complication of track selection is the track-to-renderer mapping done in MappingTrackSelector which currently prevents the selection of different tracks in the same group for different renderers (=it's impossible to set an override that selects the thumbnail track for the image renderer).
5. Adjust LoadControl to prevent excessive preloading while showing a trick-play track. The reason is that we don't want to reload images again and again during fast scrubbing. But we also need to load enough data to not get stuck when playing in trick play (see Support I-frame only playback (trick play) in player and buffering logic #7171).
6. Potentially adjust caching. I think the default caching components of ExoPlayer are already sufficient to handle all practical cases, but maybe some adjustments or special handling are necessary.
7. Optimize seeking. Repeated seeking (=scrubbing) currently overrides the existing pending seek, so that the track is never able to actually show a frame. This needs to be optimized for scrubbing by stashing new seek requests until at least one new frame has been shown.
8. Add some central control flag to turn trick play on or off. All the changes above need to be triggered at the same time and a user of the library likely wants a single flag. There are also other central player properties that need to be adjusted: For example, the player should pause playback during trick play.
9. Add UI support in ExoPlayer's default components. This is probably optional, but would help for debugging and showing how the feature is used.

This list isn't necessarily exhaustive nor does it mean that we really need to make all of these changes. But as you can see, this feature is likely to be very complicated and that's why it would be helpful to focus on one specific subset first. It seems like steps 1 and 2 could be suitable for a PR? What do you think? If you agree, it would be nice to update the doc to focus on this particular part in detail and then we can go from there.

[suraneau]

Thank you @tonihei for the feedback you provided to our design proposal "Add support for thumbnail/preview images from AdaptationSets in DASH".

We are very pleased with the efforts and work done so far on the integration of trick play for HLS. Especially the references to tickets and PRs have given us a better insight into the approaches.

DASH and HLS have two different approaches to thumbnail support. HLS has its own "thumbnail video representation" and when scrubbing, the representation is completely replaced in the view, so we understand the hints about adding the track to the video track group, pausing the current stream and more. DASH's approach is that the thumbnails are rendered in a small preview window in relation to the scrubbing position on the timebar and the current video doesn't need to be paused.

Our approach is similar to how dash.js handles thumbnails. Only the metadata (like the URL) for the thumbnail is provided by exoplayer via an API. The tasks of loading, caching, crop/zoom and rendering of the (tile) images is done by the external app/UI logic.

We like the suggestion to plan the process in stages to get a better overview and understanding of the PRs. We have adapted the design proposal accordingly and divided the whole project into 4 milestones to keep track of the progress.

We would like to share the updated proposal regarding "Add support for thumbnail/preview images from AdaptationSets in DASH" (last update on 2022-06-30). As the 4th and last milestone we'll integrate the thumbnail support into the demo app.

We're looking forward to feedback, discussion and the okay for the start of implementation.

Regards
Sinh

[suraneau]

@tonihei , i would like to get in touch with you regarding CLA. How can i contact you?

[suraneau]

Hi @tonihei, i would like to get in touch with you regarding CLA. We do have a signed CLA from a previous/older project with Google/non ExoPlayer and would like to check if this can be used for this project. How can i contact you?

[gorkemg]

Hi, we created a Pull Request for this feature.
Thanks.

[FongMi]

@gorkemg
Hello, is there an example of this PR ?

[gorkemg]

@FongMi
Hi, you can checkout commit dad74276efdfc7119476a3d62746ba5a0cf3da0e, where the demo app has a functional thumbnail scrub bar implemented as an example, before it was removed in a later commit. Simply select a stream from the "Thumbnails" section on the launch screen.

[FongMi]

@FongMi Hi, you can checkout commit dad74276efdfc7119476a3d62746ba5a0cf3da0e, where the demo app has a functional thumbnail scrub bar implemented as an example, before it was removed in a later commit. Simply select a stream from the "Thumbnails" section on the launch screen.

Thanks