Generalize/enhance persistent caching functionality

[ojw28]

ExoCache is designed specifically for DASH implementations where every request can be in the form of a bounded range request (which in practice means single-segment on-demand streams containing sidx boxes). It doesn't work for anything else (DASH where streams are segmented in the manifest, SmoothStreaming, HLS, ExtractorSampleSource). There are three things to fix:

1. We don't support unbounded range requests. This is because the cache currently has no concept of the length of a piece of data. If we have a file in the cache with byte range [0-1000], and make a request for range [0-*], we currently don't know what to do after we've return the first 1000 bytes from the cache. We don't want to make an unconditional request to the network starting at offset=1000 because it's inefficient, wont work in the offline case, and because the server may return a 416 (unsatisfiable byte-range) in the case that the content really is 1000 bytes long, which would be awkward to handle.
2. It's probably more common for servers that are configured to serve every chunk from a separate URL to not support range requests. We may need the cache to support a "request the whole chunk or nothing from the upstream source" mode for this use case. I'm not sure how we'd decide when to turn it on. Awkward.
3. The cache currently indexes: contentId->Tree[byte-range->file]. For chunks that are requested from different URLs, we need the Tree to be indexed by chunk-index+byte-range, or perhaps just chunk-index depending on the answer to (2).

[alim-n]

It also should be able to cache mp3, mp4 files distributed through plain http. In my use case app plays media file from server, but URL and CDN server may change everytime file is requested. I think we should use key based caching.

[alim-n]

I'm making my own cache for now, should I modify DataSource interface or create CacheableDataSource instead?

[alim-n]

I did some work, please check it. https://github.com/nzer/ExoPlayer/commit/6c3c89073e25b7af97fc7c55e6a7d15d308a9fbc
Idea is when media is being downloaded (MP3, AAC, whatever) its written to disk at same time. If download succeeds to the end, we mark file finished and offer it to datasource to instead of loading it from network.
Key should be designed by app developer. MD5 of artist+title for music files for example.

[Arqu]

One question about the indexes (key), why not just use a hashing function of the DataSpec info, like uri+position+length? Would be unique enough for the caching structure and would also avoid having another segments chunk index overlap with some other.

[ojw28]

Because if you've cached the whole of a media stream under some hash h(uri, 0, length_of_stream), then you'll get a cache miss for every request to for the same uri unless you happen to be requesting from position 0 for the whole stream, even though you have this data cached already.

[Arqu]

Hmmm, right, however I've managed to get the cache working using a simple hash function of uri+offset+length, which works great currently because it is pretty much unique to that stream and to its requests so if a user seeks he immediately gets the stream playing if it is cached. Only a small hick up happens if the video has switched to a higher resolution meanwhile and you seek back to the stream, than it has to reload that part in another variant, but I guess that is more of a feature/expected than a bug. Also in my tests so far, it doesn't impede any streams at all, live works well with it too - it doesn't cache it. I think that it can be easily leveraged to implement additional functionality like live seeking and similar I just need to extend it to support these more or less unbounded requests.

[ojw28]

The point of this bug is to generalize support for all use cases. What you're doing only optimizes for a subset of cases (from your description, I'm assuming either HLS or a particular variant of DASH/SmoothStreaming). It likely also breaks if a read error occurs part way through a request (subsequent read attempts probably get a cache hit to the incomplete data).

[Arqu]

Yes, I'm currently using the existing implementation and only testing from my HLS use cases. I should definitely take a look what happens with incomplete data and how it handles.

[42footPaul]

Question - just to be clear, this enhancement when done will include the capability to save mp4 from http source to the device as it's playing?

our use case is to play the video from http the first time and if the file download completes as it's playing, then the file gets saved to our app cache and next time it can be played from disk instead.

I have sort of a working solution but I'm not sure it's really the correct approach.
I'm using a customized copy of the upstream.HttpDataSource and my own "caching" transfer listener.

Essentially the http data source passes the bytes (readInternal method) to a custom transfer listener which writes the bytes to a file output stream as the video downloads/plays. When the download is complete/ file is done being written, that video is marked as saved in our app.

This seems to be working as expected except when I ask the player to seek during download - it looks like the connection and input stream are closed and re-opened with a new request and the file output gets hosed.

Couple of questions:
Is there a timeframe that you think this use case will be implemented in ExoPlayer?

Is there another approach I might take in the meantime?

Is it currently possible to create my own downloader and play the video from a file input stream as it downloads?

Is it possible to create an internal http proxy server and pass the internal url to the httpdatasource?

[cmdkhh]

Anyone have a dev branch currently working on this? @Arqu @42footPaul @ojw28

[welshk91]

@cmdkhh Agreed. An update on the status of Exo caching would be most appreciated. Has anyone else found any type of success with caching?

I'm currently attempting to cache mp4 files served from a URL with CacheDataSource, but it doesn't seem to be actually cache anything from what I can tell (powering off/on the screen, releasing the player and reinitializing it, still redownloads the mp4 from the network again).

[jpshelley]

If there is a current roadmap of tasks that need to be done to implement caching, or if anyone has an official development branch started, that would be better then starting from scratch.

If any of these are available do let us know. We plan to start implementing caching ourselves but would love to provide support back to Exo.

[dekalo-stanislav]

Would like to share my investigations in this way.

I have spent some time to implement cache for exo player in my app.
Implementation is very custom and will not fit as general solution that
could be reintegrated back.

Requirements for my app:
Show small looped videos(mp4) < 3MB. Cache it on disk.

Issues that I have faced with:

1. There are class DataSpec (Defines a region of media data), it could be
   bounded (end point defined) and unbouned(end point undefined)
   exo's CacheDataSource doesn't cache unbouned requests as it doesn't know
   how.
2. Player requested to provide DataSpec [0-) - unbounded request.
   But it does not read it fully, after reading first few kilobytes (header as
   I understand) it stops and opened another unbounded request open(DataSpec
   [4012-]) (Note that end of first request reading are not equal to start of
   next bounded request)

1:
In my implementation I have no seek functionality, so I have
RandomAccessFile and just append network content to it's end.
To detect cache hit, we store file sizes in separate xml file, and cache
hit happens if file sizes on disk and in xml are same.

2:
I general implementation, with seek functionality I think player could ask
DataSource to provide any part of stream, so our cache will contain various
cached ranges with empty(not requested) regions between cached. Here we
have next issue: how to store that separate cachd data ranges in easy to
process way? Plus we need to implement detection if requested DataSpec is
partially cached, and download only missed.
I do not see other solution except local db implementation.

Let me know if I am unclear or missed something.

2015-11-10 20:15 GMT+02:00 John Shelley [email protected]:

If there is a current roadmap of tasks that need to be done to implement
caching, or if anyone has an official development branch started, that
would be better then starting from scratch.

If any of these are available do let us know. We plan to start
implementing caching ourselves but would love to provide support back to
Exo.

—
Reply to this email directly or view it on GitHub
#420 (comment).

[MaTriXy]

I will just add my interaction with the caching issue.
It's a very simple solution :
I've added my own disklrucache implementation together with creating a custom DataSource implementation where i would cache the response according to the url request issued out and saved it to later on fetch from the disklrucache if it's present.

[laurencedawson]

If you setup OkHttp it's pretty straight forward to get files to cache. I can provide details if needed.

[welshk91]

@laurencedawson an example or some direction would be awesome! I'm thoroughly lost after attempting it for a while now.

Even getting something relatively simple like the ExoPlayer Demo to cache with OkHttp would be tremendously helpful. I swapped in OkHttp and added a cache for it, but the demo still seems to re-download when that activity is recreated (rotation for example).

[laminsk]

@laurencedawson I'm using okhttpdatasource and still no luck trying to download the audio stream to disk. Any input you can provide will be greatly appreciated. Thanks

[b95505017]

Check this:
https://github.com/square/okhttp/wiki/Recipes#response-caching

Using CacheControl to set max stale, and OkHttp's Cache will honor them

[laminsk]

Thanks for the prompt reply. I've followed your suggestion and included the
header in makerequest and configured sOkHttpClient to use cache but it's
not writing to disk at all. Perhaps I'm doing it wrong and if you could
update your sample project with caching, that would be most grateful.

On Fri, Jan 29, 2016 at 11:01 AM, Yu-Hsuan Lin [email protected]
wrote:

Check this:
https://github.com/square/okhttp/wiki/Recipes#response-caching

Using CacheControl to set max stale, and OkHttp's Cache will honor them

—
Reply to this email directly or view it on GitHub
#420 (comment).

[kinsleykajiva]

@zhangvb the link you gave is dead.

[ojw28]

@kinsleykajiva the link is fine if you copy and paste the URL as text (it's just that it's linked incorrectly).
@qqli007 is correct about dev-v2. We recently pushed support for caching for regular media files, provided you specify FLAG_CACHE_UNBOUNDED_REQUESTS as in his sample code.

[calulelalu]

@qqli007 great example... caching seems to work but now seeking backward or forward is not working any more. I'm working with mp3 files.

[zhangvb]

Appending my code, it worked for me. If play without seeking cache works.

Interceptor interceptor = new Interceptor() {
                @Override
                public Response intercept(Chain chain) throws IOException {

                    Request request = chain.request();
                    request = request.newBuilder()
                            .cacheControl(CacheControl.FORCE_CACHE)
                            .build();

                    Response response = chain.proceed(request);

                    if (!response.isSuccessful()) {
                        Request.Builder requestBuilder = request.newBuilder();
                        String range = request.header("Range");
                        LogUtil.logD("Range " + range);
                        // Parse Range, if Range is "dddd-" format, remove Range to active OkHttp Cache.
                        if (!TextUtils.isEmpty(range) && range.startsWith("bytes=")) {
                            range = range.replace("bytes=", "");
                            String[] ranges = range.split(",");
                            LogUtil.logD("ranges length " + ranges.length);
                            if (ranges.length == 1) {
                                String range0 = ranges[0].trim();
                                if ((!range0.startsWith("-")) && range0.endsWith("-")) {
                                    String startString = range0.substring(0, range0.indexOf("-"));
                                    int start = -1;
                                    try {
                                        start = Integer.valueOf(startString);
                                    } catch (Throwable t) {
                                    }
                                    LogUtil.logD("Range start " + start);

                                    // if start is too large, maybe the user is seeking, keep Range.
                                    if (start > 0 && start < 1024 * 1024) {
                                        requestBuilder.removeHeader("Range");
                                    }
                                }
                            }
                        }

                        request = requestBuilder
                                .cacheControl(CacheControl.FORCE_NETWORK)
                                .build();
                        response = chain.proceed(request);
                    } else {
                        LogUtil.logD("Cache request success.");
                    }

                    CacheControl localCacheControl = new CacheControl.Builder()
                            .maxStale(Integer.MAX_VALUE, TimeUnit.DAYS)
                            .maxAge(Integer.MAX_VALUE, TimeUnit.DAYS)
                            .build();

                    return response
                            .newBuilder()
                            .header("Cache-Control", localCacheControl.toString())
                            .removeHeader("Pragma")
                            .build();
                }
            };

            File cacheFile = new File(cacheParentFile, "OkHttpCache");
            okhttp3.Cache cache = new Cache(cacheFile, 200 * 1024 * 1024);
            OK_HTTP_CLIENT = new OkHttpClient.Builder()
                    .addInterceptor(interceptor)
                    .cache(cache)
                    .build();

[Anshumansharma12]

Hello by using dev-v2 am able to create local cache of stream video. Now its showing multiple files in following format "xxxxx.v2.exo" is there any way by which i can create .mp4 file.
Or any other way by which i can cache it into ".mp4" format. As am streaming mp4 video http url.

[erdemguven]

@Anshumansharma12 it isn't possible but you can always download the file yourself and play it using a FileDataSource.

[danrossi]

Can somebody please explain how the CacheDataSource is supposed to be setup ?

All I can see is code like this but no idea how to integrate back into the demo application.

ie

 LeastRecentlyUsedCacheEvictor evictor = new LeastRecentlyUsedCacheEvictor(100 * 1024 * 1024);
        SimpleCache simpleCache = new SimpleCache(new File(getCacheDir(), "media_cache"), evictor);
        DefaultDataSource upstream = defaultMediaDataSourceFactory.createDataSource();
        return new CacheDataSource(simpleCache, upstream, false, false);

The demo application has code like this, so where does CacheDataSource go here ?

case C.TYPE_DASH:
        return new DashMediaSource(uri, buildDataSourceFactory(false),
            new DefaultDashChunkSource.Factory(mediaDataSourceFactory), mainHandler, eventLogger);

I'm hoping this will work as an offline cache ? Why is it not integrated into it already ? offline playback seems to be a common request.

[danrossi]

I've got this but I can't seem to debug if it's working. There is zero documentation and I've resorted to dev-v2-id3 as it seems the most up to date with all the refactoring.

If I start playing a dash source. Turn off remote connection and try and replay it fails to load the mpd file from the cache.

DataSource.Factory buildDataSourceFactory(boolean cache, final DefaultBandwidthMeter bandwidthMeter, final CacheDataSource.EventListener listener) {

    if (!cache) {
      return new DefaultDataSourceFactory(this, bandwidthMeter,
              buildHttpDataSourceFactory(bandwidthMeter));
    }

    return new DataSource.Factory() {
      @Override
      public DataSource createDataSource() {
        LeastRecentlyUsedCacheEvictor evictor = new LeastRecentlyUsedCacheEvictor(100 * 1024 * 1024);
        SimpleCache simpleCache = new SimpleCache(new File(getCacheDir(), "media_cache"), evictor);


        return new CacheDataSource(simpleCache, buildCachedHttpDataSourceFactory(bandwidthMeter).createDataSource(),
                new FileDataSource(), new CacheDataSink(simpleCache, 10 * 1024 * 1024),
                CacheDataSource.FLAG_BLOCK_ON_CACHE | CacheDataSource.FLAG_IGNORE_CACHE_ON_ERROR, listener);
      }
    };
  }


  private DefaultDataSource.Factory buildCachedHttpDataSourceFactory(DefaultBandwidthMeter bandwidthMeter) {
    return new DefaultDataSourceFactory(this, bandwidthMeter, buildHttpDataSourceFactory(bandwidthMeter));
  }

  HttpDataSource.Factory buildHttpDataSourceFactory(DefaultBandwidthMeter bandwidthMeter) {
    return new DefaultHttpDataSourceFactory(userAgent, bandwidthMeter);
  }

The dash source stays the same

return new DashMediaSource(uri, buildDataSourceFactory(true, false),
            new DefaultDashChunkSource.Factory(mediaDataSourceFactory), mainHandler, eventLogger);

The event logger implements this event method but nothing.

  @Override
  public void onCachedBytesRead(long cacheSizeBytes, long cachedBytesRead) {
    Log.d(TAG, "onCachedBytesRead[cachedBytes:" + cacheSizeBytes + ", cachedBytesRead: " + cachedBytesRead + "]");
  }

[danrossi]

Confirming this to be working. It is causing audio buffer underruns when playing from the cache so dropping frames. Turning off connection the fragments can play back from the cache but the main manifest still wants a connection and is not being stored into the cache. Perhaps this solution can be used for providing an offline downloader without having to re-engineer one. Just run the caching in the background ?

[danrossi]

I have both caching and persistent widevine dash working. However it's not caching the Mpd yet perhaps there is still a setup problem there ?

[danrossi]

The cache gets messed up with adaptive switching. Still can't figure out why its not caching the Mpd file. It is possible not setting the key correctly for the Dash segments.

Need to now figure out how to turn Adaptive off and choose the largest bitrate while trying to build a cached offline file.

There has been zero proper documentation, even code commenting for any of it sadly. As I mentioned FairPlay has a downloader and offline playback feature for AVPlayer out of the box with the current SDK. It would be nice to match it.

[danrossi]

This code is so convoluted. I was able to track down why the mpd is not being cached. It's because of a content length being -1. I can't work out right now where or why it's returning -1. Really hard work.

You have other options than to use the cache directory I believe like these. One will clear content when the app is removed, the other will not.

File storageDir = new File(getExternalFilesDir(Environment.DIRECTORY_MOVIES), cacheDir);

File storageDir = new File(Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_MOVIES), "media_cache");

[danrossi]

As far as I can see the manifest loader runs through a parser. No content length is given before it's opened. It will fail to store in the cache because no content length is given.

[danrossi]

https://github.com/google/ExoPlayer/blob/dev-v2/library/src/main/java/com/google/android/exoplayer2/upstream/ParsingLoadable.java#L75

This provides no length therefore the issue. the length can't be set after reading bytes. Requires some hacking elsewhere to allow writing the cache without checking for length possibly.

[ojw28]

@erdemguven - Do we cache data if even the resolved length is unknown (this is likely a result of the mpd response being gzip'd) and/or have a solution to this?

@danrossi - I think it would be fairer to call the code necessarily complex ;). There are many small nuances to caching media properly, particularly when considering things like caching of non-completed requests. As an aside, you probably shouldn't be using the caching components as-is to build a proper DASH offlining solution. A proper DASH offlining solution would more than likely involve explicitly downloading only a single representation (and not the manifest, although you might optionally recreate an offline manifest containing only the single representation downloaded). The caching components can be used as part of an implementation for this, but the actual downloading bit should probably not be implemented using a player instance.

[erdemguven]

@ojw28 @danrossi - Unfortunately not. We skip the cache if the resolved length is unknown as the content is likely to be a live stream. I don't know how we can handle gzip'd responses.

[danrossi]

@ojw28 ok. I was hoping to use this then play the locally stored sources ?

I have the custom offline widevine manager working and will load the keys. I could be interested to upload that to github.

The last past of course was hoping to switch to the largest bitrate and turn abr off and just let the internals do the downloading without having to re-engineer anything. I couldn't even find documentation how to do that sadly.

I wouldn't know where to start to. Im assuming the entire DashDataSource needs to be duplicated and modified to use a filedatasource for playback like the cache system does.

[danrossi]

I thought the latest changes were intended to deal with lack of content length issues. It still won't cache the mpd file sadly.

It seems this system is not designed for and not adequate for offline storage purposes, and an offline concurrent downloader feature needs to be built on top of the dash parser feature possibly. Then another system like the cachedatasource to play back those local files. None of this readily available.

[danrossi]

Is there an internal feature like the cachedatasource, to download parsed dash manifests and fragments then use the FileDataSource to play it back ?

It would be easier if it just concurrently downloaded everything into the cache path and CacheDataSource played them back ?

[erdemguven]

@danrossi We're working on full DASH offlining solution but there isn't anything for downloading yet. I don't know when it'll be ready.

[matclayton]

@erdemguven Any idea when this is likely to land? We need this functionality so might end up building it in the mean time, but it would be fantastic to not have to!

[erdemguven]

@matclayton if you mean the DASH downloader, basic functionality should be in github in 2 weeks time.

[matclayton]

Yes I did, fantastic!

[bnussey]

Hey @erdemguven was the DASH downloader released? Can't seem to find it.

[erdemguven]

@bnussey we decided to release it once the downloader service and manager are ready as major api changes might be necessary. It's hard to give an estimate now. You can follow it on #2643.