Support for playing Kotlin Multiplatform resources

[Woren]

Use case description

Main use case is about playing files inside APK with not only for example Uri:
Uri.parse(("file:///android_asset/music.mp3"))

But to be able to use files located in jar archive. For example Uri:
jar:file:/data/app/~~4hhLse7uFXE7V7sA==/com.example.composetest-jD6eQ3BeyvfQ==/base.apk!/composeResources/com.example.shared.resources/files/music.mp3

This approache can be used for multiplatform applications (e.i. multiplatform media players using AVPlayer on iOS and Exoplayer on Android with shared code) using Compose Multiplatform which will generated resources Uri for files inside common/shared directory. Mentioned directory "android_asset" is not accessible on iOS and generated resources Uri can't be used on Android. Media files needs to be duplicated to proper platform dependend directories.

More about Compose Multiplatform resources:
https://www.jetbrains.com/help/kotlin-multiplatform-dev/compose-images-resources.html

Proposed solution

Be able to use following code:
val mediaItem = MediaItem.Builder().setUri(Res.getUri("files/music.mp3")).build()

Alternatives considered

Only solution I can think of is using custom MediaSource and handling InputStream using method from Compose Multiplatform:
Res.readBytes("files/music.mp3")
with solution mentioned for example here (Exoplayer issue)
Similar issue is opened in Compose Multiplatform repository here and support from this side is not coming. The reasons are mentioned there.

Thank you for consideration!

[icbaker]

I can see two possible options here:

1. The proposed suggestion, which boils down to creating a DataSource implementation that can handle jar: URIs. It seems that, in general,jar: URIs might not be only jar:file:, they could also be jar:http: - see docs here: https://developer.android.com/reference/java/net/JarURLConnection

   • I suppose we could strictly only support jar:file: URIs
   • This still seems fiddly though, we will need to handle the general case of locating the JAR file on disk, and then opening it, unless we require the caller to guarantee the bit after the ! can be passed to Class.getResourceAsStream or similar (not sure how that works with e.g. split APKs on Android).

2. Define a new media3-specific URI scheme for accessing Compose resources specifically, e.g. "composeresource://files/music.mp3", and implement a DataSource that supports these by using Res.readBytes(...).

(2) seems to solve the problem described here, without the fiddliness introduced by (1).

The downside of (2) seems to be that we can only read the entire file into memory in one go, which might not be great for large media files. On the other hand, large media files probably shouldn't be baked into the APK in the first place...

I'm tempted to go with (2), and maybe in future Compose Multi-Platform will add some first-class support for reading resources 'incrementally' and we can update our implementation to use that.

[icbaker]

Ah, I didn't realise that the Res class is generated and project-specific, like R classes on Android:

After importing a project, a special Res class is generated which provides access to resources.

This makes (2) a little trickier, though we can either take a super-type of this generated Res class (if one exists) in the constructor of the new DataSource, or instead take a callback that resolves a path to a byte[].

[icbaker]

EDIT: The suggestion in this comment of creating a separate ResolvingByteArrayDataSource is no longer needed after 4dd8360 which allows passing a UriResolver to the library's ByteArrayDataSource. The rest of the code below about wiring the datasource into the player is still relevant.

In the meantime, I think you can make this work yourself by defining your own DataSource that wraps ByteArrayDataSource. The only tricky part is that you have to delay the instantiation of ByteArrayDataSource until your outer DataSource.open is called (and this causes a bit of messiness with TransferListener handling). Something like (this uses java.util.Function so you either need to desugar, or min API >= 24, but you could define your own UriResolver interface instead to avoid this). I haven't tested this:

public final class ResolvingByteArrayDataSource implements DataSource {

  public static final class Factory implements DataSource.Factory {

    private final Function<Uri, byte[]> uriResolver;

    public Factory(Function<Uri, byte[]> uriResolver) {
      this.uriResolver = uriResolver;
    }

    @Override
    public DataSource createDataSource() {
      return new ResolvingByteArrayDataSource(uriResolver);
    }
  }

  private final Function<Uri, byte[]> uriResolver;
  private final ArrayList<TransferListener> listeners;

  @Nullable private ByteArrayDataSource delegate;
  @Nullable private Uri uri;

  private ResolvingByteArrayDataSource(Function<Uri, byte[]> uriResolver) {
    this.uriResolver = uriResolver;
    this.listeners = new ArrayList<>(/* initialCapacity= */ 1);
  }

  @Override
  public long open(DataSpec dataSpec) throws IOException {
    uri = dataSpec.uri;
    delegate = new ByteArrayDataSource(uriResolver.apply(uri));
    for (TransferListener listener : listeners) {
      delegate.addTransferListener(listener);
    }
    return delegate.open(dataSpec);
  }

  @Override
  public int read(byte[] buffer, int offset, int length) throws IOException {
    return delegate.read(buffer, offset, length);
  }

  @Override
  public void addTransferListener(TransferListener transferListener) {
    if (!listeners.contains(transferListener)) {
      listeners.add(transferListener);
    }
    if (delegate != null) {
      delegate.addTransferListener(transferListener);
    }
  }

  @Nullable
  @Override
  public Uri getUri() {
    return uri;
  }

  @Override
  public void close() throws IOException {
    uri = null;
    delegate.close();
    delegate = null;
  }
}

Then you would set your player up with this (see https://developer.android.com/media/media3/exoplayer/customization) - this makes your player only work with 'kotlin multi platform' URIs:

ExoPlayer player =
    new ExoPlayer.Builder(/* context= */ this)
        .setMediaSourceFactory(
            new DefaultMediaSourceFactory(
                new ByteArrayDataSource.Factory(uri -> Res.readBytes(uri.getPath())))
        .build();

And you should then be able to play a resource like this:

player.addMediaItem(MediaItem.fromUri("files/music.mp3"));

[Woren]

First thing - great thank to you for all the help and ideas so far! I really appreciate it.

I have tested code above and it's working. I have few things if someone will reuse it too - possibly from Kotlin code:
a)
val uriResolver = { uri: Uri? -> runBlocking { Res.readBytes(uri?.path ?: "") } }

readBytes() is suspend function so it needs to be called similar to this

val player = ExoPlayer.Builder(context).setMediaSourceFactory( DefaultMediaSourceFactory(ResolvingByteArrayDataSource.Factory(uriResolver))).build()

b) Don't forget to add also media3-common and media3-datasource to Gradle dependencies
c) Convert ResolvingByteArrayDataSource to Kotlin too. I was getting strange NoClassDefFoundError, surely something wrong on my side.
d) Information about desugaring (running this on API level below 24) can be found here.

As you mentioned above about delaying instantiation of ByteArrayDataSource - mentioned runBlocking { } should do the trick if I understand your point correctly. Surely not pretty solution.

In case that option (2) is way to go - should I create new issue on Compose Multiplatform side about Res.readBytes() reading in on go as you mentioned above? Or it will be more clear what kind of interface will be needed as this will be implemented on this side? In short - let me know if I can do something more. And thanks again.

[yuroyami]

@Woren I am assuming you don't have a lot of files at hand, a little workaround approach to do is via offloading the files inside the jar to the app's contextual private files directory (Which on Android equals to filesDir, whereas on iOS it is NSHomeDirectory). At app launch, I'd perform some background task through IO coroutines to move them to filesDir for easy and fast access later with media3

[Woren]

@yuroyami Before Compose Multiplatform Resources generation (since 1.6.0) I was using mentioned file:///android_asset and compose-resources (similar but now afaik deprecated approach) directory for other platforms using expect/actual access to those assets. Then I had script which copied files to Android assets directory before build. In my opinion better approach then doing it on each device (wasting resources) and handling updates (adding, removing files etc.) which can be error prone. Also in my case it can be hundreds mostly small files so time for copying will not be on my side.

But as mentioned before workaround using uriResolver seams to be working for now so hopefully until proper solution by Media3/Exoplayer library there will be no need another one. But thanks for the idea! I will keep in mind for another possible problems in other areas. And surely there will be some.

[icbaker]

4dd8360 changes the library's ByteArrayDataSource to allow passing a UriResolver into the constructor that is called during open(), so the wrapper class described above is no longer needed (I will edit my comment).

We are not currently planning any tighter integration with Kotlin Multiplatform Resources - partly because it looks like it will require some additional dependencies that are currently tricky to resolve in the environments where we develop and deploy this library (and it would probably also need to be in a new extension module). I will leave this issue open to track doing something more closely integrated in future.

[Woren]

Thank you for changes with simplifications and more info about the future. I will check mentioned commit when the new version is out. Version 1.6.0 of Compose/Kotlin Multiplatform is one of the first versions with this support (now for example lacking handling larger files) so it will be more mature in coming months/years.

Reading big raw files, like long videos, as a stream is not supported yet. Use the getUri() function to pass separate files to a system API

[hichamboushaba]

If someone is still looking at this, and want a more optimized solution that doesn't load the whole content of the file to the memory, here is an attempt at creating a custom DataSource that streams from the Jar resource's InputStream, it's inspired by the AssetDataSource in the library:

class JarResourceDataSource : BaseDataSource(false) {
    private var uri: Uri? = null
    private var inputStream: InputStream? = null
    private var bytesRemaining = 0

    override fun read(buffer: ByteArray, offset: Int, length: Int): Int {
        if (length == 0) return 0
        if (bytesRemaining == 0) return C.RESULT_END_OF_INPUT

        val read = inputStream!!.read(buffer, offset, min(bytesRemaining, length))

        if (read == -1) {
            return C.RESULT_END_OF_INPUT
        } else {
            bytesRemaining -= read
            bytesTransferred(read)
        }

        return read
    }

    override fun open(dataSpec: DataSpec): Long {
        uri = dataSpec.uri
        transferInitializing(dataSpec)

        inputStream = URL(dataSpec.uri.toString()).openConnection().getInputStream()

        bytesRemaining = if (dataSpec.length == C.LENGTH_UNSET.toLong()) {
            inputStream!!.available() - dataSpec.position.toInt()
        } else {
            dataSpec.length.toInt()
        }

        return bytesRemaining.toLong()
    }

    override fun getUri(): Uri? = uri

    override fun close() {
        inputStream?.close()
        uri = null
        inputStream = null
    }
}

I think this is similar to the approach (1) that @icbaker initially suggested, just instead of having to think about where the Jar is located and other stuff, we just delegate the call to URL(jarUri).openConnection().inputStream, which delegates any handling to the platform itself using the resulting JarUrlConnection.

Now, we can consume this directly same as the suggestions above by passing a custom DataSource.Factory that creates it.
But if someone is using the same MediaPlayer entity for different sources, we can keep the support for the other sources in addition to this one by something like this:

class ExtendedMediaFactory(private val defaultMediaFactory: DefaultMediaSourceFactory) :
    MediaSource.Factory by defaultMediaFactory {

    private val jarResourceMediaSourceFactory = ProgressiveMediaSource.Factory {
        JarResourceDataSource()
    }

    override fun createMediaSource(mediaItem: MediaItem): MediaSource {
        return if (mediaItem.localConfiguration?.uri?.scheme == "jar") {
            jarResourceMediaSourceFactory.createMediaSource(mediaItem)
        } else {
            defaultMediaFactory.createMediaSource(mediaItem)
        }
    }
}

----

ExoPlayer.Builder(context).setMediaSourceFactory(ExtendedMediaFactory(DefaultMediaSourceFactory(context)))

Then, you should be able to play the media:

        player.addMediaItem(MediaItem.fromUri(Res.getUri("files/media.mp3")));

[icbaker]

Thanks for the code suggestions @hichamboushaba - a couple of thoughts if you or others are interested:

1. You can write a contract test for your JarResourceDataSource by subclassing DataSourceContractTest. This will help give you confidence you've correctly implemented the DataSource interface contract. You can see lots of examples of these in our code base.
2. For your integration with 'other sources', you should be able to make just a custom ExtendedDataSource instead of an entirely custom MediaSource.Factory (this is one level 'more specific' than a custom MediaSource.Factory). You can see an example in DefaultDataSource. Your ExtendedDataSource would choose to delegate between JarResourceDataSource and DefaultDataSource inside DataSource.open (instead of in MediaSource.Factory.createMediaSource as you have above).

[hichamboushaba]

Thank you @icbaker, this is quite helpful, I didn't know about these two points.

[icbaker]

No problem - some other thoughts:

One of the problems with building a DataSource directly on top of (only) InputStream is the lack of 'size' info. You're using InputStream.available(), but this doesn't return the total size of the stream, just the amount available without blocking:

Returns an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking

Instead of this, I wonder if you can use JarEntry.getSize() when calculating the size to return from DataSource.open(). Looks like you can get a JarEntry from JarURLConnection.getJarEntry().

---

You current implementation also ignores DataSpec.position in open() (or not completely ignores, but doesn't use it to offset the read from the InputStream), which means it won't work correctly when being asked to read from an offset within the resource.

[hichamboushaba]

Thanks again @icbaker

You're using InputStream.available(), but this doesn't return the total size of the stream, just the amount available without blocking:

I used InputStream#available as the implementation was inspired by the AssetDataSource class which uses it.
If someone needs to change the implementation, in addition to what you suggested, I think there is a an alternative approach, we can use URL(jarUri).openConnection()'s result by calling getContentLength on it, in my testing, this returned the correct size.

You current implementation also ignores DataSpec.position in open() (or not completely ignores, but doesn't use it to offset the read from the InputStream), which means it won't work correctly when being asked to read from an offset within the resource.

That's correct, I missed this, thanks for pointing it.

[suchiz]

@Woren Do you have a dummy project implementing this solution?

Looks like the uriResolver fails to readBytes from the uri.path resulting ByteArrayDataSource(uriResolver(uri)) to be null.
uri.path isn't null and looks like the one you gave in #1405 (comment)

Files are located into commonMain/composeResources/files and MediaItem.fromUri(Res.getUri("files/my_sound.mp3")) returns an item so I guess the Uri is okay.

Maybe did you add an additional Extractor ?

Currently it looks like this:

@UnstableApi
@ExperimentalResourceApi
actual class AudioPlayer(private val context: Context) {

    private val mediaPlayer = ExoPlayer.Builder(context)
        .setMediaSourceFactory(
            DefaultMediaSourceFactory(
                ResolvingByteArrayDataSource.Factory { uri ->
                    runBlocking { Res.readBytes(uri.path!!) }
                }
            )
        )
        .build()

    actual fun playBackgroundMusic() {
        mediaPlayer.setMediaItem(MediaItem.fromUri(Res.getUri("files/background_music2.mp3")))
        mediaPlayer.prepare()
        mediaPlayer.play()
    }
}

Thanks for helping

[icbaker]

@suchiz you seem to be both calling Res.getUri before constructing your MediaItem and then also calling Res.readBytes with the result of this.

Based on the example URIs in the top comment, it seems that Res.getUri will return a URI like jar:file:/data/app/~~4hhLse7uFXE7V7sA==/com.example.composetest-jD6eQ3BeyvfQ==/base.apk!/composeResources/com.example.shared.resources/files/music.mp3, and this means you are passing a path like /data/app/~~4hhLse7uFXE7V7sA==/com.example.composetest-jD6eQ3BeyvfQ==/base.apk!/composeResources/com.example.shared.resources/files/music.mp3 into Res.readBytes, which I don't think is what it's expecting (I think it's expecting files/background_music2.mp3).

What happens if you change your MediaItem instantiation to MediaItem.fromUri("files/background_music2.mp3")?

[suchiz]

What happens if you change your MediaItem instantiation to MediaItem.fromUri("files/background_music2.mp3")?

What happens ? Well, it happens to work... Feeling dumb on this one... Still have some problem to deal with URIs. Thanks so much !

To summarize

Here is the full, quick snippet. Don't forget the guidelines, which I obviously didn't follow here.

---

libs.version.toml

[versions]
# Android only
## Media3
media3 = "1.4.1"

[libraries]
# Android only
## Media3
androidx-media3-datasource = { group = "androidx.media3", name = "media3-datasource", version.ref = "media3" }
androidx-media3-exoplayer = { group = "androidx.media3", name = "media3-exoplayer", version.ref = "media3" }
androidx-media3-common = { group = "androidx.media3", name = "media3-common", version.ref = "media3" }

---

build.gradle.kts (composeApp)

    sourceSets {
        androidMain.dependencies {
            // Media3
            implementation(libs.androidx.media3.exoplayer)
            implementation(libs.androidx.media3.common)
            implementation(libs.androidx.media3.datasource)
        }
    }

---

AudioPlayer.kt (androidMain)

@UnstableApi
@ExperimentalResourceApi
actual class AudioPlayer(private val context: Context) {
    private val mediaPlayer = ExoPlayer.Builder(context)
        .setMediaSourceFactory(
            DefaultMediaSourceFactory(
                ResolvingByteArrayDataSource.Factory { uri ->
                    runBlocking { Res.readBytes(uri.path!!) }
                }
            )
        )
        .build()

    actual fun playBackgroundMusic() {
        mediaPlayer.setMediaItem(MediaItem.fromUri("files/background_music.mp3"))
        mediaPlayer.prepare()
        mediaPlayer.play()
    }
}

---

ResolvingByteArrayDataSource.kt (androidMain)

@UnstableApi
class ResolvingByteArrayDataSource(private val uriResolver: (Uri) -> ByteArray) : DataSource {
    private lateinit var uri: Uri
    private lateinit var byteArrayDataSource: ByteArrayDataSource
    private val listeners = ArrayList<TransferListener>(1)
    
    class Factory(private val uriResolver: (Uri) -> ByteArray) : DataSource.Factory {
        override fun createDataSource(): DataSource {
            return ResolvingByteArrayDataSource(uriResolver)
        }
    }

    override fun read(buffer: ByteArray, offset: Int, length: Int): Int {
        return byteArrayDataSource.read(buffer, offset, length)
    }

    override fun addTransferListener(transferListener: TransferListener) {
        if (transferListener !in listeners) {
            listeners.add(transferListener)
        }
    }

    override fun open(dataSpec: DataSpec): Long {
        uri = dataSpec.uri
        byteArrayDataSource = ByteArrayDataSource(uriResolver(uri))
        listeners.forEach { byteArrayDataSource.addTransferListener(it) }
        return byteArrayDataSource.open(dataSpec)
    }

    override fun getUri(): Uri = uri

    override fun close() {
        byteArrayDataSource.close()
    }
}

---

App.kt (commonMain)

@Composable
@Preview
fun App(audioPlayer: AudioPlayer) {
    MaterialTheme {
        Column(
            modifier = Modifier.fillMaxSize(),
            horizontalAlignment = Alignment.CenterHorizontally,
            verticalArrangement = Arrangement.Center
        ) {
            Button(onClick = { audioPlayer.playBackgroundMusic() }) {
                Text("Play")
            }
        }
    }
}

Big thanks to you, @icbaker. So efficient, really appreciate it!