[API Request] Advanced transcoding

[Tolriq]

Type of change

API extension

Proposal description

This is the first step to discuss the future transcoding API extension that give a lot more control over when and to what to transcode.

This is necessary to give a lot more liberty to clients to request media they can play or cast to other device with proper quality control. When you can't play DSD files for example, it's better to transcode to FLAC than low quality mp3.
Or when your device does not support sampleRate over 48Khz it's better to reduce sample rate then low quality mp3.

There is 2 different approach to this need that have different impacts on the client or the servers.

1 - Client side decision

Leave the transcoding decision and target to the client

• The client request the precise details about the media with codec, container and sub codec details
• The client decide what to do
• The client request transcoding to a target configuration (codec, container, max sample rate, max bitrate, ...)

Issues:

• Not all servers may support all the possible target configuration depending on their ffmpeg version or build, it requires error handling and fallback client side.
• Each clients needs to write their transcoding decision engine.

2 - Server side decision

Leave the transcoding decision and target to the server.

• The client send a request to the server telling I want to play that media, I support X,Y,Z profile for direct play, if you transcode I prefer A,B,C profile (in order)
• The server answer: You can direct play or you can transcode and pass an object that will need to be passed to the transcode endpoint, or I'm unable to satisfy your request.
• The client plays or cast the media.

Issues:

• Requires a precise API definition for the profiles (Need to decide the level of control to avoid changing the API too often)
• Each server needs to write the transcoding decision engine.

Other platforms

Other platforms like Emby, Plex, Jellyfin all do solution 2. But since they are a single server for a give API endpoint there's no drawback of multiple servers writing their own transcoding decision engine.

Backward compatibility impact

No response

Backward compatibility

• No backward compatibility impact.

API details

TBD

Security impacts

No response

Potential issues

No response

Alternative solutions

No response

[epoupon]

Thank you for starting this discussion.
I would go with option 2. I feel like it’s not too complicated to implement on the server side. And ideally, if everything goes well, we should have more OS clients than servers anyway, so it makes sense to place this load on the servers.

[Tolriq]

No it requires a lot more control, codec, container, maxbitrate, max sample rate, max channel number, max bitdepth at minimum .Those are not server side profiles but client profiles that vary depending on the need.
The client can't know in advance what the server support. It request what it support (the client) and the servers says if it can or not.

[sentriz]

doesn't seem backwards compatable with what we have already. why not add an endpoint for the server to respond with what it can support, client chooses from the list, and sends in the next request with format= (we already have this) and maxBitRate= (we also have this one)

for control over num channels and samplerate, what's the usecase there?

[Tolriq]

The use case is the basis of the explanation I gave on why the new endpoints.
So no it's not meant to be backward compatible it's new extension and new endpoints.

For my use case, I have TB of hi res music, the Sonos one does not support FLAC 24/96 or more.
I don't want to transcode that to mp3 or opus, I want FLAC 16/96 or 24/48.
That's specific to that device and can't be preconfigured on the server.

Same for DSD, my phone only support up to 192 so I want FLAC 24/192 and not opus or FLAC 16/44.

Same for channel count there's plenty of usage to request multi channel music to stereo without quality change.

[deluan]

I also vote for option 2, same reason as @epoupon. Don't we have a previous discussion about that already? Should we add the summary of that discussion here, or at least a link to it, for reference?

[Tolriq]

It was mixed in the discussion about the codec and container data for songs that we decided to postpone.

This one being self contained I prefer to move on that first before going back on the global song data where no one agreed on anything.

But yes I also think 2 is way better, option 1 was more if servers did not want to work on the engine decision part.