Should use GstVideoEncoder / GstVideoDecoder base classes

[sdroege]

First of all, thanks for the effort of writing all this and releasing it!

It would be good if the video encoders/decoder would use the GstVideoEncoder and GstVideoDecoder base classes. This would make sure that the elements behave exactly like video encoders/decoder should behave in GStreamer and also more special features like reverse playback, QoS handling, forcing of keyframes at specific times, etc would automatically be handled correctly.

It should also allow you to get rid of a lot of GStreamer plumbing code probably, especially if you implement all the features a complete video encoder/decoder should have according to GStreamer standards (see above for a small subset that would already be handled by the base class).

GstVideoEncoder/Decoder are successfully used for various other hardware related GStreamer elements, e.g. gstreamer-vaapi, gst-omx and the Android/iOS video codec API integration.

[dvrogozh]

Thank you for the comment. Your feedback is noted and we will consider it.

That’s not something new which you have suggested. Indeed moving to these base classes may provide better compatibility with the range of already existing plugins. At the same time I should note that whether to rely on some other code or not, including base classes we discuss here, is always a tradeoff. While I agree this will give better compatibility, at the same time this will significantly limit (ok, may limit) code flexibility: any base classes, any 3rd party code may have bugs, which will require workarounds – usually it is simpler to not rely on any code, but implement support on your own; secondly, plugins make friendship between 2 APIs, in this particular case gstreamer and mediasdk, so introduction of additional layer represented by base classes with their own API extensions/restrictions may provide difficulties in enabling features support, i.e. mapping of just native gstreamer API directly to mediasdk API may be simpler.

Making conclusions from the said above, I consider the current design simply more focused on the Proprietary plugins support, which is that, yes, we may not have compatibility with the whole gstreamer plugins universe, but right now we don’t pretend to be compatible with all of them! We focus on the best possible compatibility with the selected range of plugins and on the best support we can provide to the customers. Another approach would be to rely on some other 3rd party staff, for example, base classes – compatibility may be better in average, but some aspects may suffer. So, tradeoff.

However, in this particular case, it may be reasonable to switch to existing base classes. We will consider.

[sdroege]

Let me know if you have any questions about how to move to those base classes. They should very well support making use of all the features of the Media SDK API, they work with much worse APIs like OpenMAX IL already.

Generally our experience in GStreamer was/is that 3rd party plugins that don't use the base classes generally lack in usability (i.e. compatibility with other plugins) and features for no good reasons due to not using those base classes but reimplementing everything again. Also in general the amount of bugs was lower, and especially many bugs have to be fixed only in a single place.

As such I can only highly recommend you to moving to these base classes.

For the vpp element it will also make sense to move to GstBaseTransform btw, for exactly the same reasons.

[sdroege]

It seems like there's another plugin for the Media SDK, also made by Intel: https://github.com/01org/gstreamer-media-SDK
This one uses the base classes and looks like it's based on gstreamer-vaapi

What's the plan here?

[sdroege]

Had the wrong link above, sorry. Edited that now

[dmitryermilov]

Indeed, another team is working on a similar solution. We keep communication. The plan is to co-exist peacefully and exchange expertise. Architecture of these two solutions is completely different so it's hardly possible to merge them into one.

[rlim77]

I'm from the other team. :)

As mentioned by Dmitry, we are collaborating closely and have the same intend to extend MSDK use-ability and reach.

We have in certain use case (mostly close source RTOS) where a BSD license is preferred and this would be a perfect fit.