Example: How to create and register a subliminal provider extension.

[etherealite]

If you are like me, you want to make your own provider but you don't want to fork or wrap subliminal.

For anyone wondering how to make their own provider 'extension', you may have noticed that documentation for this is a placeholder:
https://subliminal.readthedocs.io/en/latest/api/extensions.html

**Extensions
Extension managers for the providers, refiners and language converters.

But don't fret, after many trials and tribulations I have figured out how to perform the magical incantations necessary to make an extension. Let my own suffering be your salvation and read on...

All you have to do is make an entry in your pyproject.toml like so:

[project.entry-points."subliminal.providers"]
dummy = "ihatestevedore.dummyprovider:DummyProvider"

In this example, your package name is ihatestevedore and it has a module named dummyprovider containing your provider class definition. The entry name that you assign (left side of the '=') is arbitrary and can be whatever you want. You can then create your package layout like so:

/path/to/extension/ihatestevedore
└───ihatestevedore
    │   dummyprovider.py

Where you implement the DummyProvider class in the dummyprovider module:

from subliminal import Provider

class DummyProvider(Provider):
    pass

Here is what a complete project.toml might look like:

[project]
name = "ihatestevedore"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "subliminal>=2.2.1",
]

[project.entry-points."subliminal.providers"]
dummy = "ihatestevedore.dummyprovider:DummyProvider"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

After installing your package, there should be an entry_points.txt within its home in the site-packages directory:

$ ls python/Lib/site-packages/ihatestevedore-0.1.0.dist-info
direct_url.json
entry_points.txt
INSTALLER
METADATA
RECORD
REQUESTED
WHEEL

It should look like:

[subliminal.providers]
dummy = ihatestevedore.dummyprovider:DummyProvider

Now your extension should be available for discovery by subliminal and you can follow the rest of the docs and examples concerning making your own provider.

More reading on how this works:

• making plugins with stevedore https://docs.openstack.org/stevedore/latest/user/tutorial/creating_plugins.html
• python toml entrypoints spec https://packaging.python.org/en/latest/specifications/entry-points/#entry-points
• subliminal's wrapper of stevedore https://github.com/Diaoul/subliminal/blob/2.2.1/subliminal/extensions.py

[getzze]

Thanks for bringing this up :)

About the documentation, the extensions section has more information but it is not updated on Readthedocs (@Diaoul).
But there is a section about how to write your own provider here:
https://subliminal.readthedocs.io/en/latest/user/provider_guide.html

Thish doc section explains how to write a provider but not what to do after:

• make a PR with the new provider to make it available from subliminal (it can be a long process).
• write your own python package for the provider with an entry point to be picked by subliminal.
• (a third option, that doesn't exist yet, would be to make a PR to a python package (to be created) that has several providers but with an easier and faster merge process.)

Anyway, it would be great to add the information you give to the docs actually (if you have time for a PR :))

[etherealite]

I'd be willing to submit a PR once I have my provider working.

My own take on this is that adding information to the docs about extensions might be a touch premature. The cli module currently has no support for passing arguments to a provider.

Getting args passed from the cli to my provider required some hackery:

import click

from click import get_current_context
from subliminal.cli import (
    providers_config,
)
from subliminal import cli

# pull out the function wrapped by the command and other decorators.
# Remember to unreverse the params before reattaching them to the 
# replacement command.


# Fore sure breaking the original command group in some way here.
# There might be a way to reuse the existing group.
@click.group(
    name=cli.subliminal.name,
    params=list(reversed(cli.subliminal.params)),
    context_settings=cli.subliminal.context_settings,
    epilog=cli.subliminal.epilog,
)
# this ends up creating a whole new option group in the generated docs
# for some reason
@providers_config.option(
    '--jimaku',
    type=click.STRING,
    nargs=2,
    metavar='JIMAKU_APIKEY',
    help='Jimaku API key.',
)
def root_cmd_wraper(*args, **kwargs):
    restore_kwargs = {**kwargs}
    jimaku_args = restore_kwargs.pop('jimaku', None)
    
    if jimaku_args:
        # setup provider config here
        pass

    ctx = get_current_context()
    cli.subliminal.callback.__wrapped__(
        ctx, *args, **restore_kwargs
    )

# monkey patch subliminal
cli.subliminal = root_cmd_wraper

Do we really want to add something like: "to pass args to your provider, reverse engineer click and monkey patch core" to the official docs?

The extensions part of the docs being a stub is a stern, albeit indirect, indication that they're not fully developed to anyone thinking of writing one. Were there to be a guide on how to write an extension and then omit how to pass args with the cli, that feels like I'd be dead ending any hopeful developers.

Maybe a new section of the docs like 'experimental', 'only if you dare' or a clear demarcation at the top of the extensions section would help?

I hate to be clobbering github issues with non tickets like this. For the time being, this is the only place we have to collaborate on stuff. Would you mind keeping this one open?

[getzze]

You cannot pass arguments to the providers directly from the CLI (I am still thinking what would be the best way to do it) but you can pass them through the configuration file:
https://github.com/Diaoul/subliminal/blob/main/docs/config.toml

Create a section [provider.my_provider] and add the keyword arguments you need.

The documentation lacking clarity is enough reason to open an issue ;)

We also opened a Discord channel:
https://discord.com/invite/kXW6sWte9N

[etherealite]

I've collected all the hacks I did to get my provider to actually run, check this link for details.

There is one additional problem that is not addressed by #1180 making it impossible for providers from separate packages to work. Even though provider entry points are discovered by the extension manager, they are not passed to the AsyncProviderPool.

Extensions have to be 'registered' yet again using by calling the register() method on the extension manager before they are actually used.

I monkey patched click.option() to force the registration. Focus on this part:

provider_manager.register('jimaku = sublimaku:JimakuProvider')

Within this code:

_option = click.option

def option(*args, **kwargs):
    global provider_group
    # force the download subcommand to use make the jimaku provider available
    if '--provider' in args:
        provider_manager = globals().get('provider_manager')
        if provider_manager is None:
            from subliminal import provider_manager
        provider_manager.register('jimaku = sublimaku:JimakuProvider')
        kwargs['type'] = click.Choice(sorted(provider_manager.names()))

    # save a copy of the providers_config group so we can add our own
    # options to it later
    if provider_group is None and (group := kwargs.get('group')):
        if isinstance(group, OptionGroup) and group.name == 'Providers configuration':
            provider_group = group

    return _option(*args, **kwargs)

click.option = option

I'd argue that requiring a call on extension_manager.register() isn't adding any value, nor is it adding protection, as evidenced by the above code. Keeping core extensions separate from others would certainly come to have some merit in the future, but going through all the work of adding cross package discoverability and then barricading it off seems a little bit odd.