Hook into preprocessing prior execution

[marscher]

nbconvert is very flexible, when it comes to preprocessing notebooks. One can define the preprocessors execution order and so on and so forth. Some issues like hiding certain stuff (#303, #65 etc.) could be easily implemented via a custom preprocessor.
But it does not seem to be possible to add them prior the ExecutePreprocessor does its job.
What do you think? Would it be worth adding a mechanism like this to make it more flexible for users?

[mgeier]

Yes, I think it would make sense to add some support for specifying custom preprocessors.

If you have a concrete idea how to do this, feel free to open a new PR or add some comments here.

Please note that I have hard-coded a few preprocessor settings here:

nbsphinx/src/nbsphinx.py

Lines 722 to 726 in 880bf3a

	config=traitlets.config.Config({
	'HighlightMagicsPreprocessor': {'enabled': True},
	# Work around https://github.com/jupyter/nbconvert/issues/720:
	'RegexRemovePreprocessor': {'enabled': False},
	}),

If custom preprocessors are supported, this might have to be changed, too (or not?).

[marscher]

I think the best option would be to implement this in the fashion of the nbconvert Exporter class:

https://github.com/jupyter/nbconvert/blob/a863787231f6e420d290acd98ef1e7af79b691e7/nbconvert/exporters/exporter.py#L48-L52

[marscher]

My current workaround looks like this. In order to generalize this, we would need to add a preprocessors argument to the application and pass it into the right spot.

def _preprocess_notebooks():
    """ hooks into ExecutePreprocessor.preprocess to execute our own filters."""
    import nbsphinx
    from nbconvert.preprocessors import Preprocessor
    org_method = nbsphinx.Exporter.from_notebook_node

    class MyPreprocessor(Preprocessor):
        def preprocess(self, nb, resources):
            # ... 
            return nb, resources

    class MyPreprossor2(Preprocessor):
        def preprocess(self, nb, resources):
            # ...
            return nb, resources

    def my_from_notebook_node(self, nb, resources, **kwargs):
        self.log.info('patched preprocessing method')
        filters = [MyPreprocessor(),
                   MyPreprossor2(),
                   ]
        for f in filters:
            nb, resources = f.preprocess(nb, resources=resources)

        return org_method(self, nb, resources=resources, **kwargs)

    nbsphinx.Exporter.from_notebook_node = my_from_notebook_node

[mgeier]

we would need to add a preprocessors argument

Could you please provide a few examples for how such arguments might look like?

[marscher]

I would do it as in nbsphinx, e.g. just provide a list of classes specified by fully qualified names (importable names).

[mgeier]

I'm not sure if custom pre-processors should be allowed before or after the ExecutePreprocessor, or both?

It would also be interesting to evaluate whether the extension mechanism of nbconvert could be exposed to users.

@marscher Except for hiding stuff, do you know other potential use cases?