configure magic conversion is seamless

[malayhm]

%%configure magic code conversion is not seamless, for example:

%%configure -f
{"executorMemory": "3072M", "executorCores": 4, "numExecutors":10}

gets transformed to python code & r code

# %%configure -f
{"executorMemory": "3072M", "executorCores": 4, "numExecutors":10}

and gets transformed to scala code

// %%configure -f
{"executorMemory": "3072M", "executorCores": 4, "numExecutors":10}.

I think the conversion should be completely commented similar to SQL magic %%sql.

[mwouts]

Hello @malayhm , thanks for reporting this.
This happens because Jupytext is not aware of the %%configure cell magic.

Now we could think of three ways of fixing this...
a) You add a line continuation \\ character to the configure magic cell, e.g.

%%configure -f \\
{"executorMemory": "3072M", "executorCores": 4, "numExecutors":10}

b) we add configure as an additional Jupyter language (can you tell us which package defines the %%configure magic?)
c) we always escape the full cell when a magic cell command is used.

I just gave a try to c), and I am not so sure we want to do that... For instance, for %%time` I am not sure we want to comment out the content of the cell. What do you think?

[malayhm]

@mwouts Thanks for the quick response. Let me dig deeper and find out which one of the above solutions will bring the desired behavior.

[mwouts]

Sure! Keep us posted. Let us know if know if a) works for you. Also, if you want to try b) locally, that's easy, you just need to add "configure" to _JUPYTER_LANGUAGES in jupytext/languages.py.

[malayhm]

Hello @mwouts,

Here are my findings:

a) You add a line continuation \\ character to the configure magic cell

This produces the same output, hence it will not work.

b) we add configure as an additional Jupyter language (can you tell us which package defines the %%configure magic?)

It is part of sparkmagic, https://github.com/jupyter-incubator/sparkmagic

c) we always escape the full cell when a magic cell command is used.

I think, jupytext should support %%time, we can not use the blanket approach to comment out all the magic.

There will be one more use case where someone implements custom magic, it will follow into the same category where handling has to be taken care of in the code.

Considering all the above points, I think if we come up with an option to add new languages as a configuration option through command line or notebook configuration, that would solve the issue for %%configure as well as any new such options.
Basically add support for passing new items for _JUPYTER_LANGUAGES in https://github.com/mwouts/jupytext/blob/master/jupytext/languages.py.

What do you think?

[mwouts]

Hi @malayhm , yes I agree that adding new items to the list of cell magics would be great. I'll think about how to do that.

Also, I think the continuation line approach (which is only a quick fix, I agree) should work. At least it does work in this unit test: 6b6317d. Would you like to give it a second try? Maybe in the notebook you should enter a single \?

[malayhm]

@mwouts The above approach will work when the input is controllable. I am trying to handle a case where I don't have control over the cell content hence this approach will fail.

I think there will be two options for the smooth conversion:

1. Code cleanup
2. Configuration option as suggested in my previous comment.

I think option 1 will change the user input and the reverse conversion will not retain the actual data.

What are your thoughts?

[mwouts]

Hi @malayhm , the next release (version 1.6.0) will have a new configuration option custom_cell_magics.

For instance, if you add

custom_cell_magics = "configure,local"

to your jupytext.toml file, then all the notebook in that folder and subfolders will recognize configure and local as cell magics, and hence the whole cell will be commented out.

[malayhm]

Thanks for the update @mwouts.