turn repo into a fully pip-installable package #671

jcb91 · 2016-07-18T01:26:58Z

Turn setup.py into a proper setuptools installer

fix setup.py & MANIFEST.in to ensure all files are included as part of the package
replace old installation mechanism with a jupyter contrib subcommand to install/uninstall all of the nbextensions using jupyter's machinery (or the compatibility shims from jupyter_contrib_core for notebook versions < 4.2)
fix version to be pep440-compliant
add setup.cfg to support universal wheel builds, flake8 linting and isort import sorting
add add bumpversion config for easier version management
add a module and associated app to migrate old (pre-this PR) installs

* make a proper package, which can therefore be installed * include package data * don't modify config as part of pip install, only do it as part of explicit script * add setup.cfg to support universal wheel builds

jcb91 · 2016-07-18T01:29:56Z

still to do:

fix repo name in setup.py
edit readme to reflect new installation procedure

juhasch · 2016-07-18T18:40:37Z

So what is the best way to move this to jupyter-contrib ?
Merge and then move ?

Also todo (probably not in this PR, but needs to be done):

document templates better
document pre-/postprocessors better (and rename some of them for consistency)
toc exporter using entrypoints for nbconvert ?

I could work on the first two bullet points. Also, we could autogenerate documentation from the md file. I believe you gave it a try before ?

jcb91 · 2016-07-18T19:30:01Z

we could autogenerate documentation from the md file

Not exactly sure what you mean by this - the md files generally are the documentation, as far as I can see?

juhasch · 2016-07-18T19:32:02Z

What I meant was to autogenerate a documentation like standard modules have, i.e. like HTML or PDF docs, usually generated by sphinx or so.

jcb91 · 2016-07-18T19:58:57Z

oh, right! I guess this is what gets hosted at, e.g. http://jupyter-notebook.readthedocs.io? But yes, I'm sure we could autogenerate some of the docs/ tree from the existing markdown through nbconvert or similar

juhasch · 2016-07-18T20:03:04Z

I tried to get sphinx to autogenerate documentation from md files some time ago, but got stuck in path problems, i.e. getting sphinx to read from the nbextension directories and getting the graphics imported correctly. It should be doable, however.

Now first things first, let's get this PR ready and the we will see, I guess.

juhasch · 2016-07-18T20:06:20Z

Btw. what is scripts/jupyter-contrib-nbextension for ?

jcb91 · 2016-07-18T20:13:10Z

Btw. what is scripts/jupyter-contrib-nbextension for ?

Ah, that's part of the new installation/configuration mechanism, for which I haven't yet written docs/readme. It makes the application callable as a jupyter contrib subcommand. Essentially the installation is now:

pip install jupyter_contrib_nbextensions  # install the pip-package
jupyter contrib nbextension install --user  # install all nbextensions from pip package to jupyter dirs

where the second part is basically just a wrapper around notebook's jupyter nbextension install, (plus shims from jupyter_contrib_core to provide the notebook 4.2-compatible nbextension api).

juhasch · 2016-07-18T20:34:01Z

OK, thanks for the explanation. Sounds good.
Hmm, after installation when trying the jupyter contrib command, I get this:

d:\source.local\IPython-notebook-extensions>jupyter contrib
Traceback (most recent call last):
  File "C:\Anaconda3\Scripts\jupyter-contrib-script.py", line 9, in <module>
    load_entry_point('jupyter-contrib-core==0.3.0', 'console_scripts', 'jupyter-contrib')()
  File "C:\Anaconda3\lib\site-packages\jupyter_core\application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "C:\Anaconda3\lib\site-packages\traitlets\config\application.py", line 594, in launch_instance
    app = cls.instance(**kwargs)
  File "C:\Anaconda3\lib\site-packages\traitlets\config\configurable.py", line 405, in instance
    inst = cls(*args, **kwargs)
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_core-0.3.0-py3.5.egg\jupyter_contrib_core\application.p
y", line 27, in __init__
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_core-0.3.0-py3.5.egg\jupyter_contrib_core\application.p
y", line 43, in _refresh_subcommands
  File "c:\anaconda3\envs\test\lib\site-packages\setuptools-23.0.0-py3.5.egg\pkg_resources\__init__.py", line 2229, in load
  File "c:\anaconda3\envs\test\lib\site-packages\setuptools-23.0.0-py3.5.egg\pkg_resources\__init__.py", line 2235, in resolve
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_nbextensions-0.0.0-py3.5.egg\jupyter_contrib_nbextensio
ns\application.py", line 14, in <module>
    from jupyter_contrib_nbextensions.install import install, uninstall
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_nbextensions-0.0.0-py3.5.egg\jupyter_contrib_nbextensio
ns\install.py", line 17, in <module>
    import jupyter_contrib_nbextensions.nbconvert_support
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_nbextensions-0.0.0-py3.5.egg\jupyter_contrib_nbextensio
ns\nbconvert_support\__init__.py", line 11, in <module>
    from .toc2 import TocExporter
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_nbextensions-0.0.0-py3.5.egg\jupyter_contrib_nbextensio
ns\nbconvert_support\toc2.py", line 16, in <module>
    from jupyter_contrib_nbextensions.nbconvert_support import templates_directory
ImportError: cannot import name 'templates_directory'

jcb91 · 2016-07-18T21:03:57Z

oh boo, this is something I broke in a rebase, I think. I haven't checked all my CI tests, so it's not surprising I missed something, I guess. I'll check it out now

jcb91 · 2016-07-18T21:48:01Z

Yep, seems I accidentally introduced a circular import. I'll rebase to fix this...

juhasch · 2016-07-19T20:17:05Z

This is what I see when trying jupyter contrib command after installing using python setup.py install :

Unknown option

C:\Anaconda3\envs\test\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'platform'
  warnings.warn(msg)

requests package is missing as dependency

(test) d:\source.local\IPython-notebook-extensions>jupyter contrib
Traceback (most recent call last):
  File "C:\Anaconda3\envs\test\Scripts\jupyter-contrib-script.py", line 9, in <module>
    load_entry_point('jupyter-contrib-core==0.3.0', 'console_scripts', 'jupyter-contrib')()
  File "C:\Anaconda3\envs\test\lib\site-packages\jupyter_core\application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "C:\Anaconda3\envs\test\lib\site-packages\traitlets\config\application.py", line 594, in launch_instance
    app = cls.instance(**kwargs)
  File "C:\Anaconda3\envs\test\lib\site-packages\traitlets\config\configurable.py", line 412, in instance
    inst = cls(*args, **kwargs)
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_core-0.3.0-py3.5.egg\jupyter_contrib_core\application.p
y", line 27, in __init__
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_core-0.3.0-py3.5.egg\jupyter_contrib_core\application.p
y", line 43, in _refresh_subcommands
  File "c:\anaconda3\envs\test\lib\site-packages\setuptools-23.0.0-py3.5.egg\pkg_resources\__init__.py", line 2229, in load
  File "c:\anaconda3\envs\test\lib\site-packages\setuptools-23.0.0-py3.5.egg\pkg_resources\__init__.py", line 2235, in resolve
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_nbextensions-0.0.0-py3.5.egg\jupyter_contrib_nbextensio
ns\application.py", line 14, in <module>
    from jupyter_contrib_nbextensions.install import install, uninstall
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_nbextensions-0.0.0-py3.5.egg\jupyter_contrib_nbextensio
ns\install.py", line 17, in <module>
    import jupyter_contrib_nbextensions.nbconvert_support
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_nbextensions-0.0.0-py3.5.egg\jupyter_contrib_nbextensio
ns\nbconvert_support\__init__.py", line 10, in <module>
    from .pre_svg2pdf import SVG2PDFPreprocessor
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_nbextensions-0.0.0-py3.5.egg\jupyter_contrib_nbextensio
ns\nbconvert_support\pre_svg2pdf.py", line 15, in <module>
    import requests
ImportError: No module named 'requests'

templates_directory is not found

(test) d:\source.local\IPython-notebook-extensions>jupyter contrib
Traceback (most recent call last):
  File "C:\Anaconda3\envs\test\Scripts\jupyter-contrib-script.py", line 9, in <module>
    load_entry_point('jupyter-contrib-core==0.3.0', 'console_scripts', 'jupyter-contrib')()
  File "C:\Anaconda3\envs\test\lib\site-packages\jupyter_core\application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "C:\Anaconda3\envs\test\lib\site-packages\traitlets\config\application.py", line 594, in launch_instance
    app = cls.instance(**kwargs)
  File "C:\Anaconda3\envs\test\lib\site-packages\traitlets\config\configurable.py", line 412, in instance
    inst = cls(*args, **kwargs)
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_core-0.3.0-py3.5.egg\jupyter_contrib_core\application.p
y", line 27, in __init__
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_core-0.3.0-py3.5.egg\jupyter_contrib_core\application.p
y", line 43, in _refresh_subcommands
  File "c:\anaconda3\envs\test\lib\site-packages\setuptools-23.0.0-py3.5.egg\pkg_resources\__init__.py", line 2229, in load
  File "c:\anaconda3\envs\test\lib\site-packages\setuptools-23.0.0-py3.5.egg\pkg_resources\__init__.py", line 2235, in resolve
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_nbextensions-0.0.0-py3.5.egg\jupyter_contrib_nbextensio
ns\application.py", line 14, in <module>
    from jupyter_contrib_nbextensions.install import install, uninstall
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_nbextensions-0.0.0-py3.5.egg\jupyter_contrib_nbextensio
ns\install.py", line 17, in <module>
    import jupyter_contrib_nbextensions.nbconvert_support
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_nbextensions-0.0.0-py3.5.egg\jupyter_contrib_nbextensio
ns\nbconvert_support\__init__.py", line 11, in <module>
    from .toc2 import TocExporter
  File "C:\Users\hasch\AppData\Roaming\Python\Python35\site-packages\jupyter_contrib_nbextensions-0.0.0-py3.5.egg\jupyter_contrib_nbextensio
ns\nbconvert_support\toc2.py", line 16, in <module>
    from jupyter_contrib_nbextensions.nbconvert_support import templates_directory
ImportError: cannot import name 'templates_directory'

jcb91 · 2016-07-22T15:41:18Z

@juhasch sorry for silence on this, it's presumably a bunch of bugs I've fixed between the pippify branch (this PR) and my CI branch, which actually includes travis & appveyor testing, since the CI branch seems to work ok. I'll try to investigate soon.

jcb91 · 2016-07-22T15:49:31Z

UserWarning: Unknown distribution option: 'platform' warnings.warn(msg)

This a warning resulting from setuptools/distutils differences that I don't understand properly from googling it (everything I can find seems to say that it means you should have setuptools, but I already do, so I don't see how that can be the problem). At any rate, it doesn't actually cause any problems.

requests is missing as a dependency

True, I've apparently fixed this in my CI branch through remaking the test_preprocessors, coincidentally not using requests.

templates_directory is not found

this is just a side-effect of the ImportError in pre_svg2pdf, which means that the function never gets defined...

juhasch · 2016-07-22T15:54:22Z

No worries :-)

So will you update this branch or is it better to try your CI branch ?

jcb91 · 2016-07-22T15:57:11Z

will you update this branch or is it better to try your CI branch ?

mm, I don't mind really, but I guess merging my CI commits into my pippify branch might be the simplest. I have a couple of changes to the CI one to make first though, to check that migrating from the old repo works ok...

* add function to find templates directory from nbconvert_support package * update config_scripts to use new package structure * update some readme files which refer to templates directory

based on a jupyter application calling notebook 4.2-style extension API methods Also make check for running server more robust

* tests on travis and appveyor via tox * codecov.yml for github status suppression

* consolidate common operations into functions * test imports in test_preprocessors * skip rather than fail on tests which require pandoc if we can't find it * move svg2pdf inkscape-finding into a function which doesn't raise exceptions, allowing us to skip test if inkscape can't be found, and skip accordingly * use urllib instead of requests in svg2pdf preprocessor * fix flake8 complaints

* fix repo URLs in setup.py * clarify separation of `jupyter_nbextensions_configurator` in readme * add reference to github view for viewing repository readmes * update readme with migration instructions * formatting update for notbook extension structure in readme

* fix urls for appveyor * move notes about custom installs after regualr instrucitons * number installation steps

jcb91 · 2016-07-25T10:29:34Z

ok, this seems to have all of its CI tests passing now. I've had a go at updating the readme too.

juhasch · 2016-07-25T19:51:46Z

README.md

-More information can also be found in the
-[Wiki](https://github.com/ipython-contrib/IPython-notebook-extensions/wiki).
-
-
 The install process has two steps:



I think we should first explain the installation approach, and then describe the steps in detail. How about this:

To install notebook extensions, three steps are required. First, this Python package needs to be installed. Then, the notebook extensions themselves can be copied to the Jupyter data directory.
Finally, the installed notebook extensions can be enabled, either by using built-in Jupyter commands, or more convenient by using the jupyter_nbextensions_configurator server extension.

The Python package installation step is necessary to allow painless installation of the extensions togther with additional items like nbconvert templates, pre-/postprocessors, and exporters.

Sure, sounds reasonable. So that would then replace the current bullet points, I guess? PR away 😉

juhasch · 2016-07-25T19:54:06Z

Looks good. I think we should merge and rename the repo.
It is definitely a big improvement compared to master. Great work!

jfbercher · 2016-07-25T21:30:37Z

This is really nice indeed! A big and great work!
I will have a closer look tomorrow.

I have some improvements to latex_envs and two related exporters that are almost ready. Perhaps should I wait for the new repo to PR and add entry points to the setup.py? (or PR to @jcb91 pippify?)

jcb91 · 2016-07-25T21:51:48Z

Also, I was wondering whether it might be simpler to make the migrate happen automatically on running the install command if pip is importable and lists the old package name (Python-contrib-nbextensions) as installed?

jcb91 · 2016-07-25T22:01:45Z

should I wait for the new repo to PR and add entry points to the setup.py?

Not sure what you mean by entrypoints here @jfbercher, what exactly are you attempting to do? point me to a branch if you like 😄

or PR to @jcb91 pippify?

I feel like that might get rather confusing for later merges... I guess you could open a PR against the main master, then you can always rebase your branch later to fix merges locally once this is merged, then force-push the rebased branch to update the PR automatically. I know rebases and forced updates are frowned upon from a git standpoint, but I feel like as long as you've warned people that the branch may be volatile, it's not a problem...

jfbercher · 2016-07-25T22:27:05Z

I meant adding entry points in order to register the exporters with the package, thus enabling to call them with a custom short name. This is explained eg here.

There is no urgency, I'll wait for the new repo.

juhasch · 2016-07-26T17:48:54Z

So let's merge.

Turn setup.py into a proper setuptools installer

199bfe2

* make a proper package, which can therefore be installed * include package data * don't modify config as part of pip install, only do it as part of explicit script * add setup.cfg to support universal wheel builds

jcb91 mentioned this pull request Jul 18, 2016

migrate config settings correctly #670

Closed

jcb91 force-pushed the pippify branch from 853c292 to fd30bed Compare July 18, 2016 21:48

jcb91 added 10 commits July 23, 2016 23:45

fixes for nbconvert_support package

4014c30

* add function to find templates directory from nbconvert_support package * update config_scripts to use new package structure * update some readme files which refer to templates directory

New installation method

4f23698

based on a jupyter application calling notebook 4.2-style extension API methods Also make check for running server more robust

add a module and associated app to migrate old installs

1620db1

add CI testing

84ead09

* tests on travis and appveyor via tox * codecov.yml for github status suppression

add basic tests

adb71bc

create all preprocessing test notebooks programatically.

32b71b5

add preprocessor test for collapsible headings.

018c43d

fix collapsible_headings preprocessor

19938b9

fix isort complaints

c04edfe

jcb91 force-pushed the pippify branch from fd30bed to c04edfe Compare July 24, 2016 00:14

update readme

cd1c561

jcb91 force-pushed the pippify branch from 50c947d to cd1c561 Compare July 24, 2016 01:05

jcb91 added 2 commits July 25, 2016 10:59

yet more readme updates [skip ci]

f749b16

* fix urls for appveyor * move notes about custom installs after regualr instrucitons * number installation steps

juhasch reviewed Jul 25, 2016
View reviewed changes

juhasch merged commit 630caaa into ipython-contrib:restructure Jul 26, 2016

jcb91 deleted the pippify branch July 29, 2016 23:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

turn repo into a fully pip-installable package #671

turn repo into a fully pip-installable package #671

jcb91 commented Jul 18, 2016 •

edited

Loading

jcb91 commented Jul 18, 2016

juhasch commented Jul 18, 2016

jcb91 commented Jul 18, 2016

juhasch commented Jul 18, 2016

jcb91 commented Jul 18, 2016

juhasch commented Jul 18, 2016

juhasch commented Jul 18, 2016

jcb91 commented Jul 18, 2016

juhasch commented Jul 18, 2016

jcb91 commented Jul 18, 2016

jcb91 commented Jul 18, 2016

juhasch commented Jul 19, 2016

jcb91 commented Jul 22, 2016

jcb91 commented Jul 22, 2016

juhasch commented Jul 22, 2016

jcb91 commented Jul 22, 2016

jcb91 commented Jul 25, 2016

juhasch Jul 25, 2016

jcb91 Jul 25, 2016

juhasch commented Jul 25, 2016

jfbercher commented Jul 25, 2016

jcb91 commented Jul 25, 2016

jcb91 commented Jul 25, 2016

jfbercher commented Jul 25, 2016

juhasch commented Jul 26, 2016

turn repo into a fully pip-installable package #671

turn repo into a fully pip-installable package #671

Conversation

jcb91 commented Jul 18, 2016 • edited Loading

jcb91 commented Jul 18, 2016

juhasch commented Jul 18, 2016

jcb91 commented Jul 18, 2016

juhasch commented Jul 18, 2016

jcb91 commented Jul 18, 2016

juhasch commented Jul 18, 2016

juhasch commented Jul 18, 2016

jcb91 commented Jul 18, 2016

juhasch commented Jul 18, 2016

jcb91 commented Jul 18, 2016

jcb91 commented Jul 18, 2016

juhasch commented Jul 19, 2016

jcb91 commented Jul 22, 2016

jcb91 commented Jul 22, 2016

juhasch commented Jul 22, 2016

jcb91 commented Jul 22, 2016

jcb91 commented Jul 25, 2016

juhasch Jul 25, 2016

Choose a reason for hiding this comment

jcb91 Jul 25, 2016

Choose a reason for hiding this comment

juhasch commented Jul 25, 2016

jfbercher commented Jul 25, 2016

jcb91 commented Jul 25, 2016

jcb91 commented Jul 25, 2016

jfbercher commented Jul 25, 2016

juhasch commented Jul 26, 2016

jcb91 commented Jul 18, 2016 •

edited

Loading