More robust docstring support

[chadrik]

Hi,
I was perusing the mypy repo and I discovered that mypy can parse docstrings. That's great! The support, however, is currently pretty basic. For starters, mypy currently only supports the google standard. From looking at the code, my instincts tell me that it might be somewhat fragile when exposed to a lot of real-world docstrings.

First, my motivation for using docstrings to provide typing information is the following:

1. Our company is already doing it throughout our codebase
2. It's nice to include type info alongside the description of an argument, so even with PEP484 now in python 3.5, I'll continue to enforce the use of proper docstrings
3. It works in both python 2 and 3

At this point, the python community semes to have settled on three docstring standards:

• reStructuredText
• google
• numpy

Sphinx supports the first out-of-the-box, with the latter two supported via the very nifty napolean extension, which comes bundled in sphinx-contrib. It works by converting google and numpy docstrings into reStructuredText, which is then parsed using docutils. Napolean is quite thorough, very battle-tested, and just forgiving enough on docstring authors to be usable (supports Return as well as Returns, Yield as well as Yields). Here's the relevant code. Also, handily enough, napolean already supports PEP848-style type hints. It depends on six and pockets but the latter could easily be removed.

My thought was that we could leverage napolean and docutils to get rock-solid parsing of all 3 formats. I'm assuming that adding dependencies to mypy is best avoided, so perhaps it can be installed as an extras_require. e.g. pip install mypy[docparse]. To avoid requiring all of sphinx, we could encourage the authors of napolean to provide the guts of their tool as a separate library (it used to be its own repo before it was consumed by sphinx-contrib).

What do you think? I'm glad to do all the work, including talking with the author of napolean, but I wanted to first see if this is something that the mypy team is open to.

Thanks!

[refi64]

Pretty sure this has been discussed numerous times:

• Should we support type annotations in docstrings for Python 2 code? #612
• Support annotations defined in docstrings #839
• Supporting type from docstring  #1669
• And more...

[chadrik]

The thing that confuses me is that docstring parsing is already in mypy, it's just not very good. I assumed that if the team was ideologically opposed to this that the feature would simply be absent. Should we close this issue or is there some hope?

[chadrik]

Hmmm... I see that the feature is disabled. Or more accurately that the result of parsing docstrings is always ignored.

[gvanrossum]

It looks like it was a Dropbox "hack week" project, which means it was an experiment. I am still philosophically opposed to it (just read the other issues, I don't want to have to repeat myself).

[chadrik]

I think that there will continue to be a demand for this feature because of the reasons I listed above, but I understand that you don't want to support this in mypy. So I'd like to propose a compromise solution: an optional, user-supplied callback function for processing docstrings. Something like this:

mypy --docstring-parser=mymodule.parse_docstring

This will enable those who need this feature to get it using third-party tools, which you will not have to write or support. The callback would be called in place of mypy.docstring.parse_docstring and would also allow the complete removal of the mypy.docstring module.

If the docstring-specific nature of this request is unappealing, an alternative would be to allow the passing of a custom Parser class via the command-line.

mypy --parser=mymodule.MyParser

Would either of these solutions be acceptable?

[gvanrossum]

I wonder if this could fall under the plugin design. There's an issue but I've forgotten the number.

…

--Guido (mobile)

[gvanrossum]

It's #1240.

[gvanrossum]

Core mypy just won't support this, and I'm closing this as wontfix to reflect that. You are welcome to write a script that converts docstrings to PEP 484. Once we have a plugin system you are also welcome to write a plugin for this.