Improve CLI, refactor and document stubgen

docs/source/index.rst

@@ -53,6 +53,7 @@ Mypy is a static type checker for Python 3 and Python 2.7.
    mypy_daemon
    installed_packages
    extending_mypy
+   stubgen
 
 .. toctree::
    :maxdepth: 2

docs/source/stubgen.rst

@@ -0,0 +1,143 @@
+.. _stugen:
+
+Automatic stub generation
+=========================
+
+Stub files (see `PEP 484 <https://www.python.org/dev/peps/pep-0484/#stub-files>`_)
+are files containing only type hints not the actual runtime implementation.
+They can be useful for C extension modules, third-party modules whose authors
+have not yet added type hints, etc.
+
+Mypy comes with a ``stubgen`` tool for automatic generation of
+stub files (``.pyi`` files) from Python source files. For example,
+this source file:
+
+.. code-block:: python
+
+   from other_module import dynamic
+
+   BORDER_WIDTH = 15
+
+   class Window:
+       parent = dynamic()
+       def __init__(self, width, hight):
+           self.width = width
+           self.hight = hight
+
+   def create_empty() -> Window:
+       return Window(0, 0)
+
+will be transformed into the following stub file:
+
+.. code-block:: python
+
+   from typing import Any
+
+   BORDER_WIDTH: int = ...
+
+   class Window:
+       parent: Any = ...
+       width: Any = ...
+       height: Any: ...
+       def __init__(self, width, height) -> None: ...
+
+   def create_empty() -> Window: ...
+
+In most cases, the auto-generated stub files require manual check for
+completeness. This section documents stubgen's command line interface.
+You can view a quick summary of the available flags by running
+``stubgen --help``.
+
+.. note::
+
+   Stubgen tool is still experimental and will evolve. Command line flags
+   are liable to change between releases.
+
+Specifying what to stub
+***********************
+
+By default, you can specify for what code you want to generate
+stub files by passing in the paths to the sources::
+
+    $ stubgen foo.py bar.py some_directory
+
+Note that directories are checked recursively.
+
+Stubgen also lets you specify modules for stub generation in two
+other ways. The relevant flags are:
+
+``-m MODULE``, ``--module MODULE``
+    Asks stubgen to generate stub file for the provided module. This flag
+    may be repeated multiple times.
+
+    Stubgen *will not* recursively generate stubs for any submodules of
+    the provided module.
+
+``-p PACKAGE``, ``--package PACKAGE``
+    Asks stubgen to generate stubs for the provided package. This flag may
+    be repeated multiple times.
+
+    Stubgen *will* recursively generate stubs for all submodules of
+    the provided package. This flag is identical to ``--module`` apart from
+    this behavior.
+
+.. note::
+
+   You can use either module/package mode or source code mode, these two
+   can't be mixed together in the same stubgen invocation.
+
+Specifying how to generate stubs
+********************************
+
+By default stubgen will try to import the modules and packages given.
+This has an advantage of possibility to discover and stub also C modules.
+By default stubgen will use mypy to semantically analyze the Python
+sources found. To alter this behavior, you can use following flags:
+
+``--no-import``
+    Don't try to import modules, instead use mypy's normal mechanisms to find
+    sources. This will not find any C extension modules. Stubgen also uses
+    runtime introspection to find actual value of ``__all__``, so with this flag
+    the set of re-expoted names may be incomplete. This flag will be useful if
+    importing the module causes an error.
+
+``--parse-only``
+    Don't perform mypy semantic analysis of source files. This may generate
+    worse stubs, in particular some module, class, and function aliases may
+    be typed as variables with ``Any`` type. This can be useful if semantic
+    analysis causes a critical mypy error.
+
+``--doc-dir PATH``
+    Try to infer function and class signatures by parsing .rst documentation
+    in ``PATH``. This may result in better stubs, but currently only works for
+    C modules.
+
+Additional flags
+****************
+
+``--py2``
+    Run stubgen in Python 2 mode (the default is Python 3 mode).
+
+``--ignore-errors``
+    Ignore any errors when trying to generate stubs for modules and packages.
+    This may be useful for C modules where runtime introspection is used
+    intensively.
+
+``--include-private``
+    Generate stubs for objects and members considered private (with single
+    leading underscore and no trailing underscores).
+
+``--search-path PATH``
+    Specify module search directories, separated by colons (currently only
+    used if ``--no-import`` is given).
+
+``--python-executable PATH``
+    Use Python interpreter at ``PATH`` for module finding and runtime
+    introspection (has no effect with ``--no-import``). Currently only works
+    for Python 2. In Python 3 mode only the default interpreter will be used.
+
+``-o PATH``, ``--output PATH``
+    Change the output directory. By default the stubs are written in
+    ``./out`` directory. The output directory will be created if it didn't
+    exist. Existing stubs in the output directory will be overwritten without
+    warning.

mypy/build.py

@@ -1463,7 +1463,8 @@ def __init__(self,
                  # as error reporting should be avoided.
                  temporary: bool = False,
                  ) -> None:
-        assert id or path or source is not None, "Neither id, path nor source given"
+        if not temporary:
+            assert id or path or source is not None, "Neither id, path nor source given"
         self.manager = manager
         State.order_counter += 1
         self.order = State.order_counter

mypy/newsemanal/semanal.py

@@ -245,7 +245,9 @@ def __init__(self,
         self.incomplete_namespaces = incomplete_namespaces
         self.postpone_nested_functions_stack = [FUNCTION_BOTH_PHASES]
         self.postponed_functions_stack = []
-        self.all_exports = set()  # type: Set[str]
+        self.all_exports = []  # type: List[str]
+        # Map from module id to list of explicitly exported names (i.e. names in __all__).
+        self.export_map = {}  # type: Dict[str, List[str]]
         self.plugin = plugin
         # If True, process function definitions. If False, don't. This is used
         # for processing module top levels in fine-grained incremental mode.
@@ -317,6 +319,8 @@ def visit_file(self, file_node: MypyFile, fnam: str, options: Options,
                     if name not in self.all_exports:
                         g.module_public = False
 
+            self.export_map[self.cur_mod_id] = self.all_exports
+            self.all_exports = []
             del self.options
             del self.patches
             del self.cur_mod_node
@@ -3922,7 +3926,7 @@ def add_exports(self, exp_or_exps: Union[Iterable[Expression], Expression]) -> N
         exps = [exp_or_exps] if isinstance(exp_or_exps, Expression) else exp_or_exps
         for exp in exps:
             if isinstance(exp, StrExpr):
-                self.all_exports.add(exp.value)
+                self.all_exports.append(exp.value)
 
     def check_no_global(self, n: str, ctx: Context,
                         is_overloaded_func: bool = False) -> None:

mypy/nodes.py

@@ -652,6 +652,8 @@ class Decorator(SymbolNode, Statement):
 
     func = None  # type: FuncDef                # Decorated function
     decorators = None  # type: List[Expression] # Decorators (may be empty)
+    # Some decorators are removed by semanal, keep the original here.
+    original_decorators = None  # type: List[Expression]
     # TODO: This is mostly used for the type; consider replacing with a 'type' attribute
     var = None  # type: Var                     # Represents the decorated function obj
     is_overload = False
@@ -661,6 +663,7 @@ def __init__(self, func: FuncDef, decorators: List[Expression],
         super().__init__()
         self.func = func
         self.decorators = decorators
+        self.original_decorators = decorators.copy()
         self.var = var
         self.is_overload = False
 

mypy/semanal.py

@@ -242,7 +242,9 @@ def __init__(self,
         self.missing_modules = missing_modules
         self.postpone_nested_functions_stack = [FUNCTION_BOTH_PHASES]
         self.postponed_functions_stack = []
-        self.all_exports = set()  # type: Set[str]
+        self.all_exports = []  # type: List[str]
+        # Map from module id to list of explicitly exported names (i.e. names in __all__).
+        self.export_map = {}  # type: Dict[str, List[str]]
         self.plugin = plugin
         # If True, process function definitions. If False, don't. This is used
         # for processing module top levels in fine-grained incremental mode.
@@ -314,6 +316,8 @@ def visit_file(self, file_node: MypyFile, fnam: str, options: Options,
                     if name not in self.all_exports:
                         g.module_public = False
 
+            self.export_map[self.cur_mod_id] = self.all_exports
+            self.all_exports = []
             del self.options
             del self.patches
             del self.cur_mod_node
@@ -3707,7 +3711,7 @@ def add_exports(self, exp_or_exps: Union[Iterable[Expression], Expression]) -> N
         exps = [exp_or_exps] if isinstance(exp_or_exps, Expression) else exp_or_exps
         for exp in exps:
             if isinstance(exp, StrExpr):
-                self.all_exports.add(exp.value)
+                self.all_exports.append(exp.value)
 
     def check_no_global(self, n: str, ctx: Context,
                         is_overloaded_func: bool = False) -> None:

mypy/stubdoc.py

@@ -0,0 +1,155 @@
+"""Parsing/inferring signatures from documentation.
+
+This module provides several functions to generate better stubs using
+docstrings and Sphinx docs (.rst files).
+"""
+
+from typing import Optional, MutableMapping, MutableSequence, List, Sequence, Tuple
+
+import re
+
+# Type alias for signatures in format ('func_name', '(arg, opt_arg=False)').
+Sig = Tuple[str, str]
+
+
+def parse_signature(sig: str) -> Optional[Tuple[str,
+                                                List[str],
+                                                List[str]]]:
+    """Split function signature into its name, positional an optional arguments.
+
+    The expected format is "func_name(arg, opt_arg=False)". Return the name of function
+    and lists of positional and optional argument names.
+    """
+    m = re.match(r'([.a-zA-Z0-9_]+)\(([^)]*)\)', sig)
+    if not m:
+        return None
+    name = m.group(1)
+    name = name.split('.')[-1]
+    arg_string = m.group(2)
+    if not arg_string.strip():
+        # Simple case -- no arguments.
+        return name, [], []
+
+    args = [arg.strip() for arg in arg_string.split(',')]
+    positional = []
+    optional = []
+    i = 0
+    while i < len(args):
+        # Accept optional arguments as in both formats: x=None and [x].
+        if args[i].startswith('[') or '=' in args[i]:
+            break
+        positional.append(args[i].rstrip('['))
+        i += 1
+        if args[i - 1].endswith('['):
+            break
+    while i < len(args):
+        arg = args[i]
+        arg = arg.strip('[]')
+        arg = arg.split('=')[0]
+        optional.append(arg)
+        i += 1
+    return name, positional, optional
+
+
+def build_signature(positional: Sequence[str],
+                    optional: Sequence[str]) -> str:
+    """Build function signature from lists of positional and optional argument names."""
+    args = []  # type: MutableSequence[str]
+    args.extend(positional)
+    for arg in optional:
+        if arg.startswith('*'):
+            args.append(arg)
+        else:
+            args.append('%s=...' % arg)
+    sig = '(%s)' % ', '.join(args)
+    # Ad-hoc fixes.
+    sig = sig.replace('(self)', '')
+    return sig
+
+
+def parse_all_signatures(lines: Sequence[str]) -> Tuple[List[Sig],
+                                                        List[Sig]]:
+    """Parse all signatures in a given reST document.
+
+    Return lists of found signatures for functions and classes.
+    """
+    sigs = []
+    class_sigs = []
+    for line in lines:
+        line = line.strip()
+        m = re.match(r'\.\. *(function|method|class) *:: *[a-zA-Z_]', line)
+        if m:
+            sig = line.split('::')[1].strip()
+            parsed = parse_signature(sig)
+            if parsed:
+                name, fixed, optional = parsed
+                if m.group(1) != 'class':
+                    sigs.append((name, build_signature(fixed, optional)))
+                else:
+                    class_sigs.append((name, build_signature(fixed, optional)))
+
+    return sorted(sigs), sorted(class_sigs)
+
+
+def find_unique_signatures(sigs: Sequence[Sig]) -> List[Sig]:
+    """Remove names with duplicate found signatures."""
+    sig_map = {}  # type: MutableMapping[str, List[str]]
+    for name, sig in sigs:
+        sig_map.setdefault(name, []).append(sig)
+
+    result = []
+    for name, name_sigs in sig_map.items():
+        if len(set(name_sigs)) == 1:
+            result.append((name, name_sigs[0]))
+    return sorted(result)
+
+
+def infer_sig_from_docstring(docstr: str, name: str) -> Optional[Tuple[str, str]]:
+    """Look for signature of function with given name in a docstring.
+
+    Signature is any string of the format <function_name>(<signature>) -> <return type>
+    or perhaps without the return type.
+
+    In the signature, we allow the following characters:
+    * colon/equal: to match default values, like "a: int = 1"
+    * comma/space/brackets: for type hints like "a: Tuple[int, float]"
+    * dot: for classes annotating using full path, like "a: foo.bar.baz"
+
+    Return a pair of argument list, return type, for example: '(arg: int, x=None)', 'Any',
+    or None, if there is no match.
+    """
+    if not docstr:
+        return None
+    docstr = docstr.lstrip()
+    sig_str = r'\([a-zA-Z0-9_=:, \[\]\.]*\)'
+    sig_match = r'%s(%s)' % (name, sig_str)
+
+    # First, try to capture return type; we just match until end of line
+    m = re.match(sig_match + ' -> ([a-zA-Z].*)$', docstr, re.MULTILINE)
+    if m:
+        # strip potential white spaces at the right of return type
+        return m.group(1), m.group(2).rstrip()
+
+    # If that didn't work, try to not match return type
+    m = re.match(sig_match, docstr)
+    if m:
+        return m.group(1), 'Any'
+
+    # Give up.
+    return None
+
+
+def infer_prop_type_from_docstring(docstr: str) -> Optional[str]:
+    """Check for Google/Numpy style docstring type annotation for a property.
+
+    The docstring has the format "<type>: <descriptions>".
+    In the type string, we allow the following characters:
+    * dot: because sometimes classes are annotated using full path
+    * brackets: to allow type hints like List[int]
+    * comma/space: things like Tuple[int, int]
+    """
+    if not docstr:
+        return None
+    test_str = r'^([a-zA-Z0-9_, \.\[\]]*): '
+    m = re.match(test_str, docstr)
+    return m.group(1) if m else None