Foundation for fine-grained incremental checking #2838

JukkaL · 2017-02-09T15:59:31Z

Currently fine-grained incremental checking can only be run through unit tests -- there's no interactive way to use it. It only supports a small Python subset, and it doesn't support all kinds of modifications.

mypy/server/update.py is the main entry point and has more documentation.

JukkaL · 2017-02-09T16:04:39Z

test-data/unit/merge.test

@@ -0,0 +1,495 @@
+-- Test cases for AST merge (user for fine-grained incremental checking)


The important thing here is that we test that identities of certain nodes are preserved across merge. <id> suffixes represent object identities (somewhat similar to id(x)), and they only make sense within a single test case. Two things within a single test case that share an id have the same identity, and two things with different ids have different identities.

JukkaL · 2017-02-09T17:18:43Z

The travis failures look strange. I'll investigate them tomorrow.

ilevkivskyi · 2017-02-09T20:02:47Z

mypy/server/astdiff.py

+    * If one of the types is not valid for whatever reason, they are
+      considered different.
+
+    * Sometimes require types to be structurally identical, even if the


typo "if the are" -> "if they are"

ilevkivskyi · 2017-02-09T20:03:55Z

mypy/server/aststrip.py

@@ -0,0 +1,49 @@
+"""Strip AST from from semantic and type information."""


typo: "from from" -> "from"

JukkaL · 2017-03-02T17:42:23Z

Rebased and squashed most commits to help rebasing. Still trying to get a passing travis build.

gvanrossum · 2017-03-08T22:46:54Z

mypy/checker.py

        self.pass_num += 1
-        todo = self.deferred_nodes
+        if not todo:
+            todo = self.deferred_nodes


Hm, what if todo and self.deferred_nodes are both non-empty?

gvanrossum · 2017-03-08T22:57:32Z

mypy/checker.py

@@ -568,7 +584,7 @@ def is_implicit_any(t: Type) -> bool:
                for i in range(len(typ.arg_types)):
                    arg_type = typ.arg_types[i]

-                    ref_type = self.scope.active_class()
+                    ref_type = self.scope.active_self_type()  # type: Type


Optional[Type]

gvanrossum · 2017-03-08T23:05:57Z

mypy/checker.py


    def __init__(self, module: MypyFile) -> None:
        self.stack = [module]

-    def top_function(self) -> Optional[FuncItem]:
+    def top_function(self) -> Optional[FuncDef]:
        for e in reversed(self.stack):
            if isinstance(e, FuncItem):


gvanrossum · 2017-03-08T23:15:51Z

mypy/errors.py

+        self.pop_target_component()
+
+    def push_target_component(self, name: str) -> None:
+        if self.target and not self.function_or_member[-1]:


Can you explain the relationship between target and function_and_member?

gvanrossum · 2017-03-08T23:21:14Z

mypy/semanal.py

+    def refresh_top_level(self, file_node: MypyFile) -> None:
+        """Reanalyze a stale module top-level in fine-grained incremental mode."""
+        for d in file_node.defs:
+            if not isinstance(d, (FuncItem, ClassDef)):


TODO: analyze class bodies?

gvanrossum · 2017-03-22T22:43:25Z

test-data/unit/fine-grained.test

+class B(C): pass
+[file m.py.2]
+class C: pass
+class B: pass


Preserve C.g.

gvanrossum · 2017-03-22T22:58:39Z

mypy/semanal.py

+            defn.defs.accept(self)
+
+    @contextmanager
+    def analyze_class_body(self, defn: ClassDef) -> Iterator[None]:


Better name??

* Add AST diff * Add fine-grained dependency generation * Add AST merge for merging old and new ASTs for a module * Update AST dump to show various object identities * Add various tests

In fine-grained incremental mode keep reporting errors for targets with errors even if the targets weren't explicitly triggered during successive iterations. We currently repeatedly type check these targets. Alternatively, we could perhaps just keep generating the errors without actually re-type-checking the targets, but there's a risk that it will be hard to always do this correctly, so we are starting with this simple-minded approach which is good enough for now.

* Use up-to-date type maps instead of the global type map which will get out of sync. * Update dependencies of reprocessed targets.

Also change order of processing stale nodes in fine-grained incremental to be more similar to normal mode.

A class target covers the entire class body and all definitions within it.

Also fix cases where multiple namespaces cause additional steps.

Relative imports are not yet supported.

This is no longer necessary since runtests was recently fixed to not spawn multiple parallel pytest jobs.

emmatyping · 2017-04-04T17:58:03Z

mypy/test/testfinegrained.py

@@ -71,6 +71,9 @@ def run_case(self, testcase: DataDrivenTestCase) -> None:
            a.append('==')
            a.extend(new_messages)

+        # Normalize paths in test output (for Windows).
+        a = [line.replace('\\', '/') for line in a]


I guess this doesn't really matter as much, but I am always hesitant to use unix forward slash on Windows, I think it is okay here as long as it is internal and the file itself is never written to disk, but to avoid bugs I always favor the platform specific separator. Just a thought.

JukkaL commented Feb 9, 2017

View reviewed changes

ilevkivskyi reviewed Feb 9, 2017

View reviewed changes

JukkaL force-pushed the fine-grained branch from 0af6e27 to d19fa40 Compare March 2, 2017 17:41

gvanrossum reviewed Mar 22, 2017

View reviewed changes

gvanrossum mentioned this pull request Mar 28, 2017

Fix #2308: Keep suppressed imports in the suppressed list #3065

Merged

JukkaL added 22 commits April 3, 2017 15:18

Add foundation for fine-grained incremental type checking

d126ce4

* Add AST diff * Add fine-grained dependency generation * Add AST merge for merging old and new ASTs for a module * Update AST dump to show various object identities * Add various tests

Attempt to fix tests on Windows

c690fec

Try to work around travis failure

38d69aa

Add missing import

3b98a1d

Another attempt to fix the travis build

4f3b192

Fix typos

da6631c

Support multiple rounds of event propagation

2799278

Refactor and continue reporting error if no changes

f38024b

Fix bugs

01ed71c

* Use up-to-date type maps instead of the global type map which will get out of sync. * Update dependencies of reprocessed targets.

Create dependencies for inheritance

709740a

Detect differences in MRO

3a67189

Merge base classes and MRO

42a77b6

Add support for minimal debug output

093d17d

Add test cases

5a6eb05

Fix to attributes, inheritance and fine-grained incremenal

2763794

Also change order of processing stale nodes in fine-grained incremental to be more similar to normal mode.

Fix handling changes to attributes in base classes

d9cfc3f

Fixes to dependency generation

07bc54b

Support classes as fine-grained incremental targets

d95864b

A class target covers the entire class body and all definitions within it.

Fix inheritance test case

baeb421

Add minimal package support

41c81aa

Add tests for __init__ modules

82afe22

JukkaL added 17 commits April 3, 2017 15:21

Add test cases for module attributes

e5199da

Fix test case

84b7a62

Implement multiple propagation steps for module attributes

67a54d7

Also fix cases where multiple namespaces cause additional steps.

Support constructors for fine-grained incremental

54c800d

Support from m import with fine-grained incremental

770cc61

Relative imports are not yet supported.

Support nested classes with fine-grained incremental

2f0c0a7

Fix merge test case

b480807

Remove debug print

0636545

More nested class test cases

e271844

Fixes to classes with fine-grained incremental

291286f

Minor fixes

25a2846

Add review feedback

2b9104c

Address more feedback and fix a bug

ef97a75

Add additional debug output

23ca75b

Fix issues caused by rebase

027f79b

Remove travis CI workaround

dbd5d67

This is no longer necessary since runtests was recently fixed to not spawn multiple parallel pytest jobs.

Fix another issue caused by the rebase

a47ecb8

JukkaL force-pushed the fine-grained branch from a6a5388 to a47ecb8 Compare April 3, 2017 15:11

JukkaL added 3 commits April 3, 2017 18:19

Fix flaky test case

ba17fe4

Fix flaky test by making processing order deterministic

d3bf923

Attempt to fix tests on Windows

5c03d7e

emmatyping reviewed Apr 4, 2017

View reviewed changes

JukkaL added 3 commits April 5, 2017 13:21

Fix deferred lambdas

89adec4

Fix self check failure

287f831

Merge branch 'master' into fine-grained

43d16e4

JukkaL changed the title ~~[WIP] Fine-grained incremental checking~~ Foundation for fine-grained incremental checking Apr 5, 2017

gvanrossum merged commit 737434b into master Apr 5, 2017

gvanrossum deleted the fine-grained branch April 5, 2017 15:20

pkch mentioned this pull request Apr 13, 2017

Handle types.ModuleType #3107

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Foundation for fine-grained incremental checking #2838

Foundation for fine-grained incremental checking #2838

JukkaL commented Feb 9, 2017

JukkaL Feb 9, 2017

JukkaL commented Feb 9, 2017

ilevkivskyi Feb 9, 2017

ilevkivskyi Feb 9, 2017

JukkaL commented Mar 2, 2017

gvanrossum Mar 8, 2017

gvanrossum Mar 8, 2017

gvanrossum Mar 8, 2017

gvanrossum Mar 8, 2017

gvanrossum Mar 8, 2017

gvanrossum Mar 22, 2017

gvanrossum Mar 22, 2017

emmatyping Apr 4, 2017

		@@ -0,0 +1,495 @@
		-- Test cases for AST merge (user for fine-grained incremental checking)

		@@ -0,0 +1,49 @@
		"""Strip AST from from semantic and type information."""

Foundation for fine-grained incremental checking #2838

Foundation for fine-grained incremental checking #2838

Conversation

JukkaL commented Feb 9, 2017

Choose a reason for hiding this comment

JukkaL commented Feb 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JukkaL commented Mar 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment