Fix issues related to indirect imports in import cycles #4695

JukkaL · 2018-03-07T16:15:43Z

This adds supports for some cases of importing an imported name
within an import cycle. Originally they could result in false positives
or false negatives.

The idea is to use a new node type ImportedName in semantic
analysis pass 1 to represent an indirect reference to a name in another
module. It will get resolved in semantic analysis pass 2.

ImportedName is not yet used everywhere where it could make
sense and this doesn't fix all related issues with import cycles.

Also did a bit of refactoring of type semantic analysis to avoid passing
multiple callback functions.

Fixes #4049.
Fixes #4429.
Fixes #4682.

Borrowed test cases from #4495 by @carljm. This PR is an alternative
for #4495.

Supersedes #4495

Previously we could fall back to the `Any` type if we imported an imported name. This depended on the order in which an SCC was processed. This fixes the behavior for `from m import x` and also makes the behavior consistent with fine-grained incremental mode. I'll fix other kinds of imports separately.

This makes it easier to add additional functions to the interface.

…lity The previous, more correct behavior caused too many breakages in an internal Dropbox repository. I'll enable this later after the breakages have been fixed.

ilevkivskyi

Thanks! I have few questions/suggestions.

ilevkivskyi · 2018-03-07T19:32:52Z

mypy/semanal.py

@@ -1573,6 +1603,14 @@ def process_import_over_existing_name(self,

    def normalize_type_alias(self, node: SymbolTableNode,
                             ctx: Context) -> Optional[SymbolTableNode]:
+        """If node refers to a built-intype alias, normalize it.


Typo buit-intype (space mising).

ilevkivskyi · 2018-03-07T19:38:02Z

mypy/semanal.py

-                             existing.node != node.node) and existing.kind != UNBOUND_IMPORTED:
+            if (existing
+                    and (not isinstance(node.node, MypyFile) or existing.node != node.node)
+                    and existing.kind != UNBOUND_IMPORTED


I would expect that the goal is to get rid of UNBOUND_IMPORTED and replace all such checks with isinstance(node.node, ImportedName), what I am missing here?

That's the end goal. This PR doesn't remove all instance of UNBOUND_IMPORTED. In particular, import m statements generate them still. I'll fix them in a separate PR.

I'll fix them in a separate PR.

OK, having smaller PRs makes sense.

ilevkivskyi · 2018-03-07T19:43:37Z

mypy/semanal_pass3.py

        self.errors.report(ctx.get_line(), ctx.get_column(), msg)

    def fail_blocker(self, msg: str, ctx: Context) -> None:
        self.fail(msg, ctx, blocker=True)

+    def note(self, msg: str, ctx: Context) -> None:
+        self.sem.note(msg, ctx)


What is the reason we can reuse self.sem.note here but not self.sem.fail above?

These were different originally, but it seems like there's no good reason. I'll unify them.

ilevkivskyi · 2018-03-07T19:46:13Z

mypy/types.py

@@ -284,6 +284,8 @@ class TypeOfAny(Enum):
    special_form = 'special_form'
    # Does this Any come from interaction with another Any?
    from_another_any = 'from_another_any'
+    # Does this Any come from an implementation limitation/bug?
+    implementation_artifact = 'implementation_artifact'


Big +1 for adding this. I think I have seen other cases where this would be appropriate (but can't remember them now).

ilevkivskyi · 2018-03-07T19:49:38Z

test-data/unit/check-modules.test

+[case testIndirectFromImportWithinCycle]
+import a
+[file a.py]
+from b import f


Would it make sense to check that the result is independent on the order of import statements (just in case) here and/or in other tests?

I'll add a few test cases where we reverse the shape of the import cycle relative to the main module.

ilevkivskyi · 2018-03-07T19:53:37Z

(I updated the description to mention that this also fixes #4682 since I found the relevant code and tests.)

ilevkivskyi

Looks good! I also edited the description to add Supersedes #4495, this should automatically close the old PR when this is merged (cool new GitHub feature :-))

carljm · 2018-03-08T17:51:25Z

Confirmed that this fixes the problems in our codebase. Thanks @JukkaL and @ilevkivskyi !

@carljm

This adds supports for some cases of importing an imported name within an import cycle. Originally they could result in false positives or false negatives. The idea is to use a new node type ImportedName in semantic analysis pass 1 to represent an indirect reference to a name in another module. It will get resolved in semantic analysis pass 2. ImportedName is not yet used everywhere where it could make sense and this doesn't fix all related issues with import cycles. Also did a bit of refactoring of type semantic analysis to avoid passing multiple callback functions. Fixes python#4049. Fixes python#4429. Fixes python#4682. Inspired by (and borrowed test cases from) python#4495 by @carljm. Supersedes python#4495

JukkaL added 13 commits March 5, 2018 17:19

Fix crash on ignored __init__.py and add test cases

80b9e20

Fix type forward references in properties

92d7cfc

Refactoring: use abstract class instead of passing multiple callbacks

e14578c

This makes it easier to add additional functions to the interface.

Fix forward reference to typing alias import

de24b3c

Also handle "from m import *"

c1b3629

Remove type forward reference handling to maintain backward compatibi…

68736fd

…lity The previous, more correct behavior caused too many breakages in an internal Dropbox repository. I'll enable this later after the breakages have been fixed.

Add dereferencing on module attribute references like m.x

d138ec8

Handle more indirect references in type annotations and add tests

aedd220

Update AST strip

1625832

Fix type check

ef7681f

Fix str() of ImportedName

8091724

Also fix #4429

85129b1

JukkaL mentioned this pull request Mar 7, 2018

Don't error on re-exported imports in cycles. Fixes #4049. #4495

Closed

ilevkivskyi reviewed Mar 7, 2018

View reviewed changes

Update based on feedback

842b1c7

ilevkivskyi approved these changes Mar 8, 2018

View reviewed changes

JukkaL merged commit 0b7597c into master Mar 8, 2018

This was referenced Mar 8, 2018

Any type inferred for circular reference due to import cycle #3277

Closed

Unexpected Any for complex import structure #4110

Closed

Named tuples in import cycles still fail sometimes #4442

Closed

gvanrossum deleted the imports-in-cycles branch March 15, 2018 03:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix issues related to indirect imports in import cycles #4695

Fix issues related to indirect imports in import cycles #4695

JukkaL commented Mar 7, 2018 •

edited by ilevkivskyi

Loading

ilevkivskyi left a comment

ilevkivskyi Mar 7, 2018

ilevkivskyi Mar 7, 2018

JukkaL Mar 8, 2018

ilevkivskyi Mar 8, 2018

ilevkivskyi Mar 7, 2018

JukkaL Mar 8, 2018

ilevkivskyi Mar 7, 2018

ilevkivskyi Mar 7, 2018

JukkaL Mar 8, 2018

ilevkivskyi commented Mar 7, 2018

ilevkivskyi left a comment

carljm commented Mar 8, 2018

Fix issues related to indirect imports in import cycles #4695

Fix issues related to indirect imports in import cycles #4695

Conversation

JukkaL commented Mar 7, 2018 • edited by ilevkivskyi Loading

ilevkivskyi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilevkivskyi commented Mar 7, 2018

ilevkivskyi left a comment

Choose a reason for hiding this comment

carljm commented Mar 8, 2018

JukkaL commented Mar 7, 2018 •

edited by ilevkivskyi

Loading