Optionally allow redefinition of variable with different type

[JukkaL]

If the flag --allow-redefinition is used, it's now possible to redefine a
variable with a different type. For example, this code will be okay:

x = 0  # Define 'x' with type 'int'
print(x)
x = 'a'  # Define another 'x' with type 'str'
print(x)

The variable needs to be read before redefinition is possible. This is mainly
to allow these idioms:

x: List[int]
x = []  # Not a redefinition

y = None  # type: List[int]  # Strict optional disabled
y = []  # Not a redefinition

It's also okay to use a type annotation in any of the assignments. This is
fine:

x: int = 0
print(x)
x: str = 'a'
print(x)

Redefinition is only allowed to happen in the same block and the nesting level
as the original definition. This lets us keep the implementation relatively simple,
and there will hopefully be less of a readability impact, since it's less likely that
users accidentally redefine a variable or use redefinition in confusing ways. We
may want to lift the restriction in the future, however. Function arguments can
be redefined in the body but only in non-nested blocks.

The implementation does a simple static analysis to rename variables to make
them distinct when we detect redefinition. This happens before semantic
analysis pass 1.

The new behavior is not fully backward compatible with the old behavior. For
example, here we lose type context in the second assignment:

x: List[int] = []
f(x)
x = []  # Need type annotation since this creates an independent variable

Internally we use a sequence of single quotes as a variable suffix
to represent renamed variants. The implementation provides an unmangle()
function to get the original name from a potentially mangled name.

It's still impossible to redefine final names, classes and certain other things.
The main use case is simple variables, either in single or multiple assignments.

I did some benchmarking (not with the latest version though) and the performance
impact of the new pass was minor (under 2% IIRC).

Things to do in additional PRs:

• Add documentation.
• Make sure we unmangle names everywhere.
• Consider allowing redefinition to happen sometimes in another block.

[ilevkivskyi]

Thanks for this huge amount of work! This will make mypy more "pythonic" and closer to the spirit of PEP 526 (annotation is not a declaration, but just a hint). Overall PR looks very good, I have a bunch of small comments (mostly requests for documentation).

I would say it is important to add docs before this will be released. Supporting "conditional" redefinition is less important (because it may require some non-trivial work). But also note that it will solve one of the current downsides:

x: object
x
x = 42
if bool():
    x = "hi"  # error here

Also an idea about

x: List[str] = []
x
x  = []  # error here

Maybe when we detect UninhabitedType, instead of immediately giving an error we can try to re-infer type of r.h.s. using type of the "previous" x (with one prime less) as a context?

[ilevkivskyi]

Maybe add a comment documenting this attribute?

[ilevkivskyi]

This one is pretty clear, but some details would be helpful (is it for "continuous" nesting of loops or total, ie. including while (...): if(...): while (...): ... etc).

[ilevkivskyi]

Suggested change

	# List of scopes; each scope maps short name to block id.
	# List of scopes; each scope maps short (i.e. unsuffixed) name to block id.

[ilevkivskyi]

Suggested change

	# References to variables the we may need to rename. List of
	# References to variables that we may need to rename. List of

[ilevkivskyi]

What about 'cls'?

[JukkaL]

We special case self since it can define new attributes. cls doesn't have this property, so it doesn't need to be special cased. Added comment.

[ilevkivskyi]

Suggested change

	from typing import Optional, List, Callable, Dict, Set
	from typing import Optional, List, Callable

[ilevkivskyi]

is_final only used once. I would just write s.is_final_def = self.unwrap_final(s).

[ilevkivskyi]

This looks a bit mysterious. A comment explaining why name with two primes or more are "bad".

[JukkaL]

Refactored and added a comment.

[ilevkivskyi]

If one removes the assignment above, will this become an error?

[JukkaL]

Yes. Added a test case.

[ilevkivskyi]

I would add test cases for

e = 1
e
try:
    ...
except Exception as e:
    ...

and

with open('file') as f:  # IO[str]
    ...
with open('file', 'b') as f:  # IO[bytes]
    ...

(maybe plus some variations).

[JukkaL]

Added test cases.

[JukkaL]

I think that I've now addressed all the feedback.

Agreed that this will need to be documented by the next release (or we should hide the option).

Maybe when we detect UninhabitedType, instead of immediately giving an error we can try to re-infer type of r.h.s. using type of the "previous" x (with one prime less) as a context?

I thought about this but I couldn't come up with a simple implementation, so I decided to postpone this. Calculating the previous 'x' is more complicated, since the rules for renaming are different for globals and locals, for example. This is a potential future improvement. I'll create a follow-up issue about this.

[ilevkivskyi]

Thanks for the update! LGTM.

Please don't forget to create follow-up issues for the things we discussed: allowing more (conditional) redefinitions, problem with empty containers and using "previous" type, adding docs. (You also mentioned "Make sure we unmangle names everywhere" but I am not sure what is missing in this PR).

[neighthan]

Is there a way to only allow redefinition if an explicit type is given? So that

x = 0
print(x)
x: str = 'a'
print(x)

is valid but

x = 0
print(x)
x = 'a'
print(x)

is not? This would prevent any accidental redefinition and make it clear that the type is supposed to be changing (in the example above it's clear anyway, but if we had x = some_func() it might not be obvious anymore).

[oakkitten]

this doesn't seem to work for def:

from typing import Callable, cast

foo: str = "a"
foo: int = cast(int, foo)

bar: Callable[[], int] = lambda: 1
bar: int = cast(int, bar)

def baz() -> int:
    return 1
baz: int = cast(int, baz)

11: error: Name 'baz' already defined on line 9

any workarounds?