Add interactions between Literal and Final

[Michael0x2a]

This pull request adds logic to handle interactions between Literal and Final: for example, inferring that foo has type Literal[3] when doing foo: Final = 3.

A few additional notes:

1. This unfortunately had the side-effect of causing some of the existing tests for Final become noiser. I decided to mostly bias towards preserving the original error messages by modifying many of the existing variable assignments to explicitly use things like Final[int].

   I left in the new error messages in a few cases -- mostly in cases where I was able to add them in a relatively tidy way.

   Let me know if this needs to be handled differently.

2. Since mypy uses 'Final', this means that once this PR lands, mypy itself will actually be using Literal types (albeit somewhat indirectly) for the first time.

   I'm not fully sure what the ramifications of this are. For example, do we need to detour and add support for literal types to mypyc?

3. Are there any major users of Final other then mypy? It didn't seem like we were really using it in our internal codebase at least, but I could be wrong about that.

   If there are some people who have already started depending on 'Final', maybe we should defer landing this PR until Literal types are more stable to avoid unnecessarily disrupting them. I had to make a few changes to mypy's own source code to get it to type check under these new semantics, for example.

[ilevkivskyi]

Before doing the full review, I wanted to point out an important thing. A pattern like this (using a constant as a default value) is very common:

BORDER_SIZE: Final = 10
class Window:
    def __init__(self) -> None:
        self.border = BORDER_SIZE

w = Window()
w.border = 20

START: Final = 42
def count() -> int:
    counter = START
    while True:
        counter += 1
        ...

Currently this works, but with this PR it will emit (quite mysterious) errors. I think the logic of interactions between Literal and Final is not exactly how I imagined it. I think it should work like this:

NUM: Final = 42

x = NUM  # type of x is just int, because there is no literal context

def test(x: Literal[42]) -> None:
    ...
test(NUM)  # this works however

This can be summarized in short as everything should work as if final names are replaced with their values (including chaining).

Note that this cannot be achieved by simply "erasing" the literal type if there is no literal context, because explicit literal types should "propagate":

small: Literal[1, 2, 3] = 1
smalls = [small]  # type here is List[Literal[1, 2, 3]]

I think this should be implemented very roughly like this: final assignments still infer instance types, then analyze_ref_expr can construct a literal type (depending on context) using the final_value (if known).

This may be more tricky with "intelligent indexing", unless we set a literal context there (more like in your option 2, that we recently discussed).

[ilevkivskyi]

Also note that all your three questions may become solved/irrelevant if you implement this PR the way I propose.

[Michael0x2a]

Hmm, I'll try experimenting with that idea. I agree that preserving the use-case you pointed out is important: I wasn't very happy about having to make changes to mypy itself after breaking that use-case.

I do have some new questions about this though:

1. If we do x: Final = 4, what should the output of reveal_type(x) be? I think it would be misleading to print out either int or Literal[4] -- maybe we need to come up with some new representation? I guess maybe Final[Literal[4]], but that also seems misleading since that isn't actually a proper type.
2. What should happen if the user does x: Final[int] = 4 or x: Final[Literal[4]] = 4? I think the two reasonable options are to either disable the "replace final names with their values" behavior if the type is explicitly provided like this, or to disallow users from writing things like this at all. Neither solution feels satisfactory though.
3. With this change, there's no longer going to be a convenient short-hand way of declaring some variable to be exactly a Literal[...] type. Is it worth introducing back the x: Literal = "foobar" notation to address this gap?

[ilevkivskyi]

OK, here are my thoughts:

1. I think we shouldn't do anything special here, but just reveal the truth. For example:

   MODE: Final = 'rb'
   
   data_mode = reveal_type(MODE)  # str
   open('/home/data', reveal_type(MODE))  # Literal['rb']

   different revealed type depending on context can already happen e.g. for generic functions, reveal_type() just reveals what do we have in type map for a given expression.

2. I think nothing special should happen for x: Final[int] = 42, x: Final[object] = 42, or x: Final[Literal[42]] = 42. In a non-literal context we use the explicit or inferred type, in a literal context, we use the final_value (if known).

3. Final (and ClassVar but there is a bug) is a qualifier, while Literal (like Tuple) is a type constructor. I don't think this (presumably very rare) use case justifies breaking consistency here.
   Just to clarify, union of literals still counts as a literal context, so this example will work:

   NO_DICE: Final = 0
   
   next_dice: Literal[0, 1, 2, 3, 4] = NO_DICE

As a general comment, I am not worried about interference between literals and explicit types, since their use cases are almost non-overlapping. A typical use case for explicit type is a "global container": REGISTRY: Final[Dict[str, Type[Model]]] = {}, which can't be used in literal context anyway.

[Michael0x2a]

Ok, this should be ready for a second look whenever.

@ilevkivskyi -- I basically went with the approach you suggested (and removed the hacks I added to mypy + the tests.)

[ilevkivskyi]

Thanks! Overall looks good, I have few minor comments.

[ilevkivskyi]

Why can't you just write:

final_value = t.final_value.accept(self)
assert isinstance(final_value, LiteralType)

thus avoiding the extra variable.

[Michael0x2a]

It's because t.final_value.accept(self) has a return type of Type, which means we wouldn't be able to assign it to final_value without raising an error or without casting.

I can replace this entire if statement with a cast if you want, similar to what we're doing in visit_tuple_type or visit_typeddict_type below, but I wanted to add a runtime check mostly for peace of mind.

[ilevkivskyi]

Will my version work if you remove the type comment on final_value?

(This is not important however.)

[Michael0x2a]

Alas, it doesn't: I get a 'Argument "final_value" to "Instance" has incompatible type "Optional[Type]"; expected "Optional[LiteralType]"` error a little later on when we use the variable.

[ilevkivskyi]

Hm, I am not sure I totally understand why x: Final = 1 and x: Final[int] = 1 appear in the diff. Shouldn't we infer exactly the same type Instance('builtins.int', final_value=Literal[1]) for both of them?

[Michael0x2a]

No -- we infer Instance('builtins.int', final_value=None) for the latter. So, since those two declarations have subtly different behavior, I thought it would be important to test both.

(I also went ahead and expanded this test a little. I forgot to update it to test the new semantics earlier.)

[ilevkivskyi]

What is the reason for inferring different type? Is this because of situations like x: Final[float] = 42? It looks like this is similar to #2008, so I am not going to argue here.

[Michael0x2a]

Basically, yeah.

In short, I thought it was more important to make sure that statements like x: Final[float] = 42 and x: float = 42 behaves in the same way as much as possible. This helps ensure that mypy's behavior stays consistent no matter what, even if we lose track of the original literal value (for example, if the user is messing around with generics).

This does mean that x: Final[int] = 9 and x: Final = 9 appear to behave "inconsistently", but I thought that would be less surprising then if x: Final[int] = 9 and x: int = 9 were to behave "inconsistently".

[Michael0x2a]

On second thought though, I wonder if it might be worth making x: Final[int] = 4 and x: Final = 4 behave in the same way only if the declared and inferred types are equivalent. That way, we don't run into any issues with expressions like x: Final[float] = 4 since we'd be disabling this "substitution" behavior in that case.

That might be a little too magical though, not sure. LMK if this is an idea you think might be worth exploring.

[ilevkivskyi]

LMK if this is an idea you think might be worth exploring.

Maybe just open an issue about this. We can decide later when we will have more experience.

[ilevkivskyi]

In principle we can support this at some point, but not now. IIRC there is already an issue about this (for common fields in named tuples).

[Michael0x2a]

@ilevkivskyi -- ok, this should be ready for a second look whenever! I made the changes you asked and also added in a few more tests.

[ilevkivskyi]

I think this LG now, just a couple of optional questions.

[ilevkivskyi]

Will my version work if you remove the type comment on final_value?

(This is not important however.)

[ilevkivskyi]

What is the reason for inferring different type? Is this because of situations like x: Final[float] = 42? It looks like this is similar to #2008, so I am not going to argue here.