Arguments aren't always evaluated left to right

[gilch]

For example,

=> (setv foo 42)
foo = 42
None
=> [foo (del foo)]
del foo
[foo, None]
Traceback (most recent call last):
  File "c:\users\me\documents\github\hy\hy\importer.py", line 198, in hy_eval
    return eval(ast_compile(expr, "<eval>", "eval"), namespace)
  File "<eval>", line 1, in <module>
NameError: name 'foo' is not defined

The expected result was [42, None].

[Kodiologist]

This is unquestionably counterintuitive, but what else would you have [foo (del foo)] compile to? Hy's general strategy for expression-ifying a statement is to move the statement before the expression. For example,

(f (g) (with [(E)] (h)))

compiles to

with E():
    _hy_anon_var_1 = h()
f(g(), _hy_anon_var_1)

; notice that h gets called before g. Would you instead have something like

_hy_anon_var_2 = g()
with E():
    _hy_anon_var_1 = h()
f(_hy_anon_var_2, _hy_anon_var_1)

and for [foo (del foo)], something like

_hy_anon_var_1 = foo
del foo
[_hy_anon_var_1, None]

? It's not obvious to me that this strategy will work in general. Think hard about what you want.

[gilch]

This also happens to setv

=> [(setv foo 1) foo (setv foo 2) foo]
foo = 1
foo = 2
[None, foo, None, foo]
[None, 2, None, 2]

The expected answer was [None, 1, None, 2].

[gilch]

This is unquestionably counterintuitive

Python guarantees the sequential order of evaluation in displays and function calls. Programs may rely on this for correct behavior. If Hy is allowed to reorder things this must, at least, be documented. But how do we even document this?

I notice these things quickly because I almost always have the --spy option on in the repl, and I understand Python well. But other users would be very confused.

Now that we don't have let, I was thinking about how to clean up gensysms using del. But what if the answer you want the expression to return is in a gensym? I thought maybe if we had PROG1 from Common Lisp, it could be done as (PROG1 ~g!result (del ~g!result)). (A PROG1 returns the result of the first expression, and executes the rest for side-effects.) But we don't have PROG1, just Clojure's do, which is a PROGN, which returns the last result. Clojure doesn't have this either, but there are various approaches to emulate it. One of the simplest is PROG1 with (do (first [...])). So I tried that in Hy and ran into this ordering problem.

what else would you [...] compile to?

I don't have to know the answer to that to report it as a problem.

But perhaps we should pull everything out of a display expression or function call if we pull one thing out. Examples:

[(setv foo 1) foo (setv foo 2) foo]

foo = 1
_hy_anon_var_1 = None
_hy_anon_var_2 = foo
foo = 2
_hy_anon_var_3 = None
_hy_anon_var_4 = foo
[_hy_anon_var_1, _hy_anon_var_2, _hy_anon_var_3, _hy_anon_var_4]

[foo (del foo)]

_hy_anon_var_1 = foo
del foo
_hy_anon_var_2 = None
[_hy_anon_var_1, _hy_anon_var_2]

(f (g) (with [(E)] (h)))

_hy_anon_var_1 = g()
with E():
    _hy_anon_var_2 = h()
f(_hy_anon_var_1, _hy_anon_var_2)

[Kodiologist]

what else would you [...] compile to?

I don't have to know the answer to that to report it as a problem.

Certainly, but it's a question we have to answer if we want to ever fix a bug.

[gilch]

It's not obvious to me that this strategy will work in general.

I think it would always work in the case of display expressions and function calls, since things should be evaluated once each in sequence. We can fix that at least.

When things should be evaluated some other number of times (like zero in the if branch not taken, or more than once in a loop) then we might have to do something else. I have a solution for what a genexpr should expand to #588, at least in Python3.

Are there other loop or branch cases that have this problem? Or do you have a counterexample for displays or function calls?

[gilch]

Thinking about this some more, I think it would suffice if we pulled out everything before the statement, instead of just everything. Any expressions afterwards should work fine directly in the display or call, so

[(setv foo 1) foo (setv foo 2) foo]

could compile to

foo = 1
_hy_anon_var_1 = None
_hy_anon_var_2 = foo
foo = 2
_hy_anon_var_3 = None
[_hy_anon_var_1, _hy_anon_var_2, _hy_anon_var_3, foo]

It would save assignments, but I'm not sure if it would be harder to implement.

[gilch]

We'd definitely want to put nested cases in the tests, to make sure things still happen in strict left-to-right order. But it could totally work.

[0 (setv x 1) x [1 (setv x 2) x 3 4] x]

_hy_anon_var_1 = 0
x = 1
_hy_anon_var_2 = None
_hy_anon_var_3 = x
_hy_anon_var_5 = 1
x = 2
_hy_anon_var_6 = None
_hy_anon_var_4 = [_hy_anon_var_5 _hy_anon_var_6 x 3 4]
[_hy_anon_var_1 _hy_anon_var_2 _hy_anon_var_3 _hy_anon_var_4 x]

[0, None, 1, [1, None, 2, 3, 4], 2]

[gilch]

@kirbyfan64 this seems similar to what we did for and/or short-circuiting when they contained statements. #824 You were able to detect when the form had a statement and compile the whole thing differently (as if/else statements). Couldn't something similar be done here?

[refi64]

... probably. It would have to be a bit different, but I'm pretty sure it would work.

[gilch]

Thinking about this some more, the compiler should be able to detect if an element is a simple constant literal (like a quoted symbol or an int). With the possible exception of f-strings (which aren't really constant), literals are not going to change due to side effects. So my last example [0 (setv x 1) x [1 (setv x 2) x 3 4] x] could compile to the simplified:

x = 1
_hy_anon_var_1 = None
_hy_anon_var_2 = x
x = 2
_hy_anon_var_4 = None
_hy_anon_var_3 = [1 _hy_anon_var_4 x 3 4]
[0 _hy_anon_var_1 _hy_anon_var_2 _hy_anon_var_3 x]

Furthermore, certain special forms are known to the compiler to always return None. I don't think we're allowed to override these with custom functions. If these were known never to change, it could be further simplified by detecting the special form at compile time and throwing out the expression context in those cases while inserting a None in its place:

x = 1
_hy_anon_var_1 = x
x = 2
_hy_anon_var_2 = [1 None x 3 4]
[0 None _hy_anon_var_1 _hy_anon_var_2 x]

But these are just optimizations. Pulling out everything when there's a statement will work correctly.