-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
break-with-value and for/else implementation #23260
Conversation
This commit enables a combination of two language features that dovetail nicely together (as remarked by @StefanKarpinski in [1]): for/else and break-with-value, which together allow something like: name = for person in people if person.id == id break person.name end else generate_name() end The parsing patch is pretty straightforward. As for the code lowering part, I opted to make a second version of `'break-block`, called `'break-block-with-value`, which passes a target variable for the return value onto the `break-labels` stack. That's where the `'break` operation finds it. An alternative approach would be something more similar to the existing `replace-return` function: Instead of having an intermediate node representing `'break-block-with-value`,we could traverse the expression tree and replace `break-with-value` by an assignment followed by a break. I have no opinion either way; the current commit seemed like the obvious implementation to me, but that was before I saw `replace-return`'s prior art. This patch still has a few places marked `TODO`, where scoping issues for the `else` block need to be resolved. I'll tackle that after awaiting feedback. [1] JuliaLang#22891
Because a new language feature should probably be used in the standard library, to encourage cargo-culting.
Wow, very nice first contribution! I'm against this change but I want to commend your work here. |
Thanks for your remark @ararslan ! I thought it is a nice language feature but I certainly see how opinions can be different. If it turns out to have just been a little learning project for me, that still makes me happy :) |
Thanks, @tkluck, this is great work! I haven't reviewed it in detail yet but it seems basically good. I imagine the main issue here will be whether people want this feature :) |
Noting that this relates to #22659. Interestingly, this feature doesn't provide a way to get the last value of the iteration variable, so that might mean we should add both this and |
base/array.jl
Outdated
@@ -1251,10 +1251,11 @@ julia> findnext(A,3) | |||
function findnext(A, start::Integer) | |||
for i = start:length(A) | |||
if A[i] != 0 | |||
return i |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not related to the feature itself, but I find explicit returns easier to reason about.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They are, but sometimes you don't want to factor a single loop into a function just so that you can use a return
to get a value out. This basically lets you have that without a function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do like the for else
feature but isn't v = for ...; break i; else j end
always equivalent to for ...; v = i; break; else v = j end
(with the correct scope of v
)?
The latter seems much easier to understand than the former. It is similar to the return value of if else
but in that case at least the return value is easier to find.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with @KristofferC that the findnext
example isn't the best use-case for this feature because a return
suffices; I just included it for the sake of show-casing the working patch.
@yuyichao , yes that is indeed equivalent, and it's probably a matter of taste which is easier to understand.
IMHO, in a language where there is no distinction between statements and expressions, it is a bit wasteful and arguably unexpected (e.g. when compared to your if
/else
example) not to be able to specify a value for a loop expression. That's why I personally like this feature.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Afterthought: it's also a bit more 'functional' because it's easier to reason about whether the resulting variable v
is mutable or not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's why I do like the for else
and my example above includes the else
block.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops, I misread that. Got it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do agree, however, that these cases are clearer with return
; but they do serve as an in-situ test of the new syntax, which is useful for a WIP pull request.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A nice example is intersect
, which can be improved into:
function intersect(s::Set, sets...)
i = similar(s)
for x in s
for t in sets
!in(x, t) && break false
else true
end && push!(i, x)
end
return i
end
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great example @rfourquet ! It could also be
for x in s
for t in sets
!in(x, t) && break
else
push!(i, x)
end
end
which doesn't use break-with-value, but does use the else
feature.
I personally like options with all
or with comprehensions best, but that's a matter of taste and may not give the same performance (yet).
I can't build this branch – am I missing something here or is this work in progress?
|
@StefanKarpinski that's odd; I tried a clean build before pushing. Might still be a leftover build artifact somewhere; I'll look into it tomorrow (am in CEST). Thanks for trying it out! |
I understand the build issue now; the change to |
Bootstrapping is tricky. The system is continually building itself and things you're used to being able to do are at various points not defined yet. So using |
Before 976ff84, a parsed `break` expression `'(break)` was being augmented to `'(break <label>)` by the function `expand-forms`. Since the latter is supposed to be idempotent [1], it used to check (pair? e) before deciding on replacing it. In 976ff84 however, the parsed break can have a value expression, so `(pair? e)` can be true for that reason as well. Thinking I was being smart, I replaced the check by (symbol? (cadr e)) not realizing that a lone symbol can be a valid expression too! This commit fixes that by *always* compiling the break expression to a *triple* '(break <expression> <label>) where <expression> may be `'(null)`. This should have no big implications for the code being generated, because `'(null)` generates no code when, in the function `compile`, `(and value tail)` is false, which is likely the typical case. [1] At least, that's what I'm guessing is the reason.
Just pushed a fix for the issue I found, and confirmed that I think it's possible to get rid of As for scoping, I'm tempted to keep the
So unless anyone objects, that's what I'll implement. |
…ck and break-block-with-value Instead, we give every `break-block` an `else`-block, potentially equal to `'(null)`. This gives us one less atom to worry about. This commit also makes a small modification to `compile`, which now passes the value of `tail` for a `break` onto the `break-labels` stack. That means a `break` can be compiled to a `return` if the `break-block` is in the last position of a function.
I get a lot of "invalid AST" now, e.g.: julia> f(n) = for i = 1:n
isprime(i) && break i
ERROR: syntax: invalid AST
julia> f(n) = for i = 1:n
isprime(i) && break(i)
ERROR: syntax: invalid AST
julia> f(n) = for i = 1:n
isprime(i) && break
ERROR: syntax: invalid AST
julia> for i = 1:n
isprime(i) && break
ERROR: syntax: invalid AST
julia> for i = 1:n
isprime(i) && break i
ERROR: syntax: invalid AST
julia>
[ stefan ] ./julia KARPINSKI-MAC:~/projects/julia
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: https://docs.julialang.org
_ _ _| |_ __ _ | Type "?help" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.7.0-DEV.1331 (2017-08-15 22:22 UTC)
_/ |\__'_|_|_|\__'_| | tkluck-break-with-value/3abd4d9a8d (fork: 4 commits, 5 days)
|__/ | x86_64-apple-darwin16.1.0
julia> for i = 1:n
isprime(i) && break i
ERROR: syntax: invalid AST
julia> for i = 1:n
isprime(i) && break
ERROR: syntax: invalid AST
julia> 1 + 2
3
julia> for i = 1:10
println(i)
ERROR: syntax: invalid AST |
The following issue: Test Failed Expression: Base.incomplete_tag(parse(str, raise=false)) == tag Evaluated: none == block was due to returning a different error message string in case of parsing the incomplete string "for i=1;". This commit fixes it by delegating error handling to `expect-end`.
This was triggered by Error in testset inference: Test Failed Expression: all(isleaftype, (ast12474[1])[1].slottypes) which had, as a root cause, the unused new-mutable-var being of type Any. Now, that isn't really what this doctest is trying to avoid, but I'm guessing it's worthwhile avoiding an unneeded mutable var nonetheless.
This is an artifact from when I wasn't sure whether anything else would write to the break-labels stack apart from the code point that I had updated. I'm now confident that doesn't happen, so no need for special cases.
@StefanKarpinski ai, thanks for finding that! I can't reproduce that anymore with what I just pushed. It's possible that this commit fixed it. My hypothesis would be that the REPL relies on the incompleteness error from the parser to decide whether to give a new prompt or wait for more instructions. That would also explain why I didn't see it before, because I was testing with code in a source file. If that sounds believable to you, would you mind trying the latest update? Thanks for your patience! |
That is correct. I'll give it another try. It's always amazing how different people's workflows are an how that can lead to triggering bugs or completely missing them. |
Thinking about scoping: The current patch initiates a
My scoping expectation would be that every iteration starts a new scope. This is particularly relevant because julia seems to want to move freely between the two constructions I use above, e.g. with the This comment here lists a few similar scoping tickets. Any thoughts on how it should relate to what we're doing here? (BTW, in addition to this, this pull request still has a few TODO places for collecting variable info from the else body.) |
I postponed this mostly because the names of the methods sounded scary and I thought I'd need to think about it; turns out it's just a tree traversal.
(Use `git blame -w` to see the most recent functional changes to this code.)
CI failure seems unrelated, and a quick inspection of https://github.com/JuliaLang/julia/commits/8eca27df422089a8c21301764c6485fb86d0fee8 suggests that is was introduced by #30442 and merged into this pull request when merging master. |
Some new conflicts with master branch have just appeared as a result of #30656. Is now maybe a good time to make a decision on this feature? If we put it in master now it will have maximal time in the I'll gladly and swiftly fix the merge conflict in case of a positive decision. |
I think this is @JeffBezanson's call. He is Julia's Syntax Czar. |
Hey @JeffBezanson -- anything I can do to help you review this? |
For what it's worth, reading the documentation changes here made me like this feature much more — I had been on the fence before. It's introduced quite well and sensibly. Perhaps we should discuss on the next triage call? |
Agree. The documentation is really lovely and motivates this feature quite nicely. |
Triage has mixed reactions to this. Personally I'd say I'm weakly in favor of it; it's a useful pattern and I can't think of a really strong reason not to have it. Points for:
Points against:
It was also suggested that we look up python's experience with this and see if they regret it. |
At this point Rust has had break with values for long enough that we might be able to learn from their experience with that part of this feature. |
Test failure logs give me very little to go on. Should I assume these are just noisy test failures like we've seen a few times in this PR? |
I'm going to close this pull request and give myself some mental closure. If anyone is willing to make the effort to shepherd this through, feel free to go ahead. |
Sorry about that @tkluck. It was excellent work and I'm sorry that it wasn't used. I hope you don't feel badly about it. Part of the experimental design process is that sometimes one does a lot of work on PRs only to decide not to use them. It is always still here if we decide to go ahead with it. |
Thanks for the kind words @StefanKarpinski. Of course I do feel bad about it, but better to cut my losses. I wouldn't even mind chasing master and dealing with CI false positives if some ending were in sight, but upon reflection, I just don't see this making further progress after the relatively unactionable outcome from the triage meeting. |
Yes, sorry, it's a bit unsatisfying. We did discuss for a long time and just could not collectively or even individually come to a conclusion. |
Regardless of the outcome, I do hope you can feel proud of the quality of the PR though. It's incredibly impressive work, and we all really appreciate you going through the effort. Major features like this can be the source of a tremendous amount of discussion and deliberation, but every time someone proposes something new, we all learn from it, and Julia gets better as a result, even if the feature doesn't land. So again, I hope you can feel proud of this work! |
Since the main argument against this seems to be the use of
i.e. just let the loop return |
Looks like a great idea. It's one half of this PR which looks indeed quite less controversial, although there are still mixed feelings about it: quoting #23260 (comment):
I really like this "break with value" feature, which makes loops more "expression-like", like most things in the language. Paraphrasing: they allow to have the Note that even though the function intersect(s, sets...)
i = similar(s)
for x in s
intersects = for t in sets
!in(x, t) && break false
end
intersects === nothing && push!(i, x)
end
return i
end |
It's kind of already implied by @StefanKarpinski's remark #22891 but you can actually add " ifnothing(f) = x -> ifnothing(f, x)
ifnothing(f, x) = x
ifnothing(f, ::Nothing) = f()
name = for person in people
if person.id == id
break person.name
end
end |> ifnothing() do
generate_name()
end (of course, in this case, it's probably better to write So supporting only |
If we're going to have the break with value part, I'd be inclined to just do the whole thing—it's weirder and more awkward to not be able to do the I've thought of a way to look at this that reconciles what I consider to have been the most significant objection, which was this point made by @JeffBezanson:
Namely, that we just consider the natural exit of a loop to implicitly do while cond()
# body
end loop # loop construct that loops forever
cond() || break # this is the return value of the loop
# body
end From this perspective, it is always a |
In case this ever gains traction again:
values = rand(1:100, 10)
evens = for i in values::Array{Int64} # optional return type
if iseven(i)
continue i # append to returned array
else if i == 13
continue "not Int64" # throw an exception
end
default # previously `else`
newvalues = values * 2 # do stuff
return newvalues # explicit return
end
|
Hi @jakobjpeters ! Thanks for engaging with the topic. If you'd want your ideas to fit within julia 1.x you have to find a way to stay backwards compatible:
is valid (even though
Please open a separate bug for your |
Thank you for the educational response! May I ask a couple clarifying questions to streamline the new issue?
evens::Array{Int64} = for i in values
|
@jakobjpeters happy to answer those questions in their own issue. There's no need to streamline it before opening it. |
This commit enables a combination of two language features that dovetail nicely together (as remarked by @StefanKarpinski in [1]): for/else and break-with-value, which together allow something like:
The parsing patch is pretty straightforward. As for the code lowering part, I opted to make a second version of
'break-block
, called'break-block-with-value
, which passes a target variable for the return value onto thebreak-labels
stack. That's where the'break
operation finds it.An alternative approach would be something more similar to the existing
replace-return
function: Instead of having an intermediate node representing'break-block-with-value
,we could traverse the expression tree and replacebreak-with-value
by an assignment followed by a break.I have no opinion either way; the current commit seemed like the obvious implementation to me, but that was before I saw
replace-return
's prior art.This patch still has a few places marked
TODO
, where scoping issues for theelse
block need to be resolved. I'll tackle that after awaiting feedback. It also doesn't parse while/else yet, only for/else. That would be a trivial addition.[1] #22891
edit: re-flow paragraphs.