fix: Handle newlines and comments in the middle of expressions #217
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
susliko
force-pushed
the
less-automated-semicolons
branch
5 times, most recently
from
b224541
to
60243c0
Compare
Problem: ------- Code like ```scala if true () else () ``` or ```scala a .b // comment .c ``` is not parsed correctly, for extra `$._automated_semicolon` tokens, which are produced after newlines Solution: ------- Tweak `scanner.c` to stop producing AUTOMATED_SEMICOLON tokens when encountering comments, else/catch clauses, etc.
|
@eed3si9n this one is ready for review |
eed3si9n
reviewed
May 24, 2023
|
|
||
| lexer->mark_end(lexer); | ||
| lexer->result_symbol = AUTOMATIC_SEMICOLON; | ||
|
|
||
| if (newline_count > 1) return true; | ||
| // Probably, a multi-line field expression, e.g. |
There was a problem hiding this comment.
Thanks for this comment here.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #188, #199
Problem:
Code like
or
a .b // comment .cis not parsed correctly, for extra
$._automated_semicolontokens,which are produced after newlines
Solution:
Tweak
scanner.cto stop producing AUTOMATED_SEMICOLON tokens whenencountering comments, else/catch clauses, etc.
Extra changes:
corpus/expressions.txt formatted with
tree-sitter test -uin the first commitNote:
The following code still won't be parsed
a /* comment */ .bas I haven't figured out a way both to parse it and code like
where AUTOMATED_SEMICOLON is needed to distinguish consecutive blocks