You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Lexer currently makes too many decisions about valid syntax when it should just be collecting valid tokens. It shouldn't care that some@ thing isn't a valid fragment.
Additionally, if it allowed some@ thing then it would emit the same tokens as some @thing since it scrubs whitespace, so there should most likely be a Token::Whitespace to represent any contiguous span of spaces and/or tabs.
It probably shouldn't also add a Newline unless there is one.
The text was updated successfully, but these errors were encountered:
Lexer currently makes too many decisions about valid syntax when it should just be collecting valid tokens. It shouldn't care that
some@ thing
isn't a valid fragment.Additionally, if it allowed
some@ thing
then it would emit the same tokens assome @thing
since it scrubs whitespace, so there should most likely be aToken::Whitespace
to represent any contiguous span of spaces and/or tabs.It probably shouldn't also add a
Newline
unless there is one.The text was updated successfully, but these errors were encountered: