posts/pl/lexer/

[utterances-bot]

Starting with a Lexer - in Rust

Building a lexer in Rust, following the book 'Writing an Interpreter'

https://mohitkarekar.com/posts/pl/lexer/

[marlalain]

read_identifier and read_digit have a bug. It's not while self.read_position < self.input.len(), it's while self.read_position <= self.input.len(). Testing something like 1 + 1 will show the bug.

[HavenSelph]

I think Vec<char> is an interesting approach vs str.chars however storing a Vec<char> inside of a variant is needless complexity and a waste of allocation. Instead store a slice of the source array. While you're at it you might as well go use str.char_indicies and just take &self.input[start..stop] as that would need no allocation at all and could be generalized across a token struct rather than only being part of select variants.

enum TokenKind {
  Identifier,
  Plus,
  Minus,
  ...
}

struct Token<'s> {
  kind: TokenKind,
  text: &'source str
}

struct Lexer<'s> {
  source: &'s str,
  chars: Peekable<CharIndicies<'s>>,
  current: Option<(usize, usize)>
}