Implement Input Cursor

[victor-pogor]

Background and Motivation

A cursor in the context of a lexer helps manage the current position within the input text being processed, it can look ahead (peek) for the next characters, advance the cursor, etc.

Proposed API

internal sealed class TextCursor
{
    // Constructors
    public TextCursor(ReadOnlySpan<byte> text);
    public TextCursor(ReadOnlySpan<char> text);
    public TextCursor(ReadOnlySpan<string> text);

    // Properties
    public Option<byte> Current { get; }
    public int Length { get; }
    public bool IsAtEnd { get; }

    // Methods
    public Option<byte> Peek(int delta = 0);
    public void Advance(int delta = 1);
    public void Advance(Func<byte, bool> predicate);
    public bool TryAdvance(byte b);
    public bool TryAdvance(ReadOnlySpan<byte> subtext);
}

Usage Examples

ITextCursor cursor = new Cursor("test string"u8);
Option<byte> firstChar = cursor.Peek(); // 't'

Acceptance criteria

• Do not use null, use an Option type
• The cursor must work with UTF-8 string literal as an input
• Unit Tests
• Mutation Testing

Open questions

• Naming: Cursor, Scanner?
• How do different languages handle the scanning phase?
  • Roslyn uses a separate SlidingTextWindow class
  • Typescript has scanning and lexing phase in one file
  • Go has scanning and lexing phase in one file
  • Rust has a Cursor impl