lexer errors shouldn't abort compilation

[timotheecour]

proposal

turn lexer error into a nkLexerError PNode that doesn't stop compilation.
This will make it easier to evolve nim when we need to make backward incompatible changes.

example 1

since (1,5,3):
  proc `'big`(number: string): int = ...
  echo 1234'big

this code should compile in nim < 1.5.3 and nim >= nim-lang/Nim#17020 (user-defined number suffixes)

example 2

see nim-lang/Nim#14253 (comment)

since (1,3,5):
  let foo1 {.importc.}: int  # works since lexer has always allowed it

since (1,5,3):
  const foo2 {.importc.}: int  # lexer error, but would work under this RFC

implementation

the simplest is to, upon encountering a lexer error, tokenize based on whitespace (or perhaps other delimiters) and produce a (nkLexerError) nkError PNode for each unrecognized token. An error would then be generated if semantic pass encounters it in a non-disabled context (eg when false)

note

• this should be backported to 1.0 branch
• nkError: refs outline of how I imagine nkError to work Nim#16915

[Araq]

I don't understand why we cannot use the more general (upcoming...) nkError node for this.

[timotheecour]

i didn't want to tie up this RFC to nkError since it doesn't exist yet, but yes, nkError makes sense to reuse for this, I've replaced nkLexerError by nkError in proposal