-
Notifications
You must be signed in to change notification settings - Fork 35
GenericLexerExtension
Olivier Duhart edited this page Feb 21, 2018
·
15 revisions
For performance purpose the generic lexer limits the lexems definitions. Nevertheless an extension mechanism provides a way to add new lexem pattern relying on the FSM backing the generic lexer. To extend a lexer we need to add transitions and nodes to the underlying FSM.
New generic tokens are denoted with the specifice GenericToken.Extension
value. For instance we can define a date lexem like this
public enum Extensions {
[Lexeme(GenericToken.Extension)]
DATE,
[Lexeme(GenericToken.Double)]
DOUBLE,
}
The extensions are build in a callback function called for every token badged with the GenericToken.Extension
value.
The callback signature is a delegate
public delegate void BuildExtension<IN>( IN token, LexemeAttribute lexem, GenericLexer<IN> lexer) where IN : struct;
where :
- IN is the enum name used for the lexer (
Extensions
above) - token is the token to build the extension for
- lexem is the lexem attribute : it allows to get optional parameters for the lexem through
lexem.GenericTokenParameters
(string[]
) - lexer is the lexer.
diagram representing the maximal predifined FSM. list of named nodes.
API goto transition* + precondition end callback
BuildExtension<Extensions> extensionBuilder = (Extensions token, LexemeAttribute lexem, GenericLexer<Extensions> lexer) => {
if (token == Extensions.DATE) {
// do something interesting here
}
}
var lexerRes = LexerBuilder.BuildLexer<Extensions>(new BuildResult<ILexer<Extensions>>(), extensionBuilder);