Define a Hex Literal

[utech626]

Can you please help defining a Hex Literal using the Generic Lexer. Example [a-fA-F0-9]+

thanks

[b3b00]

Hello,

I will look at it tomorrow. In the meantime you can look at lexer extension that is the way to do it

[b3b00]

Here is a quick answer.

using sly.lexer;
using sly.lexer.fsm;
using sly.i18n;

namespace hexa
{
    public enum hexalexer
    {
        [Extension]
        HEX,
    }

    public class Extendedhexalexer
    {
        public static void Extendhexalexer(hexalexer token, LexemeAttribute lexem, GenericLexer<hexalexer> lexer)
        {
            if (token == hexalexer.HEX)
            {
                NodeCallback<GenericToken> callbackHEX = (FSMMatch<GenericToken> match) =>
                {
                    // this store the token id the the FSMMatch object to be later returned by GenericLexer.Tokenize 
                    match.Properties[GenericLexer<hexalexer>.DerivedToken] = hexalexer.HEX;
                    return match;
                };
                var builder = lexer.FSMBuilder;
                builder.GoTo("start").Transition('0')
                .Transition('x')
                .MultiRangeTransition(('0', '9'), ('a', 'f'), ('A', 'F'))
                .Mark("mark#1")
                .MultiRangeTransitionTo("mark#1", ('0', '9'), ('a', 'f'), ('A', 'F'))
                .End(GenericToken.Extension)
                .CallBack(callbackHEX);
            }
        }
    }
}

[utech626]

Thanks so much

[b3b00]

i am now thinking to incorporate it to CSLY. This also makes me found some issues in CSLY CLI that I've used to generate the answer, so thanks for it :)

[b3b00]

I have published a new csly version that provides a New hexadécimal token type.

    public enum HexaLexer
    {
        [Hexa("6x")]
        H,
        [Int]
        I,
    }

This lexer définies an hexadecimal token H starting with 6x and followed by a séquence of hexadecimal digits.

You can try it with nuget version 3.1.8-alpha3