Feature Request: Dynamic tokens in the pest file

[NickLarsenNZ]

I would like to use tera for templating files, but I want to be able to use different delimeters.

I see they have hard coded the delimeters https://github.com/Keats/tera/blob/93c92b97bf54799dc14aa859cc8628fe6e8493b5/src/parser/tera.pest#L148-L155 which is preventing Keats/tera#343 from progressing.

I thought it would be nice so that the application can provide some dynamic tokens to pest at runtime (eg: from user config).

token1 = { $t1 }
token2 = { $t2 }

#[derive(Parser)]
#[grammar = "parser/tera.pest"]
pub struct TeraParser;

...

let dynamic_tokens = HashMap::from([
  ("t1", "~{ | {~"),
  ("t2", "}~ | ~}"),
]);

TeraParser::parse_with_tokens(Rule::template, input, dynamic_tokens);

I hope the above illustrates the point, even if not using desirable syntax.

[NickLarsenNZ]

I understand the Parse struct(s) is built at compile time, but maybe there can be something added to the generated code to inject the overriden tokens.

[tomtau]

I'm not sure if it's easy to retrofit that functionality in a non-breaking way to pest 2.X.
It's been in consideration for pest3: #885 (comment)
@TheVeryDarkness made some progress on a pest3 prototype by using pest-typed in https://github.com/TheVeryDarkness/pest3
After the basic prototype is done, we can look into refactoring and extending it to potentially include these other features, such as dynamically changing tokens.

BTW I think @Keats mentioned the new version of tera would use its custom parser, so it shouldn't be limited by this I think?

[Keats]

Yep new version will have a new parser. It doesn't allow custom delimiters right now but it should not be hard to add