What format/form is being used for Cpp2 grammar?

[DyXel]

I started reading into Kate Editor's documentation for syntax highlighting (KSyntaxHighlighting, which is also used by Qt Creator) since I wanted to write one for cpp2.

After a while of reading said docs I noticed that the way the context switching works in this syntax highlighter is pretty much how a PEG behaves, which in turn are usually written formally with a grammar syntax, for which there seems to be a lot of tooling available today.

The main idea was to see if I could generate all (or most of) the highlighter definition from the grammar, and then continue from there, so I wrote a small Python script to fetch the grammar from cppfront's source code but then... I couldn't find any tool that would parse that grammar syntax? even without obvious stuff like the Any ... placeholders, and I can't figure out which format is this grammar supposed to follow.

I checked BNF, EBNF, ABNF, CDL, but none look like the one.. I also checked some python libraries to see which one look similar, but none of them do (or maybe I am just tired). At the moment I am contemplating if I should write my own parser in lieu of an existing solution or just do the highlighter manually but that seems very tedious.

Thoughts on this? I looked into using what was proposed at #910 but it seems to be out of date already...

Script used to extract the grammar (in case anybody is interested)

#!/usr/bin/env python3
from os import path, listdir
from sys import argv, stderr


def read_grammar_from_directory(file_dir: str) -> str:
    grammar = ""
    for file_name in listdir(file_dir):
        full_path = path.join(file_dir, file_name)
        if not path.isfile(full_path):
            continue
        with open(full_path, "r") as f:
            for l in f:
                if "//G" not in l:
                    continue
                grammar += l.lstrip()
    return grammar.rstrip()


def clean_up_grammar(g: str) -> str:
    grammar = ""
    big_todo = False
    for l in g.splitlines():
        if "//G TODO" in l:
            big_todo = True
            continue
        if "//GT" in l or "//GTODO" in l in l:
            continue
        stripl = l.removeprefix("//G")[1:]
        if big_todo:
            if not stripl:
                big_todo = False
            continue
        grammar += f"{stripl}\n"
    return grammar


def remove_comments_from_grammar(g: str) -> str:
    grammar = ""
    comments = ["#", "[Note", "//"]
    for l in g.splitlines(keepends=True):
        for c in comments:
            idx = l.rfind(c)
            if idx >= 0:
                l = f"{l[:idx].rstrip()}\n"
                # If line ended with just space as a result of replacement,
                # just get rid of it.
                if l.isspace():
                    l = ""
        grammar += l
    return grammar


def main():
    if len(argv) <= 1:
        print(f"Usage: {argv[0]} path_to_cppfront_source_folder", file=stderr)
        exit(1)
    grammar = read_grammar_from_directory(argv[1])
    grammar = clean_up_grammar(grammar)
    print(remove_comments_from_grammar(grammar))


if __name__ == "__main__":
    main()

[JohelEGP]

I started reading into Kate Editor's documentation for syntax highlighting (KSyntaxHighlighting, which is also used by Qt Creator) since I wanted to write one for cpp2.

[JohelEGP]

I do not know the format or form,
but I do know a rule from https://github.com/hsutter/cppfront/wiki/Design-note%3A-Unambiguous-parsing:

First match wins

[DyXel]

Yep, that sounds like what PEGs do and how it works in this highlighter as well.

[hsutter]

Following up on this in the context of #1031: Without rewriting the grammar, are there any blockers right now to mechanically generate PEG grammar from the //G comment grammar I'm using now (and which likely still has a few oversight corrections needed)? If so is there anything incremental I can do to unblock that mechanical transformation?

I'll ask the same question on the sister issues.

Thanks!

[DyXel]

Unfortunately it seems like grammars of this format are not "mechanically transformable" to a PEG, and subsequently, to the highlighting engine of KSyntaxHighlighting. I transformed the entirety of the grammar to YAML in order to do more mechanical transformations to it, but I didn't get anywhere (I'll attach it here in case its helpful to you or anyone else).

The main issue that makes creating a good highlighter for Cpp2 not trivial is that it needs to take into account all the contextual keywords that are used, its not enough to just dumbly highlight the keywords. For example, override is highlighted on C++ mode everywhere, despite technically being a contextual keyword.

grammar.zip as ZIP since GitHub doesn't like YAMLs for some reason.

[hsutter]

Thanks!

The main issue that makes creating a good highlighter for Cpp2 not trivial is that it needs to take into account all the contextual keywords that are used, its not enough to just dumbly highlight the keywords.

Right -- but I didn't think that was a problem in practice? Are you encountering cases where taking that shortcut was really noisy and a problem in Cpp1 code, so the highlighter doesn't feel "good"?

And speaking of which:

For example, override is highlighted on C++ mode everywhere, despite technically being a contextual keyword.

You can mostly blame that on me too... I'm the one who led the charge for making override and final be C++'s first two contextual keywords (way better than attributes).

But as you can see in Cpp2, I'm totally unrepentant about it. 😉 Even in a new syntax, I think it's important to do that, because it avoids a long-term problem: Globally reserved words always create a tension between the user and the language over who gets the "nice" words, and because after 1.0 ships the user always wins. So with lots of globally reserved keywords, the language always ends up becoming inconsistent over time because it will have some "nice" keywords (that were in 1.0, like Cpp1 virtual, so the language could take them) and "not-nice" ones (that weren't in 1.0, like Cpp1 co_await and soon contract_assert).

[DyXel]

Right -- but I didn't think that was a problem in practice? Are you encountering cases where taking that shortcut was really noisy and a problem in Cpp1 code, so the highlighter doesn't feel "good"?

It's really not, I just wanted to do it right, only highlight in and such where it matters for example (same with type and other short common words).

I totally get from where you come, I am not against context sensitive keywords, but it does complicate things a bit for highlighters is my point, especially if you want to do an outstanding job, like I wanted to...

Might dedicate few hours to do the dumb highlighting I mentioned to not be empty handed here, but I haven't really needed it since the C++ highlighter covers most things.

[DyXel]

Also, before I forget it: Its not just a Cpp2 highlighter in isolation, you also need the logic to detect C++ code since they both can be interleaved in the same file (like other languages such as HTML+CSS+JS), doing the "fallback" is a bit of a weird process that I am not sure how its supposed to work yet 😛

[hsutter]

Ah, got it!

For the fallback, you could check the io.h file for how I do it... basically, start the top of the file and:

• while not at end of file
  • if it's a comment or preprocessor statement, skip this line
  • otherwise, if it starts with an identifier followed by a colon, I treat it as a Cpp2 section until the end of the declaration
  • otherwise, treat it as Cpp1 and do brace-matching to skip that Cpp1 section (this checks for braces in literals, but can't handle unbalanced braces added by macros which fortunately seem to be very rare)

[HALL9kv0]

@DyXel Have you managed to get highlighting and clangd working on Qt Creator for cpp2? I've been trying for a long time, but no luck.
If you have found the way, would you mind sharing a guide similar to Herb's guide on how to set up cpp2 for Visual Studio?

Thank you in advance!

[DyXel]

Not really. What I have done is simply use the C++ highlighter for now, it seems to be good enough. One funny thing is that since $ is not yet part of the basic character set, the highlighter marks it red, but it actually ends up helping me find where I am using captures in my code.