Highlighting error for concatenated strings on multiple lines

[jhoussay]

Description

For concatenated strings declared on multiple lines, an highlighting error occurs.
This is quite a minor issue as it does not prevent defect detection in the rest of the source file, it only generates invalid highlighting in the source file and pollutes sonar-scanner outputs with warnings.

Sample code to reproduce the problem

#include <string>

// Works !
std::string ok = "foo" "-bar";

// Does not work...
std::string fail = "foo"
                   "-bar";

sonar-scanner warning

WARN: Highligthing error in file '.../Test.cpp' at line:7, column:19

Source visualization in SonarQube

As expected with the warning, highlighting fails on the second string declaration.
However, highlighting is also incorrect for the last two characters on the first string declaration.

[guwirth]

@jhoussay this is an known issue. Problem is the two step mechanism to parse the code. First step is the preprocessor doing the string concatenation (intermediate result) and in a second step the C++ parser is parsing and creating the syntax highlighting.

Good ideas/solutions are welcome.

[ivangalkin]

Hi @guwirth, @jhoussay and all,

I suggest to store the original Tokens, merged during the preprocessing as Trivia. Please see PR #1480 for proposed solution.

Regards

[ivangalkin]

Hi @guwirth,

although my change (#1480) is consistent with the previous highlighting logic (comments and preprocessed preprocessor statements are also stored in Trivia), I am wondering why is highlighting is done after preprocessing at all?

Usually AST is not required for highlighting at all, only lexer is used. So maybe the long-term (?) solution is to split our parser or/and put highlighting between the lexer and later parsing steps (e.g. preprocessor etc)?

[guwirth]

closed with #1480