Error when we add a spécific GCC include path in sonar.c.includeDirectories

[david2053]

Description

We have Lexer error: and java.lang.IllegalArgumentException when we add a spécific GCC include path in sonar.c.includeDirectories

Related information

We are using sonarqube on an ARM project in C.

Sonarqube Version 6.7.3 (build 38370)
C (Community) plugin version 1.1.0
sonar.language=c
sonar.sourceEncoding=UTF-8

We are using gcc-arm-none-eabi-4_9-2015q3 as compiler.

Our project compile without trouble on the same machine (Linux debian)

Trouble description

For each include from the gcc-arm-none-eabi-4_9-2015q3 librairy, sonarqube give a warning like this :
11:51:53.477 WARN: [/home/gcc/jenkins/workspace/G5.c/common/debug.h:27]: cannot find the sources for '#include <stddef.h>'

It seems missing GCC includes create many "C++ Parser can't read code. Declaration is skipped.", for exemple uint16_t, size_t, ... are unknown and create "C++ Parser can't read code. Declaration is skipped."

To solve this, we have tried to had librairy include path in sonar.c.includeDirectories.

If we add :
/home/gcc/gcc-arm-none-eabi-4_9-2015q3/arm-none-eabi/include/sys/, /home/gcc/gcc-arm-none-eabi-4_9-2015q3/lib/gcc/arm-none-eabi/4.9.3/include/
it supresse some warning " cannot find the sources for ..."

but to complet include's path we need to add :
/home/gcc/gcc-arm-none-eabi-4_9-2015q3/arm-none-eabi/include

and when we add this last path, we have a lots off error like :
11:42:58.068 ERROR: Unable to parse file: /home/gcc/jenkins/workspace/G5.c/common/quickSort.c 11:42:58.071 ERROR: Lexer error: Unable to lex url: file:/home/gcc/jenkins/workspace/G5.c/common/quickSort.c 11:42:57.437 DEBUG: global settings for: '/home/gcc/jenkins/workspace/G5.c/common/quickSort.c' 11:42:57.438 DEBUG: Parse '/home/gcc/jenkins/workspace/G5.c/common/quickSort.c' as C file, ends in '*.c' 11:42:58.068 ERROR: Unable to parse file: /home/gcc/jenkins/workspace/G5.c/common/quickSort.c 11:42:58.071 ERROR: Lexer error: Unable to lex url: file:/home/gcc/jenkins/workspace/G5.c/common/quickSort.c 11:42:58.072 DEBUG: API File: quickSort.c 11:42:58.072 DEBUG: Header file suffixes: [.h] 11:42:58.073 DEBUG: finished preprocessing '/home/gcc/jenkins/workspace/G5.c/common/quickSort.c' 11:42:58.073 DEBUG: API File: quickSort.c 11:42:58.074 DEBUG: Header file suffixes: [.h, .hh, .hpp, .H] 11:42:58.081 DEBUG: CxxFileLinesVisitor: '/home/gcc/jenkins/workspace/G5.c/common/quickSort.c' 11:42:58.084 DEBUG: 'common/quickSort.c' generated metadata with charset 'UTF-8' 11:42:58.085 DEBUG: lines: '429' 11:42:58.085 DEBUG: executableLines: '[]' 11:42:58.085 DEBUG: linesOfCode: '[]' 11:42:58.086 DEBUG: linesOfComments: '[]' 11:42:58.086 DEBUG: Not enough content in 'common/quickSort.c' to have CPD blocks, it will not be part of the duplication detection 11:42:58.087 DEBUG: global settings for: '/home/gcc/jenkins/workspace/G5.c/common/quickSort.h' 11:42:58.087 DEBUG: Parse '/home/gcc/jenkins/workspace/G5.c/common/quickSort.h' as C++ file 11:42:58.100 DEBUG: API File: quickSort.h 11:42:58.100 DEBUG: Header file suffixes: [.h] 11:42:58.101 DEBUG: finished preprocessing '/home/gcc/jenkins/workspace/G5.c/common/quickSort.h' 11:42:58.101 DEBUG: API File: quickSort.h 11:42:58.102 DEBUG: Header file suffixes: [.h, .hh, .hpp, .H] 11:42:58.105 DEBUG: 'common/quickSort.h' generated metadata with charset 'UTF-8' 11:42:58.108 WARN: Highligthing error in file '/home/gcc/jenkins/workspace/G5.c/common/quickSort.h' at line:29, column:0 11:42:58.110 DEBUG: highlighing exception {} java.lang.IllegalArgumentException: Unable to highlight file common/quickSort.h at org.sonar.api.batch.sensor.highlighting.internal.DefaultHighlighting.highlight(DefaultHighlighting.java:97) at org.sonar.api.batch.sensor.highlighting.internal.DefaultHighlighting.highlight(DefaultHighlighting.java:37) at org.sonar.cxx.sensors.visitors.CxxHighlighterVisitor.highlight(CxxHighlighterVisitor.java:187) at org.sonar.cxx.sensors.visitors.CxxHighlighterVisitor.visitToken(CxxHighlighterVisitor.java:178) at com.sonar.sslr.impl.ast.AstWalker.visitToken(AstWalker.java:107) at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:86) at com.sonar.sslr.impl.ast.AstWalker.walkAndVisit(AstWalker.java:69) at org.sonar.squidbridge.AstScanner.scanFiles(AstScanner.java:110) at org.sonar.cxx.sensors.squid.CxxSquidSensor.execute(CxxSquidSensor.java:173) at org.sonar.scanner.sensor.SensorWrapper.analyse(SensorWrapper.java:53) at org.sonar.scanner.phases.SensorsExecutor.executeSensor(SensorsExecutor.java:88) at org.sonar.scanner.phases.SensorsExecutor.execute(SensorsExecutor.java:82) at org.sonar.scanner.phases.SensorsExecutor.execute(SensorsExecutor.java:68) at org.sonar.scanner.phases.AbstractPhaseExecutor.execute(AbstractPhaseExecutor.java:88) at org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:180) at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:135) at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:121) at org.sonar.scanner.scan.ProjectScanContainer.scan(ProjectScanContainer.java:288) at org.sonar.scanner.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:283) at org.sonar.scanner.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:261) at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:135) at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:121) at org.sonar.scanner.task.ScanTask.execute(ScanTask.java:48) at org.sonar.scanner.task.TaskContainer.doAfterStart(TaskContainer.java:84) at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:135) at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:121) at org.sonar.scanner.bootstrap.GlobalContainer.executeTask(GlobalContainer.java:121) at org.sonar.batch.bootstrapper.Batch.doExecuteTask(Batch.java:116) at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:71) at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60) at com.sun.proxy.$Proxy0.execute(Unknown Source) at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:171) at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:128) at org.sonarsource.scanner.cli.Main.execute(Main.java:111) at org.sonarsource.scanner.cli.Main.execute(Main.java:75) at org.sonarsource.scanner.cli.Main.main(Main.java:61) Caused by: java.lang.IllegalArgumentException: 6 is not a valid line offset for pointer. File common/quickSort.h has 0 character(s) at line 29 at org.sonar.api.internal.google.common.base.Preconditions.checkArgument(Preconditions.java:145) at org.sonar.api.batch.fs.internal.DefaultInputFile.checkValid(DefaultInputFile.java:312) at org.sonar.api.batch.fs.internal.DefaultInputFile.newPointer(DefaultInputFile.java:246) at org.sonar.api.batch.fs.internal.DefaultInputFile.newRange(DefaultInputFile.java:262) at org.sonar.api.batch.sensor.highlighting.internal.DefaultHighlighting.highlight(DefaultHighlighting.java:95) ... 40 common frames omitted

Investigation tests

Without the last include path, the same files are parsed with some warning, but without error.

If we remove the first 2 path and keep the third, we have the same error. This is not due to a sonar.c.includeDirectories line too long.

We create a new project to check some analyse history effect. With the new project without history, we have the same error.

We have uninstalled C++ (Community) plugin, we have the same error.

We have check that all files are in UTF8.

Questions

Is default compiler include path in sonarqube, like GCC include ?
Is there another way than sonar.c.includeDirectories to manage compiler include path?
If not, anyone have an idea about this trouble?

[ivangalkin]

Hi @david2053 and thank you for reaching out!
@guwirth I suggest to mark this issue as duplicate to #1230 and think once again about the modification of predecessor:

• soften the warnings about the missing includes (WARN -> INFO) ?
• disable include processing at least partially (simplify preprocessor if squid checks are disabled #1528) ?

or/and

• create a FAQ, so that we don't have to explain each time, that missing includes do not play any (significant) role.

more ideas are welcome

[david2053]

It seems missing GCC includes create many "C++ Parser can't read code. Declaration is skipped.", for exemple uint16_t, size_t, ... are unknown and create "C++ Parser can't read code. Declaration is skipped."

[ivangalkin]

W.r.t. to the issue itself:

WARN: Highligthing error in file '/home/gcc/jenkins/workspace/G5.c/common/quickSort.h' at line:29, column:0 11:42:58.110 DEBUG: highlighing exception {} java.lang.IllegalArgumentException: Unable to highlight file common/quickSort.h at  ...

@david2053

1. I suggest you to ignore the warnings à la "missing includes". They do not play any role at all if you import static analysis reports from 3rd party tools. They might have very limited impact on the size/complexity metrics or internal checks, but

• you are most probably interested in the dynamic of the metrics (is there fall/rise off some metric) rather then in its absolute value
• we suggest to use external static analysis tools instead of internal checks anyway

2. Highlighting problems are caused most probably by our preprocessor. Please remove the standard includes (/home/gcc/gcc-arm-none-eabi-4_9-2015q3/arm-none-eabi/include/sys/, /home/gcc/gcc-arm-none-eabi-4_9-2015q3/lib/gcc/arm-none-eabi/4.9.3/include/) from your include paths and try to run the analysis again. Please ignore the include warnings for now (as I mentioned they don't play any significant role). Does the highlighting error still appear?

Thank you!

[david2053]

We have many "C++ Parser can't read code. Declaration is skipped." like this 👍

We thougth that it's due to the fact that time_t, uint16_t, bool are unknow due to "GCC missing include"

[david2053]

without gcc-arm-none-eabi-4_9-2015q3 include path, quickSort.h is parsed without error, no "Highligthing error in file"

[david2053]

It seems we have done a wrong interpretation, I juste try to replace uint16_t an so, "C++ Parser can't read code. Declaration is skipped." are still here:

[ivangalkin]

I would highly recommend you to disable all parser related warnings. E.g.

• C++ parser failure
• C++ preprocessor unable to locate file referenced by #include directive
• C++ skip parser error

They cause many false positives and are pretty much redundant: Your compiler is the best tool for checking the syntax, missing includes etc.

Also we recommend to use external static analysis reports instead of internal checks. Internal checks of sonar-cxx provide a very limited number of functionality.

Please take a look at the tools, which reports sonar-cxx can import and visualize: https://github.com/SonarOpenCommunity/sonar-cxx/wiki/Running-tools

[david2053]

I found the source of the trouble.
If we use PRIu64 or equivalent like in this code :
fprintf(stdout, "% 23" PRIu64 ":%-16s %4" PRIu16 ":", xTaskGetTickCount() * portTICK_PERIOD_MS, pFileName, ligne);

The GCC include is not found and PRIu64 is not define and it generate many "C++ skip parser error" in the code. Only one use at the end of a file generate many tens of "C++ skip parser error" to the top of the file.

It seem there is a bug when a define not define use in an ASCII concatenation.

[ivangalkin]

Hi @david2053,

great, that you've succeeded with your investigation. However it's not really a bug but something I've predicted in my previous answer:

Highlighting problems are caused most probably by our preprocessor.

1. Preprocessor couldn't find the definition of format macro constants (PRI<type>).
2. It couldn't perform the macro expansion
3. Without macro expansion the code is syntactically incorrect: It was impossible to build the AST, perform the highlighting, execute some checks etc

The correct solution would be to provide all possible header files or to parametrize the defines. You can intercept your build by Bear and use the resulting JSON compiler database for the passing of all paths and defines. Please see sonar.cxx.jsonCompilationDatabase. Some build systems (e.g. cmake) are able to generate the compiler database too.

but please keep in mind, that

1. Our preprocessor is not as mighty as the real compiler's preprocessor (and I doubt it ever will). There will be always parsing errors, just because of the complexity of C/C++ syntax.
2. There is no sense to rely on the parser of sonar-cxx: just compile your source code by your favorite compiler and you'll get the best syntax check. You might want to compile the code with multiple compilers if the portability is important for you (e.g. clang is known for strict implementation of language standards)
3. Parsability is mostly important for the internal checks. The main advantage of sonar-cxx is however the import of external reports. They are far more reliable, especially the ones, which are based on compiler toolchain (e.g. clang-tidy, clang-sa etc).

@guwirth we can hard-code these format macro constants (https://en.cppreference.com/w/c/types/integer) in the similar way we hard-code __FILE__, __LINE__ etc. I don't think, that it makes much sense, since there will be always some missing includes/macros and other preprocessor pitfalls

[guwirth]

@ivangalkin

@guwirth we can hard-code these format macro constants (https://en.cppreference.com/w/c/types/integer) in the similar way we hard-code FILE, LINE etc. I don't think, that it makes much sense, since there will be always some missing includes/macros and other preprocessor pitfalls

Would not hard-code it. The macros from https://en.cppreference.com/w/c/types/integer are part of an header file (<stdint.h>). __FILE__, __LINE__, ... are predefined macros of a tool. Think the better way for @david2053 is to forceinclude a header file and add them there. It's also possible to add other includes. e.g. <stdint.h>. If you start hard-coding it where do you start and where do you stop. Next one is using MFC, next ATL, next ...

• https://github.com/SonarOpenCommunity/sonar-cxx/wiki/Force-Include
• https://github.com/SonarOpenCommunity/sonar-cxx/wiki/Dealing-with-compiler-specific-code-pieces

[guwirth]

@ivangalkin other idea: we could read the .cfg files from cppcheck to read the types and macros. Would avoid maintenance effort on our side.

[ivangalkin]

@guwirth I don't like the hard-coding either. The idea with cppcheck's .cfg is more or less the the same as JSON compilation database, which (if generated and parsed correctly) already contains all includes and user defines. It doesn't contain the standard defines, but having all includes should be sufficient (#1304?).

Our preprocessor is important for building an AST, otherwise the visitors won't function. I am aware of that. However since we favor external analyzers over the internal ones, I don't see much perspective in pleasing the preprocessor again and again.

The C/C++ standard and our preprocessor implement that following rule: if define was not set, it's evaluated to 0. E.g. #ifdef EXPERIMENTAL is evaluated to 0 by default. What if we could extend the JoinStringsPreprocessor in the similar way?

Currently the chained literals are concatenated:

"Hello" " world" "!" // becomes "Hello world!"

Maybe we could soften the parsing error and interpret the constructs like that as chained literals too?

"Hello" UNRESOLVABLE_MACRO "!" // becomes "Hello <unresolved>!"

It's not the standard C/C++ anymore, but rather an improvement of parser robustness.

[david2053]

Thanks to you to investigate soutions about our probleme.

I want to had a comment, for us, if the line with the undefiend define is not parsed and give a WARN, it is not a probleme.
For us the main probleme is this line, with the undefiend define, generate tens lines with "C++ skip parser error" above in the file. With many anoying effect like :

• many lines are not analysed
• many lines are not marked with code coverage indicator

If your parser could avoid to generate "false" WARN in the rest of the file, it could be very interessant. But I have no idea about the complexity of this kind of solution.