Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recursive parsing does not seem to propagate compilation unit settings #1489

Closed
apriori opened this issue Jun 8, 2018 · 4 comments · Fixed by #1984
Closed

Recursive parsing does not seem to propagate compilation unit settings #1489

apriori opened this issue Jun 8, 2018 · 4 comments · Fixed by #1984
Assignees
Labels
Milestone

Comments

@apriori
Copy link

apriori commented Jun 8, 2018

Description

When parsing a .cpp file that includes another in bracket form, the parser fails to locate headers included
in the second file, because the include path information provided by the compilation unit of the root file are not recursively propagated.

Steps to reproduce the problem

Have a .cpp file like this:

(CLI.cpp)

#include <ApplicationController.h>

(ApplicationController.h)

#include <QCoreApplication> 

(compile_commands.json)

[
    {
        "directory": "/root",
        "command": "/usr/bin/clang++-6.0 -DQT_NO_DEBUG -I/opt/Qt/5.10.0/gcc_64/include -I. -I/opt/Qt/5.10.0/gcc_64/include/QtCore -I/opt/Qt/5.10.0/gcc_64/mkspecs/linux-g++ -O3 -DNDEBUG -Wall -Wextra -Wpedantic -Werror -fPIC -std=gnu++1z -o CLI.cpp.o -c CLI.cpp -I/usr/include/c++/7 -I/usr/include/x86_64-linux-gnu/c++/7 -I/usr/include/c++/7/backward -I/usr/lib/gcc/x86_64-linux-gnu/7/include -I/usr/local/include -I/usr/lib/gcc/x86_64-linux-gnu/7/include-fixed -I/usr/include/x86_64-linux-gnu -I/usr/include",
        "file": "CLI.cpp"
    }
]

Expected behavior

Upon analysis of CLI.cpp, Squid should parse the file, hit the include line, and then recursively continue, always propagating root include path configuration.

Actual behavior

As indicated in the logs, instead, when the "ApplicationController.h" is parsed, Squid falls back to only use the global settings. Which is clearly wrong.

LOG file

11:23:30.494 DEBUG: 'CoreApplication/src/CLI.cpp' indexed with language 'c++'
11:23:30.495 DEBUG: 'CoreApplication/src/ApplicationController.cpp' indexed with language 'c++'
11:23:30.499 DEBUG: 'CoreApplication/src/include_public/ApplicationController.h' indexed with language 'c++'
1:23:32.421 DEBUG: compilation unit settings for: '$root/CoreApplication/src/CLI.cpp'
11:23:32.421 DEBUG: storing include root: '$root/CoreApplication/src/include_public'
11:23:32.421 DEBUG: storing include root: '$root/CoreApplication/src'
11:23:32.421 DEBUG: storing include root: '/opt/Qt/5.10.0/gcc_64/include'
11:23:32.421 DEBUG: storing include root: '/opt/Qt/5.10.0/gcc_64/include/QtCore'
11:23:32.421 DEBUG: storing include root: '/opt/Qt/5.10.0/gcc_64/mkspecs/linux-g++'
11:23:32.421 DEBUG: storing include root: '/usr/include/c++/7'
11:23:32.421 WARN: the include root '$root/Clang-Release-Linux/CoreApplication/src/CoreApplication_autogen/include' doesn't exist
11:23:32.421 DEBUG: storing include root: '/usr/include/x86_64-linux-gnu/c++/7'
11:23:32.421 DEBUG: storing include root: '/usr/include/c++/7/backward'
11:23:32.421 DEBUG: storing include root: '/usr/lib/gcc/x86_64-linux-gnu/7/include'
11:23:32.421 DEBUG: storing include root: '/usr/local/include'
11:23:32.421 DEBUG: storing include root: '/usr/lib/gcc/x86_64-linux-gnu/7/include-fixed'
11:23:32.421 DEBUG: storing include root: '/usr/include/x86_64-linux-gnu'
11:23:32.421 DEBUG: storing include root: '/usr/include'
11:23:32.651 WARN: [/usr/include/x86_64-linux-gnu/gnu/stubs.h:7]: cannot find the sources for '# include <gnu/stubs-32.h>'
11:23:33.015 DEBUG: API File: CLI.cpp
11:23:33.016 DEBUG: Header file suffixes: [.hxx, .hpp, .hh, .h]
11:23:33.016 DEBUG: finished preprocessing '$root/CoreApplication/src/CLI.cpp'
11:23:33.016 DEBUG: API File: CLI.cpp
11:23:33.016 DEBUG: Header file suffixes: [.h, .hh, .hpp, .H]
11:23:33.017 DEBUG: 'CoreApplication/src/CLI.cpp' generated metadata  with charset 'UTF-8'
11:23:33.019 DEBUG: CxxFileLinesVisitor: '$root/CoreApplication/src/CLI.cpp'
11:23:33.019 DEBUG:    lines:           '26'
11:23:33.019 DEBUG:    executableLines: '[18, 8, 24]'
11:23:33.019 DEBUG:    linesOfCode:     '[18, 21, 6, 8, 24, 11, 15]'
11:23:33.019 DEBUG:    linesOfComments: '[17, 23]'
11:23:33.020 DEBUG: global settings for: '$root/CoreApplication/src/include_public/ApplicationController.h'
11:23:33.020 DEBUG: Parse '$root/CoreApplication/src/include_public/ApplicationController.h' as C++ file
11:23:33.020 WARN: [$root/CoreApplication/src/include_public/ApplicationController.h:3]: cannot find the sources for '#include <QCoreApplication>'
11:23:33.022 DEBUG: API File: ApplicationController.h
11:23:33.022 DEBUG: Header file suffixes: [.hxx, .hpp, .hh, .h]
11:23:33.022 DEBUG: finished preprocessing '$root/CoreApplication/src/include_public/ApplicationController.h'
11:23:33.022 DEBUG: API File: ApplicationController.h
11:23:33.022 DEBUG: Header file suffixes: [.h, .hh, .hpp, .H]
11:23:33.023 DEBUG: 'CoreApplication/src/include_public/ApplicationController.h' generated metadata  with charset 'UTF-8'
11:23:33.023 WARN: [$root/CoreApplication/src/include_public/ApplicationController.h:9]: skip declaration: class ApplicationController : public QObject {
11:23:33.023 WARN: [$root/CoreApplication/src/include_public/ApplicationController.h:11]:    syntax error: Q_OBJECT public : 
11:23:33.023 DEBUG: ***** Node: initDeclaratorList tokenValue='ApplicationController' tokenLine=15 tokenColumn=4
11:23:33.024 DEBUG: Doc: //! Constructs a new ApplicationController based on the framework-\p qtApp.
11:23:33.024 DEBUG: Public API: ApplicationController
11:23:33.024 DEBUG: node: declaratorId line: 15 id: 'ApplicationController' documented: true
11:23:33.024 DEBUG: ***** Node: initDeclaratorList tokenValue='ApplicationController' tokenLine=15 tokenColumn=4
11:23:33.024 DEBUG: Doc: //! Constructs a new ApplicationController based on the framework-\p qtApp.
11:23:33.028 DEBUG: Public API: ApplicationController
11:23:33.028 DEBUG: node: declaratorId line: 15 id: 'ApplicationController' documented: true
11:23:33.029 DEBUG: ***** Node: initDeclaratorList tokenValue='~' tokenLine=18 tokenColumn=4
11:23:33.029 DEBUG: Doc: //! Destructor
11:23:33.029 DEBUG: Public API: ~
11:23:33.029 DEBUG: node: declaratorId line: 18 id: '~' documented: true
11:23:33.029 DEBUG: ***** Node: initDeclaratorList tokenValue='~' tokenLine=18 tokenColumn=4
11:23:33.029 DEBUG: Doc: //! Destructor
11:23:33.029 DEBUG: Public API: ~
11:23:33.029 DEBUG: node: declaratorId line: 18 id: '~' documented: true
11:23:33.029 DEBUG: ***** Node: initDeclaratorList tokenValue='parseCommandLineParameters' tokenLine=22 tokenColumn=9
11:23:33.029 DEBUG: Doc: //! Parse command line parameters that were committed to the \qtApp.
11:23:33.029 DEBUG: Public API: parseCommandLineParameters
11:23:33.029 DEBUG: node: declaratorId line: 22 id: 'parseCommandLineParameters' documented: true
11:23:33.030 DEBUG: ***** Node: initDeclaratorList tokenValue='parseCommandLineParameters' tokenLine=22 tokenColumn=9
11:23:33.030 DEBUG: Doc: //! Parse command line parameters that were committed to the \qtApp.
11:23:33.030 DEBUG: Public API: parseCommandLineParameters
11:23:33.030 DEBUG: node: declaratorId line: 22 id: 'parseCommandLineParameters' documented: true
11:23:33.030 DEBUG: Not enough content in 'CoreApplication/src/include_public/ApplicationController.h' to have CPD blocks, it will not be part of the duplication detection
11:23:33.031 DEBUG: CxxFileLinesVisitor: '$root/CoreApplication/src/include_public/ApplicationController.h'
11:23:33.031 DEBUG:    lines:           '24'
11:23:33.031 DEBUG:    executableLines: '[]'
11:23:33.031 DEBUG:    linesOfCode:     '[18, 20, 22, 9, 11, 13, 15]'
11:23:33.031 DEBUG:    linesOfComments: '[17, 5, 21, 14]'
11:23:33.033 DEBUG: global settings for: '$root/CoreApplication/src/include_public/CLI.h'
11:23:33.033 DEBUG: Parse '$root/CoreApplication/src/include_public/CLI.h' as C++ file
11:23:33.034 WARN: [$root/CoreApplication/src/include_public/CLI.h:3]: cannot find the sources for '#include <QObject>'
11:23:33.037 DEBUG: API File: CLI.h
11:23:33.037 DEBUG: Header file suffixes: [.hxx, .hpp, .hh, .h]
11:23:33.037 DEBUG: finished preprocessing '$root/CoreApplication/src/include_public/CLI.h'
11:23:33.037 DEBUG: API File: CLI.h
11:23:33.037 DEBUG: Header file suffixes: [.h, .hh, .hpp, .H]
11:23:33.039 DEBUG: 'CoreApplication/src/include_public/CLI.h' generated metadata  with charset 'UTF-8'
11:23:33.039 WARN: [$root/CoreApplication/src/include_public/CLI.h:13]: skip declaration: class CLI : public QObject {
11:23:33.039 WARN: [$root/CoreApplication/src/include_public/CLI.h:15]:    syntax error: Q_OBJECT public : 

Related information

  • cxx plugin version: 1.0.0
  • SonarQube version 6.7.1 LTS
@ivangalkin
Copy link
Contributor

Hi @apriori,

thank you for the ticket. Indeed, it looks like the preprocessor applies the include paths only to the first-level includes. I will double-check it.

Does this issues affects you in any other way, other then the console warnings? If you have a squid-rule MissingIncludeFile enabled and see too many false-positives, please consider disabling this rule as a workaround (you will still notice the real missing includes when you compile your sources).

@guwirth the maintenance of existing preprocessor code contradicts to #1484. Please comment, which strategy should we follow.

Regards

@guwirth guwirth added the bug label Jun 9, 2018
@guwirth
Copy link
Collaborator

guwirth commented Jun 9, 2018

Hi @ivangalkin,

I also realized that there are problems with the preprocessor. There is also #1225 to fix. For me it's not interesting because we use anyway external tools for issue validation.

he maintenance of existing preprocessor code contradicts to #1484. Please comment, which strategy should we follow.

Think the discussion how to proceed in general with the parser will take longer. My proposal is to fix #1489 and #1225.

Regards,

@ivangalkin
Copy link
Contributor

Hi @apriori,

I tried to reproduce your issue and didn't succeed (I used different include nestings, the quote includes, angle-bracket includes etc). The preprocessor code looks correct to me, too.

Is it possible, that your compilation database is not consistent? E.g. the following lines indicate, that some include paths do not exist or were not expanded properly:

11:23:32.421 DEBUG: storing include root: '$root/CoreApplication/src/include_public'
11:23:32.421 DEBUG: storing include root: '$root/CoreApplication/src'
11:23:32.421 WARN: the include root '$root/Clang-Release-Linux/CoreApplication/src/CoreApplication_autogen/include' doesn't exist

Could you please check your JSON file and/or provide an example, which could help us to reproduce the issue? Thank you!

@apriori
Copy link
Author

apriori commented Jun 11, 2018

@ivangalkin I just manually replaced parts of the as with "$root" to denote, that this is the project root part, so the content should not be considered literally. I'll try to come up with an example.

And yes, it makes the analysis virtually useless, because classes of depending headers are not parsed and then considered parser errors in the .cpp lacking its definitions.

I'll recheck the compilation database. It would not surprise me that there is something "odd" with it, because frankly, this whole procedure with the database is a mere hack.

@guwirth guwirth modified the milestones: 1.1, 1.2 Jun 19, 2018
@guwirth guwirth removed this from the 1.2 milestone Oct 29, 2018
@guwirth guwirth added this to the 1.2.1 milestone Nov 14, 2018
@guwirth guwirth modified the milestones: 1.2.1, 1.2.2 Dec 23, 2018
@guwirth guwirth removed this from the 1.2.2 milestone Feb 9, 2019
@guwirth guwirth self-assigned this Dec 1, 2020
@guwirth guwirth added this to the 2.0.0 milestone Dec 1, 2020
guwirth added a commit to guwirth/sonar-cxx that referenced this issue Dec 2, 2020
- support Microsoft Quoted form search (close SonarOpenCommunity#1225):
1) In the same directory as the file that contains the #include statement.
2) In the directories of the currently opened include files, in the reverse order in which they were opened. The search begins in the directory  of the parent include file and continues upward through the directories of any grandparent include files.
3) fallback to use include paths instead of local folder (same as Angle-bracket form)

- add #include unit tests
- test for compilation database settings propagation (close SonarOpenCommunity#1489)
- refactoring
guwirth added a commit to guwirth/sonar-cxx that referenced this issue Dec 3, 2020
- support Microsoft Quoted form search (close SonarOpenCommunity#1225):
1) In the same directory as the file that contains the #include statement.
2) In the directories of the currently opened include files, in the reverse order in which they were opened. The search begins in the directory  of the parent include file and continues upward through the directories of any grandparent include files.
3) fallback to use include paths instead of local folder (same as Angle-bracket form)

- add #include unit tests
- test for compilation database settings propagation (close SonarOpenCommunity#1489)
- refactoring
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging a pull request may close this issue.

3 participants