Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WARN: Cannot find the file '...' in module '...' base dir '...', skipping violations. #1651

Closed
rudolfgrauberger opened this issue Jan 3, 2019 · 6 comments · Fixed by #1653
Assignees
Milestone

Comments

@rudolfgrauberger
Copy link
Contributor

Description

If the source code is in subdirectories, the comparison of the paths is case-sensitive. (Also on Windows)

The output of MSBuild contains all paths in lowercase (see buildlog.log in log section). However, the project contains a subdirectory "Sources" in which all source files are located. This means that the files cannot be found.

Steps to reproduce the problem

PS > git clone https://github.com/rudolfgrauberger/sonar-cxx-examples.git
PS > cd sonar-cxx-examples/FolderStructureCpp
PS > MSBuild.exe .\FolderStructureCpp.vcxproj /t:rebuild /p:Configuration=Release /fileLogger "/fileLoggerParameters:LogFile=buildlog.log;Verbosity=normal;Encoding=UTF-8"
PS > sonar-scanner

Expected behavior

That the compiler sensor finds the file and analyzes it. Under Windows the upper and lower case "actually" doesn't matter.

If I have seen this correctly here (#155), then it was already a topic once. Could it be that it was lost during the last refactoring (#1180)?

Actual behavior

Because the file is not found, it is not analyzed.

Known workarounds

Not good, but works...

Rename folder to lower case

PS > mv Sources sources_temp
PS > mv sources_temp sources

LOG file

stdout

...
INFO: Searching reports by relative path with basedir 'E:\Dev\Source\Repos\sonar-cxx-samples\FolderStructureCpp' and search prop 'sonar.cxx.vc.reportPath'
INFO: Parser will parse '1' report file(s)
INFO: Processing report 'E:\Dev\Source\Repos\sonar-cxx-samples\FolderStructureCpp\buildlog.log'
INFO: Parsing 'Visual C++' initialized with report 'E:\Dev\Source\Repos\sonar-cxx-samples\FolderStructureCpp\buildlog.log', Charset= 'UTF-8'
INFO: Using pattern : '(.*>)?\x20*(?<file>.*)\((?<line>\d+)\):\x20warning\x20(?<id>C\d+):(?<message>.*)'
WARN: Cannot find the file 'e:\dev\source\repos\sonar-cxx-samples\folderstructurecpp\sources\main.cpp' in module 'FolderStructureCpp' base dir 'E:\Dev\Source\Repos\sonar-cxx-samples\FolderStructureCpp', skipping violations.
INFO: CXX-COMPILER-VC processed = 0
...

buildlog.log

Der Buildvorgang wurde am 03.01.2019 01:47:41 gestartet.
Projekt "E:\Dev\Source\Repos\sonar-cxx-samples\FolderStructureCpp\FolderStructureCpp.vcxproj" auf Knoten "1", rebuild Ziel(e).
_PrepareForClean:
  Die Datei "Release\FolderSt.67F13409.tlog\FolderStructureCpp.lastbuildstate" wird gelöscht.
InitializeBuildStatus:
  "Release\FolderSt.67F13409.tlog\unsuccessfulbuild" wird erstellt, da "AlwaysCreate" angegeben wurde.
ClCompile:
  C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.16.27023\bin\HostX86\x86\CL.exe /c /IIncludes /Zi /nologo /W4 /WX- /diagnostics:classic /sdl /O2 /Oi /Oy- /GL /D CODE_ANALYSIS /D _MBCS /Gm- /EHsc /MD /GS /Gy /fp:precise /permissive- /Zc:wchar_t /Zc:forScope /Zc:inline /Fo"Release\\" /Fd"Release\vc141.pdb" /Gd /TP /analyze /analyze:quiet /analyze:plugin"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.16.27023\bin\HostX86\x86\EspXEngine.dll" /analyze:plugin"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.16.27023\bin\HostX86\x86\localespc.dll" /FC /errorReport:queue  /analyze:ruleset"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Team Tools\Static Analysis Tools\Rule Sets\NativeRecommendedRules.ruleset" Sources/main.cpp Sources/TestClass.cpp
  main.cpp
e:\dev\source\repos\sonar-cxx-samples\folderstructurecpp\sources\main.cpp(3): warning C4100: "argv": Unreferenzierter formaler Parameter [E:\Dev\Source\Repos\sonar-cxx-samples\FolderStructureCpp\FolderStructureCpp.vcxproj]
e:\dev\source\repos\sonar-cxx-samples\folderstructurecpp\sources\main.cpp(3): warning C4100: "argc": Unreferenzierter formaler Parameter [E:\Dev\Source\Repos\sonar-cxx-samples\FolderStructureCpp\FolderStructureCpp.vcxproj]
  TestClass.cpp
  Kompilieren...
Link:
  C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.16.27023\bin\HostX86\x86\link.exe /ERRORREPORT:QUEUE /OUT:"E:\Dev\Source\Repos\sonar-cxx-samples\FolderStructureCpp\Release\FolderStructureCpp.exe" /NOLOGO kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /MANIFEST /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /DEBUG:FULL /PDB:"E:\Dev\Source\Repos\sonar-cxx-samples\FolderStructureCpp\Release\FolderStructureCpp.pdb" /OPT:REF /OPT:ICF /LTCG:incremental /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"E:\Dev\Source\Repos\sonar-cxx-samples\FolderStructureCpp\Release\FolderStructureCpp.lib" /MACHINE:X86 /SAFESEH Release\main.obj
  Release\TestClass.obj
  Code wird generiert.
  All 2 functions were compiled because no usable IPDB/IOBJ from previous compilation was found.
  Codegenerierung ist abgeschlossen.
  FolderStructureCpp.vcxproj -> E:\Dev\Source\Repos\sonar-cxx-samples\FolderStructureCpp\Release\FolderStructureCpp.exe
FinalizeBuildStatus:
  Die Datei "Release\FolderSt.67F13409.tlog\unsuccessfulbuild" wird gelöscht.
  Aktualisieren des Timestamps von "Release\FolderSt.67F13409.tlog\FolderStructureCpp.lastbuildstate".
Die Erstellung von Projekt "E:\Dev\Source\Repos\sonar-cxx-samples\FolderStructureCpp\FolderStructureCpp.vcxproj" ist abgeschlossen (rebuild Ziel(e)).


Der Buildvorgang wurde erfolgreich ausgeführt.

"E:\Dev\Source\Repos\sonar-cxx-samples\FolderStructureCpp\FolderStructureCpp.vcxproj" (rebuild Ziel) (1) ->
(ClCompile Ziel) -> 
  e:\dev\source\repos\sonar-cxx-samples\folderstructurecpp\sources\main.cpp(3): warning C4100: "argv": Unreferenzierter formaler Parameter [E:\Dev\Source\Repos\sonar-cxx-samples\FolderStructureCpp\FolderStructureCpp.vcxproj]
  e:\dev\source\repos\sonar-cxx-samples\folderstructurecpp\sources\main.cpp(3): warning C4100: "argc": Unreferenzierter formaler Parameter [E:\Dev\Source\Repos\sonar-cxx-samples\FolderStructureCpp\FolderStructureCpp.vcxproj]

    2 Warnung(en)
    0 Fehler

Verstrichene Zeit 00:00:03.43

Related information

  • Windows version: 10 x64 (1809)
  • cxx plugin version: V1.2.1
  • SonarQube (in Docker) version: 7.4.0.18908
  • Visual Studio version: 2017 Enterprise
@guwirth
Copy link
Collaborator

guwirth commented Jan 3, 2019

@rudolfgrauberger thanks for reporting the issue.

If the source code is in subdirectories, the comparison of the paths is case-sensitive. (Also on Windows)

This is a known issue:

  • MSBuild creates always lower case paths.
  • SQ core is using a database internally. The comparison of paths depends on the database setting (and not the OS).
  • I'm solving this with a script re-creating upper/lowercase afterwards before forwarding it to scanner (not the best solution)

Someone another solution? @Bertk think you are also using this. How you are doing this?

@ivangalkin
Copy link
Contributor

@rudolfgrauberger @guwirth short comment from my side:

for the implementation of the check, whether the given path lies inside of the give project/module, we rely completely on the SonarQube plugin API.

  public InputFile getInputFileIfInProject(SensorContext sensorContext, String path) {
    ...
    final InputFile inputFile = sensorContext.fileSystem().inputFile(sensorContext.
      fileSystem().predicates().hasPath(path));
    if (inputFile == null) {
      LOG.warn("Cannot find the file '{}' in module '{}' base dir '{}', skipping violations.",
        path, sensorContext.module().key(), sensorContext.fileSystem().baseDir());
      notFoundFiles.add(path);
    }
    ...
  }

The internal implementation of the search predicates is (as far I see) string based. See DefaultFileSystem for more details. This might explain why expected equality doesn't work in case of symbolic links, case-insensitive file systems etc. You might want to file an issue against https://github.com/SonarSource/sonarqube (probably somewhere in the depths of https://jira.sonarsource.com/).

P.S. I believe, we shouldn't implement any workarounds as part of sonar-cxx.

@guwirth
Copy link
Collaborator

guwirth commented Jan 3, 2019

@ivangalkin problem is that VS on Windows never works without correction. So not sure if we should not correct it inside the VS sensor?

@rudolfgrauberger
Copy link
Contributor Author

rudolfgrauberger commented Jan 3, 2019

The internal implementation of the search predicates is (as far I see) string based. See DefaultFileSystem for more details. This might explain why expected equality doesn't work in case of symbolic links, case-insensitive file systems etc.
You might want to file an issue against https://github.com/SonarSource/sonarqube (probably somewhere in the depths of https://jira.sonarsource.com/).

When I see this here and see that they did it in the scanners themselves (Pull Request and Commit), I don't think they're changing anything.

P.S. I believe, we shouldn't implement any workarounds as part of sonar-cxx.

I don't think there's a realistic alternative. If we solve this similar to their commit with Path.toRealPath(), we would have solved both problems (symbolic links and case-insensitive file systems).

Edit: I'm definitely gonna try it. It sounds like a cleaner solution/workaround than renaming the paths with a script before.

Are you interested? If yes, then I can make a pull request afterwards. Then you can still see if it suits you and you want to have it.

@ivangalkin
Copy link
Contributor

@rudolfgrauberger sonar-cxx is not a stand-alone application; it is a plugin, which is executed inside of the framework; so there are two things running independently

  • sonar-scanner framework walks through the project metadata and build-ups the internal structure (project -1:n-> modules -1:n-> directories -1:n-> files)
  • for each module they build a set of file paths
  • obviously they don't use canonical path representation. This is a simplification, which allows to avoid e.g. symbolic links and therefore files shared between modules. In terms of SonarQube such situation is just impossible.
  • every plugin is invoked for all created modules
  • sonar-cxx e.g. parses the external reports
  • it extracts the paths and the corresponding warnings
  • the most important part is to check if the path is applicable to this particular module (external report can cover multiple modules, some path might be excluded or just wrong)
  • it is impossible to publish an issue for a path, which SonarQube doesn't tracks as a part of this particular module
  • so there is no use to have your own real/canonical paths, as long as they are not known to the framework

We had this very issue with symbolic links. Please see #1574/ #1575.

I believe the solution with Path::toRealPath() might work, but we definitely have to use Path::toRealPath(LinkOption.NOFOLLOW_LINKS). Also I have my doubts w.r.t. the relative paths. In that case we have to absolutize the given sub-path by means of the module base-directory. What if SonarQube framework (see 1) used non-real base-directory in its structure? The result of our normalization will never be found.

I did a quick draft of the possible solution. Please take a look at the implementation (#1653). Not sure, if we always have to double-check the result of searching-for-realpath with searching-for-just-given-path.

@guwirth what do you think?

@rudolfgrauberger could you please test the solution on you setup. You can find the build artifacts under https://ci.appveyor.com/project/SonarOpenCommunity/sonar-cxx/builds/21369802/artifacts

@rudolfgrauberger
Copy link
Contributor Author

@ivangalkin Thank you very much for this explanation, it is very helpful. I will test this solution and post my answer to #1653

ivangalkin added a commit to ivangalkin/sonar-cxx that referenced this issue Jan 5, 2019
* introduce the fallback for the lookup of `InputFile`s:
  try it with the Path::realPath()

resolves SonarOpenCommunity#1651
@guwirth guwirth added this to the 1.2.2 milestone Jan 6, 2019
Bertk pushed a commit to Bertk/sonar-cxx that referenced this issue Jun 22, 2019
* introduce the fallback for the lookup of `InputFile`s:
  try it with the Path::realPath()

resolves SonarOpenCommunity#1651
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging a pull request may close this issue.

3 participants