allow to specify absolute paths for reports. to support msbuild runner #683

jmecosta · 2015-11-19T20:30:32Z

support absolute path of report paths

guwirth · 2015-11-21T11:38:42Z

sonar-cxx-plugin/src/main/java/org/sonar/plugins/cxx/utils/CxxReportSensor.java

@@ -187,6 +196,10 @@ protected String getStringProperty(String name, String def) {
        }

        for (String relPath : relPaths) {
+          CxxUtils.LOG.debug("Report '{}'", relPath);
+        }
+


@jmecosta it's iterating twice over the paths now? Is there a reason for this?

will take this, left for debugging. forgot to remove

@guwirth would you be ok to test this?

jmecosta · 2015-11-21T12:10:47Z

comments done

guwirth · 2015-11-24T19:55:00Z

@jmecosta will try to test it this week

guwirth · 2015-11-24T19:56:54Z

Related to #425

guwirth · 2015-11-28T22:43:57Z

@jmecosta not sure if I understand the feature in the right way? Following it's not working;

sonar-project.properties at D:\Sonar\Test

sonar.cxx.other.reportPath=D:/xxx/reports/externalrules-*.xml

log:

23:31:54.840 INFO  - Base dir: D:\Sonar\Test
23:31:54.840 INFO  - Working dir: D:\Sonar\Test\.sonar
...
23:31:55.830 INFO  - Sensor CxxExternalRulesSensor
23:31:55.830 DEBUG - Using pattern 'D:\xxx\reports\externalrules-*.xml' to find reports
23:31:55.830 DEBUG - Search string transformed from 'D:\xxx\reports\externalrules-*.xml'
23:31:55.830 DEBUG - To ':\xxx\reports\externalrules-*.xml' to find reports, using base Path 'D:\Sonar\Test'
23:31:55.830 WARN  - Cannot find a report for 'sonar.cxx.other.reportPath=:\xxx\reports\externalrules-*.xml'
...

jmecosta · 2015-11-29T06:22:47Z

I will change it, don't think there any reasons not to use reports outside
project root. In this change you can only specify absolute paths that are
inside project root

On Sun, Nov 29, 2015, 00:43 Günter Wirth [email protected] wrote:

@jmecosta https://github.com/jmecosta not sure if I understand the
feature in the right way? Following it's not working;

sonar-project.properties at D:\Sonar\Test

sonar.cxx.other.reportPath=D:/xxx/reports/externalrules-*.xml

log:

23:31:54.840 INFO - Base dir: D:\Sonar\Test
23:31:54.840 INFO - Working dir: D:\Sonar\Test.sonar
...
23:31:55.830 INFO - Sensor CxxExternalRulesSensor
23:31:55.830 DEBUG - Using pattern 'D:\xxx\reports\externalrules-.xml' to find reports
23:31:55.830 DEBUG - Search string transformed from 'D:\xxx\reports\externalrules-.xml'
23:31:55.830 DEBUG - To ':\xxx\reports\externalrules-.xml' to find reports, using base Path 'D:\Sonar\Test'
23:31:55.830 WARN - Cannot find a report for 'sonar.cxx.other.reportPath=:\xxx\reports\externalrules-.xml'
...

—
Reply to this email directly or view it on GitHub
#683 (comment)
.

guwirth · 2015-11-29T09:08:43Z

Still a problem:

09:54:01.296 INFO  - Sensor CxxExternalRulesSensor
09:54:01.296 DEBUG - Using pattern 'D:\xxx\reports\externalrules-*.xml' to find reports
09:54:01.296 DEBUG - Search string transformed from 'D:\xxx\reports\externalrules-*.xml'
09:54:01.296 DEBUG - To 'D:\xxx\reports\externalrules-*.xml' to find reports, using base Path 'D:\Sonar\Test'
09:54:01.296 WARN  - Cannot find a report for 'sonar.cxx.other.reportPath=D:\xxx\reports\externalrules-*.xml'

guwirth · 2015-11-29T09:19:03Z

@jmecosta Think would be better you write some unit tests. In my example above this should work:

 Base dir: D:\Sonar\Test

inside root absolute and relative inside root

sonar.cxx.other.reportPath=reports/externalrules-*.xml
sonar.cxx.other.reportPath=D:/Sonar/Test/reports/externalrules-*.xml

outside root absolute and relative outside root

sonar.cxx.other.reportPath=../../xxx/reports/externalrules-*.xml
sonar.cxx.other.reportPath=D:/xxx/reports/externalrules-*.xml

outside root on other drive

sonar.cxx.other.reportPath=C:/xxx/reports/externalrules-*.xml

guwirth · 2015-11-29T09:21:06Z

sonar-cxx-plugin/src/main/java/org/sonar/plugins/cxx/utils/CxxReportSensor.java

        DirectoryScanner scanner = new DirectoryScanner();
        String[] includes = new String[1];
        includes[0] = reportPath;
-        scanner.setIncludes(includes);
-        String baseDirPath = baseDirPath1;
        scanner.setBasedir(new File(baseDirPath));


is old baseDirPath

jmecosta · 2015-11-29T10:26:52Z

@guwirth i will write a integration test, i think its a better solution than unit test for this case. anyway this is still incomplete. thanks for the initial test, i will update and create the it and let you know its ready

jmecosta · 2015-11-29T15:53:37Z

@guwirth do you know why we have 2 basedir paths in the getReports function in CxxReportSensor?

can get the use for 2 paths from the tests..

edit: ok, so one is the root of the project and the other is the root of the module

guwirth · 2015-12-01T19:50:32Z

@jmecosta will test and if positive merge it this evening.

guwirth · 2015-12-01T21:06:36Z

@jmecosta I'm sorry but my example is still not working 👎

22:00:17.399 INFO  - Sensor CxxExternalRulesSensor
22:00:17.399 DEBUG - Using pattern 'D:\xxx\reports\externalrules-*.xml' to find reports
22:00:17.399 DEBUG - Unprocessed root directory 'D:\Sonar\Test'
22:00:17.399 DEBUG - Unprocessed report file 'D:\xxx\reports\externalrules-*.xml'
22:00:17.399 DEBUG - Processed root directory '\'
22:00:17.399 DEBUG - Processed report file 'D:\xxx\reports\externalrules-*.xml'
22:00:17.399 DEBUG - Unprocessed root directory 'D:\Sonar\Test'
22:00:17.399 DEBUG - Unprocessed report file 'D:\xxx\reports\externalrules-*.xml'
22:00:17.399 DEBUG - Processed root directory '\'
22:00:17.399 DEBUG - Processed report file 'D:\xxx\reports\externalrules-*.xml'
22:00:17.400 WARN  - Cannot find a report for 'sonar.cxx.other.reportPath=D:\xxx\reports\externalrules-*.xml'
22:00:17.400 INFO  - Sensor CxxExternalRulesSensor (done) | time=1ms

jmecosta · 2015-12-02T04:35:09Z

Ok suppose we need the it after all.

On Tue, Dec 1, 2015, 23:06 Günter Wirth [email protected] wrote:

@jmecosta https://github.com/jmecosta I'm sorry but my example is still
not working [image: 👎]

22:00:17.399 INFO - Sensor CxxExternalRulesSensor
22:00:17.399 DEBUG - Using pattern 'D:\xxx\reports\externalrules-.xml' to find reports
22:00:17.399 DEBUG - Unprocessed root directory 'D:\Sonar\Test'
22:00:17.399 DEBUG - Unprocessed report file 'D:\xxx\reports\externalrules-.xml'
22:00:17.399 DEBUG - Processed root directory ''
22:00:17.399 DEBUG - Processed report file 'D:\xxx\reports\externalrules-.xml'
22:00:17.399 DEBUG - Unprocessed root directory 'D:\Sonar\Test'
22:00:17.399 DEBUG - Unprocessed report file 'D:\xxx\reports\externalrules-.xml'
22:00:17.399 DEBUG - Processed root directory ''
22:00:17.399 DEBUG - Processed report file 'D:\xxx\reports\externalrules-.xml'
22:00:17.400 WARN - Cannot find a report for 'sonar.cxx.other.reportPath=D:\xxx\reports\externalrules-.xml'
22:00:17.400 INFO - Sensor CxxExternalRulesSensor (done) | time=1ms

—
Reply to this email directly or view it on GitHub
#683 (comment)
.

jmecosta · 2015-12-05T13:45:44Z

@guwirth actually it almost works.

DEBUG: Processed report file 'e:\reports\reports-xunit*.xml'
INFO: Cannot find a report for 'sonar.cxx.xunit.reportPath=e:\reports\reports-xunit*.xml'
DEBUG: No reports found, nothing to process
INFO: Sensor CxxXunitSensor (done) | time=1ms
INFO: Sensor CxxCoverageSensor
DEBUG: Parsing coverage reports
INFO: Processing report 'e:\reports\bullseye-coverage-results\bullseyecoverage-result-0.xml'
INFO: Parsing 'Cobertura' format
INFO: Parsing 'Bullseye' format
INFO: Added report 'e:\reports\bullseye-coverage-results\bullseyecoverage-result-0.xml' (parsed by: BullseyeParser) to the coverage data

if path is real file then is able to read it. need to figure out how that directory scanner works with these cases.

jmecosta · 2015-12-05T17:04:26Z

@guwirth i am really surprised with the behaviour of this directory scanner... i wrote a test to check for existing paths... the log shows this:

18:58:10.808 [main] DEBUG CxxPlugin - Unprocessed root directory 'E:\src\data'
18:58:10.810 [main] DEBUG CxxPlugin - Unprocessed report file 'E:\reports\reports-rats*.xml'
18:58:10.812 [main] DEBUG CxxPlugin - Processed root directory 'E:'
18:58:10.812 [main] DEBUG CxxPlugin - Processed report file 'reports\reports-rats*.xml'

In test it asserts that
assertThat(relPaths.length).isEqualTo(1615);

When i run the analysis i get:

DEBUG: Using pattern 'E:\reports\reports-rats*.xml' to find reports
DEBUG: Unprocessed root directory 'E:\src\data'
DEBUG: Unprocessed report file 'E:\reports\reports-rats*.xml'
DEBUG: Processed root directory 'E:'
DEBUG: Processed report file 'reports\reports-rats*.xml'
DEBUG: Number of Report Files Found '0'

Exactly the same output, one detects 1615 reports the other one 0.

What i am missing here? Can you cofirm this behaviour in your side? the test is this testWithRealDataInTest

this solution should work, as per test. i am ensure why its not working

jmecosta · 2015-12-05T17:14:59Z

drop directory scanner and use

http://wilddiary.com/list-files-matching-a-naming-pattern-java/

guwirth · 2015-12-05T20:27:39Z

@jmecosta did some tests removing reactor. Maybe you can take this into account => #695

jmecosta · 2015-12-06T08:00:28Z

are you suggesting to remove the reactor. for absolute paths the reactor does not apply, the base dir should be basepath or the padron is defined.

jmecosta · 2015-12-06T10:21:37Z

@guwirth i have something that works now for your case, hopefully it works also for other cases but the implementation is now totally different and tests need to be adjusted a lot.

also in my opinion we should isolate the tests for path lookup, reports parsing and metrics saving (there should not be any unit tests really but integration tests), having unit tests that do all requires loads of maintenance if we need to refactor some parts as we are doing it now

wdyt?

jmecosta · 2015-12-06T10:28:45Z

the implementation is like this:

absolute Path: x:\abc\dfe\abc.xml
. baseroot -> x:\abc\dfe
. search pattern -> abc.xml
. recursive -> false

absolute Path: c:\abc\dfe*.xml (or anytype of https://docs.oracle.com/javase/tutorial/essential/io/fileOps.html#glob)
. baseroot -> x:\abc\dfe
. search pattern -> *.xml or anytype of glob
. recursive -> false

relative Paths: abc.xml
. baseroot -> Reactor base Path, or Module Base Path if not found in reactor
. search pattern -> abc.xml
. recursive -> false

relative Paths: *.xml (or anytype of https://docs.oracle.com/javase/tutorial/essential/io/fileOps.html#glob)
. baseroot -> Reactor base Path, or Module Base Path if not found in reactor
. search pattern -> *.xml (or anytype of https://docs.oracle.com/javase/tutorial/essential/io/fileOps.html#glob)
. recursive -> false

relative Paths: **/abc.xml
. baseroot -> Reactor base Path only because in fact all abc,xml in all folders will be found
. search pattern -> abc.xml
. recursive -> true

the code is much simpler now. also a notice the normalizepath function is really slow and we need to find some alternative also during this refactoring

guwirth · 2015-12-11T15:35:37Z

@jmecosta Travis build is working again. You can squash and rebase to get new results.

jmecosta · 2015-12-11T15:51:23Z

@guwirth its done, but this is not expected to pass for now. ive disable many unit tests. if we agree on a way to go then i will refactor the tests also

however i think you can test this to see if it supports your use case

jmecosta · 2015-12-11T15:53:23Z

@guwirth appveyour down?

guwirth · 2015-12-11T18:09:53Z

@jmecosta Travis & AppVeyor failing, problem in multi-module IT.

SonarOpenCommunityAdmin · 2015-12-11T18:39:37Z

appveyour should now pass. i need to check that it once it runs again.

jmecosta · 2015-12-12T07:48:43Z

@guwirth i added a new it test, its a complex multimodule project. it can be used to test the process of reports relative to module. its currently testing reports outside project folder, however you might need to create a python function to change the contents of the reports if we are in windows or linux

jmecosta · 2015-12-12T11:58:43Z

@guwirth tests are now passing in windows, and ITs are also passing in all configurations. there is 1 test failing in unix only that i am ensure whats the issue. not sure why it fails. ive push a commit to improve the logging of the test.

i supose review is ready, and please try also running with with your test data that was failing earlier

jmecosta · 2015-12-12T13:16:56Z

All green now

guwirth · 2015-12-12T13:48:35Z

@jmecosta it's working now 👍
@Bertk: do you also wanna test it?

test with Windows 7 x64, SQ 5.1.2, this PR
reports absolute outside of baseDir
path in reports: absolute

sonar.cxx.other.reportPath=D:/xxx/reports/externalrules-*.xml
sonar.cxx.coverage.reportPath=D:/xxx/reports/coverage-*.xml

<?xml version="1.0"?>
<results>
<error file="D:\Sonar\Test\error_recovery\pi\Pi.cpp" line="11" ... />
<error file="d:\sonar\test\error_recovery\pi\pi.cpp" line="12" ... />
</results>

guwirth · 2015-12-12T14:49:48Z

integration-tests/testdata/cpp-multimodule-project-2/A/2/sonar-project.properties

@@ -0,0 +1,2 @@
+sonar.projectName=2
+sonar.sources=./


Why is sources this ./ and not only . ?

guwirth · 2015-12-13T08:19:50Z

@jmecosta could you squash and rebase that please that we can merge it.
@Bertk any comments from your side?

jmecosta · 2015-12-13T08:35:47Z

done

Bertk · 2015-12-13T10:54:46Z

@jmecosta I started an analysis with the PR this morning and results are OK. By the way I also used PR #695 😉

guwirth · 2015-12-13T11:14:28Z

@Bertk what is your main use case:

path to reports in config file relative or absolute?
reports inside or outside baseDir?
file items in report files relative or absolute?

…te-paths-in-report allow to specify absolute paths for reports. to support msbuild runner

Bertk · 2015-12-13T19:00:31Z

@guwirth

I use relative and absolute paths for the reports
additional I use multiple test reports for one component ( comma separate) . The ant pattern is useless because the files located in different folders and I use absolute paths for the test results
the reports for pc-lint, cppcheck, Vera++, RATS are located in basedir. I use post-build events to create this reports within CI builds
I use a mixture of relative or absolute paths for the files within the reports (vc compiler and pc-lint with absolute paths, Vera++ and CppCheck or RATS with relative paths)

guwirth reviewed Nov 21, 2015
View reviewed changes

jmecosta force-pushed the feature/allow-to-specify-absolute-paths-in-report branch from 53daaa4 to 82097da Compare November 21, 2015 12:09

guwirth added the enhancement label Nov 24, 2015

guwirth added this to the M 0.9.5 milestone Nov 24, 2015

guwirth assigned jmecosta Nov 24, 2015

jmecosta mentioned this pull request Nov 29, 2015

CppCheck result is not shown in Sonar jmecosta/CxxSonarQubeRunner#5

Closed

guwirth reviewed Nov 29, 2015
View reviewed changes

jmecosta force-pushed the feature/allow-to-specify-absolute-paths-in-report branch from 2275078 to 6b2fd4c Compare November 29, 2015 19:25

guwirth mentioned this pull request Dec 5, 2015

Multi module support #697

Closed

jmecosta mentioned this pull request Dec 6, 2015

improved multi module support #695

Merged

This was referenced Dec 6, 2015

Use java.nio.file.path instead of org.apache.commons.io.FilenameUtils #621

Closed

Report files outside of project base directory not read in #425

Closed

jmecosta force-pushed the feature/allow-to-specify-absolute-paths-in-report branch from e4f717b to 3e4f489 Compare December 11, 2015 15:49

jmecosta force-pushed the feature/allow-to-specify-absolute-paths-in-report branch from 1f3afb3 to 2e2012e Compare December 11, 2015 18:38

jmecosta force-pushed the feature/allow-to-specify-absolute-paths-in-report branch from 6c89e46 to 4eae78e Compare December 12, 2015 07:45

jmecosta force-pushed the feature/allow-to-specify-absolute-paths-in-report branch from 5f09764 to a663b63 Compare December 12, 2015 11:24

guwirth reviewed Dec 12, 2015
View reviewed changes

allow to specify absolute paths for reports. to support msbuild runner

d2038d6

jmecosta force-pushed the feature/allow-to-specify-absolute-paths-in-report branch from 89262b0 to d2038d6 Compare December 13, 2015 08:34

guwirth added a commit that referenced this pull request Dec 13, 2015

Merge pull request #683 from jmecosta/feature/allow-to-specify-absolu…

0ecf39e

…te-paths-in-report allow to specify absolute paths for reports. to support msbuild runner

guwirth merged commit 0ecf39e into SonarOpenCommunity:master Dec 13, 2015

This was referenced Jan 16, 2016

CxxReportSensor.java: file not found for issue #693

Closed

CxxXunitSensor in a multimodule project #612

Closed

		@@ -0,0 +1,2 @@
		sonar.projectName=2
		sonar.sources=./

allow to specify absolute paths for reports. to support msbuild runner #683

allow to specify absolute paths for reports. to support msbuild runner #683

Conversation

jmecosta commented Nov 19, 2015

guwirth Nov 21, 2015

Choose a reason for hiding this comment

jmecosta Nov 21, 2015

Choose a reason for hiding this comment

jmecosta Nov 21, 2015

Choose a reason for hiding this comment

jmecosta commented Nov 21, 2015

guwirth commented Nov 24, 2015

guwirth commented Nov 24, 2015

guwirth commented Nov 28, 2015

jmecosta commented Nov 29, 2015

guwirth commented Nov 29, 2015

guwirth commented Nov 29, 2015

guwirth Nov 29, 2015

Choose a reason for hiding this comment

jmecosta commented Nov 29, 2015

jmecosta commented Nov 29, 2015

guwirth commented Dec 1, 2015

guwirth commented Dec 1, 2015

jmecosta commented Dec 2, 2015

jmecosta commented Dec 5, 2015

jmecosta commented Dec 5, 2015

jmecosta commented Dec 5, 2015

guwirth commented Dec 5, 2015

jmecosta commented Dec 6, 2015

jmecosta commented Dec 6, 2015

jmecosta commented Dec 6, 2015

guwirth commented Dec 11, 2015

jmecosta commented Dec 11, 2015

jmecosta commented Dec 11, 2015

guwirth commented Dec 11, 2015

SonarOpenCommunityAdmin commented Dec 11, 2015

jmecosta commented Dec 12, 2015

jmecosta commented Dec 12, 2015

jmecosta commented Dec 12, 2015

guwirth commented Dec 12, 2015

guwirth Dec 12, 2015

Choose a reason for hiding this comment

guwirth commented Dec 13, 2015

jmecosta commented Dec 13, 2015

Bertk commented Dec 13, 2015

guwirth commented Dec 13, 2015

Bertk commented Dec 13, 2015