Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add coverage sensor 'Testwell CTC++' for textural reports #1486

Merged
merged 17 commits into from
Jun 14, 2018

Conversation

rufinio
Copy link
Contributor

@rufinio rufinio commented May 30, 2018

This change is Reviewable

@guwirth guwirth requested review from Bertk and guwirth May 31, 2018 05:34
@guwirth
Copy link
Collaborator

guwirth commented May 31, 2018

@rufinio thx for providing this extension.

Could you provide some information please.

  1. A link to the tool page.
  2. How to call / use the tool to generate a report for the cxx plugin?
  3. Wondering why you are parsing an ASCII report? Does the tool not provide XML?

Did not have the time to do a code review yet. Points should work:

@guwirth guwirth added this to the 1.1 milestone May 31, 2018
Copy link
Contributor

@Bertk Bertk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice to add support for another commercial coverage tool 👍

sensor = new CxxCoverageSensor(new CxxCoverageCache(), language, context);
sensor.execute(context);

assertThat(context.lineHits("ProjectKey:HGBuildNumberLookup.cpp", 42)).isEqualTo(10);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use one assertThat statement per test. You can also use SoftAssertions (see http://joel-costigliola.github.io/assertj/core/api/org/assertj/core/api/SoftAssertions.html)

sensor = new CxxCoverageSensor(new CxxCoverageCache(), language, context);
sensor.execute(context);

assertThat(context.lineHits("ProjectKey:test-wildmatch.c", 3)).isEqualTo(209);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use one assertThat statement per test. You can also use SoftAssertions (see http://joel-costigliola.github.io/assertj/core/api/org/assertj/core/api/SoftAssertions.html)

}
}

private void addEachLine(CoverageMeasures coverageMeasures) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please reduce complexity of this method (e.g. extract methods). The Cognitive Complexity of this method is 57.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you recommend a tool for Cognitive Complexity?
Which value range is good?

if((new File(filename)).isAbsolute() == false) {
normalFilename = FilenameUtils.normalize("./" + filename);
}
else {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style: Move this "else" on the same line that the previous closing curly brace (4 times in this file)

Please align with the current style.

@rufinio
Copy link
Contributor Author

rufinio commented Jun 1, 2018

A link to the tool page.

https://www.verifysoft.com/en_ctcpp.html

How to call / use the tool to generate a report for the cxx plugin?

Adapted from Testwell CTC++ User's Guid (Basic Use of CTC++)
Suppose we have the following five source files that form a complete program: prime.c io.c calc.c. This program can be compiled and linked in many ways, one way being the following
cl –Feprime.exe prime.c io.c calc.c
which results in the prime.exe program.
Now, we wish to apply CTC++ on our program, that is, we want to measure the files prime.c, io.c and calc.c and find out how thoroughly they were exercised in our test runs. First we need to instrument the files we wish to measure. Assume we wish to measure multicondition coverage. This can be done as follows:
ctc -i m cl –Feprime.exe prime.c io.c calc.c
As a result we get the instrumented prime.exe program. Here 'ctc' is the CTC++ Preprocessor utility, which makes the instrumentation on the given C and C++ source files and drives compiling/linking of the new instrumented target. The '-i m' command-line options to ctc mean "instrument for multicondition".
when ctc instruments source files, it maintains descriptions what the files contain (what interesting code there is to ctc, on what lines, etc.). This file is called symbolfile, and when it is not specified (like here) it will be MON.sym in current directory.
All right, now prime.exe is the instrumented version of the program. Let's do one test run.
After this test run we notice that the file MON.dat has born in the current directory (same directory as the symbolfile MON.sym was created to). It is a datafile, containing the collected execution counters when the code in the instrumented files was executed.
Now we wish to see the results of our test, i.e. what parts of the program the above run has executed. We use the CTC++ Postprocessor utility as follows:
ctcpost MON.sym MON.dat -p profile.txt
What we are asking here is that ctcpost takes the symbolfile MON.sym and the datafile MON.dat as input and produces an Execution Profile Listing to the file profile.txt.
To get the results of our test to sonarqube we add the follwoing line to configuration file sonar-project.properties:
sonar.cxx.coverage.reportPath=profile.txt

Wondering why you are parsing an ASCII report? Does the tool not provide XML?

Testwell CTC++ also provides a XML output as well as JSON output. I mainly use TXT and therefore I know better about it than XML.

Handle empty reports.

Tested by Test shouldConsumeEmptyReport

Support of sonar.cxx.errorRecoveryEnabled

I wonder if that is applicable for this extension: In my optinion invalid file references or line numbers has to be handled by calling layer (CxxCoverageSensor).
In case of errors in a report file: This extension can't deside between tolerant and strict mode. No data and corrupt data have the same effect that regex doesn't match and no coverage data is collected.

@guwirth
Copy link
Collaborator

guwirth commented Jun 2, 2018

@rufinio thx for your answer. Which version of the tool are you using / testing? What should we write in the docu? Supporting version x and later...

@guwirth
Copy link
Collaborator

guwirth commented Jun 2, 2018

@rufinio found also this page http://www.testwell.fi/ctcdesc.html. Is the tool from Testwell or Verisoft?

Copy link
Collaborator

@guwirth guwirth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Only minor findings.

*/
@Override
public void processReport(final SensorContext context, File report, final Map<String, CoverageMeasures> coverageData) {
LOG.debug("Parsing 'Testwell CTC++' format");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To make clear that ‚only‘ textual format is supported:
Parsing 'Testwell CTC++' textual format

@@ -67,6 +67,7 @@ public CxxCoverageSensor(CxxCoverageCache cache, CxxLanguage language, SensorCon
parsers.add(new CoberturaParser());
parsers.add(new BullseyeParser());
parsers.add(new VisualStudioParser());
parsers.add(new TestwellCtcParser());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe better TestwellCtcTxtParser?

@guwirth guwirth changed the title Add coverage sensor 'Testwell CTC++' Add coverage sensor 'Testwell CTC++' for textural reports Jun 2, 2018
@ivangalkin
Copy link
Contributor

@rufinio Technical feedback: I am not sure, if your parser fits into the existing framework. Please see the CxxCoverageSensor::processReports(): it iterates over all registered CoverageParsers and gives each of them a chance to process the coverage report. If parser doesn't recognize the format it must either throw an exception XMLStreamException/EmptyReportException or produce an empty map Map<String, CoverageMeasures> (please see CxxCoverageSensor::parseCoverageReport()). Not sure about the map emptiness, but your parser never throws as far as I see. That's suspicious. Please consider using EmptyReportException as a signal, that the file format was not recognized.

@guwirth w.r.t. #1465 and the generic coverage format, wouldn't it be more foresighted to create an external (?) converter CTC++ -> XML and import the coverage by means of sonar.coverageReportPaths?

@ivangalkin
Copy link
Contributor

one more side remark: TestwellCtcParser.java has common parts with https://github.com/Londran/sonar-ctc/blob/master/src/main/java/org/sonar/plugins/ctc/api/parser/CtcTextParser.java (among others some logging messages are 100% identical).

The original file is copyrighted

/*
 * Testwell CTC++ Plugin
 * Copyright (C) 2014 Verifysoft Technology GmbH
...
*/

Not sure if you can just skip the original copyright of LGPL3 code. Please double-check.

@guwirth
Copy link
Collaborator

guwirth commented Jun 3, 2018

Not sure if you can just skip the original copyright of LGPL3 code. Please double-check.

Source is LGPL and as long derivative work is again under license LGPL this should be no problem. But for later documentation it makes sense to add a comment below our header with a link to the original source.

@rufinio
Copy link
Contributor Author

rufinio commented Jun 4, 2018

@guwirth:

Which version of the tool are you using / testing? What should we write in the docu? Supporting version x and later...

Testwell CTC++

Prerequisites

Testwell CTC++ Version 7.2 and later.

1. Generate instrumented executable

Suppose we have the following source files that form a complete program: prime.c io.c calc.c. This program can be compiled and linked in many ways, one way being the following
cl –Feprime.exe prime.c io.c calc.c
or
gcc -o prime prime.c io.c calc.c
which results in the prime.exe / prime program.
Now, we wish to apply CTC++ on our program, that is, we want to measure the files prime.c, io.c and calc.c and find out how thoroughly they were exercised in our test runs. First we need to instrument the files we wish to measure. Assume we wish to measure multicondition coverage. This can be done as follows:
ctc -i m cl –Feprime.exe prime.c io.c calc.c
or
ctc -i m gcc -o prime prime.c io.c calc.c
As a result we get the instrumented prime.exe program. Here 'ctc' is the CTC++ Preprocessor utility, which makes the instrumentation on the given C and C++ source files and drives compiling/linking of the new instrumented target. The '-i m' command-line options to ctc mean "instrument for multicondition".
when ctc instruments source files, it maintains descriptions what the files contain (what interesting code there is to ctc, on what lines, etc.). This file is called symbolfile, and when it is not specified (like here) it will be MON.sym in current directory.
All right, now prime.exe is the instrumented version of the program. Let's do one test run.

2. Generate coverage report

After this test run we notice that the file MON.dat has born in the current directory (same directory as the symbolfile MON.sym was created to). It is a datafile, containing the collected execution counters when the code in the instrumented files was executed.
Now we wish to see the results of our test, i.e. what parts of the program the above run has executed. We use the CTC++ Postprocessor utility as follows:
ctcpost MON.sym MON.dat -p profile.txt
What we are asking here is that ctcpost takes the symbolfile MON.sym and the datafile MON.dat as input and produces an Execution Profile Listing to the file profile.txt.

3. Read coverage file with SonarQube

To get the results of our test to sonarqube we add the following line to configuration file sonar-project.properties:
sonar.cxx.coverage.reportPath=profile.txt

Testwell or Verifysoft

Verifysoft Technology GmbH (Offenburg/Germany) has acquired all intellectual property rights for the software test and analysis tools of Testwell (Finland).
The aquisition includes Testwell CTC++, CMT++/CMTJava und CTA++ as well as tools for the ADA programming language.
Source: https://www.verifysoft.com/en_verifysoft_acquires_testwell_ipr.html

@ivangalkin:
If scanner can't recognize file format, it's signaled by an empty map Map<String, CoverageMeasures>. You can check this behavior by running test shouldConsumeEmptyReport.

…port can have TXT, XML or JSON format. This parser is aligned to TXT format.
@guwirth guwirth requested a review from ivangalkin June 6, 2018 15:23
Copy link
Contributor

@ivangalkin ivangalkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some minor comments w.r.t. to the style etc

FILE_COND("^\\Q***TER\\E +\\d+ % \\( *(\\d+)/ *(\\d+)\\) of FILE (?:.*)$"),
FILE_STMT("^ {6} +\\d+ % \\( *(\\d+)/ *(\\d+)\\) statement.*$");

public static final Pattern REPORT_HEADER = Pattern.compile(MON_SYM.patternString + "\\s+" + MON_DAT.patternString
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here and below the joining can be simplified IMHO

String.join("\s+", MON_SYM.patternString, MON_DAT.patternString, LIS_DTE.patternString, COV_VIW.patternString);

public static final Pattern REPORT_FOOTER = Pattern.compile(SRC_FLS.patternString + "\\s+" + HDR_EXT.patternString + "\\s+"
+ FKT_EXT.patternString + "\\s*" + SRC_LNS.patternString, MULTILINE);
public static final Pattern FILE_HEADER = Pattern.compile(FILE_MONI.patternString + "\\s+" + FILE_INST.patternString, MULTILINE);
public static final Pattern SECTION_SEP = compile("^-{77}|={77}$", MULTILINE);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here and below: non-consistent compile() invocation: Pattern.compile() vs just compile()
I personally would prefer not to import like import static java.util.regex.Pattern.compile; but that's a matter of taste; the code just has to be consistent

private static final Logger LOG = Loggers.get(TestwellCtcTxtParser.class);

private Scanner scanner;
private Matcher matcher;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keeping Matcher object as a state doesn't look correct to me. I believe it wold be easier to read and to understand if Matcher will be created in scope of each function. Especially because of permanent calls like matcher.reset() or matcher.usePattern()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using matcher.reset() is better for performance at the cost of being less readable. In the case of a loop it is one less object to create in each loop iteration, and correspondingly one less object to be garbage collected per loop iteration.
After I moved matcher object in scope of each method I saw in my tests with report_big.txt about 5% decrease in performance. Hope that is compensated by the better readability.


private Scanner scanner;
private Matcher matcher;
private State state;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here: having a 'state' only makes the logic more complex IMHO

the main parsing loop is just something like that:

if ( parseReportHead() )
{
      while( parseUnit(coverageData) )
      {
      }
}

where each function returns true if parsing was successful

private void parseFileUnit(final Map<String, CoverageMeasures> coverageData) {
LOG.debug("Parsing file section...");

String normalFilename;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here and below: there is no need to declare variable in advance, it's not C89

int lineHitsFalse;
boolean conditionDetected;

lineIdPrev = 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here and below: please combine declaration and initialization (although '0' and 'false' are actually set by default, but maybe it is more readable)


private void addEachLine(CoverageMeasures coverageMeasures) {

int lineIdCur;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

int lineIdCur = Integer.parseInt(matcher.group(LINE_NR_GROUP)); belong into the do-loop

please double-check the following variable declaration; the scope of some of them must be reduced;
only loop-invariants or some states (accumulators) should be declared here

} else {
// multicondition

if (conditionDetected == true) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if (conditionDetected) {

}

private void setLinehits(CoverageMeasures coverageMeasures, int lineIdPrev, int lineIdCur, int lineHits) {
int lineIdNext;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please reduce the scope

    if (lineIdPrev > 0) {
      int lineIdNext = lineIdPrev + 1;
      while (lineIdNext < lineIdCur) {
....

private final MapSettings settings = new MapSettings();

@Before
public void setUp() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is very error-prone to share mutable state between test cases
maybe it's convenient (?) to declare ...

  • private CxxCoverageSensor sensor;
  • private DefaultFileSystem fs;
  • private SensorContextTester context;

... and permanently reset them in @Before...

but sharing of...

  • private final MapSettings settings = new MapSettings();

... is just wrong IMHO

private static MapSettings singletonMapSettings( String key, String value )
{
     MapSettings settings = new MapSettings();
     settings.set( key, value );
     return settings;
}

@Test
public void myWhateverTest() {
  MapSettings settings = singletonMapSettings( "some key", "some value );
}

P. S. I grepped in other tests for settings and these pattern seems to be common (which still doesn't make it correct). Feel free to ignore my comment.

@guwirth
Copy link
Collaborator

guwirth commented Jun 10, 2018

P. S. I grepped in other tests for settings and these pattern seems to be common (which still doesn't make it correct). Feel free to ignore my comment.

@ivangalkin maybe we should fix this in an extra refactoring task for all unit tests?

@guwirth
Copy link
Collaborator

guwirth commented Jun 12, 2018

@rufinio do you have time to fix the review comments from @ivangalkin?

Copy link
Collaborator

@guwirth guwirth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, ready to merge.

Copy link
Contributor

@ivangalkin ivangalkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for applying of the review comments. The current ones are less important: The one with try-with-resources or the scope of Matcher headerMatcher could be still improved IMHO, but the remaining comments are about style. You decide. Great job BTW

@@ -79,113 +71,88 @@ public TestwellCtcTxtParser() {
@Override
public void processReport(final SensorContext context, File report, final Map<String, CoverageMeasures> coverageData) {
LOG.debug("Parsing 'Testwell CTC++' textual format");


try {
this.scanner = new Scanner(report).useDelimiter(SECTION_SEP);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed this during the previous comment: in order to avoid this.scanner.close() calls (e. g. in case of exception), one could use so called "try with resources".

try ( this.scanner = new Scanner(report).useDelimiter(SECTION_SEP) )
{
    ...
}
catch ( ... )
{
}

Since Scanner is AutoCloseable it will be automatically closed when control flow lefts the try scope (see documentation)

} else {
state = State.END;
scanner.close();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

scanner.close() will be not necessary if try-with-resources will be used

while (!matcher.reset(scanner.next()).usePattern(FILE_RESULT).find()) {
parseLineSection(coverageMeasures);
String nextLine = scanner.next();
while (!FILE_RESULT.matcher(nextLine).find()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like a classic use-case for a for loop to me:

for (String nextLine = scanner.next(); FILE_RESULT.matcher(nextLine).find(); nextLine = scanner.next())
{
   parseLineSection(coverageMeasures, nextLine);
}

... but it's a matter of taste

filename = matcher.group(1);
if((new File(filename)).isAbsolute() == false) {
String filename = headerMatcher.group(1);
if (!(new File(filename)).isAbsolute()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parenthesis are not necessary if (!new File(filename).isAbsolute()) {

@guwirth
Copy link
Collaborator

guwirth commented Jun 14, 2018

@rufinio for me the code is ok. If you like to work-in the last points from @ivangalkin tell me.
Otherwise I would merge it now.

@rufinio
Copy link
Contributor Author

rufinio commented Jun 14, 2018

Yes, I'd like to apply these last points.

@guwirth
Copy link
Collaborator

guwirth commented Jun 14, 2018

@rufinio thanks a lot for your contribution!

@guwirth guwirth merged commit b28b697 into SonarOpenCommunity:master Jun 14, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

4 participants