Update remaining directories

README.md

@@ -2,29 +2,31 @@
 This repository contains the replication package for the application of Cling (automated class integration testing tool) and EvoSuite (automated unit testing tool) on 140 pairs of classes from five different open-source projects.
 
 This document contains the following instructions:
+- [Cling Application](#cling-application)
 - [Docker image](#docker-image)
 - [Run case filtering](#run-case-filtering)
 - [Run test generation tools](#run-test-generation-tools)
-  * [Prepare input CSV file](#prepare-input-csv-file)
-  * [Run test generation script](#run-test-generation-script)
-  * [Output](#output)
-- [RQ1: CBC Coverage](#rq1--cbc-coverage)
-  * [Collect CBC Coverage](#collect-cbc-coverage)
-  * [Data Analysis](#data-analysis)
-- [RQ2: Mutation Score](#rq2--mutation-score)
-  * [Run PIT](#run-pit)
-  * [Collect Mutation Scores](#collect-mutation-scores)
-  * [Collect the mutants Killed  By Cling](#collect-the-mutants-killed--by-cling)
-  * [Data Analysis](#data-analysis-1)
-- [RQ3: Captured Faults](#rq3--captured-faults)
-  * [Collect Captured Stack Traces](#collect-captured-stack-traces)
-  * [Collect Interesting Stack Traces](#collect-interesting-stack-traces)
-  * [Manual Analysis](#manual-analysis)
+  - [Prepare input CSV file](#prepare-input-csv-file)
+  - [Run test generation script](#run-test-generation-script)
+  - [Output](#output)
+- [RQ1: CBC Coverage](#rq1-cbc-coverage)
+  - [Collect CBC Coverage](#collect-cbc-coverage)
+  - [Data Analysis](#data-analysis)
+- [RQ2: Mutation Score and line coverage](#rq2-mutation-score-and-line-coverage)
+  - [Run PIT](#run-pit)
+  - [Collect Line coverages](#collect-line-coverages)
+  - [Collect Mutation Scores](#collect-mutation-scores)
+  - [Collect the mutants Killed  By Cling](#collect-the-mutants-killed--by-cling)
+  - [Data Analysis](#data-analysis-1)
+- [RQ3: Captured Faults](#rq3-captured-faults)
+  - [Collect Captured Stack Traces](#collect-captured-stack-traces)
+  - [Collect Interesting Stack Traces](#collect-interesting-stack-traces)
+  - [Manual Analysis](#manual-analysis)
 
 # Docker image
 Docker image of this replication package is available in [DockerHub](https://hub.docker.com/repository/docker/pderakhshanfar/cling-application).
 
-Also, you can build and image by running `build-docker` bash file:
+Also, you can build and image by running `build-image` bash file:
 
 ```
 . scripts/docker/build-image.sh
@@ -61,7 +63,9 @@ Each row in this CSV file contains a `caller_class` and `calee_class` (one of th
 The rows in this CSV file is categorized by their value for tool column:
 * `cling` executes class integration testing for `caller_class` and `callee_class` for 5 minutes.
 * `evosuite-caller5` executes EvoSuite (unit testing) on `caller_class` for 5 minutes.
-* `evosuite-callee5`  executes EvoSuite (unit testing) on `callee_class` for 5 minutes.
+* `evosuite-callee5` executes EvoSuite (unit testing) on `callee_class` for 5 minutes.
+* `randoop-caller5` executes Randoop (unit testing) on `caller_class` for 5 minutes.
+* `randoop-callee5` executes Randoop (unit testing) on `callee_class` for 5 minutes.
 
 
 To generate this CSV file, you need to execute `generate.py` python file:
@@ -78,7 +82,7 @@ To change the number of execution times, you need to change this python file (li
 
 ## Run test generation script
 
-Run `test-geeration.sh` bash file with one input parameter, which is **number of parallel processes**:
+Run `test-generation.sh` bash file with one input parameter, which is **number of parallel processes**:
 
 ```
 . scripts/main/test-generation.sh <number of parallel processes>
@@ -126,7 +130,7 @@ Rscript data_analysis/r-scripts/cbc-coverage.r
 The output is two boxplots indicating the CBC coverage of EvoSuite and Cling. These boxplots are saved as `data_analysis/figures/cbc-total.pdf` and `data_analysis/figures/cbc-per-project.pdf`.
 
 
-# RQ2: Mutation Score
+# RQ2: Mutation Score and line coverage
 ## Run PIT
 To calculate the mutation score of the generated test cases by Cling and EvoSuite, we need to execute PIT. to run PIT on all of the test cases, generated in this study, run the following bash file:
 
@@ -140,6 +144,17 @@ docker exec -it cling-application-container bash scripts/main/rq2-mutation-score
 ```
 The output of the PIT tool will be saved in the `data/rq2/pit/` directory. Also, the execution logs of PIT runs will be stored in the `logs/pit/` directory.
 
+## Collect Line coverages
+To collect all of the line coverages achieved by each generated test case, run the following python script:
+
+```
+python scripts/python/line-coverage/collect-line-coverages.py
+```
+**docker**
+```
+docker exec -it cling-application-container python scripts/python/line-coverage/collect-line-coverages.py
+```
+
 ## Collect Mutation Scores
 To collect all of the mutation scores reported in PIT outputs, run the following python script:
 
@@ -167,19 +182,15 @@ To collect all of this information, run the following bash script:
 ```
 docker exec -it cling-application-container bash scripts/main/rq2-killed-mutants.sh 
 ```
-This script creates the following CSV files in `data/rq2`:
-
-* **cling-evoe-evor-final:** List of mutants killed only by Cling
-* **cling-evoe:** List of mutants killed by Cling but now by TE
-* **cling-evor:** List of mutants killed by Cling but now by TR
-* **killedM-C-E-R:** Number of mutants killed only by Cling on each case
-* **killedM-C-E:** Number of mutants killed by Cling but now by TE on each case
-* **killedM-C-R:** Number of mutants killed by Cling but now by TR on each case
-* **final-mutants-table:** A large CSV file containing the mutation score improvements achieved by Cling
+This script creates multiple CSV files in `data/rq2`. Each CSV file indicates the mutants which are killed by cling but not with the other tools and congifurations.
 
 
 ## Data Analysis
 After generating all of these CSV files, you can run the following R scripts to create tables and figures reported in the paper.
+For line coverage:
+```
+Rscript data_analysis/r-scripts/line-coverage.r 
+```
 
 For mutation scores:
 ```

dockerfile

@@ -1,7 +1,7 @@
 ARG experiment
 
 
-FROM	  	  fedora:latest
+FROM	  	  fedora:30
 
 RUN         dnf upgrade --assumeyes
 RUN         dnf install findutils --assumeyes
@@ -16,12 +16,10 @@ RUN         dnf install procps --assumeyes
 
 WORKDIR /experiment
 #
-COPY logs /experiment/logs
+# COPY logs /experiment/logs
 # COPY results /experiment/results
 COPY defects4j /experiment/defects4j
 # COPY data /experiment/data
 COPY projects /experiment/projects
-COPY scripts /experiment/scripts
-COPY tools /experiment/tools
+# COPY tools /experiment/tools
 COPY libs /experiment/libs
-COPY consoleLog /experiment/consoleLog

scripts/cbc/cbc-coverage.sh

@@ -1,5 +1,5 @@
 # Cling CBC Coverage
-python scripts/python/cbc/cling-cbc.py
+# python scripts/python/cbc/cling-cbc.py
 
 # EvoSuite CB Coverage
 # Check input CSV file
@@ -22,48 +22,78 @@ do
         continue
     fi
 
-
     if [[ "$tool" == "cling" ]]; then
         continue
         # echo "Applying CBC calculator on $resultDir"
     elif [[ "$tool" == "evosuite-caller5" ]]; then
         resultDir="results/evosuite5/$project-$caller_class-$execution_id"
         # echo "Applying CBC calculator on $resultDir"
+
+    elif [[ "$tool" == "randoop-caller5" ]]; then
+        resultDir="results/randoop5/$project-$caller_class-$execution_id"
+        # echo $resultDir
     else
-        # CBC coverage is only for Cling and evosuite on caller class
+        # CBC coverage is only for Cling and evosuite or Randoop on caller class
         continue
     fi
 
     if [[ -d $resultDir ]]; then
+
+        exists=$(python scripts/python/cbc/is-data-available.py $execution_id $project $caller_class $callee_class $tool)
+        if [[ "$exists" == "1" ]]; then
+            echo "CBC data is already available: tool: $tool, CallerClass: $caller_class, CalleeClass: $callee_class, executionId: $execution_id"
+            continue
+        fi
+
         # Prepare CP entries
         CPEntriesDir="projects/$project/cp-entries.txt"
         CPEntriesContent=$( cat $CPEntriesDir)
         preparedCPs=$( python scripts/python/reassemble-cps.py $CPEntriesContent "$project")
         classPaths="$preparedCPs:$(cat libs/test_execution/classpath.txt)$resultDir"
+        if [[ "$tool" == "evosuite-caller5" ]]; then
+            # Prepare testSuite name
+            for mainTest in `find $resultDir -name "*_ESTest.java" -type f`; do
+                filename=$(basename -- "$mainTest")
+                filename="${filename%.*}"
+                # echo "Filename: "$filename
+                callerClassName=$caller_class"_ESTest"
+                callerClassName="${callerClassName##*.}"
+                # echo "Caller class name: "$callerClassName
+                if [ "$filename" = "$callerClassName" ]; then
+                    testClass=$caller_class"_ESTest"
+                else
+                    testClass=$callee_class"_ESTest"
+                fi
+            done
+            echo "Test case is $testClass"
 
-        # Prepare testSuite name
-        for mainTest in `find $resultDir -name "*_ESTest.java" -type f`; do
-            filename=$(basename -- "$mainTest")
-            filename="${filename%.*}"
-            # echo "Filename: "$filename
-            callerClassName=$caller_class"_ESTest"
-            callerClassName="${callerClassName##*.}"
-            # echo "Caller class name: "$callerClassName
-            if [ "$filename" = "$callerClassName" ]; then
-                testClass=$caller_class"_ESTest"
+            echo "Run EvoSuite CBC coverage. testClass: $testClass, CallerClass: $caller_class, CalleeClass: $callee_class, executionId: $execution_id"
+            java -Djavax.accessibility.assistive_technologies=" " -jar tools/cbc.jar -project_cp "$classPaths" -test_suite "$testClass" -caller "$caller_class" -callee "$callee_class" > "logs/cbc/$tool-$project-$caller_class-$callee_class-$execution_id-out.txt" 2> "logs/cbc/$tool-$project-$caller_class-$callee_class-$execution_id-err.txt" &
+        else
+            # Randoop
+            if [[ -f "$resultDir/RegressionTest.java" ]]; then
+                mainRegresstionTest="RegressionTest"
             else
-                testClass=$callee_class"_ESTest"
+                mainRegresstionTest=""
             fi
-        done
 
-        echo "Test case is $testClass"
+            if [[ -f "$resultDir/ErrorTest.java" ]]; then
+                mainErrorTest="ErrorTest"
+            else
+                mainErrorTest=""
+            fi 
+
+            echo "Run Randoop CBC coverage. testClass: "$mainRegresstionTest":"$mainErrorTest", CallerClass: $caller_class, CalleeClass: $callee_class, executionId: $execution_id"
+            java -Djavax.accessibility.assistive_technologies=" " -jar tools/cbc.jar -project_cp "$classPaths" -test_suite "$mainRegresstionTest:$mainErrorTest" -caller "$caller_class" -callee "$callee_class" > "logs/cbc/$tool-$project-$caller_class-$callee_class-$execution_id-out.txt" 2> "logs/cbc/$tool-regression-$project-$caller_class-$callee_class-$execution_id-err.txt" &
+
+        fi
 
-        echo "Run EvoSuite CBC coverage. testClass: $testClass, CallerClass: $caller_class, CalleeClass: $callee_class, executionId: $execution_id"
-        java -Djavax.accessibility.assistive_technologies=" " -jar tools/cbc.jar -project_cp "$classPaths" -test_suite "$testClass" -caller "$caller_class" -callee "$callee_class" > "logs/cbc/$tool-$project-$caller_class-$callee_class-$execution_id-out.txt" 2> "logs/cbc/$tool-$project-$caller_class-$callee_class-$execution_id-err.txt" &
+
         pid=$!
         . scripts/cbc/observer.sh $pid $execution_id $project $caller_class $callee_class $tool &
     else
         echo "XXXXXXXX $resultDir"
+        python scripts/python/cbc/write_on_csv_file.py $execution_id $project $caller_class $callee_class "" $tool &
     fi
 
     while (( $(pgrep -l java | wc -l) >= $LIMIT ))

scripts/cbc/observer.sh

@@ -9,7 +9,16 @@ filename="logs/cbc/$tool-$project-$caller_class-$callee_class-$execution_id-out.
 
 
 while kill -0 "$pid"; do
-            sleep 1
+            sleep 30
+            modifiedTime=$(date -r "$filename" "+%s")
+            currentTime=$(date "+%s")
+            inActiveTime=$((currentTime-modifiedTime))
+            echo "Process $pid is inactive for $inActiveTime seconds"
+            if [[ "$inActiveTime" -gt "120" ]]; then
+                kill -9 "$pid"
+                echo "killing process $pid"
+            fi
+
 done
 
 python scripts/python/cbc/write_on_csv_file.py $execution_id $project $caller_class $callee_class $filename $tool &

scripts/docker/remove-image.sh

@@ -3,4 +3,4 @@ docker stop cling-application-container
 # Remove old containers
 docker rm cling-application-container
 # Remove previous docker image
-docker rmi cling-application
+# docker rmi cling-application

scripts/docker/run-container.sh

@@ -5,10 +5,11 @@ docker run -dit --name cling-application-container  \
 --mount type=bind,source="$(pwd)/results",target=/experiment/results \
 --mount type=bind,source="$(pwd)/data",target=/experiment/data \
 --mount type=bind,source="$(pwd)/subject_generator",target=/experiment/subject_generator \
+--mount type=bind,source="$(pwd)/scripts",target=/experiment/scripts \
+--mount type=bind,source="$(pwd)/tools",target=/experiment/tools \
 cling-application
 
 
 # --mount type=bind,source="$(pwd)/defects4j",target=/experiment/defects4j \
 # --mount type=bind,source="$(pwd)/projects",target=/experiment/projects \
-# --mount type=bind,source="$(pwd)/scripts",target=/experiment/scripts \
-# --mount type=bind,source="$(pwd)/tools",target=/experiment/tools \
+

scripts/main/collect-line-coverages.sh

@@ -0,0 +1,5 @@
+# Collect analyze the covered lines in each case for each given tool
+# Output: 1- a csv file containing the line coverages for each tool
+#         2- a CSV file containing: Tool, Caller, Callee, line number, Covered (boolean value that 1 indicates the successful coverage)
+
+python scripts/python/line-coverage/collect-line-coverages.py

scripts/main/rq1-cbc-coverage.sh

@@ -7,4 +7,8 @@ echo "compiling the generated tests"
 . scripts/test/compile_tests.sh
 
 echo "Runing CBC analyzer"
-. scripts/cbc/cbc-coverage.sh $LIMIT
+. scripts/cbc/cbc-coverage.sh $LIMIT
+
+# Replace -1 values from cbc-coverage csv file to their main value according to the other tools' results.
+
+python scripts/python/cbc/correct_values.py

scripts/main/rq2-killed-mutants.sh

@@ -4,15 +4,28 @@ python scripts/python/mutation/find-mutants-killed-by-evosuite.py
 # Detect mutants killed by Cling
 python scripts/python/mutation/find-mutants-killed-by-cling.py
 
+# Detect mutants killed by Randoop
+python scripts/python/mutation/find-mutants-killed-by-randoop.py
+
 # Detect mutants killed by Cling but not by TE
 python scripts/python/mutation/cling-evoe.py
 
 # Detect mutants killed by Cling but not by TR
 python scripts/python/mutation/cling-evor.py
 
-# Detect mutants killed only by Cling
+# Detect mutants killed only by Cling but not with EvoSuite
 python scripts/python/mutation/cling-evoe-evor.py
 
+# Detect mutants killed by Cling but not by RandoopE
+python scripts/python/mutation/cling-randoope.py
+
+# Detect mutants killed by Cling but not by RandoopR
+python scripts/python/mutation/cling-randoopr.py
+
+# Detect mutants killed only by Cling but not with Randoop
+python scripts/python/mutation/cling-randoop.py
+
+
 # Save the final csv file which contains the information about the killed mutants by Cling
 Rscript data_analysis/r-scripts/killedM.r 
 python scripts/python/mutation/generate-final-mutants-table.py

scripts/main/rq2-mutation-score.sh

@@ -1,19 +1,28 @@
 
 LIMIT=$1
+toolTyoe=$2
 
-echo "Set class separator to false"
-. scripts/test/prepare_tests.sh
-
-# compile the generated tests
-echo "compiling the generated tests"
-. scripts/test/compile_tests.sh
+# if [[ "$toolTyoe" != "randoop" ]]; then
+#     echo "Set class separator to false"
+#     . scripts/test/prepare_tests.sh
+#     # compile the generated tests
+#     echo "compiling the generated tests"
+#     . scripts/test/compile_tests.sh
+# fi
 
 # detect and remove flaky tests from the generated test suites
-echo "Detecting the flaky tests"
-. scripts/test/detect_flaky_tests.sh
+# echo "Detecting the flaky tests"
+# . scripts/test/detect_flaky_tests.sh
+
+
+# if [[ "$toolTyoe" != "randoop" ]]; then
+#     echo "Re-compile the tests"
+#     . scripts/test/compile_tests.sh
+# else
+#     echo "Re-compile the modified tests"
+#     . scripts/test/compile_modified_randoop_tests.sh
+# fi
 
-echo "Re-compile the modified tests"
-. scripts/test/compile_tests.sh
 
 echo "Running pitest"
 . scripts/mutation/mutation-score-calculator.sh $LIMIT