Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal Report Viewer: host and use the the report viewer locally #1176

Merged
merged 55 commits into from
Feb 20, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
fd89367
Testing local report viewer
dfuchss Jul 5, 2023
739fa34
Update to Java JDK 20
dfuchss Jul 16, 2023
6807daf
Merge branch 'develop' into feature/internal-web-viewer
dfuchss Jul 16, 2023
a52b059
Merge branch 'develop' into feature/internal-web-viewer
tsaglam Sep 29, 2023
5d2c8df
Implemented routing for internal report viewer.
TwoOfTwelve Oct 12, 2023
7e2d1bd
Implemented server side for local mode.
TwoOfTwelve Oct 12, 2023
e27f632
Fixed 404 response codes for local mode.
TwoOfTwelve Oct 12, 2023
3085de0
frontend logic for local mode
Kr0nox Oct 12, 2023
0defe37
Removed accidental changes from package.json
TwoOfTwelve Oct 12, 2023
5a27848
remove unneeded dependencies
Kr0nox Oct 16, 2023
3a5b0be
Merge remote-tracking branch 'origin/develop' into feature/internal-w…
Kr0nox Oct 16, 2023
853ff7f
Changed local report viewer to return index html for every path, that…
TwoOfTwelve Oct 18, 2023
c794eff
Merge remote-tracking branch 'origin/feature/internal-web-viewer' int…
TwoOfTwelve Oct 18, 2023
4c8ea19
Fixed small error for local mode.
TwoOfTwelve Oct 18, 2023
30ac721
fix new tabs on comparison
Kr0nox Oct 21, 2023
288716f
Moved compiling the report-viewer into a profile.
TwoOfTwelve Oct 31, 2023
d65afab
Merge remote-tracking branch 'origin/feature/internal-web-viewer' int…
TwoOfTwelve Oct 31, 2023
e9f902d
fix tests
Kr0nox Nov 8, 2023
e36eb01
Merge remote-tracking branch 'origin/develop' into feature/internal-w…
Kr0nox Nov 30, 2023
4f055d0
fix report viewer build
Kr0nox Nov 30, 2023
daede42
fix new tabs
Kr0nox Nov 30, 2023
71d094c
Added javadoc to internal report viewer and moved the resource declar…
TwoOfTwelve Dec 1, 2023
a207694
Added logging for not existing resource url.
TwoOfTwelve Dec 5, 2023
dc58ad6
Merge branch 'develop' into feature/internal-web-viewer
dfuchss Dec 6, 2023
31b818b
Changed port to 1996.
TwoOfTwelve Dec 11, 2023
aadeae3
Merge remote-tracking branch 'origin/feature/internal-web-viewer' int…
TwoOfTwelve Dec 11, 2023
f9d32d2
Merge remote-tracking branch 'origin/develop' into feature/internal-w…
Kr0nox Dec 11, 2023
10a9453
Added port option for internal report viewer
TwoOfTwelve Dec 12, 2023
3c84512
Merge remote-tracking branch 'origin/feature/internal-web-viewer' int…
TwoOfTwelve Dec 12, 2023
e4fa97f
Merge remote-tracking branch 'origin/develop' into feature/internal-w…
dfuchss Dec 14, 2023
86f0716
Build report viewer only if needed.
dfuchss Dec 14, 2023
b69cd36
Merge branch 'develop' into feature/internal-web-viewer
TwoOfTwelve Jan 23, 2024
52bf30b
fix prettier
Kr0nox Jan 23, 2024
1dcc4b5
fix loading index html file
Kr0nox Jan 23, 2024
4b9c925
file name in tab
Kr0nox Jan 23, 2024
d4920db
fix anonymous propagation
Kr0nox Jan 23, 2024
7956543
Fixed some minor code style issues in the internal report viewer.
TwoOfTwelve Feb 6, 2024
ab6ae74
Merge branch 'develop' into feature/internal-web-viewer
TwoOfTwelve Feb 6, 2024
34c177b
Merge remote-tracking branch 'origin/feature/internal-web-viewer' int…
TwoOfTwelve Feb 6, 2024
ccaddce
add example zip name again
Kr0nox Feb 6, 2024
7b724bb
add node js link to docs
Kr0nox Feb 6, 2024
70a449f
Written tests for internal server.
TwoOfTwelve Feb 9, 2024
24fba7e
Merge remote-tracking branch 'origin/feature/internal-web-viewer' int…
TwoOfTwelve Feb 9, 2024
889cc05
Improved clarity for HttpRequestMethod
TwoOfTwelve Feb 9, 2024
0464d4f
Merge remote-tracking branch 'origin/develop' into feature/internal-w…
Kr0nox Feb 13, 2024
93b6655
Added port lookup.
TwoOfTwelve Feb 13, 2024
1f27130
spotless
TwoOfTwelve Feb 13, 2024
6fe98ad
Merge remote-tracking branch 'origin/develop' into feature/internal-w…
Kr0nox Feb 14, 2024
13d40b6
Clean up logging for internal server and remove unnecessary index.html
TwoOfTwelve Feb 14, 2024
c198274
Fixed error due to renaming of enum constant.
TwoOfTwelve Feb 14, 2024
c3fcf62
Added more tests for RoutingTree
TwoOfTwelve Feb 16, 2024
00d7d8c
Fixed log message if port is not available.
TwoOfTwelve Feb 16, 2024
cec3c6d
Sonarcloud issues
TwoOfTwelve Feb 20, 2024
54d8474
Merge remote-tracking branch 'origin/develop' into feature/internal-w…
TwoOfTwelve Feb 20, 2024
d4303bc
Code style and documentation
TwoOfTwelve Feb 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions .github/workflows/maven.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: Build

on:
push:
push:
paths:
- ".github/workflows/maven.yml"
- "**/pom.xml"
Expand All @@ -14,7 +14,7 @@ on:
- "**/pom.xml"
- "**.java"
- "**.g4"

# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:

Expand Down Expand Up @@ -43,17 +43,21 @@ jobs:
with:
java-version: 21
distribution: 'temurin'


- uses: actions/setup-node@v4
with:
node-version: "18"
Kr0nox marked this conversation as resolved.
Show resolved Hide resolved

- name: Run Tests
run: mvn verify -B -U

- name: Build Assembly
run: mvn clean package assembly:single
run: mvn -Pwith-report-viewer clean package assembly:single

- name: Upload Assembly
uses: actions/upload-artifact@v4
with:
name: "JPlag"
path: "jplag.cli/target/jplag-*-jar-with-dependencies.jar"


6 changes: 5 additions & 1 deletion .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,12 @@ jobs:
with:
java-version: '21'
distribution: 'temurin'
- uses: actions/setup-node@v4
with:
node-version: "18"
Kr0nox marked this conversation as resolved.
Show resolved Hide resolved

- name: Build JPlag
run: mvn -U -B clean package assembly:single
run: mvn -Pwith-report-viewer -U -B clean package assembly:single

- name: Attach CLI to Release on GitHub
uses: softprops/action-gh-release@v1
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/sonarcloud-branch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
path: ~/.m2
key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
restore-keys: ${{ runner.os }}-m2

- name: Build and analyze
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/sonarcloud-pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ jobs:
path: ~/.m2
key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
restore-keys: ${{ runner.os }}-m2

- name: Build and analyze (PR)
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/spotless.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ on:
- "**/pom.xml"
- "**.java"
- "**.g4"

# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:

Expand Down Expand Up @@ -43,9 +43,9 @@ jobs:
with:
java-version: 21
distribution: 'temurin'

- name: Check with Spotless
run: mvn clean spotless:check



3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,8 @@ JPlag is released on [Maven Central](https://search.maven.org/search?q=de.jplag)
1. Download or clone the code from this repository.
2. Run `mvn clean package` from the root of the repository to compile and build all submodules.
Run `mvn clean package assembly:single` instead if you need the full jar which includes all dependencies.
5. You will find the generated JARs in the subdirectory `cli/target`.
Run `mvn -P with-report-viewer clean package assembly:single` to build the full jar with the report viewer. In this case, you'll need [Node.js](https://nodejs.org/en/download) installed.
3. You will find the generated JARs in the subdirectory `cli/target`.

## Usage
JPlag can either be used via the CLI or directly via its Java API. For more information, see the [usage information in the wiki](https://github.com/jplag/JPlag/wiki/1.-How-to-Use-JPlag). If you are using the CLI, you can display your results via [jplag.github.io](https://jplag.github.io/JPlag/). No data will leave your computer!
Expand Down
53 changes: 53 additions & 0 deletions cli/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
<version>${revision}</version>
</parent>
<artifactId>cli</artifactId>

<dependencies>
<!-- IMPORTANT: For Coverage testing, you have to add dependencies to the coverage-report project ! -->

Expand Down Expand Up @@ -167,4 +168,56 @@
</plugin>
</plugins>
</build>

<profiles>
<profile>
<id>with-report-viewer</id>
<build>
<resources>
<resource>
<targetPath>report-viewer</targetPath>
<directory>../report-viewer/dist</directory>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.3.2</version>
<executions>
<execution>
<id>npm install</id>
<goals>
<goal>exec</goal>
</goals>
<phase>generate-resources</phase>
<configuration>
<executable>npm</executable>
<workingDirectory>../report-viewer</workingDirectory>
<arguments>
<argument>install</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>npm build</id>
<goals>
<goal>exec</goal>
</goals>
<phase>generate-resources</phase>
<configuration>
<executable>npm</executable>
<workingDirectory>../report-viewer</workingDirectory>
<arguments>
<argument>run</argument>
<argument>build</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
38 changes: 31 additions & 7 deletions cli/src/main/java/de/jplag/cli/CLI.java
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,11 @@
import static picocli.CommandLine.Model.UsageMessageSpec.SECTION_KEY_OPTION_LIST;
import static picocli.CommandLine.Model.UsageMessageSpec.SECTION_KEY_SYNOPSIS;

import java.awt.*;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.URI;
import java.security.SecureRandom;
import java.util.Arrays;
import java.util.HashSet;
Expand All @@ -23,6 +26,7 @@
import de.jplag.Language;
import de.jplag.cli.logger.CollectedLoggerFactory;
import de.jplag.cli.logger.TongfeiProgressBarProvider;
import de.jplag.cli.server.ReportViewer;
import de.jplag.clustering.ClusteringOptions;
import de.jplag.clustering.Preprocessing;
import de.jplag.exceptions.ExitException;
Expand Down Expand Up @@ -81,15 +85,14 @@ public static void main(String[] args) {
ParseResult parseResult = cli.parseOptions(args);

if (!parseResult.isUsageHelpRequested() && !(parseResult.subcommand() != null && parseResult.subcommand().isUsageHelpRequested())) {
JPlagOptions options = cli.buildOptionsFromArguments(parseResult);
ProgressBarLogger.setProgressBarProvider(new TongfeiProgressBarProvider());
JPlagResult result = JPlag.run(options);
ReportObjectFactory reportObjectFactory = new ReportObjectFactory(new File(cli.getResultFilePath()));
reportObjectFactory.createAndSaveReport(result);

OutputFileGenerator.generateCsvOutput(result, new File(cli.getResultFileBaseName()), cli.options);
switch (cli.options.mode) {
case RUN -> cli.runJPlag(parseResult);
case VIEW -> cli.runViewer(null);
case RUN_AND_VIEW -> cli.runViewer(cli.runJPlag(parseResult));
}
}
} catch (ExitException | FileNotFoundException exception) { // do not pass exceptions here to keep log clean
} catch (ExitException | IOException exception) { // do not pass exceptions here to keep log clean
if (exception.getCause() != null) {
logger.error("{} - {}", exception.getMessage(), exception.getCause().getMessage());
} else {
Expand Down Expand Up @@ -124,6 +127,27 @@ public CLI() {
this.commandLine.setAllowSubcommandsAsOptionParameters(true);
}

public File runJPlag(ParseResult parseResult) throws ExitException, FileNotFoundException {
JPlagOptions jplagOptions = buildOptionsFromArguments(parseResult);
JPlagResult result = JPlag.run(jplagOptions);
File target = new File(getResultFilePath());
ReportObjectFactory reportObjectFactory = new ReportObjectFactory(target);
reportObjectFactory.createAndSaveReport(result);
OutputFileGenerator.generateCsvOutput(result, new File(getResultFileBaseName()), this.options);
return target;
}

public void runViewer(File zipFile) throws IOException {
ReportViewer reportViewer = new ReportViewer(zipFile, this.options.advanced.port);
int port = reportViewer.start();
logger.info("ReportViewer started on port http://localhost:{}", port);
Desktop.getDesktop().browse(URI.create("http://localhost:" + port + "/"));

System.out.println("Press Enter key to exit...");
System.in.read();
reportViewer.stop();
}

private List<CommandSpec> buildSubcommands() {
return LanguageLoader.getAllAvailableLanguages().values().stream().map(language -> {
CommandSpec command = CommandSpec.create().name(language.getIdentifier());
Expand Down
51 changes: 29 additions & 22 deletions cli/src/main/java/de/jplag/cli/CliOptions.java
Original file line number Diff line number Diff line change
Expand Up @@ -20,47 +20,51 @@
public class CliOptions implements Runnable {
public static final Language defaultLanguage = new JavaLanguage();

@Parameters(paramLabel = "root-dirs", description = "Root-directory with submissions to check for plagiarism%n", split = ",")
@Parameters(paramLabel = "root-dirs", description = "Root-directory with submissions to check for plagiarism", split = ",")
public File[] rootDirectory = new File[0];

@Option(names = {"--new",
"-new"}, split = ",", description = "Root-directory with submissions to check for plagiarism (same as the root directory)%n")
"-new"}, split = ",", description = "Root-directory with submissions to check for plagiarism (same as the root directory)")
public File[] newDirectories = new File[0];

@Option(names = {"--old", "-old"}, split = ",", description = "Root-directory with prior submissions to compare against%n")
@Option(names = {"--old", "-old"}, split = ",", description = "Root-directory with prior submissions to compare against")
public File[] oldDirectories = new File[0];

@Option(names = {"--language",
"-l"}, arity = "1", converter = LanguageConverter.class, completionCandidates = LanguageCandidates.class, description = "Select the language to parse the submissions (default: ${DEFAULT-VALUE}). The language names are the same as the subcommands.%n")
"-l"}, arity = "1", converter = LanguageConverter.class, completionCandidates = LanguageCandidates.class, description = "Select the language to parse the submissions (default: ${DEFAULT-VALUE}). The language names are the same as the subcommands.")
public Language language = defaultLanguage;

@Option(names = {"-bc", "--bc",
"--base-code"}, description = "Path of the directory containing the base code (common framework used in all submissions)%n")
"--base-code"}, description = "Path of the directory containing the base code (common framework used in all submissions)")
public String baseCode;

@Option(names = {"-t", "--min-tokens"}, description = "Tunes the comparison sensitivity by adjusting the minimum token required to be counted "
+ "as a matching section. A smaller <n> increases the sensitivity but might lead to more " + "false-positives%n")
+ "as a matching section. A smaller <n> increases the sensitivity but might lead to more " + "false-positives")
public Integer minTokenMatch = null;

@Option(names = {"-h", "--help"}, usageHelp = true, description = "display this help and exit")
public boolean help;

@Option(names = {"-n",
"--shown-comparisons"}, description = "The maximum number of comparisons that will be shown in the generated report, if set "
+ "to -1 all comparisons will be shown (default: ${DEFAULT-VALUE})%n")
+ "to -1 all comparisons will be shown (default: ${DEFAULT-VALUE})")
public int shownComparisons = JPlagOptions.DEFAULT_SHOWN_COMPARISONS;

@Option(names = {"-r",
"--result-file"}, description = "Name of the file in which the comparison results will be stored (default: ${DEFAULT-VALUE}). Missing .zip endings will be automatically added.%n")
"--result-file"}, description = "Name of the file in which the comparison results will be stored (default: ${DEFAULT-VALUE}). Missing .zip endings will be automatically added.")
public String resultFile = "results";

@ArgGroup(heading = "Advanced%n", exclusive = false)
@Option(names = {
"--mode"}, description = "The mode of JPlag: either only run analysis, only open the viewer, or do both (default: ${DEFAULT_VALUE})")
public JPlagMode mode = JPlagMode.RUN;

@ArgGroup(heading = "%nAdvanced%n", exclusive = false)
public Advanced advanced = new Advanced();

@ArgGroup(validate = false, heading = "Clustering%n")
@ArgGroup(validate = false, heading = "%nClustering%n")
public Clustering clustering = new Clustering();

@ArgGroup(validate = false, heading = "Merging of neighboring matches to increase the similarity of concealed plagiarism:%n")
@ArgGroup(validate = false, heading = "%nMerging of neighboring matches to increase the similarity of concealed plagiarism:%n")
public Merging merging = new Merging();

@Option(names = {"--normalize"}, description = "Activate the normalization of tokens. Supported for languages: Java, C++.")
Expand All @@ -75,30 +79,33 @@ public void run() {
}

public static class Advanced {
@Option(names = {"-d", "--debug"}, description = "Debug parser. Non-parsable files will be stored (default: ${DEFAULT-VALUE})%n")
@Option(names = {"-d", "--debug"}, description = "Debug parser. Non-parsable files will be stored (default: ${DEFAULT-VALUE})")
public boolean debug;

@Option(names = {"-s", "--subdirectory"}, description = "Look in directories <root-dir>/*/<dir> for programs%n")
@Option(names = {"-s", "--subdirectory"}, description = "Look in directories <root-dir>/*/<dir> for programs")
public String subdirectory;

@Option(names = {"-p", "--suffixes"}, split = ",", description = "comma-separated list of all filename suffixes that are included%n")
@Option(names = {"-p", "--suffixes"}, split = ",", description = "comma-separated list of all filename suffixes that are included")
public String[] suffixes = new String[0];

@Option(names = {"-x",
"--exclusion-file"}, description = "All files named in this file will be ignored in the comparison (line-separated list)%n")
"--exclusion-file"}, description = "All files named in this file will be ignored in the comparison (line-separated list)")
public String exclusionFileName;

@Option(names = {"-m",
"--similarity-threshold"}, description = "Comparison similarity threshold [0.0-1.0]: All comparisons above this threshold will "
+ "be saved (default: ${DEFAULT-VALUE})%n")
+ "be saved (default: ${DEFAULT-VALUE})")
public double similarityThreshold = JPlagOptions.DEFAULT_SIMILARITY_THRESHOLD;

@Option(names = {"-P", "--port"}, description = "The port used for the internal report viewer.")
TwoOfTwelve marked this conversation as resolved.
Show resolved Hide resolved
public int port = 1996;

@Option(names = "--csv-export", description = "If present, a csv export will be generated in addition to the zip file.")
public boolean csvExport = false;
}

public static class Clustering {
@Option(names = {"--cluster-skip"}, description = "Skips the clustering (default: ${DEFAULT-VALUE})%n")
@Option(names = {"--cluster-skip"}, description = "Skips the clustering (default: ${DEFAULT-VALUE})")
public boolean disable;

@ArgGroup
Expand All @@ -109,25 +116,25 @@ public static class ClusteringEnabled {
"--cluster-algorithm"}, description = "Which clustering algorithm to use. Agglomerative merges similar submissions bottom up. "
+ "Spectral clustering is combined with Bayesian Optimization to execute the k-Means "
+ "clustering algorithm multiple times, hopefully finding a \"good\" clustering "
+ "automatically. (default: ${DEFAULT-VALUE})%n")
+ "automatically. (default: ${DEFAULT-VALUE})")
public ClusteringAlgorithm algorithm = new ClusteringOptions().algorithm();

@Option(names = {
"--cluster-metric"}, description = "The metric used for clustering. AVG is intersection over union, MAX can expose some "
+ "attempts of obfuscation. (default: ${DEFAULT-VALUE})%n")
+ "attempts of obfuscation. (default: ${DEFAULT-VALUE})")
public SimilarityMetric metric = new ClusteringOptions().similarityMetric();
}
}

public static class Merging {
@Option(names = {"--match-merging"}, description = "Enables match merging (default: ${DEFAULT-VALUE})%n")
@Option(names = {"--match-merging"}, description = "Enables match merging (default: ${DEFAULT-VALUE})")
public boolean enabled = MergingOptions.DEFAULT_ENABLED;

@Option(names = {"--neighbor-length"}, description = "Defines how short a match can be, to be considered (default: ${DEFAULT-VALUE})%n")
@Option(names = {"--neighbor-length"}, description = "Defines how short a match can be, to be considered (default: ${DEFAULT-VALUE})")
public int minimumNeighborLength = MergingOptions.DEFAULT_NEIGHBOR_LENGTH;

@Option(names = {
"--gap-size"}, description = "Defines how many token there can be between two neighboring matches (default: ${DEFAULT-VALUE})%n")
"--gap-size"}, description = "Defines how many token there can be between two neighboring matches (default: ${DEFAULT-VALUE})")
public int maximumGapSize = MergingOptions.DEFAULT_GAP_SIZE;

}
Expand Down
19 changes: 19 additions & 0 deletions cli/src/main/java/de/jplag/cli/JPlagMode.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
package de.jplag.cli;

/**
* The mode JPlag runs in. This influences which steps JPlag will execute.
*/
public enum JPlagMode {
TwoOfTwelve marked this conversation as resolved.
Show resolved Hide resolved
/**
* Only run JPlag and create a results.zip
*/
RUN,
/**
* Only start the report viewer
*/
VIEW,
/**
* Run JPlag and open the result in report viewer
*/
RUN_AND_VIEW
}
Loading
Loading