From 119a4861c3ca564f3dc5af5dfb7d4743448be657 Mon Sep 17 00:00:00 2001 From: Alexander Vogt Date: Thu, 12 Sep 2024 10:48:24 +0200 Subject: [PATCH 1/4] first draft of e2e test readme --- report-viewer/tests/e2e/README.md | 62 +++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 report-viewer/tests/e2e/README.md diff --git a/report-viewer/tests/e2e/README.md b/report-viewer/tests/e2e/README.md new file mode 100644 index 000000000..3b027a556 --- /dev/null +++ b/report-viewer/tests/e2e/README.md @@ -0,0 +1,62 @@ +# Complete System e2e Tests + +The e2e tests are executed by the [complete e2e tests workflow](../../../.github/workflows/complete-e2e.yml) and are meant to check the entire process from building and executing JPlag to viewing the report in the report viewer. +The tests get run on 3 different operating systems: Windows, Ubuntu and MacOS. + +## Structure + +First in the `build_jar` job the JPlag jar is built. + +Then in the `run_jplag` job the JPlag jar is executed with the test data. Here using matrix, JPlag is run for each dataset on each operating system. + +Finally in the `e2e_test` job the playwright tests specified here are run. The tests are run on each operating system. + +### Open Comparison Tests + +The `OpenComparisonTest.spec.ts` test, loads each of the specified reports and tries to open its top comparison. This is done to ensure that basic functionality of JPlag is working. +We test that the most used languages are working correctly and that the report viewer does not throw an error opening them. We also test that all ways to give files as single files or folders are exported into their respective reports. + +### Other Tests + +The other tests are testing the functionality of the report viewer. Each of them tests one view and makes sure all the features are working correctly. + +## Running the tests locally + +1) To run the tests locally get the zips of the datasets from the [GitHub](../../../.github/workflows/files/) and execute JPlag on its contents. +2) Build the report viewer using `npm run build` +3) Run the e2e tests using `npm run test:e2e` + +## Adding new tests + +If you want to add new tests we suggest doing the following tests: + +1) Create a dataset you can upload to GitHub + - Copy the zip of the dataset into [the workflow files folder](../../../.github/workflows/files/) + - Execute it on your device, so you can test your new test locally + - If you want to add the dataset to `OpenComparisonTest.spec.ts` make sure there is a clear top comparison and you do not have multiple comparisons with the same percentage as the top comparison +2) Add the test to the matrix in the [complete e2e tests workflow](../../../.github/workflows/complete-e2e.yml) + - zip: The name of the zip file in the files folder + - name: The name of the dataset. This name should be unique + - folder: This is the main folder of the dataset, that gets passed to JPlag as a positional argument + - language: The language JPlag should use. This should be the same name passed to the `-l` parameter + - cliArgs: Additional arguments to pass to JPlag. This could be used to specify basecode or give JPlag more folders over `--new`/`--old` + +3) Add the test to the playwright e2e tests. + - Adding a test to `OpenComparisonTest`: + - Add the dataset name to the `datasets` array + - Specify the name of the zip that should be opened. They follow the pattern `DATASET_NAME-report.zip` + - Specify the names of the submissions of the top comparisons. These are given as regexes + - Adding a completly new Test: + - Create a new file in this folder with the file ending `.spec.ts` + - Add a new test according to the playwright documentation (examples are in the other tests) + - The test should start like this + ```typescript + test('Name of the test', async ({ page }) => { + await page.goto('/') + await uploadFile('YOUR_DATASET_NAME-report.zip', page) + // Your test code + }); + ``` + This will start you on the overview page with the dataset loaded + +4) Run the tests locally to make sure they are working \ No newline at end of file From 87554d005f684dbb817e39d275f44d7457a8c507 Mon Sep 17 00:00:00 2001 From: Alexander Vogt Date: Thu, 12 Sep 2024 10:50:47 +0200 Subject: [PATCH 2/4] add complete e2e tests to wiki --- docs/5.-End-to-End-Testing.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/5.-End-to-End-Testing.md b/docs/5.-End-to-End-Testing.md index be2cbd59d..db9080e80 100644 --- a/docs/5.-End-to-End-Testing.md +++ b/docs/5.-End-to-End-Testing.md @@ -1 +1,8 @@ +JPlag has two different types of end-to-end tests. + +# Maven end-to-end-tests +There is a module inside the maven project, that runs e2e tests on the core of Jplag. The basic structure of the end-to-end testing module is discussed in the [corresponding readme file](https://github.com/jplag/JPlag/blob/master/endtoend-testing/README.md). + +# Complete end-to-end-tests +The complete end-to-end tests are executed by the [complete e2e tests workflow](../../../.github/workflows/complete-e2e.yml) and are meant to check the entire process from building and executing JPlag to viewing the report in the report viewer. Details are specified in the [corresponding readme file](../report-viewer/tests/e2e/README.md). \ No newline at end of file From 8406624dd8edad2f9b66406dc06b83b8a0891c23 Mon Sep 17 00:00:00 2001 From: Alex | Kronox <39801116+Kr0nox@users.noreply.github.com> Date: Thu, 12 Sep 2024 10:53:09 +0200 Subject: [PATCH 3/4] Fix indentation of bullet points --- report-viewer/tests/e2e/README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/report-viewer/tests/e2e/README.md b/report-viewer/tests/e2e/README.md index 3b027a556..c2338de5e 100644 --- a/report-viewer/tests/e2e/README.md +++ b/report-viewer/tests/e2e/README.md @@ -35,11 +35,11 @@ If you want to add new tests we suggest doing the following tests: - Execute it on your device, so you can test your new test locally - If you want to add the dataset to `OpenComparisonTest.spec.ts` make sure there is a clear top comparison and you do not have multiple comparisons with the same percentage as the top comparison 2) Add the test to the matrix in the [complete e2e tests workflow](../../../.github/workflows/complete-e2e.yml) - - zip: The name of the zip file in the files folder - - name: The name of the dataset. This name should be unique - - folder: This is the main folder of the dataset, that gets passed to JPlag as a positional argument - - language: The language JPlag should use. This should be the same name passed to the `-l` parameter - - cliArgs: Additional arguments to pass to JPlag. This could be used to specify basecode or give JPlag more folders over `--new`/`--old` + - zip: The name of the zip file in the files folder + - name: The name of the dataset. This name should be unique + - folder: This is the main folder of the dataset, that gets passed to JPlag as a positional argument + - language: The language JPlag should use. This should be the same name passed to the `-l` parameter + - cliArgs: Additional arguments to pass to JPlag. This could be used to specify basecode or give JPlag more folders over `--new`/`--old` 3) Add the test to the playwright e2e tests. - Adding a test to `OpenComparisonTest`: @@ -59,4 +59,4 @@ If you want to add new tests we suggest doing the following tests: ``` This will start you on the overview page with the dataset loaded -4) Run the tests locally to make sure they are working \ No newline at end of file +4) Run the tests locally to make sure they are working From 5de139c230461dd5a2beb369fb314e7564603a6d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Timur=20Sa=C4=9Flam?= Date: Fri, 13 Sep 2024 15:49:52 +0200 Subject: [PATCH 4/4] Update report-viewer/tests/e2e/README.md --- report-viewer/tests/e2e/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/report-viewer/tests/e2e/README.md b/report-viewer/tests/e2e/README.md index c2338de5e..c5e07d590 100644 --- a/report-viewer/tests/e2e/README.md +++ b/report-viewer/tests/e2e/README.md @@ -46,7 +46,7 @@ If you want to add new tests we suggest doing the following tests: - Add the dataset name to the `datasets` array - Specify the name of the zip that should be opened. They follow the pattern `DATASET_NAME-report.zip` - Specify the names of the submissions of the top comparisons. These are given as regexes - - Adding a completly new Test: + - Adding a completely new Test: - Create a new file in this folder with the file ending `.spec.ts` - Add a new test according to the playwright documentation (examples are in the other tests) - The test should start like this