System Usability Scale Analysis Toolkit

The System Usability Scale (SUS) Analysis Toolkit is a web-based python application that provides a compilation of useful insights and contextualisation approaches based on findings from the scientific literature for the System Usability Scale questionnaire that was originally developed by John Brooke. It allows researchers and practisionaires to calculate comparative, iterative and benchmarking SUS usability study datasets. Furthermore, it provides utility to contextualize the meaning of calculated scores, compare them against scores gathered in meta-analyses, calculate SUS scores conclusiveness and analysing the contribution of specific questions of the 10-item questionnaire to the SUS study scores.

An in-depth explanation of the toolkits features and their scientific basis can be found in our conference publication about the initial release and preliminary evaluation of the toolkit. It is also included in the repository: "A Web-Based Analysis Toolkit for the System Usability Scale" - Blattgerste et al. (2022). More exemplary use cases and visualizations are shown on the Mixality Lab homepage of the University of Applied Sciences Emden/Leer, where the tool was originally developed. Since then, the tool has developed into a more comprehensive solution, which can be found at https://sus.tools/.

Preview: Multi-Variable SUS Analysis

Preview: Single-Variable Usability Benchmarking

Using the SUS Analysis Toolkit

Online versions of the toolkit are hosted at: https://sus.mixality.de/ and https://analysis.sus.tools/

Quickstart - Running the SUS Analysis Toolkit locally

Run it locally using python

1. Install Python 3.9. On Windows, make sure to check "Add Pythong 3.9 to PATH" during installation
2. Clone this repository, or manually download it as a zip file and extract it
3. Open the sus-analysis-toolkit folder
4. Open the terminal (E.g. Windows Terminal) in the sus-analysis-toolkit folder
5. Run the following commands to automatically install all dependencies:

python -m pip install --upgrade pip setuptools

python -m pip install -r requirements.txt

5. Alternatively, install the requirements manually by installing all packages specified in the requirements.txt:

plotly==5.3.1
dash==2.0.0
numpy==1.19.5
pandas==1.3.1
kaleido==0.2.1

6. After successful installation, run the toolkit locally with the command:

python dashApp.py

7. Copy the displayed address the toolkit is running on into the web browser to access the toolkit:

8. To stop the toolkits local server, close the terminal window

Run it locally using Docker

You can also download the latest Docker container and run it locally, using e.g. Docker Desktop.

Running the SUS Analysis Toolkit on a server

To run the SUS Analysis Toolkit on your own server, we recommend hosting the latest Docker container provided under Packages in our repository. You can also build a Docker Image and host the Docker Container yourself. The required Dockerfile is included in this repository.

Contributing to this project

As the SUS Analysis Toolkit is an ongoing project, we are happy to receive feedback, suggestions and bug reports through Email or GitHub Issues.

Additionally, we are planning to develop the tool further towards including:

• item-Level benchmarks for each of the 10 questions of the SUS questionnaire according to Lewis & Sauro 2018. This would allow to compare the achieved average of individual items of the SUS study scores to item-level benchmarks calculted from linear regressions based on comparable SUS study scores, the average SUS study score of 68 or the industry benchmark of 80.
• more statistical data analysis tools, ranging from simple t-test and variance analysis to more advanced statistical decision helpers
• automated SUS data processing (E.g. feeding SUS questionnaire data through APIs or directly interpreting filled out questionnaires from the SUS PDF Generator)
• improving its accessibility and usability.

Therefore, we are happy to collaborate with developers and experts in these areas. If you are interested in working on and contributing towards one of the topics, feel free to get in touch.

Acknowledgement

The tool is freely accesible for commercial and non-commercial use under the MIT license and does not require acknowledgement. Nonetheless, if you use our tool for scientific publications or presentations, we would appreciate an acknowledgement in form of a citation to our tool:

@inproceedings{10.1145/3529190.3529216,
author = {Blattgerste, Jonas and Behrends, Jan and Pfeiffer, Thies},
title = {A Web-Based Analysis Toolkit for the System Usability Scale},
year = {2022},
isbn = {9781450396318},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3529190.3529216},
doi = {10.1145/3529190.3529216},
pages = {237–246},
numpages = {10},
location = {Corfu, Greece},
series = {PETRA '22}
}