Skip to content

Commit

Permalink
Merge pull request #290 from ryanohoro/test_tastes
Browse files Browse the repository at this point in the history 
Tests For Mime/Yara Tastes and Scanner Assignments
  • Loading branch information
@phutelmyer
phutelmyer authored Jan 20, 2023
2 parents 321284d + d7b1472 commit 333ce89
Show file tree
Hide file tree
Showing 17 changed files with 904 additions and 440 deletions.
51 changes: 41 additions & 10 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,22 +26,53 @@ We rely on contributors to test any changes before they are submitted as pull re

New scanners should be accompanied by a [pytest](https://docs.pytest.org/) based test in `src/python/strelka/tests`, along with **non-malicous** and reasonably sized sample files in `src/python/strelka/tests/fixtures`.

pytest is run when the docker container is built to assure scanners will work at runtime.
New fixtures should also be accompanied by updates to the configuration tests in `tests_configuration`. Changes to tastes or scanner assignments will require updates to these tests.

Run pytest manually:
The best way to run Strelka's test suite is to build the docker containers. Some of Strelka's scanners have OS level dependencies which make them unsuitable for individual testing.

```
docker-compose -f build/docker-compose.yaml build
```bash
cd src/python/strelka/
python -m pytest tests/
============================= test session starts ==============================
platform linux -- Python 3.10.7, pytest-7.2.0, pluggy-1.0.0
rootdir: /strelka/src/python
plugins: mock-3.10.0
collected 9 items
platform linux -- Python 3.10.6, pytest-7.2.0, pluggy-1.0.0
rootdir: /strelka
plugins: mock-3.10.0, unordered-0.5.2
collected 92 items
tests/test_required_for_scanner.py .
tests/test_scan_base64.py .
tests/test_scan_base64_pe.py .
tests/test_scan_batch.py .
tests/test_scan_bmp_eof.py .
...
============================== 9 passed in 0.48s ===============================
tests/test_scan_upx.py .
tests/test_scan_url.py ..
tests/test_scan_vhd.py ..
tests/test_scan_x509.py ..
tests/test_scan_xml.py .
tests/test_scan_yara.py .
tests/test_scan_zip.py ..
======================= 92 passed, 29 warnings in 27.93s =======================
```

If you're testing with the default backend.yaml and taste.yara, enable `CONFIG_TESTS` to assure the configuration works as expected.

```
docker-compose -f build/docker-compose.yaml build --build-arg CONFIG_TESTS=true backend
============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.2.0, pluggy-1.0.0
rootdir: /strelka
plugins: mock-3.10.0, unordered-0.5.2
collected 155 items
tests_configuration/test_scanner_assignment.py .............................................................................
tests_configuration/test_taste.py ..............................................................................
======================= 155 passed, 4 warnings in 8.55s ========================
```

## Style Guides
Expand Down
9 changes: 6 additions & 3 deletions build/python/backend/Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ FROM ubuntu:22.04
ARG DEBIAN_FRONTEND=noninteractive
LABEL maintainer="Target Brands, Inc. [email protected]"

ARG CONFIG_TESTS=false

ARG USERNAME=strelka
ARG USER_UID=1001
ARG USER_GID=$USER_UID
Expand Down Expand Up @@ -185,12 +187,13 @@ RUN cd /strelka/ && \
chmod -R g=u /var/log/strelka/

# Run tests as non-root user
USER $username
USER $USERNAME

# Run build checks
RUN echo '[+] Run checks' && \
RUN echo '[+] Run build checks' && \
cd /strelka/strelka/ && \
python3 -m pytest -s tests/ && \
if $CONFIG_TESTS; then python3 -m pytest -s tests_configuration/; fi && \
echo '[+] Done'

USER root
Expand All @@ -202,4 +205,4 @@ RUN cd /strelka/ && \
# Remove config directory (will bind mount once built)
RUN rm -rf /etc/strelka/

USER $username
USER $USERNAME
71 changes: 37 additions & 34 deletions docs/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -613,50 +613,53 @@ The table below describes each scanner and its options. Each scanner has the hid
| ScanZlib | Decompresses gzip files | N/A

## Tests
As Strelka consists of many scanners and dependencies for those scanners, Pytests are particularly valuable for testing the ongoing functionality of Strelka and it's scanners. Tests allow users to write test cases that verify the correct behavior of Strelka scanners to ensure that the scanners remain reliable and accurate. Additionally, using pytests can help streamline the development process, allowing developers to focus on writing new features and improvements for the scanners. The following section details how to setup Pytests.
As Strelka consists of many scanners and dependencies for those scanners. Pytests are particularly valuable for testing the ongoing functionality of Strelka and it's scanners. Tests allow users to write test cases that verify the correct behavior of Strelka scanners to ensure that the scanners remain reliable and accurate. Additionally, using pytests can help streamline the development process, allowing developers to focus on writing new features and improvements for the scanners. Strelka contains a set of standard test fixture files that represent the types of files Strelka ingests.
If using Strelka on Github, this repository supports Github Actions which runs on Pull Requests
### Tests Setup
Here are the steps for setting up a virtualenv virtual environment, installing requirements from src/python/requirements.txt, and running pytest:
1. Install virtualenv, if it is not already installed:
```
pip install virtualenv
```
2. Create a new virtual environment:
```
virtualenv <environment-name>
```
3. Activate the virtual environment:
The best way to run Strelka's test suite is to build the docker containers. Some of Strelka's scanners have OS level dependencies which make them unsuitable for individual testing.
```
source <environment-name>/bin/activate
docker-compose -f build/docker-compose.yaml build backend
============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.2.0, pluggy-1.0.0
rootdir: /strelka
plugins: mock-3.10.0, unordered-0.5.2
collected 92 items
tests/test_required_for_scanner.py .
tests/test_scan_base64.py .
tests/test_scan_base64_pe.py .
tests/test_scan_batch.py .
tests/test_scan_bmp_eof.py .
...
tests/test_scan_upx.py .
tests/test_scan_url.py ..
tests/test_scan_vhd.py ..
tests/test_scan_x509.py ..
tests/test_scan_xml.py .
tests/test_scan_yara.py .
tests/test_scan_zip.py ..
======================= 92 passed, 29 warnings in 27.93s =======================
```
4. Install the requirements from src/python/requirements.txt:
If you're testing with the default backend.yaml and taste.yara, enable `CONFIG_TESTS` to assure the configuration works as expected.

```
pip install -r src/python/requirements.txt
```
docker-compose -f build/docker-compose.yaml build --build-arg CONFIG_TESTS=true backend
5. Run pytest to execute the test cases:
============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.2.0, pluggy-1.0.0
rootdir: /strelka
plugins: mock-3.10.0, unordered-0.5.2
collected 155 items
```
pytest
```
Upon execution, you will be provided the successes and failures for any available scanner test.
tests_configuration/test_scanner_assignment.py .............................................................................
tests_configuration/test_taste.py ..............................................................................
```
Some tests (e.g., ScanCapa, ScanDmg, ScanOCR) may fail on local host testing as they rely on
additional executables to run via `subprocess` that are not installed via `pip`. If you wish
to verify these tests, either install the relevant executable (which can be observed in Backend
Dockerfile - build/python/backend/Dockerfile) or simply build Strelka - of which the docker build
logs show test outcomes.
======================= 155 passed, 4 warnings in 8.55s ========================
```

## Use Cases
Expand Down
Loading

0 comments on commit 333ce89

Please sign in to comment.