Update the documentation, and add a developer docker-compose

.gitlab-ci.yml

@@ -90,22 +90,30 @@ test_types:
   image: python:3.6
   script:
   - cp src/config.example.py src/config.py
-  - pip3 install mypy
+  - pip3 install mypy==0.770
   - mypy src
 
 test_style:
   stage: pretest
   image: python:3.6
   script:
   - cp src/config.example.py src/config.py
-  - pip3 install black
+  - pip3 install flake8==3.7.9
+  - flake8 "src/"
+
+test_black:
+  stage: pretest
+  image: python:3.6
+  script:
+  - cp src/config.example.py src/config.py
+  - pip3 install black==2.0.2
   - black --check "src/"
 
 test_style_js:
   stage: pretest
   image: node:13
   script:
-  - npm install -g prettier
+  - npm install -g prettier@2.0.2
   - prettier --tab-width=4 --check "src/mqueryfront/src/**/*.js"
 
 deploy_mquery:

CONTRIBUTING.md

@@ -0,0 +1,83 @@
+# CONTRIBUTING.md
+
+## How to start?
+
+Great, so you want to join the development!
+
+First, [set up a development environment](INSTALL.md). Since you're going to
+write new code, use the `docker-compose.dev.yml` method.
+
+If everything went right, the system should be accessible at `localhost:80`.
+
+## Development workflow
+
+We use a standard [github fork workflow](
+https://gist.github.com/Chaser324/ce0505fbed06b947d962).
+
+1. Fork the repository.
+
+2. Create a new branch. The name does not matter, but the recommended format
+  is `feature/xxx` or `fix/yyy`.
+
+3. Work on your changes!
+
+4. If your changes are significant, consider adding a test or two
+   to the `src/tests/` directory. We test every change on our Gitlab instance,
+   but you can run them locally too:
+
+```bash
+$ docker build -t mquery_tests -f ./src/tests/Dockerfile .
+$ docker run --net mquery_default -v $(readlink -f ./samples):/mnt/samples mquery_tests
+```
+
+5. Run autoformatters and linters on your code to speed-up review (we run
+them automatically on every commit, but currently only on our internal
+GitLab instance):
+
+- Important: we use [black](https://pypi.org/project/black/) for Python:
+
+```bash
+$ pip3 install black==19.10b0
+$ black src/
+```
+
+- Important: we use [prettier](httpss://prettier.io/) for Javascript/React:
+
+```bash
+$ npm install -g [email protected]
+$ prettier --tab-width=4 --write "src/mqueryfront/src/**/*.js"
+```
+
+- Verify that there are no type errors with [mypy](http://mypy-lang.org/):
+
+```bash
+$ pip install mypy==0.770
+$ mypy src
+```
+
+- Find other styly issues with [flake8](https://flake8.pycqa.org):
+
+```bash
+$ pip install flake8==3.7.9
+$ flake8 src
+```
+
+(Lifehack: you can also plug them into your editor as on-save action).
+
+6. When you feel like you're done, commit the files:
+
+```bash
+$ git add -A
+$ git status  # check if included files match your expectations
+$ git diff --cached  # check the diff for forgotten debug prints etc
+$ git commit  # commit the changes (don't forget to add a commit message)
+```
+
+7. Push changes to your fork:
+
+```
+$ git push origin [your_branch_name]
+```
+
+8. Create a pull request with your changes from the GitHub interface and
+   wait for review.

INSTALL.md

@@ -0,0 +1,86 @@
+# INSTALL.md
+
+Supported installation and deployment methods:
+
+- [docker-compose.yml](#Docker_compose)
+- [docker-compose.dev.yml](#Docker_compose_(dev))
+- [bare metal](#bare_metal)
+- [kubernetes](kubernetes)
+
+## Docker compose
+
+Quick build&run with [docker compose](https://docs.docker.com/compose/).
+
+```
+git clone --recurse-submodules https://github.com/CERT-Polska/mquery.git
+cd mquery
+# Copy your malware samples to ./samples directory in the cloned repository
+docker-compose up --scale daemon=3  # this will take a while
+```
+
+- Good for testing mquery and production deployments on a single server
+- Poor for development
+
+## Docker compose (dev)
+
+Docker compose dedicated for developers.
+
+```
+git clone --recurse-submodules https://github.com/CERT-Polska/mquery.git
+cd mquery
+# Optionally copy test files to ./samples directory
+docker-compose -f docker-compose.dev.yml up  # this will take a while
+```
+
+- Good for development - all file changes will be picked up automatically.
+- Poor for production
+
+## Bare metal
+
+You can also compile and run everything manually.
+
+```
+sudo apt install libzmq3-dev cmake gcc g++ make python3 git npm redis-server
+
+git clone --recurse-submodules https://github.com/CERT-Polska/mquery.git
+
+cd mquery
+pip install -r requirements.txt  # this may take a few minutes
+
+cd src/mqueryfront
+npm install
+npm run build
+
+cd ../../ursadb
+mkdir build; cd build
+cmake -D CMAKE_BUILD_TYPE=Release ..  # requires gcc 7+
+make
+```
+
+Create a new database:
+
+```
+./ursadb/build/ursadb_new ~/db.ursa
+```
+
+And start everything:
+
+```
+project_dir/mquery/src$ flask run  # web server
+project_dir/mquery/src$ python3 daemon.py  # job daemon
+project_dir/ursadb/build$ ./ursadb ~/db.ursa  # backend database
+```
+
+Web interface should be available at `http://localhost:5000`.
+
+- Good for production - the most flexible method.
+- Good for development, but setting up a proper environment is tricky.
+    Just use docker compose.
+
+## Kubernetes
+
+Not strictly supported, but we use it internally so it's battle-tested.
+Take a look at the the `./deploy/k8s` directory for hints.
+
+- Good for production - it's webscale!
+- Terrible for development.

README.md

@@ -15,7 +15,7 @@ Take a look at [https://mquery.tailcall.net](https://mquery.tailcall.net) for a
 
 Unfortunately, you won't find any actual malware there. For demo purposes we
 have indexed the sources of this project - for example, you can find all exceptions
-in our source code by using this yara rule:
+in our source code by using this Yara rule:
 
 ```
 rule find_exceptions: trojan
@@ -29,115 +29,68 @@ rule find_exceptions: trojan
 }
 ```
 
-## How does it work?
+## Quickstart
 
-YARA is pretty fast, but searching through large dataset for given signature can take a lot of time. To countermeasure this, we have implemented a custom database called UrsaDB. It is able to pre-filter the results, so it is only necessary to run YARA against a small fraction of binaries:
-
-![mquery flowchart](docs/mquery-flowchart.png?raw=1)
-
-## Quick start (docker)
-
-Easiest way start is with `docker-compose`:
+Easiest way start is `docker-compose` from source:
 
 ```
 git clone --recurse-submodules https://github.com/CERT-Polska/mquery.git
 cd mquery
-# change `./samples` in `./samples:/mnt/samples` to the path where you keep
-# your malware samples. You can also keep it as it is, and copy malware samples
-# to ./samples directory before indexing.
-vim docker-compose.yml
+# Copy your malware samples to ./samples directory in the cloned repository
 docker-compose up --scale daemon=3  # building the images will take a while
 ```
 
-Web interface should be available at `http://localhost`.
-
-## Quick start (manual installation)
+The web interface should be available at `http://localhost`.
 
-You can also build the system from source:
-
-```
-sudo apt install libzmq3-dev cmake gcc g++ make python3 git npm redis-server
-
-git clone --recurse-submodules https://github.com/CERT-Polska/mquery.git
+For more options see [INSTALL.md](./INSTALL.md).
 
-cd mquery
-pip install -r requirements.txt  # this may take a few minutes
-cp src/config.example.py src/config.py
-
-cd src/mqueryfront
-npm install
-cp src/config.dist.js src/config.js
-npm run build
-
-cd ../../ursadb
-mkdir build; cd build
-cmake -D CMAKE_BUILD_TYPE=Release ..  # requires gcc 7+
-make
-```
-
-Create a new database:
-
-```
-./ursadb/build/ursadb_new ~/db.ursa
-```
+## Next steps
 
-And start everything manually (systemd services recommended here):
+After you start the system, you should index some files. Use ursadb-cli:
 
 ```
-project_dir/mquery/src$ flask run  # web server
-project_dir/mquery/src$ python3 daemon.py  # job daemon (can be scaled horisontally)
-project_dir/ursadb/build$ ./ursadb ~/db.ursa  # backend database
+sudo docker-compose run ursadb-cli tcp://ursadb:9281
+index "/mnt/samples";  # /mnt/samples refers to ./samples in the repo
 ```
 
-Web interface should be available at `http://localhost:5000`.
+The command will track the progress.
+Wait until it's finished (this can take a while).
 
-## Quick start (kubernetes)
-
-Dealing with Kubernetes is never quick. If you're up for a challenge, take a look
-at the `./deploy/k8s` directory.
-
-## Next steps
-
-After you start the system, you should index some files.
-
-Right now it can only be done with ursadb-cli tool. Start it:
-
-- for bare metal setup: `python3 ursadb-cli/ursaclient.py`
-- for docker: `sudo docker-compose run ursadb-cli tcp://ursadb:9281`
-
-Enter the `index` command (for more complicated options, see ursadb docs):
-
-- for bare metal setup `index "/path/to/your/malware/samples";`
-- for docker: `index "/mnt/samples";` (remember, this is mount configured in `docker-compose.yml`)
-
-The command should print the progress. Wait until it's finished (this can take a while).
-
-Now your files should be searchable - try with the following yara rule (or any other):
+Now your files should be searchable - try with the following Yara rule (or any other):
 
 ```
-rule emotet4_basic: trojan
+rule emotet4_basic
 {
     meta:
         author = "cert.pl"
     strings:
-        $emotet4_rsa_public = { 8d ?? ?? 5? 8d ?? ?? 5? 6a 00 68 00 80 00 00 ff 35 [4] ff 35 [4] 6a 13 68 01 00 01 00 ff 15 [4] 85 }
+        $emotet4_rsa_public = {
+            8d ?? ?? 5? 8d ?? ?? 5? 6a 00 68 00 80 00 00 ff 35 [4] ff
+            35 [4] 6a 13 68 01 00 01 00 ff 15 [4] 85
+        }
         $emotet4_cnc_list = { 39 ?? ?5 [4] 0f 44 ?? (FF | A3)}
     condition:
         all of them
 }
 ```
 
-## Contributing
+## Learn more
 
-If you want to contribute, ensure that your PR passes through the CI pipeline. Ideally:
+See [internals.md](./docs/internals.md) to learn about:
+
+ - How mquery works on a high level.
+ - Known limitations and design decisions.
+ - How to create efficient yara rules.
+
+## Contributing
 
- - check your code with `flake8`
- - autoformat your python code with `black`
- - autoformat your html/js/jsx with `prettier --tab-width=4`
+If you want to contribute, see [CONTRIBUTING.md](./CONTRIBUTING.md).
 
 ## Contact
 
-In case of any questions, feel free to contact:
+If you have any problems, bugs or feature requests related to mquery, you're
+encouraged to create a GitHub issue. If you have other questions or want to
+contact the developers directly, you can email:
 
 - Michał Leszczyński ([email protected])
 - Jarosław Jedynak ([email protected])

deploy/docker/dev.daemon.Dockerfile

@@ -0,0 +1,9 @@
+FROM python:3.7
+
+WORKDIR /usr/src/app/src
+
+RUN apt update; apt install -y cmake
+COPY requirements.txt requirements.txt
+RUN pip install --no-cache-dir -r requirements.txt
+# ./src is expected to be mounted with a docker volume
+CMD ["./autoreload", "python3 daemon.py"]

deploy/docker/dev.frontend.Dockerfile

@@ -0,0 +1,7 @@
+FROM node:8 AS build
+
+RUN npm install -g serve
+COPY src/mqueryfront /app
+WORKDIR /app
+RUN npm install
+CMD ["npm", "start"]

deploy/docker/dev.web.Dockerfile

@@ -0,0 +1,10 @@
+FROM python:3.7
+
+WORKDIR /usr/src/app/src
+
+RUN apt update; apt install -y cmake
+COPY requirements.txt requirements.txt
+RUN pip install --no-cache-dir -r requirements.txt
+# ./src is expected to be mounted with a docker volume
+ENV FLASK_DEBUG=1
+CMD ["flask", "run", "--host", "0.0.0.0", "--port", "5000"]

deploy/docker/web.Dockerfile

@@ -2,7 +2,6 @@ FROM node:8 AS build
 
 RUN npm install -g serve
 COPY src/mqueryfront /app
-COPY src/mqueryfront/src/config.dist.js /app/src/config.js
 WORKDIR /app
 RUN npm install && npm run build