PIP Server Framework (#56)

* Prepare pip server file

* Updating base analyzer

* Setup server CLI ,docs and building things

* Use tmpdir in testing

* Typo files

* Fix comment and devtools

.gitignore

@@ -573,5 +573,6 @@ cython_debug/
 duetector-dbcollector.sqlite3*
 dev-tools/duetector-dbcollector.sqlite3*
 dev-tools/config.toml
+dev-tools/duetector_server*
 docs/usercases/tracking-mljob-in-kata-containers/cifar-10-batches-py/*
 docs/usercases/tracking-mljob-in-kata-containers/cifar-10-python.tar.gz

README.md

@@ -87,6 +87,10 @@ docker pull dataucon/duetector:v0.0.1a
 
 ## 快速开始
 
+> 更多文档和例子可以在[这里](./docs/)找到。
+
+### 启动探测器
+
 使用命令行启动monitor，由于bcc需要root权限，所以我们使用 `sudo` 命令，这将启动所有的探测器，并将探测内容收集到当前目录下的 `duetector-dbcollector.sqlite3`文件中
 
 ```bash
@@ -154,7 +158,51 @@ Commands:
   stop    Stop the process.
 ```
 
-更多文档和例子可以在[这里](./docs/)找到。
+### 使用Analyzer进行分析
+
+我们提供了一个[Analyzer](https://duetector.readthedocs.io/en/latest/analyzer/index.html)，它可以对存储中的数据进行查询，在这里我们提供了一个[入门案例](./docs/usercases/simplest-open-count/README.md)
+
+### 使用Duetector Server
+
+我们提供了一个Duetector Server，作为外部PIP服务和控制接口
+
+使用`duectl-server`可以启动一个Duetector Server，默认将监听`0.0.0.0:8120`，你可以使用`--host`和`--port`来修改它。
+
+```bash
+$ duectl-server start --help
+Usage: duectl-server start [OPTIONS]
+
+  Start duetector server
+
+Options:
+  --config TEXT       Config file path, default:
+                      ``~/.config/duetector/config.toml``.
+  --load_env BOOLEAN  Weather load env variables, Prefix: ``DUETECTOR_``,
+                      Separator:``__``, e.g. ``DUETECTOR_config__a`` means
+                      ``config.a``, default: True
+  --workdir TEXT      Working directory, default: ``.``.
+  --host TEXT         Host to listen, default: ``0.0.0.0``.
+  --port INTEGER      Port to listen, default: ``8120``.
+  --workers INTEGER   Number of worker processes, default: ``1``.
+  --help              Show this message and exit.
+```
+
+在服务启动后，访问`http://{ip}:{port}/docs`可以查看API文档。
+
+同样的，使用`duectl-server-daemon start`可以在后台运行一个Duetector Server，你可以使用`duectl-server-daemon stop`来停止它
+
+```bash
+$ duectl-server-daemon
+Usage: duectl-server-daemon [OPTIONS] COMMAND [ARGS]...
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  start   Start a background process of command ``duectl-server start``.
+  status  Show status of process.
+  stop    Stop the process.
+```
 
 ## API文档与配置文档
 

README_en.md

@@ -83,6 +83,10 @@ For more details on running with docker images see [here](./docs/how-to/run-with
 
 ## Quick start
 
+> More documentation and examples can be found [here](. /docs/).
+
+### Start detector
+
 Start monitor using the command line, since bcc requires root privileges, we use the `sudo` command, which will start all probes and collect the probes into the `duetector-dbcollector.sqlite3` file in the current directory
 
 ```bash
@@ -151,7 +155,51 @@ Commands:
   stop    Stop the process.
 ```
 
-More documentation and examples can be found [here](. /docs/).
+### Analyzing with analyzer
+
+We provide an [Analyzer](https://duetector.readthedocs.io/en/latest/analyzer/index.html) that can query the data in storage, here we provide a [user case](./docs/usercases/simplest-open-count/README.md)
+
+### Using duetector server
+
+We provide a Duetector Server as an external PIP service and control interface
+
+A Duetector Server can be started using `duectl-server` and will listen on `0.0.0.0:8120` by default, you can modify it using `--host` and `--port`.
+
+```bash
+$ duectl-server start --help
+Usage: duectl-server start [OPTIONS]
+
+  Start duetector server
+
+Options:
+  --config TEXT       Config file path, default:
+                      ``~/.config/duetector/config.toml``.
+  --load_env BOOLEAN  Weather load env variables, Prefix: ``DUETECTOR_``,
+                      Separator:``__``, e.g. ``DUETECTOR_config__a`` means
+                      ``config.a``, default: True
+  --workdir TEXT      Working directory, default: ``.``.
+  --host TEXT         Host to listen, default: ``0.0.0.0``.
+  --port INTEGER      Port to listen, default: ``8120``.
+  --workers INTEGER   Number of worker processes, default: ``1``.
+  --help              Show this message and exit.
+```
+
+After the service has started, visit `http://{ip}:{port}/docs` to see the API documentation.
+
+Similarly, using `duectl-server-daemon start` you can run a Duetector Server in the background, and you can stop it using `duectl-server-daemon stop`
+
+```bash
+$ duectl-server-daemon
+Usage: duectl-server-daemon [OPTIONS] COMMAND [ARGS]...
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  start   Start a background process of command ``duectl-server start``.
+  status  Show status of process.
+  stop    Stop the process.
+```
 
 ## API documentation
 

dev-tools/entrypoint-server.py

@@ -0,0 +1,19 @@
+import os
+
+os.chdir(os.path.dirname(os.path.abspath(__file__)))
+os.environ["DUETECTOR_LOG_LEVEL"] = "DEBUG"
+
+import re
+import sys
+from pathlib import Path
+
+from pkg_resources import load_entry_point
+
+db_file = Path("./duetector-dbcollector.sqlite3")
+config_file = Path("./config.toml")
+
+if __name__ == "__main__":
+    sys.argv[0] = re.sub(r"(-script\.pyw?|\.exe)?$", "", sys.argv[0])
+    sys.argv.append("start")
+    sys.argv.extend(["--config", config_file.resolve().as_posix()])
+    sys.exit(load_entry_point("duetector", "console_scripts", "duectl-server")())

docker/start.sh

@@ -1,6 +1,7 @@
 #!/usr/bin/env bash
 mount -t debugfs debugfs /sys/kernel/debug # enable debugfs
 duectl-daemon start --loglevel=DEBUG
+duectl-server-daemon start --loglevel=DEBUG
 
 cd /home/application
 # Config user's local path for pip install some scripts

docs/source/cli/index.rst

@@ -1,9 +1,14 @@
 duetector.cli
 =========================================
 
-``duectl``: CLI for Start Monitor, generate config.
+``duectl``: CLI for start monitor, generate config.
 
-``duectl-daemon``: Allow to run as daemon, and run as a service.
+``duectl-daemon``: Allow to run monitors and server as daemon.
+
+
+``duectl-server``: CLI for start duetector server.
+
+``duectl-server-daemon``: Allow to run server as daemon.
 
 
 .. toctree::
@@ -12,3 +17,5 @@ duetector.cli
 
    duectl <main>
    duectl-daemon <daemon>
+   duectl-server <server>
+   duectl-server-daemon <server-daemon>

docs/source/cli/server-daemon.rst

@@ -0,0 +1,3 @@
+.. click:: duetector.cli.server_daemon:cli
+   :prog: duectl-server-daemon
+   :nested: full

docs/source/cli/server.rst

@@ -0,0 +1,3 @@
+.. click:: duetector.cli.server:cli
+   :prog: duectl-server
+   :nested: full

docs/usercases/simplest-open-count/README.md

@@ -20,8 +20,10 @@ Alternatively, you can run `duetector` in a kata container, this gives you more
 docker run -it --rm \
 --privileged \
 -p 8888:8888 \
+-p 8120:8120 \
 -v /lib/modules:/lib/modules \
 -e DUETECTOR_DAEMON_WORKDIR=/duetector-kata \
+-e DUETECTOR_SERVER_DAEMON_WORKDIR=/duetector-kata \
 -v $(pwd)/duetector-kata:/duetector-kata \
 -v /sys/kernel/debug:/sys/kernel/debug \
 dataucon/duetector
@@ -34,6 +36,7 @@ Note:
 - You can use `--entrypoint bash` to enter the container and run `duetector` manually.
   - In kata container, you need to mount debugfs manually: `mount -t debugfs debugfs /sys/kernel/debug`
 - `/lib/modules` contains kernel modules, more details can be found in [run-with-docker](../../how-to/run-with-docker.md).
+- `8888` is the port of JupyterLab, `8120` is the port of duetector server. Access `http://localhost:8888` in your browser to use JupyterLab, and access `http://localhost:8120/docs` to see the API docs of duetector server.
 
 ## Use JupyterLab to write some code
 

duetector/analyzer/base.py

@@ -1,5 +1,5 @@
 from datetime import datetime
-from typing import List, Optional
+from typing import Any, Dict, List, Optional
 
 from duetector.analyzer.models import AnalyzerBrief, Tracking
 from duetector.config import Configuable
@@ -42,25 +42,64 @@ def get_all_collector_ids(self) -> List[str]:
 
     def query(
         self,
-        tracer: Optional[str] = None,
-        collector_id: Optional[str] = None,
+        tracers: Optional[List[str]] = None,
+        collector_ids: Optional[List[str]] = None,
         start_datetime: Optional[datetime] = None,
         end_datetime: Optional[datetime] = None,
         start: int = 0,
-        limit: int = 20,
+        limit: int = 0,
+        columns: Optional[List[str]] = None,
+        where: Optional[Dict[str, Any]] = None,
+        distinct: bool = False,
+        order_by_asc: Optional[List[str]] = None,
+        order_by_desc: Optional[List[str]] = None,
     ) -> List[Tracking]:
         """
-        Query tracking data from storage.
+        Query all tracking records from database.
+
+        Args:
+            tracers (Optional[List[str]], optional): Tracer's name. Defaults to None, all tracers will be queried.
+            collector_ids (Optional[List[str]], optional): Collector id. Defaults to None, all collector id will be queried.
+            start_datetime (Optional[datetime], optional): Start time. Defaults to None.
+            end_datetime (Optional[datetime], optional): End time. Defaults to None.
+            start (int, optional): Start index. Defaults to 0.
+            limit (int, optional): Limit of records. Defaults to 20. ``0`` means no limit.
+            columns (Optional[List[str]], optional): Columns to query. Defaults to None, all columns will be queried.
+            where (Optional[Dict[str, Any]], optional): Where clause. Defaults to None.
+            distinct (bool, optional): Distinct. Defaults to False.
+            order_by_asc (Optional[List[str]], optional): Order by asc. Defaults to None.
+            order_by_desc (Optional[List[str]], optional): Order by desc. Defaults to None.
+        Returns:
+            List[duetector.analyzer.models.Tracking]: List of tracking records.
         """
         raise NotImplementedError
 
     def brief(
         self,
+        tracers: Optional[List[str]] = None,
+        collector_ids: Optional[List[str]] = None,
         start_datetime: Optional[datetime] = None,
         end_datetime: Optional[datetime] = None,
+        with_details: bool = True,
+        distinct: bool = False,
     ) -> AnalyzerBrief:
         """
-        Get brief of analyzer.
+        Get a brief of this analyzer.
+
+        Args:
+            tracers (Optional[List[str]], optional):
+                Tracers. Defaults to None, all tracers will be queried.
+                If a specific tracer is not found, it will be ignored.
+            collector_ids (Optional[List[str]], optional):
+                Collector ids. Defaults to None, all collector ids will be queried.
+                If a specific collector id is not found, it will be ignored.
+            start_datetime (Optional[datetime], optional): Start time. Defaults to None.
+            end_datetime (Optional[datetime], optional): End time. Defaults to None.
+            with_details (bool, optional): With details. Defaults to True.
+            distinct (bool, optional): Distinct. Defaults to False.
+
+        Returns:
+            AnalyzerBrief: A brief of this analyzer.
         """
         raise NotImplementedError
 

duetector/analyzer/db.py

@@ -12,9 +12,11 @@ class DBAnalyzer(Analyzer):
     """
     A analyzer using database.
 
-    As a top model, it will init a ``SessionManager`` and pass it to submodels.
+    We design this analyzer to be a top module, so it can be used as a standalone tools.
 
-    Config scope is ``db_analyzer``.
+    In this analyzer, we use ``SessionManager`` to manage database session.
+
+    Config scope is ``db_analyzer``. ``db_analyzer.db`` is the scope for ``SessionManager``.
 
     Example:
 

duetector/cli/daemon.py

@@ -7,6 +7,7 @@
 
 WORKDIR_ENV = "DUETECTOR_DAEMON_WORKDIR"
 DEFAULT_WORKDIR = "/tmp/duetector"
+APPLICATION = "duetector-daemon"
 
 
 @click.command(
@@ -36,7 +37,7 @@ def start(ctx, workdir, loglevel, rotate_log):
     Example:
         ``duectl-daemon start -- --config /path/to/config``
     """
-    cmd = ["duectl", "start"]
+    cmd = ["duectl", "start", "--config_dump_dir", workdir]
     cmd_args = ctx.args
     if cmd_args:
         cmd.extend(cmd_args)
@@ -47,6 +48,7 @@ def start(ctx, workdir, loglevel, rotate_log):
         f"rotate_log: {rotate_log}"
     )
     Daemon(
+        application=APPLICATION,
         cmd=cmd,
         workdir=workdir,
         env_dict={"DUETECTOR_LOG_LEVEL": loglevel},
@@ -68,6 +70,7 @@ def status(workdir):
     """
     if Daemon(
         workdir=workdir,
+        application=APPLICATION,
     ).poll():
         click.echo("Running")
     else:
@@ -88,6 +91,7 @@ def stop(workdir):
     """
     Daemon(
         workdir=workdir,
+        application=APPLICATION,
     ).stop()
     click.echo("Daemon stopped.")