Skip to content

Commit

Permalink
More on dependences and docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@Wh1isper
Wh1isper committed Sep 17, 2023
1 parent ad3e040 commit 39322d9
Show file tree
Hide file tree
Showing 4 changed files with 305 additions and 305 deletions.
154 changes: 72 additions & 82 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<h2 align="center">duetector🔍: 支持eBPF的可扩展数据使用探测器</h2>
<h2 align="center">duetector🔍: Data Usage Extensible detector(eBPF Support)</h2>
<p align="center">
<a href="https://github.com/hitsz-ids/duetector/actions"><img alt="Actions Status" src="https://github.com/hitsz-ids/duetector/actions/workflows/python-package.yml/badge.svg"></a>
<a href='https://duetector.readthedocs.io/en/latest/?badge=latest'><img src='https://readthedocs.org/projects/duetector/badge/?version=latest' alt='Documentation Status' /></a>
Expand All @@ -11,111 +11,107 @@
</p>

<p align="center">
<a href="./README.md">中文</a> | <a href="./README_en.md">English</a>
<a href="./README.md">English</a> | <a href="./README_zh.md">中文</a>
</p>

## 简介
## Introduction

> duetector是DataUCON项目中的组件之一，DataUCON项目旨在为数据使用控制提供支持。
>
> [查看DataUCON的网页](https://dataucon.idslab.io/)
>
> [深入了解并部署DataUCON](https://github.com/hitsz-ids/dataucon)
> duetector is one of the components in the DataUCON project, which is designed to provide support for data usage control. [Intro DataUCON](https://dataucon.idslab.io/).
duetector🔍是一个基于可扩展的的数据使用探测器，它可以在Linux内核中对数据使用行为进行探测（基于eBPF），从而为数据使用控制提供支持。
duetector🔍 is an extensible data usage control detector that provides support for data usage control by probing for data usage behavior in the Linux kernel(based on eBPF).

**🐛🐞🧪 项目正在大力开发中，期待任何Bug报告、功能请求、合并请求**
**🐛🐞🧪 The project is under heavy development, looking forward to any bug reports, feature requests, pull requests!**

在[ABAUC控制模型](https://github.com/hitsz-ids/dataucon)当中，duetector可作为PIP（Policy Information Point）来获取数据使用行为，从而为PDP（Policy Decision Point）提供数据使用行为的信息。[快速了解用户案例](./docs/usercases/)
In the [ABAUC control model](https://github.com/hitsz-ids/dataucon), duetector can be used as a PIP (Policy Information Point) to obtain data usage behavior, so as to provide information about data usage behavior for PDP (Policy Decision Point). Provide information on data usage behavior to PDP (Policy Decision Point).

## 目录
## Table of Contents

- [主要特性](#主要特性)
- [安装](#安装)
- [快速开始](#快速开始)
- [API文档与配置文档](#API文档与配置文档)
- [维护者](#维护者)
- [如何贡献](#如何贡献)
- [许可证](#许可证)
- [Features](#Features)
- [Installation](#Installation)
- [Quick Start](#quick-start)
- [API](#API)
- [Maintainers](#Maintainers)
- [contribute](#contribute)
- [License](#License)

## 主要特性
## Feature

- [X] 插件化系统
- [X] 支持自定义tracer
- [X] 支持自定义filter
- [X] 支持自定义collector
- [X] [自定义插件示例](./examples/)
- [ ] 配置管理
- [X] 使用单一配置文件配置
- [X] 支持生成插件配置
- [ ] 支持动态加载配置
- [ ] 基于eBPF的数据使用探测器
- [X] 文件打开操作
- [ ] ……
- [ ] 基于Shell命令的探测器
- [X] 内核信息探测
- [ ] ……
- [X] 支持SQL数据库的数据收集器
- [X] CLI工具
- [ ] PIP服务
- [X] Plug-in system
- [X] Customized tracer support
- [X] Support for custom filters
- [X] Custom collector support
- [X] [Custom Plugin Examples](./examples/)
- [ ] Configuration Management
- [X] Configuration using a single configuration file
- [X] Generate Plugin Configuration
- [ ] Support for dynamically loading configurations
- [ ] eBPF-based data usage probes
- [X] File Open Operation
- [ ] ......
- [ ] Shell command probes
- [X] Kernel Information Probe
- [ ] ......
- [X] Data collector with SQL database support
- [X] CLI Tools
- [ ] PIP Service

eBPF程序需要内核支持，详见[内核支持](./docs/kernel_config.md)
The eBPF program requires kernel support, see [Kernel Support](./docs/kernel_config.md)

## 安装
## Installation

代码通过Pypi分发，你可以通过以下命令安装
The code is distributed via Pypi, and you can install it with the following command

```bash
pip install duetector
```

目前，代码依赖[BCC](https://github.com/iovisor/bcc)对eBPF代码进行即时编译，推荐[安装最新的BCC编译器](https://github.com/iovisor/bcc/blob/master/INSTALL.md)
Currently, the code relies on [BCC](https://github.com/iovisor/bcc) for on-the-fly compilation of eBPF code, we recommend [installing the latest BCC compiler](https://github.com/iovisor/bcc/blob/master/INSTALL.md)

或使用我们提供的Docker镜像，其使用[JupyterLab](https://github.com/jupyterlab/jupyterlab)作为**示例**用户应用，您也可以自行修改[Dockerfile](./docker/Dockerfile)和[启动脚本](./docker/start.sh)来自定义用户程序
Or use the Docker image that we provide, which uses [JupyterLab](https://github.com/jupyterlab/jupyterlab) as the **example** user application, or you can modify the [Dockerfile](./docker/Dockerfile) and [startup script](./docker/start.sh) to customize the user application.

```bash
docker pull dataucon/duetector:latest
```

预发布版本将不会更新到 `latest`上，您可以指定tag进行拉取，如 `v0.0.1a`
Pre-releases will not be updated to `latest`, you can specify the tag to pull, e.g. `v0.0.1a`

```bash
docker pull dataucon/duetector:v0.0.1a
```

使用docker镜像运行的更多细节请参考[这里](./docs/how-to/run-with-docker.md)
For more details on running with docker images see [here](./docs/how-to/run-with-docker.md)

## 快速开始
## Quick start

> 更多文档和例子可以在[这里](./docs/)找到。
> More documentation and examples can be found [here](. /docs/).
### 启动探测器
### Start detector

使用命令行启动monitor，由于bcc需要root权限，所以我们使用 `sudo` 命令，这将启动所有的探测器，并将探测内容收集到当前目录下的 `duetector-dbcollector.sqlite3`文件中
Start monitor using the command line, since bcc requires root privileges, we use the `sudo` command, which will start all probes and collect the probes into the `duetector-dbcollector.sqlite3` file in the current directory

```bash
sudo duectl start
```

按下 `CRTL+C`可以退出监测，你将看到屏幕上输出了一段总结
Press `CRTL+C` to exit monitoring and you will see a summary output on the screen

```
{'DBCollector': {'OpenTracer': {'count': 31, 'first at': 249920233249912, 'last': Tracking(tracer='OpenTracer', pid=641616, uid=1000, gid=1000, comm='node', cwd=None, fname='SOME-FILE', timestamp=249923762308577, extended={})}}}
{'DBCollector': {'OpenTracer': {'count': 31, 'first at': 249920233249912, 'last': Tracking(tracer='OpenTracer', pid=641616, uid=1000, gid= 1000, comm='node', cwd=None, fname='SOME-FILE', timestamp=249923762308577, extended={})}}}
```

启动 `DEBUG`日志
Enable `DEBUG` log

```bash
sudo DUETECTOR_LOG_LEVEL=DEBUG duectl start
```

启动时，配置文件将自动生成，对应路径为 `~/.config/duetector` ，修改这个配置文件可以修改数据库地址等内容，可以使用 `--config`指定使用的配置文件
At startup, the configuration file will be automatically generated at `~/.config/duetector`, and you can specify the configuration file to use with `--config`.

```bash
sudo duectl start --config <config-file-path>
```

也支持使用环境变量进行配置:
Configuration using environment variables is also supported:

```bash
Usage: duectl start [OPTIONS]
Expand All @@ -130,21 +126,22 @@ Options:
...
```

当使用插件时，默认的配置文件不会包含插件的配置内容，使用动态生成配置的指令生成带有插件配置的配置文件，这个指令也支持合并当前已有的配置文件和环境变量

When using a plugin, the default configuration file will not contain the plugin's configuration, use the dynamically-generated configuration directive to generate a configuration file with the plugin's configuration, this directive also supports merging existing configuration files and environment variables.

```bash
duectl generate-dynamic-config --help
```

当配置文件出错时，可以使用 `generate-config` 恢复默认状态
Use `generate-config` to restore the default state in case of configuration file errors.

```bash
duectl generate-config
```

更进一步的，后台运行可以使用 `duectl-daemon start`命令，这将会在后台运行一个守护进程，你可以使用 `duectl-daemon stop`来停止它
Going a step further, running in the background you can use the `duectl-daemon start` command, which will run a daemon in the background, which you can stop using `duectl-daemon stop`

使用 `duectl-daemon --help` 获取更多细节：
Use `duectl-daemon --help` for more details:

```bash
Usage: duectl-daemon [OPTIONS] COMMAND [ARGS]...
Expand All @@ -158,15 +155,15 @@ Commands:
stop Stop the process.
```

### 使用Analyzer进行分析
### Analyzing with analyzer

我们提供了一个[Analyzer](https://duetector.readthedocs.io/en/latest/analyzer/index.html)，它可以对存储中的数据进行查询，在这里我们提供了一个[入门案例](./docs/usercases/simplest-open-count/README.md)
We provide an [Analyzer](https://duetector.readthedocs.io/en/latest/analyzer/index.html) that can query the data in storage, here we provide a [user case](./docs/usercases/simplest-open-count/README.md)

### 使用Duetector Server
### Using duetector server

我们提供了一个Duetector Server，作为外部PIP服务和控制接口
We provide a Duetector Server as an external PIP service and control interface

使用`duectl-server`可以启动一个Duetector Server，默认将监听`0.0.0.0:8120`，你可以使用`--host`和`--port`来修改它。
A Duetector Server can be started using `duectl-server` and will listen on `0.0.0.0:8120` by default, you can modify it using `--host` and `--port`.

```bash
$ duectl-server start --help
Expand All @@ -187,9 +184,9 @@ Options:
--help Show this message and exit.
```

在服务启动后，访问`http://{ip}:{port}/docs`可以查看API文档。
After the service has started, visit `http://{ip}:{port}/docs` to see the API documentation.

同样的，使用`duectl-server-daemon start`可以在后台运行一个Duetector Server，你可以使用`duectl-server-daemon stop`来停止它
Similarly, using `duectl-server-daemon start` you can run a Duetector Server in the background, and you can stop it using `duectl-server-daemon stop`

```bash
$ duectl-server-daemon
Expand All @@ -204,29 +201,22 @@ Commands:
stop Stop the process.
```

## API文档与配置文档

我们在readthedocs上为开发者和用户提供了API与配置文档，你可以在[这里](https://duetector.readthedocs.io/)查看

## 维护者

本项目由**哈尔滨工业大学（深圳）数据安全研究院**发起，若您对本项目以及DataUCON项目感兴趣并愿意一起完善它，欢迎加入我们的开源社区。
## API documentation

## 如何贡献
See [docs of duetector](https://duetector.readthedocs.io/)

非常欢迎您的加入！[我们欢迎任何类型的Issue](https://github.com/hitsz-ids/duetector/issues/new)，同时也期待您的PR
## Maintainers

我们提供了以下资料让您更快了解项目
This project is initiated by **Institute of Data Security, Harbin Institute of Technology (Shen Zhen)**, if you are interested in this project and [DataUCON](https://dataucon.idslab.io/) project and willing to work together to improve it, welcome to join our open source community.

- 开发环境配置和其他注意事项请参考：[开发者文档](./CONTRIBUTING.md)
- 在这里了解本项目的设计思路和架构：[设计文档](./docs/design/README.md)
## How to contribute

# 如何开发插件
You are very welcome to join! [Raise an Issue](https://github.com/hitsz-ids/duetector/issues/new) or submit a Pull Request.

目前，tracer、filter、collector都支持自定义插件开发，以Python包作为单个插件或多个插件，可以查看[自定义插件示例](./examples/)了解开发步骤
Please refer to the [Developer Documentation](./CONTRIBUTING.md).

TODO: 提供一个插件的cookiecutter模板
Learn about the design ideas and architecture of this project here: [DESIGN DOCUMENTS](./docs/design/README.md).

## 许可证
## License

本项目使用 Apache-2.0 license，有关协议请参考[LICENSE](https://github.com/hitsz-ids/duetector/blob/main/LICENSE)。
This project uses Apache-2.0 license, please refer to [LICENSE](https://github.com/hitsz-ids/duetector/blob/main/LICENSE).
Loading

0 comments on commit 39322d9

Please sign in to comment.