Add docs for oTel(jager backend) (#90)

* Updating docs

* Fix some bugs on otel collector

README.md

@@ -1,3 +1,7 @@
+<p align="center">
+<a href="https://github.com/hitsz-ids/dataucon"><img alt="DataUCon" src="https://raw.githubusercontent.com/hitsz-ids/dataucon/main/img/white-icon-simple.png"></a>
+</p>
+
 <h2 align="center">duetector🔍: Data Usage Extensible detector(eBPF Support)</h2>
 <p align="center">
 <a href="https://github.com/hitsz-ids/duetector/actions"><img alt="Actions Status" src="https://github.com/hitsz-ids/duetector/actions/workflows/python-package.yml/badge.svg"></a>

README_zh.md

@@ -1,3 +1,7 @@
+<p align="center">
+<a href="https://github.com/hitsz-ids/dataucon"><img alt="DataUCon" src="https://raw.githubusercontent.com/hitsz-ids/dataucon/main/img/white-icon-simple.png"></a>
+</p>
+
 <h2 align="center">duetector🔍: 支持eBPF的可扩展数据使用探测器</h2>
 <p align="center">
 <a href="https://github.com/hitsz-ids/duetector/actions"><img alt="Actions Status" src="https://github.com/hitsz-ids/duetector/actions/workflows/python-package.yml/badge.svg"></a>

docs/how-to/intergration-with-otel.md

@@ -0,0 +1,115 @@
+# Integration with OpenTelemetry
+
+[OpenTelemetry](https://opentelemetry.io/) is a collection of APIs, SDKs, and tools that allow development teams to generate, process, and transmit telemetry data in a unified single format.
+
+`duetector` support integration with OpenTelemetry. Through [otel collector](../../duetector/collectors/otel.py), we can export traces to any backend supported by OpenTelemetry.
+
+## Deploy a bancend
+
+**If you already have a backend and familiar with OpenTelemetry, you can skip this section.**
+
+Before we start, we need a backend to receive traces. Here we use [jaeger](https://www.jaegertracing.io/) as an example, which is a popular open source tracing backend and supports OpenTelemetry directly.
+
+Deploy [jaeger all in one](https://www.jaegertracing.io/docs/1.50/getting-started/#all-in-one) with docker:
+
+> [Latest jaeger all in one guide](https://www.jaegertracing.io/docs/latest/getting-started/#all-in-one)
+
+```bash
+docker run --rm --name jaeger \
+  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
+  -p 6831:6831/udp \
+  -p 6832:6832/udp \
+  -p 5778:5778 \
+  -p 16686:16686 \
+  -p 4317:4317 \
+  -p 4318:4318 \
+  -p 14250:14250 \
+  -p 14268:14268 \
+  -p 14269:14269 \
+  -p 9411:9411 \
+  jaegertracing/all-in-one:1.50
+```
+
+Then you can then navigate to http://localhost:16686 to access the Jaeger UI. The port `4317` is for OTLP gRPC receiver, and `4318` is for OTLP http receiver.
+
+### Use otel collector to export traces
+
+If your backend is **NOT** supported by OpenTelemetry, you can deploy a collector to receive traces and export them to your backend.
+
+A simple way is to deploy a collector in [Agent mode](https://opentelemetry.io/docs/collector/deployment/agent/):
+
+Use docker-compose to deploy a collector:
+
+```yaml
+otel-collector:
+  image: otel/opentelemetry-collector-contrib
+  volumes:
+    - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
+  ports:
+    - 1888:1888 # pprof extension
+    - 8888:8888 # Prometheus metrics exposed by the collector
+    - 8889:8889 # Prometheus exporter metrics
+    - 13133:13133 # health_check extension
+    - 4317:4317 # OTLP gRPC receiver
+    - 4318:4318 # OTLP http receiver
+    - 55679:55679 # zpages extension
+```
+
+Here is an example config file:
+
+```yaml
+receivers:
+  otlp: # the OTLP receiver the app is sending traces to
+    protocols:
+      grpc:
+
+processors:
+  batch:
+
+exporters:
+  otlp/jaeger: # Jaeger supports OTLP directly
+    endpoint: https://jaeger.example.com:4317
+
+service:
+  pipelines:
+    traces/dev:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [otlp/jaeger]
+```
+
+For more deployment options, such as load balance, please refer to [OpenTelemetry Collecotr Deployment](https://opentelemetry.io/docs/collector/deployment/).
+
+## Enable Duetector's Otel Collector
+
+To enable duetector's otel collector, we need to set `collector.otelcollector.disabled` to `false` in config file, and set `exporter` and its `exporter_kwargs` to specify the backend. Also, you can use environment variables to set the config.
+
+Here is an example config file:
+
+```toml
+[collector.otelcollector]
+disabled = false
+statis_id = "demo-service"
+exporter = "otlp-grpc"
+
+[collector.otelcollector.backend_args]
+max_workers = 10
+
+[collector.otelcollector.exporter_kwargs]
+endpoint = "localhost:4317"
+insecure = true
+
+# disable dbcollector as we don't rely on db to store traces
+[collector.dbcollector]
+disabled = true
+```
+
+Then start duetector with the config file:
+
+```bash
+duectl start --config config.toml
+```
+
+Access jaeger UI(localhost:16686), you should see traces like this:
+
+![jaeger](./images/jaeger.png)

duetector/collectors/models.py

@@ -66,6 +66,16 @@ def normalize_field(cls, field, data):
             data = get_boot_time_duration_ns(data)
         return field, data
 
+    @classmethod
+    def serialize_field(cls, field, data):
+        """
+        Serialize filed to one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types
+        """
+        if field == "dt":
+            field = "timestamp"
+            data = datetime.timestamp(data)
+        return field, data
+
     @staticmethod
     def from_namedtuple(tracer, data: NamedTuple) -> Tracking:  # type: ignore
         """
@@ -108,10 +118,15 @@ def set_span(self, span):
                 continue
             v = getattr(self, k)
             if v is not None:
+                k, v = self.serialize_field(k, v)
                 span.set_attribute(k, v)
         for k, v in self.extended.items():
             span.set_attribute(k, v)
 
+    @property
+    def span_name(self):
+        return self.tracer
+
 
 if __name__ == "__main__":
     Tracking(tracer="test", dt=datetime.now())

duetector/collectors/otel.py

@@ -140,18 +140,22 @@ def endpoint(self) -> Optional[str]:
     def exporter_kwargs(self) -> Dict[str, Any]:
         return self.config.exporter_kwargs
 
+    @property
+    def service_name(self) -> str:
+        return f"duetector-{self.id}"
+
     def __init__(self, config: Optional[Dict[str, Any]] = None, *args, **kwargs):
         super().__init__(config, *args, **kwargs)
         self.otel = OTelInitiator()
         self.otel.initialize(
-            service_name="duetector",
+            service_name=self.service_name,
             exporter=self.exporter,
-            exporter_kwargs=self.exporter_kwargs,
+            exporter_kwargs=self.exporter_kwargs._config_dict,
         )
 
     def _emit(self, t: Tracking):
         tracer = trace.get_tracer(self.id)
-        with tracer.start_as_current_span(t.tracer) as span:
+        with tracer.start_as_current_span(t.span_name) as span:
             t.set_span(span)
 
     def summary(self) -> Dict:

duetector/tracers/base.py

@@ -132,7 +132,10 @@ def _convert_data(self, data) -> NamedTuple:
         for k in self.data_t._fields:  # type: ignore
             v = getattr(data, k)
             if isinstance(v, bytes):
-                v = v.decode("utf-8")
+                try:
+                    v = v.decode("utf-8")
+                except UnicodeDecodeError:
+                    pass
 
             args[k] = v