Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

example: add kubernetes logs example #2606

Merged
merged 4 commits into from
Mar 16, 2021
Merged

example: add kubernetes logs example #2606

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5758105
example: add example for parsing kubernetes logs
Mar 8, 2021
90c9a13
example: kubernetes: add result of performance test to README
Mar 8, 2021
21c5fb8
example: kubernetes: add example daemonset
Mar 10, 2021
b85326c
examples: kubernetes: add more explanation about inclusion/exclusion …
Mar 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions examples/kubernetes/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# OpenTelemetry Collector Demo

This demo is a sample app to build the collector and exercise its kubernetes logs scrapping functionality.

## Build and Run

### Kubernetes

Switch to this directory and run following command: `kubectl apply -n <namespace> -f otel-collector.yaml`

#### Include/Exclude Specific Logs

Kubernetes logs are being stored in `/var/log/pods`.
Path to container logs is constructed using following pattern:

`/var/log/pods/<namespace>_<pod_name>_<pod_uid>/<container>/<run_id>.log`

You can use it to manage from which containers do you want to include and exclude logs.

For example, to include all logs from `default` namespace, you can use following configuration:

```yaml
include:
- /var/log/pods/default_*/*/*.log
exclude: []
```

### Docker Compose

Two steps are required to build and run the demo:

1. Build latest docker image in main repository directory `make docker-otelcontribcol`
1. Switch to this directory and run `docker-compose up`
tigrannajaryan marked this conversation as resolved.
Show resolved Hide resolved

#### Description

`varlogpods` contains example log files placed in kubernetes-like directory structure.
Each of the directory has different formatted logs in one of three formats (either `CRI-O`, `CRI-Containerd` or `Docker`).
This directory is mounted to standard location (`/var/log/pods`).

`otel-collector-config` is a configuration to autodetect and parse logs for all of three mentioned formats

## Performance Tests

There are multiple tests for various configurations in [`testbed`](../../testbed/tests/log_test.go).
Following table shows result of example run:

Test |Result|Duration|CPU Avg%|CPU Max%|RAM Avg MiB|RAM Max MiB|Sent Items|Received Items|
----------------------------------------|------|-------:|-------:|-------:|----------:|----------:|---------:|-------------:|
Log10kDPS/OTLP |PASS | 15s| 15.2| 15.7| 69| 73| 149900| 149900|
Log10kDPS/filelog |PASS | 15s| 16.5| 18.0| 61| 74| 150000| 150000|
Log10kDPS/kubernetes_containers |PASS | 15s| 42.3| 44.0| 66| 80| 150000| 150000|
Log10kDPS/k8s_CRI-Containerd |PASS | 15s| 36.7| 38.0| 64| 78| 150000| 150000|
Log10kDPS/k8s_CRI-Containerd_no_attr_ops|PASS | 15s| 28.9| 29.7| 64| 77| 150000| 150000|
Log10kDPS/CRI-Containerd |PASS | 15s| 19.0| 21.0| 63| 77| 150000| 150000|

## ToDo

To cover kubernetes system logs, logs from journald should be supported as well.
8 changes: 8 additions & 0 deletions examples/kubernetes/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
version: "3"
services:
opentelemetry-collector-contrib:
image: otelcontribcol
command: ["--config=/etc/otel-collector-config.yml"]
volumes:
- ./otel-collector-config.yml:/etc/otel-collector-config.yml
- ./varlogpods:/var/log/pods
78 changes: 78 additions & 0 deletions examples/kubernetes/otel-collector-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
receivers:
filelog:
include:
- /var/log/pods/*/*/*.log
exclude:
# Exclude logs from all containers named otel-collector
- /var/log/pods/*/otel-collector/*.log
start_at: beginning
include_file_path: true
include_file_name: false
operators:
# Find out which format is used by kubernetes
- type: router
id: get-format
routes:
- output: parser-docker
expr: '$$record matches "^\\{"'
- output: parser-crio
expr: '$$record matches "^[^ Z]+ "'
- output: parser-containerd
expr: '$$record matches "^[^ Z]+Z"'
# Parse CRI-O format
- type: regex_parser
id: parser-crio
regex: '^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) (?P<log>.*)$'
output: extract_metadata_from_filepath
timestamp:
parse_from: time
layout_type: gotime
layout: '2006-01-02T15:04:05.000000000-07:00'
# Parse CRI-Containerd format
- type: regex_parser
id: parser-containerd
regex: '^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) (?P<log>.*)$'
output: extract_metadata_from_filepath
timestamp:
parse_from: time
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
# Parse Docker format
- type: json_parser
id: parser-docker
output: extract_metadata_from_filepath
timestamp:
parse_from: time
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
# Extract metadata from file path
- type: regex_parser
id: extract_metadata_from_filepath
regex: '^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]{36})\/(?P<container_name>[^\._]+)\/(?P<run_id>\d+)\.log$'
tigrannajaryan marked this conversation as resolved.
Show resolved Hide resolved
parse_from: $$attributes.file_path
# Move out attributes to Attributes
- type: metadata
attributes:
stream: 'EXPR($.stream)'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to keep stream as an attribute? It seems like is it only marginally useful. If we want to have it we need to come up with some idea for a semantic convention. Otel spec spec does not encourage attribute names like that: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/common/attribute-and-label-naming.md#recommendations-for-opentelemetry-authors

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any semantic convention for file_path, file_name and so on? I think stream should be added there. I consider it rather important. As end user I would like to be able to manage logs differently depending on a stream type

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I don't think we have semantic conventions like that. Can you please open an issue here in this repo to decide what we want to do with this attribute and we can address it in a future PR.

k8s.container.name: 'EXPR($.container_name)'
k8s.namespace.name: 'EXPR($.namespace)'
k8s.pod.name: 'EXPR($.pod_name)'
run_id: 'EXPR($.run_id)'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is run_id? Is it something that requires a semantic convention in Otel spec?

Copy link
Member Author

@sumo-drosiek sumo-drosiek Mar 9, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if its really needed. It defines from which run of a pod logs are coming. In case pod is restarted the run_id is incremented. It could be sth like k8s.container.run_number, k8s.container.run_id

Even more like container.* as specified in semantic conventions

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please create an issue for this and for stream and add to Basic Logs milestone in this repo so that we can decide what to do about it.

k8s.pod.uid: 'EXPR($.uid)'
# Clean up log record
- type: restructure
id: clean-up-log-record
ops:
- remove: logtag
- remove: stream
- remove: container_name
- remove: namespace
- remove: pod_name
- remove: run_id
- remove: uid
exporters:
logging:
loglevel: debug
service:
pipelines:
logs:
receivers: [filelog]
exporters: [logging]
152 changes: 152 additions & 0 deletions examples/kubernetes/otel-collector.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
---
tigrannajaryan marked this conversation as resolved.
Show resolved Hide resolved
apiVersion: v1
kind: ConfigMap
metadata:
name: otel-collector-config
data:
config.yaml: |-
receivers:
filelog:
include:
- /var/log/pods/*/*/*.log
exclude:
# Exclude logs from all containers named otel-collector
- /var/log/pods/*/otel-collector/*.log
tigrannajaryan marked this conversation as resolved.
Show resolved Hide resolved
start_at: beginning
include_file_path: true
include_file_name: false
operators:
# Find out which format is used by kubernetes
- type: router
id: get-format
routes:
- output: parser-docker
expr: '$$record matches "^\\{"'
- output: parser-crio
expr: '$$record matches "^[^ Z]+ "'
- output: parser-containerd
expr: '$$record matches "^[^ Z]+Z"'
# Parse CRI-O format
- type: regex_parser
id: parser-crio
regex: '^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) (?P<log>.*)$'
output: extract_metadata_from_filepath
timestamp:
parse_from: time
layout_type: gotime
layout: '2006-01-02T15:04:05.000000000-07:00'
# Parse CRI-Containerd format
- type: regex_parser
id: parser-containerd
regex: '^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) (?P<log>.*)$'
output: extract_metadata_from_filepath
timestamp:
parse_from: time
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
# Parse Docker format
- type: json_parser
id: parser-docker
output: extract_metadata_from_filepath
timestamp:
parse_from: time
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
# Extract metadata from file path
- type: regex_parser
id: extract_metadata_from_filepath
regex: '^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]{36})\/(?P<container_name>[^\._]+)\/(?P<run_id>\d+)\.log$'
parse_from: $$attributes.file_path
# Move out attributes to Attributes
- type: metadata
attributes:
stream: 'EXPR($.stream)'
k8s.container.name: 'EXPR($.container_name)'
k8s.namespace.name: 'EXPR($.namespace)'
k8s.pod.name: 'EXPR($.pod_name)'
run_id: 'EXPR($.run_id)'
k8s.pod.uid: 'EXPR($.uid)'
# Clean up log record
- type: restructure
id: clean-up-log-record
ops:
- remove: logtag
- remove: stream
- remove: container_name
- remove: namespace
- remove: pod_name
- remove: run_id
- remove: uid
exporters:
logging:
loglevel: debug
service:
pipelines:
logs:
receivers: [filelog]
exporters: [logging]

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: otel-collector
labels:
app: opentelemetry
component: otel-collector
spec:
selector:
matchLabels:
app: opentelemetry
component: otel-collector
template:
metadata:
labels:
app: opentelemetry
component: otel-collector
spec:
containers:
- name: otel-collector
image: otel/opentelemetry-collector-contrib:0.22.0
resources:
limits:
cpu: 100m
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- mountPath: /var/log
name: varlog
readOnly: true
- mountPath: /var/lib/docker/containers
name: varlibdockercontainers
readOnly: true
- mountPath: /etc/otel/config.yaml
name: data
subPath: config.yaml
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: data
configMap:
name: otel-collector-config

---
apiVersion: v1
kind: Service
metadata:
name: otel-collector
labels:
app: opentelemetry
component: otel-collector
spec:
ports:
- name: metrics # Default endpoint for querying metrics.
port: 8888
selector:
component: otel-collector
7 changes: 7 additions & 0 deletions ...s/kubernetes/varlogpods/containerd_logs-0_000011112222333344445555666677778888/logs/0.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
2021-02-16T09:21:15.518430714Z stdout F example: 10 Tue Feb 16 09:21:15 UTC 2021
2021-02-16T09:21:16.519721603Z stdout F example: 11 Tue Feb 16 09:21:16 UTC 2021
2021-02-16T09:21:17.521307236Z stdout F example: 12 Tue Feb 16 09:21:17 UTC 2021
2021-02-16T09:21:18.522445945Z stdout F example: 13 (part of log) Tue Feb 16 09:21:18 UTC 2021
2021-02-16T09:21:18.522445945Z stdout F example: 13 (end of log) Tue Feb 16 09:21:18 UTC 2021
2021-02-16T09:21:19.523759881Z stdout F example: 14 Tue Feb 16 09:21:19 UTC 2021
2021-02-16T09:21:20.545525544Z stdout F example: 15 Tue Feb 16 09:21:20 UTC 2021
6 changes: 6 additions & 0 deletions examples/kubernetes/varlogpods/crio_logs-0_111122223333444455556666777788889999/logs/0.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
2021-02-16T08:59:31.252009327+00:00 stdout F example: 11 Tue Feb 16 08:59:31 UTC 2021
2021-02-16T08:59:32.253038336+00:00 stdout F example: 12 Tue Feb 16 08:59:32 UTC 2021
2021-02-16T08:59:33.254364766+00:00 stdout P example: 13 (part of log) Tue Feb 16 08:59:33 UTC 2021
2021-02-16T08:59:33.254364766+00:00 stdout F example: 13 (end of log) Tue Feb 16 08:59:33 UTC 2021
2021-02-16T08:59:34.255644946+00:00 stdout F example: 14 Tue Feb 16 08:59:34 UTC 2021
2021-02-16T08:59:35.261801788+00:00 stdout F example: 15 Tue Feb 16 08:59:35 UTC 2021
7 changes: 7 additions & 0 deletions examples/kubernetes/varlogpods/docker_logs-0_222233334444555566667777888899990000/logs/0.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{"log":"example: 12 Tue Feb 16 09:15:12 UTC 2021\n","stream":"stdout","time":"2021-02-16T09:15:12.50286486Z"}
{"log":"example: 13 (part of log) Tue Feb 16 09:15:13 UTC 2021","stream":"stdout","time":"2021-02-16T09:15:13.503624173Z"}
{"log":"example: 13 (end of log) Tue Feb 16 09:15:13 UTC 2021\n","stream":"stdout","time":"2021-02-16T09:15:13.503624173Z"}
{"log":"example: 14 Tue Feb 16 09:15:14 UTC 2021\n","stream":"stdout","time":"2021-02-16T09:15:14.504446654Z"}
{"log":"example: 15 Tue Feb 16 09:15:15 UTC 2021\n","stream":"stdout","time":"2021-02-16T09:15:15.505407523Z"}
{"log":"example: 16 Tue Feb 16 09:15:16 UTC 2021\n","stream":"stdout","time":"2021-02-16T09:15:16.51048724Z"}
{"log":"example: 17 Tue Feb 16 09:15:17 UTC 2021\n","stream":"stdout","time":"2021-02-16T09:15:17.511829776Z"}
7 changes: 7 additions & 0 deletions ...kubernetes/varlogpods/otel_otel_888877776666555544443333222211110000/otel-collector/0.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
2021-02-16T09:21:15.518430714Z stdout F otel-collector: 10 Tue Feb 16 09:21:15 UTC 2021
2021-02-16T09:21:16.519721603Z stdout F otel-collector: 11 Tue Feb 16 09:21:16 UTC 2021
2021-02-16T09:21:17.521307236Z stdout F otel-collector: 12 Tue Feb 16 09:21:17 UTC 2021
2021-02-16T09:21:18.522445945Z stdout F otel-collector: 13 (part of log) Tue Feb 16 09:21:18 UTC 2021
2021-02-16T09:21:18.522445945Z stdout F otel-collector: 13 (end of log) Tue Feb 16 09:21:18 UTC 2021
2021-02-16T09:21:19.523759881Z stdout F otel-collector: 14 Tue Feb 16 09:21:19 UTC 2021
2021-02-16T09:21:20.545525544Z stdout F otel-collector: 15 Tue Feb 16 09:21:20 UTC 2021