Skip to content

Commit

Permalink
[pkg/stanza] container: move k8s metadata into resource attributes (#…
Browse files Browse the repository at this point in the history 
…33353)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
Container parser should add k8s metadata as resource attributes and not
as log record attributes.

**Link to tracking Issue:** <Issue number if applicable> Fixes
#33341

**Testing:** <Describe what testing was performed and which tests were
added.>
Manual testing on local k8s cluster:

```console
2024-06-04T06:40:08.219Z	info	ResourceLog #0
Resource SchemaURL: 
Resource attributes:
     -> k8s.pod.uid: Str(d5ecc924-e255-4525-b5be-6437939b1e4d)
     -> k8s.container.name: Str(busybox)
     -> k8s.namespace.name: Str(default)
     -> k8s.pod.name: Str(daemonset-logs-dhzcq)
     -> k8s.container.restart_count: Str(0)
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 2024-06-04 06:40:08.007370503 +0000 UTC
Timestamp: 2024-06-04 06:40:07.855932421 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str(otel logs at 06:40:07)
Attributes:
     -> logtag: Str(F)
     -> key2: Map({"key_in":"val2"})
     -> log.file.path: Str(/var/log/pods/default_daemonset-logs-dhzcq_d5ecc924-e255-4525-b5be-6437939b1e4d/busybox/0.log)
     -> time: Str(2024-06-04T06:40:07.855932421Z)
     -> log.iostream: Str(stdout)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #1
ObservedTimestamp: 2024-06-04 06:40:08.007451031 +0000 UTC
Timestamp: 2024-06-04 06:40:07.957875321 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str(otel logs at 06:40:07)
Attributes:
     -> log.file.path: Str(/var/log/pods/default_daemonset-logs-dhzcq_d5ecc924-e255-4525-b5be-6437939b1e4d/busybox/0.log)
     -> log.iostream: Str(stdout)
     -> time: Str(2024-06-04T06:40:07.957875321Z)
     -> key2: Map({"key_in":"val2"})
     -> logtag: Str(F)
Trace ID: 
Span ID: 
Flags: 0
```

**Documentation:** <Describe the documentation added.> ~

---------

Signed-off-by: ChrsMark <[email protected]>
  • Loading branch information
@ChrsMark
ChrsMark authored Jun 4, 2024
1 parent 982c728 commit 6395ff8
Show file tree
Hide file tree
Showing 4 changed files with 104 additions and 54 deletions.
28 changes: 28 additions & 0 deletions .chloggen/container_parser_resource_attributes.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: bug_fix

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: filelogreceiver

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: "Container parser should add k8s metadata as resource attributes and not as log record attributes"


# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [33341]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: []
74 changes: 43 additions & 31 deletions pkg/stanza/docs/operators/container.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,17 +31,13 @@ will produce the following k8s metadata:

```json
{
"attributes": {
"k8s": {
"container": {
"name": "kube-controller",
"restart_count": "1"
}, "pod": {
"uid": "49cc7c1fd3702c40b2686ea7486091d6",
"name": "kube-controller-kind-control-plane"
}, "namespace": {
"name": "some-ns"
}
"resource": {
"attributes": {
"k8s.pod.name": "kube-controller-kind-control-plane",
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
"k8s.container.name": "kube-controller",
"k8s.container.restart_count": "1",
"k8s.namespace.name": "some"
}
}
}
Expand Down Expand Up @@ -84,12 +80,16 @@ Note: in this example the `format: docker` is optional since formats can be auto
"attributes": {
"time": "2024-03-30T08:31:20.545192187Z",
"log.iostream": "stdout",
"k8s.pod.name": "kube-controller-kind-control-plane",
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
"k8s.container.name": "kube-controller",
"k8s.container.restart_count": "1",
"k8s.namespace.name": "some",
"log.file.path": "/var/log/pods/some_kube-controller-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d6/kube-controller/1.log"
},
"resource": {
"attributes": {
"k8s.pod.name": "kube-controller-kind-control-plane",
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
"k8s.container.name": "kube-controller",
"k8s.container.restart_count": "1",
"k8s.namespace.name": "some"
}
}
}
```
Expand Down Expand Up @@ -129,12 +129,16 @@ Configuration:
"time": "2024-04-13T07:59:37.505201169-05:00",
"logtag": "F",
"log.iostream": "stdout",
"k8s.pod.name": "kube-controller-kind-control-plane",
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
"k8s.container.name": "kube-controller",
"k8s.container.restart_count": "1",
"k8s.namespace.name": "some",
"log.file.path": "/var/log/pods/some_kube-controller-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d6/kube-controller/1.log"
},
"resource": {
"attributes": {
"k8s.pod.name": "kube-controller-kind-control-plane",
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
"k8s.container.name": "kube-controller",
"k8s.container.restart_count": "1",
"k8s.namespace.name": "some"
}
}
}
```
Expand Down Expand Up @@ -174,12 +178,16 @@ Configuration:
"time": "2023-06-22T10:27:25.813799277Z",
"logtag": "F",
"log.iostream": "stdout",
"k8s.pod.name": "kube-controller-kind-control-plane",
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
"k8s.container.name": "kube-controller",
"k8s.container.restart_count": "1",
"k8s.namespace.name": "some",
"log.file.path": "/var/log/pods/some_kube-controller-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d6/kube-controller/1.log"
},
"resource": {
"attributes": {
"k8s.pod.name": "kube-controller-kind-control-plane",
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
"k8s.container.name": "kube-controller",
"k8s.container.restart_count": "1",
"k8s.namespace.name": "some"
}
}
}
```
Expand Down Expand Up @@ -227,12 +235,16 @@ Configuration:
"time": "2023-06-22T10:27:25.813799277Z",
"logtag": "F",
"log.iostream": "stdout",
"k8s.pod.name": "kube-controller-kind-control-plane",
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
"k8s.container.name": "kube-controller",
"k8s.container.restart_count": "1",
"k8s.namespace.name": "some",
"log.file.path": "/var/log/pods/some_kube-controller-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d6/kube-controller/1.log"
},
"resource": {
"attributes": {
"k8s.pod.name": "kube-controller-kind-control-plane",
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
"k8s.container.name": "kube-controller",
"k8s.container.restart_count": "1",
"k8s.namespace.name": "some"
}
}
}
```
Expand Down
2 changes: 1 addition & 1 deletion pkg/stanza/operator/parser/container/parser.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -300,7 +300,7 @@ func (p *Parser) extractk8sMetaFromFilePath(e *entry.Entry) error {
}

for originalKey, attributeKey := range k8sMetadataMapping {
newField := entry.NewAttributeField(attributeKey)
newField := entry.NewResourceField(attributeKey)
if err := newField.Set(e, parsedValues[originalKey]); err != nil {
return fmt.Errorf("failed to set %v as metadata at %v", originalKey, attributeKey)
}
Expand Down
54 changes: 32 additions & 22 deletions pkg/stanza/operator/parser/container/parser_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -157,16 +157,18 @@ func TestProcess(t *testing.T) {
},
&entry.Entry{
Attributes: map[string]any{
"time": "2029-03-30T08:31:20.545192187Z",
"log.iostream": "stdout",
"time": "2029-03-30T08:31:20.545192187Z",
"log.iostream": "stdout",
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
},
Body: "INFO: log line here",
Resource: map[string]any{
"k8s.pod.name": "kube-scheduler-kind-control-plane",
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d3",
"k8s.container.name": "kube-scheduler44",
"k8s.container.restart_count": "1",
"k8s.namespace.name": "some",
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
},
Body: "INFO: log line here",
Timestamp: time.Date(2029, time.March, 30, 8, 31, 20, 545192187, time.UTC),
},
},
Expand Down Expand Up @@ -212,17 +214,19 @@ func TestRecombineProcess(t *testing.T) {
[]*entry.Entry{
{
Attributes: map[string]any{
"time": "2024-04-13T07:59:37.505201169-10:00",
"log.iostream": "stdout",
"logtag": "F",
"time": "2024-04-13T07:59:37.505201169-10:00",
"log.iostream": "stdout",
"logtag": "F",
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
},
Body: "standalone crio line which is awesome!",
Resource: map[string]any{
"k8s.pod.name": "kube-scheduler-kind-control-plane",
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d3",
"k8s.container.name": "kube-scheduler44",
"k8s.container.restart_count": "1",
"k8s.namespace.name": "some",
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
},
Body: "standalone crio line which is awesome!",
Timestamp: time.Date(2024, time.April, 13, 7, 59, 37, 505201169, time.FixedZone("", -10*60*60)),
},
},
Expand All @@ -246,15 +250,17 @@ func TestRecombineProcess(t *testing.T) {
[]*entry.Entry{
{
Attributes: map[string]any{
"time": "2024-04-13T07:59:37.505201169Z",
"log.iostream": "stdout",
"logtag": "F",
"time": "2024-04-13T07:59:37.505201169Z",
"log.iostream": "stdout",
"logtag": "F",
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
},
Resource: map[string]any{
"k8s.pod.name": "kube-scheduler-kind-control-plane",
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d3",
"k8s.container.name": "kube-scheduler44",
"k8s.container.restart_count": "1",
"k8s.namespace.name": "some",
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
},
Body: "standalone containerd line which is awesome!",
Timestamp: time.Date(2024, time.April, 13, 7, 59, 37, 505201169, time.UTC),
Expand Down Expand Up @@ -286,15 +292,17 @@ func TestRecombineProcess(t *testing.T) {
[]*entry.Entry{
{
Attributes: map[string]any{
"time": "2024-04-13T07:59:37.505201169-10:00",
"log.iostream": "stdout",
"logtag": "P",
"time": "2024-04-13T07:59:37.505201169-10:00",
"log.iostream": "stdout",
"logtag": "P",
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
},
Resource: map[string]any{
"k8s.pod.name": "kube-scheduler-kind-control-plane",
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d3",
"k8s.container.name": "kube-scheduler44",
"k8s.container.restart_count": "1",
"k8s.namespace.name": "some",
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
},
Body: "standalone crio line which is awesome!",
Timestamp: time.Date(2024, time.April, 13, 7, 59, 37, 505201169, time.FixedZone("", -10*60*60)),
Expand Down Expand Up @@ -326,17 +334,19 @@ func TestRecombineProcess(t *testing.T) {
[]*entry.Entry{
{
Attributes: map[string]any{
"time": "2024-04-13T07:59:37.505201169Z",
"log.iostream": "stdout",
"logtag": "P",
"time": "2024-04-13T07:59:37.505201169Z",
"log.iostream": "stdout",
"logtag": "P",
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
},
Body: "standalone containerd line which is awesome!",
Resource: map[string]any{
"k8s.pod.name": "kube-scheduler-kind-control-plane",
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d3",
"k8s.container.name": "kube-scheduler44",
"k8s.container.restart_count": "1",
"k8s.namespace.name": "some",
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
},
Body: "standalone containerd line which is awesome!",
Timestamp: time.Date(2024, time.April, 13, 7, 59, 37, 505201169, time.UTC),
},
},
Expand Down

0 comments on commit 6395ff8

Please sign in to comment.