-
Notifications
You must be signed in to change notification settings - Fork 23
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
38 changed files
with
2,804 additions
and
706 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| ## `azure_event_hub_input` operator | ||
|
|
||
| The `azure_event_hub_input` operator reads logs from Azure Event Hub using [Azure's SDK](https://github.com/Azure/azure-event-hubs-go) | ||
|
|
||
| The `azure_event_hub_input` operator will use the `EnqueuedTime` field of the event as the parsed entry's timestamp. If `EnqueuedTime` is not set, `azure_event_hub_input` will use `IoTHubEnqueuedTime` if it is set. All other fields are added to the entry's record. | ||
|
|
||
| ### Configuration Fields | ||
|
|
||
| | Field | Default | Description | | ||
| | --- | --- | --- | | ||
| | `id` | `azure_event_hub_input` | A unique identifier for the operator | | ||
| | `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | | ||
| | `namespace` | required | The Event Hub Namespace | | ||
| | `name` | required | The Event Hub Name | | ||
| | `group` | required | The Event Hub Consumer Group | | ||
| | `connection_string` | required | The Event Hub [connection string](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string) | | ||
| | `prefetch_count` | `1000` | Desired number of events to read at one time | | ||
| | `start_at` | `end` | At startup, where to start reading events. Options are `beginning` or `end` | | ||
|
|
||
| ### Example Configurations | ||
|
|
||
| #### Simple Azure Event Hub input | ||
|
|
||
| Configuration: | ||
| ```yaml | ||
| pipeline: | ||
| - type: azure_event_hub_input | ||
| namespace: stanza | ||
| name: devel | ||
| group: Default | ||
| connection_string: 'Endpoint=sb://stanza.servicebus.windows.net/;SharedAccessKeyName=dev;SharedAccessKey=supersecretkey;EntityPath=devel' | ||
| start_at: end | ||
| ``` | ||
| ### Example Output | ||
| A list of potential keys and their purpose can be found [here](https://github.com/Azure/azure-event-hubs-go/blob/master/event.go). Event Hub `system_properties` documentation can be found [here](https://docs.microsoft.com/en-us/azure/data-explorer/ingest-data-event-hub-overview#event-system-properties-mapping) | ||
|
|
||
| ```json | ||
| { | ||
| "timestamp": "2021-04-19T18:44:34.619Z", | ||
| "severity": 0, | ||
| "resource": { | ||
| "event_id": "fea3c182-00a6-4951-8f6f-9331031f978f" | ||
| }, | ||
| "record": { | ||
| "message": "hello, world!", | ||
| "system_properties": { | ||
| "x-opt-enqueued-time": "2021-04-19T18:44:34.619Z", | ||
| "x-opt-offset": 6120, | ||
| "x-opt-sequence-number": 51 | ||
| } | ||
| } | ||
| } | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| ## `azure_log_analytics_input` operator | ||
|
|
||
| The `azure_log_analytics_input` operator reads Azure Log Analytics logs from Azure Event Hub using. | ||
|
|
||
| The `azure_log_analytics_input` operator will use the `timegenerated` field as the parsed entry's timestamp. The label `azure_log_analytics_table` is derived from the log's `type` field. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| You must define a Log Analytics Export Rule using Azure CLI. Microsoft has documentation [here](https://docs.microsoft.com/en-us/azure/azure-monitor/logs/logs-data-export?tabs=portal) | ||
|
|
||
| ### Configuration Fields | ||
|
|
||
| | Field | Default | Description | | ||
| | --- | --- | --- | | ||
| | `id` | `azure_log_analytics_input` | A unique identifier for the operator | | ||
| | `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | | ||
| | `namespace` | required | The Event Hub Namespace | | ||
| | `name` | required | The Event Hub Name | | ||
| | `group` | required | The Event Hub Consumer Group | | ||
| | `connection_string` | required | The Event Hub [connection string](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string) | | ||
| | `prefetch_count` | `1000` | Desired number of events to read at one time | | ||
| | `start_at` | `end` | At startup, where to start reading events. Options are `beginning` or `end` | | ||
|
|
||
| ### Example Configurations | ||
|
|
||
| #### Simple Azure Event Hub input | ||
|
|
||
| Configuration: | ||
| ```yaml | ||
| pipeline: | ||
| - type: azure_log_analytics_input | ||
| namespace: stanza | ||
| name: devel | ||
| group: Default | ||
| connection_string: 'Endpoint=sb://stanza.servicebus.windows.net/;SharedAccessKeyName=dev;SharedAccessKey=supersecretkey;EntityPath=devel' | ||
| start_at: end | ||
| ``` | ||
| ### Example Output | ||
| A list of potential fields for each Azure Log Analytics table can be found [here](https://docs.microsoft.com/en-us/azure/azure-monitor/reference/tables/tables-category). | ||
| ```json | ||
| { | ||
| "timestamp": "2021-05-07T14:01:26.105Z", | ||
| "severity": 0, | ||
| "record": { | ||
| "containerlog": { | ||
| "_internal_workspaceresourceid": "/subscriptions/000-000/resourcegroups/integration/providers/microsoft.operationalinsights/workspaces/stanza", | ||
| "_resourceid": "/subscriptions/0000-000/resourceGroups/devops/providers/Microsoft.ContainerService/managedClusters/log-analytics", | ||
| "computer": "aks-agentpool-39365618-vmss000001", | ||
| "containerid": "f5376c6972ac19630113736e7d3bf359fe67065fde3831b0502cfee33470e68f", | ||
| "logentry": "request to api failed" | ||
| "logentrysource": "stdout", | ||
| "mg": "00000000-0000-0000-0000-000000000002", | ||
| "sourcesystem": "Containers", | ||
| "tenantid": "ae0db88b-40bb-40b7-b056-57980214436c", | ||
| "timegenerated": "2021-05-07T14:01:26.1050000Z", | ||
| "timeofcommand": "2021-05-07T14:01:29.0000000Z" | ||
| }, | ||
| "system_properties": { | ||
| "x-opt-enqueued-time": "2021-05-07T14:01:37.789Z", | ||
| "x-opt-offset": 150347296000, | ||
| "x-opt-sequence-number": 125576 | ||
| } | ||
| } | ||
| } | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,116 @@ | ||
| ## `flatten` operator | ||
|
|
||
| The `flatten` operator flattens a field by moving its children up to the same level as the field. | ||
| The operator only flattens a single level deep. | ||
|
|
||
| ### Configuration Fields | ||
|
|
||
| | Field | Default | Description | | ||
| | --- | --- | --- | | ||
| | `id` | `flatten` | A unique identifier for the operator | | ||
| | `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | | ||
| | `field` | required | The [field](/docs/types/field.md) to be flattened. | | ||
| | `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | | ||
| | `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. | | ||
|
|
||
| Example usage: | ||
|
|
||
| <hr> | ||
| Flatten an object to the base of the record | ||
| <br> | ||
| <br> | ||
|
|
||
| ```yaml | ||
| - type: flatten | ||
| field: key1 | ||
| ``` | ||
| <table> | ||
| <tr><td> Input Entry </td> <td> Output Entry </td></tr> | ||
| <tr> | ||
| <td> | ||
| ```json | ||
| { | ||
| "resource": { }, | ||
| "labels": { }, | ||
| "record": { | ||
| "key1": { | ||
| "nested1": "nestedval1", | ||
| "nested2": "nestedval2" | ||
| }, | ||
| "key2": "val2" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| </td> | ||
| <td> | ||
|
|
||
| ```json | ||
| { | ||
| "resource": { }, | ||
| "labels": { }, | ||
| "record": { | ||
| "nested1": "nestedval1", | ||
| "nested2": "nestedval2", | ||
| "key2": "val2" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| </td> | ||
| </tr> | ||
| </table> | ||
|
|
||
| <hr> | ||
| Flatten an object within another object | ||
| <br> | ||
| <br> | ||
|
|
||
| ```yaml | ||
| - type: flatten | ||
| field: wrapper.key1 | ||
| ``` | ||
| <table> | ||
| <tr><td> Input Entry </td> <td> Output Entry </td></tr> | ||
| <tr> | ||
| <td> | ||
| ```json | ||
| { | ||
| "resource": { }, | ||
| "labels": { }, | ||
| "record": { | ||
| "wrapper": { | ||
| "key1": { | ||
| "nested1": "nestedval1", | ||
| "nested2": "nestedval2" | ||
| }, | ||
| "key2": "val2" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| </td> | ||
| <td> | ||
|
|
||
| ```json | ||
| { | ||
| "resource": { }, | ||
| "labels": { }, | ||
| "record": { | ||
| "wrapper": { | ||
| "nested1": "nestedval1", | ||
| "nested2": "nestedval2", | ||
| "key2": "val2" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| </td> | ||
| </tr> | ||
| </table> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,28 +1,31 @@ | ||
| module github.com/observiq/stanza | ||
|
|
||
| go 1.14 | ||
| go 1.14 | ||
|
|
||
| require ( | ||
| github.com/antonmedv/expr v1.8.9 | ||
| github.com/Azure/azure-event-hubs-go/v3 v3.3.7 | ||
| github.com/antonmedv/expr v1.8.2 | ||
| github.com/bmatcuk/doublestar/v2 v2.0.4 | ||
| github.com/cenkalti/backoff/v4 v4.1.0 | ||
| github.com/cenkalti/backoff/v4 v4.0.2 | ||
| github.com/json-iterator/go v1.1.10 | ||
| github.com/kr/text v0.2.0 // indirect | ||
| github.com/mitchellh/mapstructure v1.4.1 | ||
| github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect | ||
| github.com/modern-go/reflect2 v1.0.1 // indirect | ||
| github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e // indirect | ||
| github.com/observiq/ctimefmt v1.0.0 | ||
| github.com/observiq/nanojack v0.0.0-20201106172433-343928847ebc | ||
| github.com/stretchr/objx v0.3.0 // indirect | ||
| github.com/stretchr/testify v1.7.0 | ||
| go.etcd.io/bbolt v1.3.5 | ||
| go.uber.org/zap v1.16.0 | ||
| golang.org/x/exp v0.0.0-20210417010653-0739314eea07 // indirect | ||
| golang.org/x/lint v0.0.0-20201208152925-83fdc39ff7b5 // indirect | ||
| golang.org/x/sync v0.0.0-20210220032951-036812b2e83c | ||
| golang.org/x/sys v0.0.0-20210423082822-04245dca01da // indirect | ||
| golang.org/x/text v0.3.6 | ||
| gonum.org/v1/gonum v0.9.1 | ||
| gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect | ||
| gopkg.in/yaml.v2 v2.4.0 | ||
| honnef.co/go/tools v0.1.3 // indirect | ||
| github.com/stretchr/testify v1.6.1 | ||
| go.etcd.io/bbolt v1.3.4 | ||
| go.uber.org/zap v1.15.0 | ||
| golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6 // indirect | ||
| golang.org/x/lint v0.0.0-20200302205851-738671d3881b // indirect | ||
| golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208 | ||
| golang.org/x/sys v0.0.0-20201015000850-e3ed0017c211 // indirect | ||
| golang.org/x/text v0.3.3 | ||
| golang.org/x/tools v0.0.0-20200904185747-39188db58858 // indirect | ||
| gonum.org/v1/gonum v0.6.2 | ||
| gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f // indirect | ||
| gopkg.in/yaml.v2 v2.3.0 | ||
| honnef.co/go/tools v0.0.1-2020.1.5 // indirect | ||
| ) |
Oops, something went wrong.