-
Notifications
You must be signed in to change notification settings - Fork 460
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Enhance traefik integration to also handle JSON-formatted access logs (…
…#770) * Migrating traefik module * Formatting package files * Removing invalid path field * Adding categories * Formatting tweaks * Adding pipeline test files * Adding YAML header * Adding system tests * Renaming pipeline test case files * Fixing pipeline tests * Adding sample event for health data set * Adding system test for access data stream * Adding README * Starting to handle JSON formatted logs * Adding ARG to Dockerfile for log format * Adding sample JSON logs * Running elastic-package format * Removing host field from sample event * Fix docker compose file * Splitting in commonlog and json format pipelines * Making pipeline test pass * Updating README.md * Address TODOs in pipeline * Specify services in system test configs * Refactoring out common processors into common pipeline * Add @timestamp field * Adding service to health data stream system test * Adding CHANGELOG entries * Parsing out event.duration * Regenerating sample events * Updating README * Add allow_duplicates: false for related.* fields' append processors * Adding community_id processor * Populating url.domain * Set allow_duplicates: false for other append processors * Regenerating README
- Loading branch information
1 parent
5c07017
commit c83a14c
Showing
24 changed files
with
762 additions
and
314 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,8 @@ | ||
| ARG TRAEFIK_VERSION | ||
| FROM traefik:${TRAEFIK_VERSION}-alpine | ||
|
|
||
| COPY ./traefik.toml /etc/traefik/traefik.toml | ||
| ARG TRAEFIK_LOG_FORMAT | ||
| COPY ./traefik_format_${TRAEFIK_LOG_FORMAT}.toml /etc/traefik/traefik.toml | ||
|
|
||
| RUN apk add --no-cache curl | ||
| HEALTHCHECK --interval=1s --retries=90 CMD curl --header 'Host:backend.elastic-package-service.docker.localhost' 'http://localhost:80/' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 1 addition & 1 deletion
2
...s/traefik/_dev/deploy/docker/traefik.toml → .../deploy/docker/traefik_format_common.toml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,5 @@ | ||
| [accessLog] | ||
| filePath = "/var/log/access.log" | ||
| filePath = "/var/log/access-common.log" | ||
|
|
||
| [api] | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| [accessLog] | ||
| filePath = "/var/log/access-json.log" | ||
| format = "json" | ||
|
|
||
| [api] | ||
|
|
||
| # Docker configuration backend | ||
| [docker] | ||
| domain = "docker.localhost" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 2 additions & 0 deletions
2
packages/traefik/data_stream/access/_dev/test/pipeline/test-format-json.log
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| {"BackendAddr":"","BackendName":"Traefik","BackendURL":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"127.0.0.1:48658","ClientHost":"127.0.0.1","ClientPort":"48658","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":40356,"FrontendName":"backend not found","OriginContentSize":19,"OriginDuration":4086,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":36270,"RequestAddr":"backend.elastic-package-service.docker.localhost","RequestContentSize":0,"RequestCount":7,"RequestHost":"backend.elastic-package-service.docker.localhost","RequestLine":"GET / HTTP/1.1","RequestMethod":"GET","RequestPath":"/","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-16T18:56:54.735539596Z","StartUTC":"2021-03-16T18:56:54.735539596Z","downstream_Content-Type":"text/plain; charset=utf-8","downstream_X-Content-Type-Options":"nosniff","level":"info","msg":"","origin_Content-Type":"text/plain; charset=utf-8","origin_X-Content-Type-Options":"nosniff","request_Accept":"*/*","request_User-Agent":"curl/7.67.0","time":"2021-03-16T18:56:54Z"} | ||
| {"BackendAddr":"172.21.0.2:80","BackendName":"backend-backend-docker","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.21.0.2:80","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.21.0.1:59068","ClientHost":"172.21.0.1","ClientPort":"59068","ClientUsername":"-","DownstreamContentSize":383,"DownstreamStatus":200,"DownstreamStatusLine":"200 OK","Duration":3034764,"FrontendName":"Host-backend-docker-docker-localhost-2","OriginContentSize":383,"OriginDuration":2155389,"OriginStatus":200,"OriginStatusLine":"200 OK","Overhead":879375,"RequestAddr":"backend.docker.docker.localhost","RequestContentSize":0,"RequestCount":27,"RequestHost":"backend.docker.docker.localhost","RequestLine":"GET / HTTP/1.1","RequestMethod":"GET","RequestPath":"/","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-16T19:08:41.039598834Z","StartUTC":"2021-03-16T19:08:41.039598834Z","downstream_Content-Length":"383","downstream_Content-Type":"text/plain; charset=utf-8","downstream_Date":"Tue, 16 Mar 2021 19:08:41 GMT","level":"info","msg":"","origin_Content-Length":"383","origin_Content-Type":"text/plain; charset=utf-8","origin_Date":"Tue, 16 Mar 2021 19:08:41 GMT","request_Accept":"*/*","request_User-Agent":"curl/7.64.1","time":"2021-03-16T19:08:41Z"} |
4 changes: 4 additions & 0 deletions
4
packages/traefik/data_stream/access/_dev/test/pipeline/test-format-json.log-config.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| fields: | ||
| "@timestamp": "2020-04-28T11:07:58.223Z" | ||
| dynamic_fields: | ||
| event.ingested: ".*" |
137 changes: 137 additions & 0 deletions
137
packages/traefik/data_stream/access/_dev/test/pipeline/test-format-json.log-expected.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,137 @@ | ||
| { | ||
| "expected": [ | ||
| { | ||
| "traefik": { | ||
| "access": { | ||
| "frontend_name": "backend not found", | ||
| "backend_url": "", | ||
| "request_count": 7 | ||
| } | ||
| }, | ||
| "source": { | ||
| "port": 48658, | ||
| "address": "127.0.0.1", | ||
| "ip": "127.0.0.1" | ||
| }, | ||
| "url": { | ||
| "original": "/", | ||
| "domain": "backend.elastic-package-service.docker.localhost" | ||
| }, | ||
| "network": { | ||
| "transport": "tcp" | ||
| }, | ||
| "@timestamp": "2021-03-16T18:56:54Z", | ||
| "related": { | ||
| "ip": [ | ||
| "127.0.0.1" | ||
| ] | ||
| }, | ||
| "http": { | ||
| "request": { | ||
| "method": "GET" | ||
| }, | ||
| "version": "1.1", | ||
| "response": { | ||
| "body": { | ||
| "bytes": 19 | ||
| }, | ||
| "status_code": 404 | ||
| } | ||
| }, | ||
| "event": { | ||
| "duration": 40356, | ||
| "ingested": "2021-03-23T00:36:56.518177200Z", | ||
| "created": "2020-04-28T11:07:58.223Z", | ||
| "kind": "event", | ||
| "category": [ | ||
| "web" | ||
| ], | ||
| "type": [ | ||
| "access" | ||
| ], | ||
| "outcome": "failure" | ||
| }, | ||
| "user": { | ||
| "name": "-" | ||
| }, | ||
| "user_agent": { | ||
| "name": "curl", | ||
| "original": "curl/7.67.0", | ||
| "device": { | ||
| "name": "Other" | ||
| }, | ||
| "version": "7.67.0" | ||
| } | ||
| }, | ||
| { | ||
| "traefik": { | ||
| "access": { | ||
| "frontend_name": "Host-backend-docker-docker-localhost-2", | ||
| "backend_url": "172.21.0.2:80", | ||
| "request_count": 27 | ||
| } | ||
| }, | ||
| "destination": { | ||
| "port": 80, | ||
| "address": "172.21.0.2", | ||
| "ip": "172.21.0.2" | ||
| }, | ||
| "source": { | ||
| "port": 59068, | ||
| "address": "172.21.0.1", | ||
| "ip": "172.21.0.1" | ||
| }, | ||
| "url": { | ||
| "original": "/", | ||
| "domain": "backend.docker.docker.localhost" | ||
| }, | ||
| "network": { | ||
| "community_id": "1:DJlJOSbrvisPNQtgBIyBaYAwlz8=", | ||
| "transport": "tcp" | ||
| }, | ||
| "@timestamp": "2021-03-16T19:08:41Z", | ||
| "related": { | ||
| "ip": [ | ||
| "172.21.0.1", | ||
| "172.21.0.2" | ||
| ] | ||
| }, | ||
| "http": { | ||
| "request": { | ||
| "method": "GET" | ||
| }, | ||
| "version": "1.1", | ||
| "response": { | ||
| "body": { | ||
| "bytes": 383 | ||
| }, | ||
| "status_code": 200 | ||
| } | ||
| }, | ||
| "event": { | ||
| "duration": 3034764, | ||
| "ingested": "2021-03-23T00:36:56.518189Z", | ||
| "created": "2020-04-28T11:07:58.223Z", | ||
| "kind": "event", | ||
| "category": [ | ||
| "web" | ||
| ], | ||
| "type": [ | ||
| "access" | ||
| ], | ||
| "outcome": "success" | ||
| }, | ||
| "user": { | ||
| "name": "-" | ||
| }, | ||
| "user_agent": { | ||
| "name": "curl", | ||
| "original": "curl/7.64.1", | ||
| "device": { | ||
| "name": "Other" | ||
| }, | ||
| "version": "7.64.1" | ||
| } | ||
| } | ||
| ] | ||
| } |
5 changes: 0 additions & 5 deletions
5
packages/traefik/data_stream/access/_dev/test/system/test-default-config.yml
This file was deleted.
Oops, something went wrong.
6 changes: 6 additions & 0 deletions
6
packages/traefik/data_stream/access/_dev/test/system/test-format-common-config.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| service: traefik_format_common | ||
| vars: ~ | ||
| data_stream: | ||
| vars: | ||
| paths: | ||
| - "{{SERVICE_LOGS_DIR}}/access-common.log" |
6 changes: 6 additions & 0 deletions
6
packages/traefik/data_stream/access/_dev/test/system/test-format-json-config.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| service: traefik_format_json | ||
| vars: ~ | ||
| data_stream: | ||
| vars: | ||
| paths: | ||
| - "{{SERVICE_LOGS_DIR}}/access-json.log" |
Oops, something went wrong.