Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docker logs support to the Elastic Log Driver #19531

Merged
merged 14 commits into from
Jul 9, 2020

Conversation

fearful-symmetry
Copy link
Contributor

@fearful-symmetry fearful-symmetry commented Jun 30, 2020

What does this PR do?

See #19371 and #13990

This PR adds support for the docker logs command to the Elastic Log Driver. This basically re-implements the "default" log driver, using Docker's own libraries, inside the Elastic Log Driver. In theory this should be a simple addition, as we're just gluing existing libraries on top on what we already have.

There's still some oddities going on here:

  • We're getting interesting error messages if you run docker logs -f for an expended period of time without additional log lines: Error grabbing logs: error decoding log message: net/http: request canceled (Client.Timeout exceeded while reading body) This doesn't seem to be something in my library. Looks like it was due to this PR here: add a 10s timeout to the client object. docker/cli#1872

Why is it important?

This is a "must have" from the Cloud folks, and it's good to have in general. Also required for docker certification.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

  1. build and install the plugin via mage buildAndInstall
  2. Checkout the main readme in dockerlogbeat for an example of how to run the plugin against a container
  3. Run the docker logs command against that container with various options, make sure everything works as expected.

@fearful-symmetry fearful-symmetry added enhancement Team:Integrations Label for the Integrations team labels Jun 30, 2020
@fearful-symmetry fearful-symmetry self-assigned this Jun 30, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 30, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 30, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19531 updated]

  • Start Time: 2020-07-09T14:13:45.500+0000

  • Duration: 29 min 39 sec

Test stats 🧪

Test Results
Failed 0
Passed 3
Skipped 0
Total 3

Steps errors

Expand to view the steps failures

  • Name: Install Go 1.14.4
    • Description: .ci/scripts/install-go.sh

    • Duration: 1 min 57 sec

    • Start Time: 2020-07-09T14:38:19.642+0000

    • log

x-pack/dockerlogbeat/handlers.go Outdated Show resolved Hide resolved
x-pack/dockerlogbeat/handlers.go Outdated Show resolved Hide resolved
x-pack/dockerlogbeat/handlers.go Outdated Show resolved Hide resolved
x-pack/dockerlogbeat/handlers.go Show resolved Hide resolved
x-pack/dockerlogbeat/pipelinemanager/pipelineManager.go Outdated Show resolved Hide resolved
@urso
Copy link

urso commented Jul 1, 2020

I had to add in the bufio library, which we re-implemented for reading from the log buffer. Not sure if it's worth re-implementing it as a writer.

The writer should be fine to reuse. We only introduced our own reader to handle reading and framing correctly in the presence of errors.

x-pack/dockerlogbeat/docs/configuration.asciidoc Outdated Show resolved Hide resolved
x-pack/dockerlogbeat/docs/configuration.asciidoc Outdated Show resolved Hide resolved
x-pack/dockerlogbeat/docs/configuration.asciidoc Outdated Show resolved Hide resolved
x-pack/dockerlogbeat/handlers.go Show resolved Hide resolved
x-pack/dockerlogbeat/pipelinemanager/clientLogReader.go Outdated Show resolved Hide resolved
x-pack/dockerlogbeat/pipelinemanager/pipelineManager.go Outdated Show resolved Hide resolved
x-pack/dockerlogbeat/pipelinemanager/pipelineManager.go Outdated Show resolved Hide resolved
x-pack/dockerlogbeat/readme.md Outdated Show resolved Hide resolved
@fearful-symmetry fearful-symmetry requested review from urso and a team July 8, 2020 14:40
x-pack/dockerlogbeat/readme.md Outdated Show resolved Hide resolved
@fearful-symmetry fearful-symmetry merged commit d53cd12 into elastic:master Jul 9, 2020
v1v added a commit to v1v/beats that referenced this pull request Jul 9, 2020
* upstream/master:
  Add `docker logs` support to the Elastic Log Driver (elastic#19531)
  [Elastic Agent] Fix saving of agent configuration on Windows to have proper ACLs (elastic#19793)
  Send the config revision down to the endpoint application. (elastic#19759)
  [Elastic Agent] Add support for multiple hosts in connection to kibana (elastic#19628)
  Remove the downloadConfig and retryConfig from plugin/process.Application and plugin/service.Application. (elastic#19603)
  Update go version to 1.14.4 (elastic#19753)
  ci: set builds as skipped when they do not match the trigger (elastic#19750)
  [Auditbeat] Fix up socket dataset runaway CPU usage (elastic#19764)
  Convert cloudfoundry input to v2 (elastic#19717)
v1v added a commit to v1v/beats that referenced this pull request Jul 9, 2020
* upstream/master:
  Add `docker logs` support to the Elastic Log Driver (elastic#19531)
  [Elastic Agent] Fix saving of agent configuration on Windows to have proper ACLs (elastic#19793)
  Send the config revision down to the endpoint application. (elastic#19759)
  [Elastic Agent] Add support for multiple hosts in connection to kibana (elastic#19628)
v1v added a commit to v1v/beats that referenced this pull request Jul 9, 2020
* upstream/master:
  Add `docker logs` support to the Elastic Log Driver (elastic#19531)
  [Elastic Agent] Fix saving of agent configuration on Windows to have proper ACLs (elastic#19793)
v1v added a commit to v1v/beats that referenced this pull request Jul 9, 2020
* upstream/master:
  Add `docker logs` support to the Elastic Log Driver (elastic#19531)
  [Elastic Agent] Fix saving of agent configuration on Windows to have proper ACLs (elastic#19793)
  Send the config revision down to the endpoint application. (elastic#19759)
v1v added a commit to v1v/beats that referenced this pull request Jul 9, 2020
* upstream/master:
  Add `docker logs` support to the Elastic Log Driver (elastic#19531)
  [Elastic Agent] Fix saving of agent configuration on Windows to have proper ACLs (elastic#19793)
fearful-symmetry added a commit to fearful-symmetry/beats that referenced this pull request Jul 9, 2020
* init commit of docker logs support

* remove vendor

* fix tests

* mage fmt

* code cleanup

* change logging directories around, add code to remove containers

* remove error message on EOF

* fix config, add docs

* more fixes, migrate to writing logs to docker's own location

* add changelog entry

* docs cleanup

(cherry picked from commit d53cd12)
fearful-symmetry added a commit that referenced this pull request Jul 9, 2020
* init commit of docker logs support

* remove vendor

* fix tests

* mage fmt

* code cleanup

* change logging directories around, add code to remove containers

* remove error message on EOF

* fix config, add docs

* more fixes, migrate to writing logs to docker's own location

* add changelog entry

* docs cleanup

(cherry picked from commit d53cd12)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
* init commit of docker logs support

* remove vendor

* fix tests

* mage fmt

* code cleanup

* change logging directories around, add code to remove containers

* remove error message on EOF

* fix config, add docs

* more fixes, migrate to writing logs to docker's own location

* add changelog entry

* docs cleanup
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Team:Integrations Label for the Integrations team v7.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants