Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #19459 to 7.x: [Filebeat] Fix reference leak in TCP and Unix socket inputs #19501

Merged
merged 1 commit into from
Jul 13, 2020

Conversation

andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented Jun 29, 2020

Cherry-pick of PR #19459 to 7.x branch. Original message:

What does this PR do?

The tcp and unix input sources were leaking references causing a memory leak.
When an accepted connection ended inputsource/common.Closer was
supposed to delete the pointer that it held to the connection, but due to a code
error delete was being called on the wrong map.

Why is it important?

This fixes a memory leak with the tcp and unix inputs.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@andrewkroh andrewkroh added [zube]: In Review backport Team:Services (Deprecated) Label for the former Integrations-Services team labels Jun 29, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 29, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 29, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 29, 2020

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19501 updated]

  • Start Time: 2020-07-13T16:24:23.785+0000

  • Duration: 126 min 15 sec

Test stats 🧪

Test Results
Failed 0
Passed 3678
Skipped 677
Total 4355

Log output

Expand to view the last 100 lines of log output

[2020-07-13T18:29:30.592Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-13T18:29:30.592Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-07-13T18:29:30.592Z] + FILE=journalbeat/build/coverage/full.cov
[2020-07-13T18:29:30.592Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-07-13T18:29:31.184Z] java.lang.InterruptedException: no matches found within 10000
[2020-07-13T18:29:31.184Z] 	at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2826)
[2020-07-13T18:29:31.184Z] 	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2705)
[2020-07-13T18:29:31.184Z] 	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2686)
[2020-07-13T18:29:31.184Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3073)
[2020-07-13T18:29:31.184Z] Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from beats-ci-immutable-ubuntu-1604-1594658995265943986.c.elastic-ci-prod.internal/10.224.0.212:41184
[2020-07-13T18:29:31.184Z] 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788)
[2020-07-13T18:29:31.185Z] 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
[2020-07-13T18:29:31.185Z] 		at hudson.remoting.Channel.call(Channel.java:998)
[2020-07-13T18:29:31.185Z] 		at hudson.FilePath.act(FilePath.java:1069)
[2020-07-13T18:29:31.185Z] 		at hudson.FilePath.act(FilePath.java:1058)
[2020-07-13T18:29:31.185Z] 		at hudson.FilePath.validateAntFileMask(FilePath.java:2684)
[2020-07-13T18:29:31.185Z] 		at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:265)
[2020-07-13T18:29:31.185Z] 		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-07-13T18:29:31.185Z] 		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-07-13T18:29:31.185Z] 		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-07-13T18:29:31.185Z] 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-07-13T18:29:31.185Z] 		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-07-13T18:29:31.185Z] 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-07-13T18:29:31.185Z] 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-07-13T18:29:31.185Z] Caused: hudson.FilePath$TunneledInterruptedException
[2020-07-13T18:29:31.185Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3075)
[2020-07-13T18:29:31.185Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
[2020-07-13T18:29:31.185Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2020-07-13T18:29:31.185Z] 	at hudson.remoting.Request$2.run(Request.java:369)
[2020-07-13T18:29:31.185Z] 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
[2020-07-13T18:29:31.185Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-07-13T18:29:31.185Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-07-13T18:29:31.185Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-07-13T18:29:31.185Z] 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
[2020-07-13T18:29:31.185Z] Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
[2020-07-13T18:29:31.185Z] 	at hudson.FilePath.act(FilePath.java:1071)
[2020-07-13T18:29:31.185Z] 	at hudson.FilePath.act(FilePath.java:1058)
[2020-07-13T18:29:31.185Z] 	at hudson.FilePath.validateAntFileMask(FilePath.java:2684)
[2020-07-13T18:29:31.185Z] 	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:265)
[2020-07-13T18:29:31.185Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-07-13T18:29:31.185Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-07-13T18:29:31.185Z] 	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-07-13T18:29:31.185Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-07-13T18:29:31.185Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-07-13T18:29:31.185Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-07-13T18:29:31.185Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-07-13T18:29:31.185Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-07-13T18:29:31.185Z] No artifacts found that match the file pattern "**/build/TEST*.out". Configuration error?
[2020-07-13T18:29:31.581Z] + curl -sSLo codecov https://codecov.io/bash
[2020-07-13T18:29:31.960Z] Failed in branch Filebeat x-pack
[2020-07-13T18:30:15.995Z] Body did not finish within grace period; terminating with extreme prejudice
[2020-07-13T18:30:16.169Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats
[2020-07-13T18:30:16.480Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-13T18:30:16.492Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats/Lint
[2020-07-13T18:30:16.571Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats/Filebeat-Windows
[2020-07-13T18:30:16.641Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-13T18:30:16.718Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-07-13T18:30:16.789Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-07-13T18:30:16.859Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-13T18:30:16.928Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats/Filebeat-oss
[2020-07-13T18:30:17.282Z] + cat
[2020-07-13T18:30:17.282Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-13T18:30:17.282Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-13T18:30:23.881Z] runbld>>> runbld started
[2020-07-13T18:30:23.881Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-13T18:30:25.800Z] runbld>>> The following profiles matched the job 'Beats/beats/PR-19501' in order of occurrence in the config (last value wins).
[2020-07-13T18:30:27.198Z] runbld>>> Debug logging enabled.
[2020-07-13T18:30:27.198Z] runbld>>> Storing result
[2020-07-13T18:30:27.198Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-13T18:30:27.198Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200713183026-F9B62998
[2020-07-13T18:30:27.198Z] runbld>>> Adding system facts.
[2020-07-13T18:30:28.599Z] runbld>>> Adding vcs info for the latest commit:  bcf4477704df8cc26fbae8b97a0bd59ed9cf24c6
[2020-07-13T18:30:28.599Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-13T18:30:28.599Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-13T18:30:28.599Z] Processing JUnit reports with runbld...
[2020-07-13T18:30:28.599Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-13T18:30:28.863Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-13T18:30:28.863Z] runbld>>> DURATION: 53ms
[2020-07-13T18:30:28.863Z] runbld>>> STDOUT: 40 bytes
[2020-07-13T18:30:28.863Z] runbld>>> STDERR: 49 bytes
[2020-07-13T18:30:28.863Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-13T18:30:28.863Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats
[2020-07-13T18:30:29.811Z] runbld>>> Storing build metadata: 
[2020-07-13T18:30:29.811Z] runbld>>> Adding test report.
[2020-07-13T18:30:29.811Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats
[2020-07-13T18:30:30.756Z] runbld>>> Found 11 test output files
[2020-07-13T18:30:31.699Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 4355 Skipped: 653
[2020-07-13T18:30:31.960Z] runbld>>> Storing result
[2020-07-13T18:30:31.960Z] runbld>>> FAILURES: 0
[2020-07-13T18:30:31.960Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-13T18:30:31.960Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200713183026-F9B62998
[2020-07-13T18:30:32.221Z] runbld>>> Email notification disabled by environment variable.
[2020-07-13T18:30:32.221Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-13T18:30:38.066Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-19501
[2020-07-13T18:30:38.276Z] [INFO] getVaultSecret: Getting secrets
[2020-07-13T18:30:38.351Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-13T18:30:39.076Z] + chmod 755 generate-build-data.sh
[2020-07-13T18:30:39.076Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19501/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19501/runs/3 ABORTED 7575033
[2020-07-13T18:30:39.076Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19501/runs/3/steps/?limit=10000 -o steps-info.json
[2020-07-13T18:30:39.627Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19501/runs/3/tests/?status=FAILED -o tests-errors.json

@andrewkroh andrewkroh force-pushed the backport_19459_7.x branch from ba9ed57 to d73b07f Compare July 13, 2020 13:16
…19459)

The tcp and unix input sources were leaking references causing a memory leak.
When an accepted connection ended inputsource/common.Closer was
supposed to delete the pointer that it held to the connection, but due to a code
error `delete` was being called on the wrong map.

Instead of modifying the common.Closer I replaced it with a cancellable context.Context which
is designed to propagate signals from parent to children and requires less code.

(cherry picked from commit 61f4846)
@andrewkroh andrewkroh force-pushed the backport_19459_7.x branch from d73b07f to bcf4477 Compare July 13, 2020 16:17
@andrewkroh andrewkroh merged commit 03b4227 into elastic:7.x Jul 13, 2020
@zube zube bot removed the [zube]: Done label Oct 13, 2020
@andrewkroh andrewkroh deleted the backport_19459_7.x branch January 14, 2022 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport Team:Services (Deprecated) Label for the former Integrations-Services team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants