Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

packetbeat starts logging errors with too many open files watching processes with -beta4 #335

Closed
portante opened this issue Oct 24, 2015 · 1 comment · Fixed by #337
Closed

packetbeat starts logging errors with too many open files watching processes with -beta4 #335

portante opened this issue Oct 24, 2015 · 1 comment · Fixed by #337

Comments

@portante
Copy link

With 1.0.0-beta4, RHEL 7.1, 4 socket, 40 CPUs, 256 GB of memory ...

When I looked the open files, and this one packetbeat instance has close to 500+ /proc files opened. Here are the logs from /var/log/message from one second:

2015-10-24T03:42:58.248673+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:288: FindSocketsOfPid: Open: open /proc/58955/fd: too many open files
2015-10-24T03:42:58.249177+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:288: FindSocketsOfPid: Open: open /proc/58956/fd: too many open files
2015-10-24T03:42:58.249551+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:288: FindSocketsOfPid: Open: open /proc/58957/fd: too many open files
2015-10-24T03:42:58.256024+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.266074+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.276254+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.287356+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.297990+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.308760+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.319017+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:153: Error finding PID files for httpd: Open /proc: open /proc: too many open files
2015-10-24T03:42:58.319460+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.330209+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.341199+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.351879+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.362335+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.373360+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.383770+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.394951+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.406156+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.417313+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.427626+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.439464+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.450666+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.461742+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.472272+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.483278+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.493968+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.504809+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.515655+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.526464+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.536990+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.546974+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.557704+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.568254+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.578937+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.589710+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.600668+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.611526+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.622117+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.632873+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.643688+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.654402+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.665359+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.676022+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.686598+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.697121+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.707641+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.718433+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.729224+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.740018+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.750833+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.761442+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.772413+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.783143+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.794491+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.805493+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.816312+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.826804+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.837874+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.849814+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.859699+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.870234+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.881108+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.892319+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.902868+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
2015-10-24T03:42:58.918103+00:00 perf34 /usr/bin/packetbeat[44016]: procs.go:271: Open: open /proc/net/tcp: too many open files
@andrewkroh andrewkroh mentioned this issue Oct 24, 2015
Merged
@andrewkroh
Copy link
Member

@portante Thanks for letting us know about this. I think I have it fixed, see #337.

andrewkroh added a commit to andrewkroh/beats that referenced this issue Oct 26, 2015
@andrewkroh
Fixes a file descriptor leak in the process monitoring feature. The g…
282ebf9 
…olang garbage collector automatically closes the FD associated with os.File which is why the FD count didn't grow completely unbounded; in my testing the FD count would cyclically grow to about ~500 and then drop back down to ~5.

Close elastic#335
@andrewkroh andrewkroh mentioned this issue Oct 26, 2015
Closed
@McStork McStork mentioned this issue Nov 19, 2015
Closed
ruflin added a commit that referenced this issue Dec 2, 2015
@ruflin
Merge pull request #335 from kozlice/patch-1
bc5a088 
PHP-FPM beat
tsg pushed a commit to tsg/beats that referenced this issue Jan 20, 2016
@andrewkroh
Fixes a file descriptor leak in the process monitoring feature. The g…
0c5ec78 
…olang garbage collector automatically closes the FD associated with os.File which is why the FD count didn't grow completely unbounded; in my testing the FD count would cyclically grow to about ~500 and then drop back down to ~5.

Close elastic#335
tsg pushed a commit to tsg/beats that referenced this issue Jan 20, 2016
@ruflin
Merge pull request elastic#335 from kozlice/patch-1
d4c0006 
PHP-FPM beat
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix process monitoring FD leak andrewkroh/beats
2 participants
@portante @andrewkroh