Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connection and memory leak when pipeline is reloaded #251

Closed
jakelandis opened this issue Dec 6, 2017 · 1 comment
Closed

Connection and memory leak when pipeline is reloaded #251

jakelandis opened this issue Dec 6, 2017 · 1 comment
Labels

Comments

@jakelandis
Copy link
Contributor

Testing against 5.6.4 with 4.3.0 of this plugin.

Everytime a pipeline is reloaded a connection is leaked.

For example a healthy non-reloaded pipeline doing jdbc work:
image

Then reload the pipeline ~10 times and the open/closed connections gap widens, and the number of noncollectable objects increase too.
image

Since we only use 1 connection (by default) and as of 4.3.0 open/close connections per statement, the gap between open and close should always be within 1.

The root cause here is that the underlying library does not guarantee a connection in use will actually get disconnected. From the underlying library disconnect method: "This method has the effect of disconnecting from the database, assuming that no connections are currently being used. "

We do not offer any assurances that the pipeline thread during a reload will not try to call disconnect while the connection is use. If that happens the library attempts to disconnect and looses it reference to the connection (thinking it is closed), but when in fact it is still open (since it is in use). Hence the connection leak. All the objects associated with the connection are leaked, hence the memory leak.

This issue will only impact consumers that reload frequently since the connection and memory leak is directly related to the reload frequency.

Note - 4.3.0 did not introduce this issue.

jakelandis added a commit to jakelandis/logstash-input-jdbc that referenced this issue Dec 6, 2017
When the pipeline reload thread closes the connection, but the connection is still active (becuase is executing a query), the connection and associated memory will leak. The underlying library does not make any assurances that the connection will indeed be closed if currently in use, and observation shows this to be true. The fix here is to ensure that the a thread can not attempt close the connection (it will block) while the connection is in use. In this case that thread is the pipeline thread and the reload will now block until the current query finishes (or errors).

Fixes logstash-plugins#251
@jakelandis
Copy link
Contributor Author

Fixed in 4.3.2 of the plugin.

@gmoskovicz gmoskovicz added the bug label Dec 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants