Memory leak in Ethon::Easy #198
Comments
|
We're seeing the same memory leak after upgrading Some minor info: I tried using |
|
@tagliala Hey, thanks for the reply! I tested the |
|
@gyfis thanks Did you get a failing test on RUBY_HEAP_SLOTS_GROWTH_FACTOR=1 RUBY_GC_HEAP_INIT_SLOTS=1 RUBY_HEAP_SLOTS_INCREMENT=10 RUBY_HEAP_SLOTS_GROWTH_FACTOR=.1 ITERATIONS=10000 rspec -I lib -I profile profile/memory_leaks.rb?
Anyway, I've tried to adapt my standalone test for #194 to check this one #!/usr/bin/env ruby
# frozen_string_literal: true
ETHON_VERSION = '0.14.0' # { git: 'https://github.com/typhoeus/ethon.git' }
require 'bundler/inline'
begin
puts "Ruby version: #{RUBY_VERSION}\n\n"
gemfile(true) do
source 'https://rubygems.org'
gem 'ethon', ETHON_VERSION
end
rescue Gem::LoadError => e
puts "\nMissing Dependency:\n#{e.backtrace.first} #{e.message}"
rescue LoadError => e
puts "\nError:\n#{e.backtrace.first} #{e.message}"
puts DATA.read
exit 1
end
require 'ethon'
$cleanup_counter = 0
# Monkey patching to provide debugging information
module Ethon
module Curl
module_function
def easy_cleanup(pointer)
$cleanup_counter += 1
puts "***** Curl.easy_cleanup called with #{pointer}"
puts "Cleanup counter: #{$cleanup_counter}"
end
end
end
10000.times do
Ethon::Easy.new
endRUBY_HEAP_SLOTS_GROWTH_FACTOR=1 RUBY_GC_HEAP_INIT_SLOTS=1 RUBY_HEAP_SLOTS_INCREMENT=10 RUBY_HEAP_SLOTS_GROWTH_FACTOR=.1 ./test.rb If I run it against master branch (change So maybe it does not fix the memory leak completely but it should help |
|
Hi all 👋 I also encountered this leak in my product, right after upgrading from ethon I tried The "thread failed to start" error is likely due to FD saturation and it turns into a "Couldn't resolve host" error in libcurl or typhoeus error handling code. I tried locally with master branch and didn't reproduce the problem (I couldn't confirm about the memory though as I did not manage to reproduce it locally because I hit the FD limits too early) so it looks better in master at least. I'm a bit hesitant to try this in production though. Edit: I managed to reproduce the leak locally by raising my FD limits. After running my script (which makes 10k Typhoeus requests in a 60-threads pool), with |
See typhoeus/ethon#194 and typhoeus/ethon#198 for a bit more details The revert is in master but no gem released yet (`0.14.0` is latest when I write this, and it's impacted). FTR I tried submitting this through the form (https://www.rubymem.com/advisories/preview) but I get a 500 every time.
|
We're having the same problem in my company. I've tested the current master version (b593a9b) against 0.14.0 and 0.12.0. Locally, it seems that only the 0.12.0 does not leak. On Heroku however, we keep having a leak. I'll reduce that to a minimal exemple soon fortunately. EDIT: We solved our issue by having smaller batch size in active record, hence it may very well have been a false lead, sorry for that. On Heroku (0.14.0 then b593a9b then 0.12.0)
0.14.0b593a9b0.12.0 |
|
We encountered the same issue recently. Our memory usage graph confirmed that there is a memory leak. The memory usage constantly growth and never goes down on the graph below:
The problem appeared when we updated the gem version from 0.12.0 to 0.14.0. Downgrade gem version back to 0.12.0 or update to 0.15.0 fixes the issue. See the memory usage graph after updating the version from 0.14.0 to 0.15.0 below:
|
|
Yes, maybe 0.13 and 0.14 should be yanked though? to avoid people installing it? |
Hi, I was used to yank gems in case of major issues or unwanted breaking changes, but now I've stopped because yanking may break production deploys of third-party applications. For example, you need a urgent hotfix on your application, but the deploy fails because the locked Based on this experience, I would not yank |
|
Ah yes indeed, the sad world of production deploys re-downloading everything ;) Makes sense, fine for me. |
|
Hi, do you think this is still an issue? The memory leak reported here and confirmed in version 0.13.0 and 0.14.0 should have been fixed in 0.15.0 |
We’re running ethon 0.15.0 with no more memory leaks present, +1 on not being an issue anymore. |
|
+1 for me too, no memory problem since |
I've tracked down a memory leak in one of our applications that uses Ethon. The simplest reproduction I've been able to find is by running the following (in an irb or pry shell):
This will cause the resident set size of the process to grow and never get cleaned up. This also seems to be consistently reproducible by running the following in the project root (taken from #45):
Note: commenting out everything in
memory_leaks.rbexcept this specific test makes it much quicker as it will only run the relevant memory leak test.It seems like the source of the leak is this specific
FFI::AutoPointer, which gets created when the Easy initializer callsset_callbacks. The second argument to the AutoPointer constructor is the Curl easy_cleanup method which does seem to get called correctly.I'm unsure if this is an Ethon, FFI, or libcurl issue, so I figured I would start here. One thing I noticed is that the
Easy.cleanupmethod doesn't ever get called anywhere. Adding a call tocleanupin the calling code seems to fix the leak:This fixes the leak (and may be how we patch our applications for the time being). Is that method supposed to be automatically called through some other mechanism or are consumers expected to call it?
I was also under the impression that
FFI::AutoPointerhandles should automatically get GC'd and shouldn't require a call to free. If that's true, this seems like it might be an FFI issue (unless there's some subtle reference to that handle floating around Ethon somewhere)This reproduces on Mac OS BigSur 11.03 and on Ubuntu 16.04.6 (ruby 2.7.3, ffi 1.15.3, ethon 0.14.0)
The text was updated successfully, but these errors were encountered: