(RuntimeError) Wallaby had an internal issue with HTTPoison

[stephaniewilkinson]

Issue

When I switch to using Chrome, I get this error:

     ** (RuntimeError) Wallaby had an internal issue with HTTPoison
     stacktrace:
       (wallaby) lib/wallaby/httpclient.ex:32: Wallaby.HTTPClient.make_request/4
       (wallaby) lib/wallaby/experimental/chrome.ex:64: Wallaby.Experimental.Chrome.start_session/1
       (wallaby) lib/wallaby.ex:80: Wallaby.start_session/1
       (school) test/support/feature_case.ex:25: SchoolWeb.FeatureCase.__ex_unit_setup_0/1
       (school) test/support/feature_case.ex:1: SchoolWeb.FeatureCase.__ex_unit__/2
       test/school_web/features/view_test.exs:1: SchoolWeb.UserListTest.__ex_unit__/2

Versions:
elixir: "~> 1.4",

Test Code & HTML

Here's my test:

defmodule SchoolWeb.UserListTest do
  use SchoolWeb.FeatureCase, async: true

  import Wallaby.Query, only: [css: 2]

  test "homepage is accessible", %{session: session} do
    session
    |> visit("/")
    |> assert_has(css(".header", count: 1, text: "Welcome"))
  end
end

Here's my FeatureCase:

defmodule SchoolWeb.FeatureCase do
  use ExUnit.CaseTemplate

  using do
    quote do
      use Wallaby.DSL

      alias School.Repo
      import Ecto
      import Ecto.Changeset
      import Ecto.Query

      import SchoolWeb.Router.Helpers
    end
  end

  setup tags do
    :ok = Ecto.Adapters.SQL.Sandbox.checkout(School.Repo)

    unless tags[:async] do
      Ecto.Adapters.SQL.Sandbox.mode(School.Repo, {:shared, self()})
    end

    metadata = Phoenix.Ecto.SQL.Sandbox.metadata_for(School.Repo, self())
    {:ok, session} = Wallaby.start_session(metadata: metadata)
    {:ok, session: session}
  end
end

Here's my HTML:

<h1 class="header center">Welcome to School!</h1>
Thanks for any help you can give!

[keathley]

Thanks for the report @stephaniewilkinson. Can you confirm what version of chrome and chromedriver you have installed?

[stephaniewilkinson]

My Chrome is Version 66.0.3359.117 (Official Build) (64-bit)
My ChromeDriver is 2.37.544337 (8c0344a12e552148c185f7d5117db1f28d6c9e85)
I had previously tried on Chrome 65 with the same result.

For phantomjs I didn't customize the version, just ran this:

npm install -g phantomjs-prebuilt

I should probably mention the tests run fine before I try to switch to chrome.

[keathley]

Thanks! All of that looks correct to me. We get those httppoison errors when there’s an issue communicating with chromedriver. Based on the stack trace it looks like it’s not able to create a new session which typically indicates an issue with wallabys ability to talk to chromedriver or chromedrivers ability to talk to chrome.

[stephaniewilkinson]

Thanks! any ideas what to try next?

[keathley]

What OS are you using? Are both chrome and chromedriver in your path? Other things to try might be trying out the latest elixir and otp version.

[dsdshcym]

Got an similar error:

     ** (RuntimeError) Wallaby had an internal issue with HTTPoison
     code: |> assert_has(css("td", text: node_id_1))
     stacktrace:
       (wallaby) lib/wallaby/httpclient.ex:32: Wallaby.HTTPClient.make_request/4
       (wallaby) lib/wallaby/experimental/selenium/webdriver_client.ex:125: Wallaby.Experimental.Selenium.WebdriverClient.text/1
       (wallaby) lib/wallaby/driver/log_checker.ex:6: Wallaby.Driver.LogChecker.check_logs!/2
       (wallaby) lib/wallaby/browser.ex:910: Wallaby.Browser.matching_text?/2
       (elixir) lib/enum.ex:2872: Enum.filter_list/2
       (elixir) lib/enum.ex:2873: Enum.filter_list/2
       (wallaby) lib/wallaby/browser.ex:903: Wallaby.Browser.validate_text/2
       (wallaby) lib/wallaby/browser.ex:925: anonymous fn/3 in Wallaby.Browser.execute_query/2
       (wallaby) lib/wallaby/br
                       _build/test/lib/wallaby/priv/run_command.sh: line 7: 85966 Segmentation fault: 11  "$@"

Not sure if it's the same issue though.

I'm running macOS 10.13.5, Chrome 67.0.3396.79, ChromeDriver 2.40.565386, Elixir 1.6.5 (compiled with OTP 20).

[keathley]

We need to add more backoff and jitter to our http client. We see these issues when chromedriver starts to get overwhelmed with requests.

[calvinkw1]

@keathley any update here? Am running into the same issue:

** (RuntimeError) Wallaby had an internal issue with HTTPoison
     stacktrace:
       (wallaby) lib/wallaby/httpclient.ex:32: Wallaby.HTTPClient.make_request/4
       (wallaby) lib/wallaby/experimental/chrome.ex:64: Wallaby.Experimental.Chrome.start_session/1
       (wallaby) lib/wallaby.ex:80: Wallaby.start_session/1
       (orderbook) test/support/browser_feature_case.ex:17: Test.Support.Browser.Feature.Case.__ex_unit_setup_all_0/1
       (orderbook) test/support/browser_feature_case.ex:1: Test.Support.Browser.Feature.Case.__ex_unit__/2
       test/redacted/features/browser/redacted.exs:1: Redacted.Features.Browser.Redacted.__ex_unit__/2

I have chromedriver 2.36.540469 installed, Elixir 1.6.4 (compiled with OTP 20), Mac OS 10.13.5

[keathley]

I'm working on tracking down the root cause but so far no luck. Without a way to reliably re-produce this issue locally its very difficult to ensure we have a proper fix. If you have a test case that you can provide that would greatly assist the effort here.

[keathley]

I've opened a PR that may help to reduce some of the issues that folks are seeing. If you would like to test out that branch locally that would be very useful.

[keathley]

The PR is now on master so if you'd like to test with that it would be very appreciated.

[calvinkw1]

Sorry.. didn't work. Getting a similar error:

 1) Redacted.Features.Browser.AddressTests: failure on setup_all callback, test invalidated
     ** (RuntimeError) Wallaby had an internal issue with HTTPoison:
     %HTTPoison.Error{id: nil, reason: :econnrefused}
     stacktrace:
       (wallaby) lib/wallaby/httpclient.ex:37: Wallaby.HTTPClient.make_request/5
       (wallaby) lib/wallaby/experimental/chrome.ex:72: Wallaby.Experimental.Chrome.start_session/1
       (wallaby) lib/wallaby.ex:80: Wallaby.start_session/1
       ...

This seems to be only for me.. works fine on Circle CI and on other colleagues' machines. Not sure why this is.

I can provide whatever you need but note that I'm an Elixir newb...

[evax]

I've been having a similar issue and realized that a lot of chrome and chromedriver processes had been accumulating - killing those fixes it for me.

[keathley]

@evax Like they're being orphaned after running tests?

@calvinkw1 Can you provide any other insights into what might be different on your machine? Different OS, chromedriver version, chrome version, etc.? Based on the error you posted it looks like chromedriver isn't even starting, hence the :econnrefused. Are you sure that chromedriver is in your path? If we can narrow down why thats happening then we might be able to provide a more meaningful error.

[evax]

@keathley yes, though in that case if that helps I wasn't running the full test suite but giving mix test the path of the test file as an argument, in an umbrella.

[keathley]

I wonder if the cleanup script isn't being run correctly because of the umbrella or something similar.

[evax]

It could be because the path is relative to the app, so when you have multiple apps in an umbrella, it will likely only work for one app, the others display: Test patterns did not match any file: path/to/testfile_test.exs

[calvinkw1]

Ah shoot. This is probably something I should have mentioned as well. I was passing in a specific file path rather than running the entire test suite. Running the entire test suite passes without issue.

Mac OS High Sierra 10.13.5

Google Chrome is up to date
Version 67.0.3396.99 (Official Build) (64-bit)

chromedriver --version
ChromeDriver 2.36.540469 (1881fd7f8641508feb5166b7cae561d87723cfa8)

Watching htop, I don't see a chromedriver process run when I only pass in a file path, but I do when I run the full suite.

Also, if it matters, I work on a Ruby project that has ChromeDriver 2.40.565386 (45a059dc425e08165f9a10324bd1380cc13ca363) installed.

[keathley]

@calvinkw1 The only thing I can think in that case would be that its not running the test_helper.exs when you run that specific file. Or that file isn't using the correct FeatureCase or something similar.

[mjankowski]

After using chromedriver+wallaby successfully on an elixir test suite for a while, I saw this issue for first time today. I remembered that last week while working on other projects I had fiddled around with my chromedriver install - I had a few ruby projects and wound up using the chromedriver-helper gem in them to get local and CI working well together.

Anyway - I have the same chromedriver version installed (ChromeDriver 2.42.591059 (a3d9684d10d61aa0c45f6723b327283be1ebaad8)) via both homebrew and as an rbenv shim from the gem installs.

When I manipulate my path to get the chromedriver installed via homebrew as the chromedriver that wallaby uses, everything is fine. When I update path to use the rbenv shim, I see the "Wallaby had an internal issue with HTTPoison" sort of errors described here.

I'm not sure what if anything to do next to help debug what these installs are doing differently - but maybe this data point helps someone. Happy to do more if I can be useful.

[keathley]

Thanks @mjankowski. It seems like something has changed recently and has broken our interactions with chromedriver.

[sparta-developers]

We also see

** (RuntimeError) Wallaby had an internal issue with HTTPoison:
%HTTPoison.Error{id: nil, reason: :econnrefused}

when running Wallaby tests in our two apps. In one app, we can set the test concurrency to 2 and the error goes away. In the other app, we always get the error unless we slow things down by increasing the jitter by forking Wallaby and by passing in our own implementation of create_session that sleeps for 500ms inside a mutex:

def create_session(base_url, capabilities) do
  Mutex.under(:chrome_session_mutex, :start_session, :infinity, fn ->
    :timer.sleep(500)

    Wallaby.Experimental.Selenium.WebdriverClient.create_session(base_url, capabilities)
  end)
end

We're using chromedriver 74.0.3729.6 installed via homebrew on MacOS.

Are other people having success with chromedriver?

[AndroidOatmeal]

@mhanberg Thanks so much for putting time into this. My coworker and I tried the branch, but neither of us saw any noticeable change in the frequency of the errors on our respective machines. Bummer :(

I did an experiment where I ran our suite of 64 acceptance tests 100 times. The suite failed due to this error 43% of the time. When the suite failed, this error usually occurred on all 64 tests at once, though sometimes half of the tests would succeed and the other half would fail with the error.

The error in question:

     ** (RuntimeError) Wallaby had an internal issue with HTTPoison:
     %HTTPoison.Error{id: nil, reason: :econnrefused}
     stacktrace:
       (wallaby) lib/wallaby/httpclient.ex:37: Wallaby.HTTPClient.make_request/5
       (wallaby) lib/wallaby/experimental/chrome.ex:96: Wallaby.Experimental.Chrome.start_session/1
       (wallaby) lib/wallaby.ex:79: Wallaby.start_session/1
       (greenlight) test/support/acceptance_test_helpers.ex:74: Greenlight.Web.AcceptanceTestHelpers.build_session_with_host/2
       (greenlight) test/support/acceptance_test_helpers.ex:30: Greenlight.Web.AcceptanceTestHelpers.admin_login/1

I'm happy to provide any other information on this issue. Thanks again for your time, it's much appreciated.

[mhanberg]

@AndroidOatmeal is this happening on CI as well or only locally?

What Chrome and ChromeDriver versions are you using?

[AndroidOatmeal]

@mhanberg Yes, this issue occurs on CircleCI as well.

Version 76.0.3809.132 (Official Build) (64-bit)
ChromeDriver 76.0.3809.68 (420c9498db8ce8fcd190a954d51297672c1515d5-refs/branch-heads/3809@{#864})

[mhanberg]

@AndroidOatmeal Do you mind sharing the contents of some of your failing tests and your support code? I think I found you on the Elixir Slack, so you can share it with me there if you like. Thanks!

[AdamT]

I'm not sure if this is related but I ran into this error today:

 5) test user can create account (MyAppWeb.Integration.UserRegistrationTest)
     test/my_app_web/integration/user_registration_test.exs:8
     ** (RuntimeError) Wallaby had an internal issue with HTTPoison:
     %HTTPoison.Error{id: nil, reason: :econnrefused}
     %HTTPoison.Error{id: nil, reason: :closed}
     code: tr_fill_in_user_email(page, email)
     stacktrace:
       (wallaby) lib/wallaby/httpclient.ex:37: Wallaby.HTTPClient.make_request/5
       (wallaby) lib/wallaby/experimental/selenium/webdriver_client.ex:52: Wallaby.Experimental.Selenium.WebdriverClient.set_value/2
       (wallaby) lib/wallaby/driver/log_checker.ex:6: Wallaby.Driver.LogChecker.check_logs!/2
       (wallaby) lib/wallaby/element.ex:178: Wallaby.Element.set_value/2
       (wallaby) lib/wallaby/browser.ex:673: Wallaby.Browser.find/3
       test/my_app_web/integration/user_registration_test.exs:12: (test)

21:20:01.785 [error] GenServer Wallaby.SessionStore terminating
** (RuntimeError) Wallaby had an internal issue with HTTPoison:
%HTTPoison.Error{id: nil, reason: :econnrefused}
    (wallaby) lib/wallaby/httpclient.ex:37: Wallaby.HTTPClient.make_request/5
    (elixir) lib/enum.ex:789: anonymous fn/3 in Enum.each/2
    (stdlib) maps.erl:232: :maps.fold_1/3
    (elixir) lib/enum.ex:1964: Enum.each/2
    (stdlib) gen_server.erl:673: :gen_server.try_terminate/3
    (stdlib) gen_server.erl:858: :gen_server.terminate/10
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:DOWN, #Reference<0.1504753928.1884291075.192258>, :process, #PID<0.554.0>, :shutdown}

Chrome and Chromedriver versions are both 79.0.3945.xx

I ran mix test which I never do. I typically run mix test --trace test/.

Experimenting a little more I get:

# fails most of the time, seems to pass every once in a while
mix test

# passes every time reliably
mix test --trace

🤷‍♂

[ghost]

Any updates on this one? I also experiencing this kind of error when running
docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome:3.141.59-zirconium
locally.

[aaronrenner]

@AndroidOatmeal @mhanberg I believe #518 should fix this problem. Would you mind trying again from master?

[AndroidOatmeal]

@aaronrenner @mhanberg Thanks so much for putting this together. I updated my wallaby mix dependency to grab the latest version in master, which includes PR #518, like so:

{:wallaby, github: "elixir-wallaby/wallaby", branch: "master",  only: :test, runtime: false},

Unfortunately, I'm still experiencing the same :econnrefused error we've been seeing for a while. I also tried the wallaby fix on @mhanberg's wallaby_stress_test repo. Sadly, the issue also persists there. :\

[aaronrenner]

I just tried out @mhanberg's wallaby_stress_test and see what you mean. I also found if I switched the driver to phantomjs, there wasn't that problem. The phantom driver uses a pool of phantomjs instances where the chromedriver one uses a single instance. I wonder if the stress test is just overwhelming the underlying chromedriver server and if building a pool of chromedriver instances would help? 🤔

[mhanberg]

What is strange is that even if you reduce the number of test modules to 2, it still explodes.

I had the same thought regarding overwhelming chromedriver, but it is strange that chromedriver would crash with only two tests.

[aarongraham]

I had a go at building a pool of chromedriver instances as @aaronrenner suggested. I think I may have had some success. I tried it on both my own test suite and on @mhanberg's wallaby_stress_test project and it seems to work. I was able to run tests with a concurrency equal to the number of cpu cores (in my case 20).

I'm not at all familiar with poolboy, or the internals of wallaby, so it would be great if somebody with this experience could review my fork and make sure I'm going in the right direction. I would also be interested in hearing if this fixes the problem for everybody, or just me.

What I've done so far is rough and untested but my employer is very keen to see this problem fixed and we're happy to invest the time to get a clean PR submitted as soon as possible if people think it would be accepted?

A couple of things I definitely need clarification on:

1. In testing I found that chromedriver can take a very long time to reach a responsive state, sometimes over 5 seconds, by which time the test has failed. To combat this I made the init function in Chrome.ChromeDriver block until ReadinessChecker confirms ready. I feel like this implementation is preferable anyway as the long startup time happens before any tests begin, and poolboy isn't handing out chromedrivers until they are ready. Am I missing a reason not to do this?

2. After starting chromedrivers in parallel I found that HTTPoison was running out of connections. At the moment the hackney pool is hardcoded to 4, would there be any reason to not make it equal to the number of chromedriver instances that get created? Or have them both sit under a configuration option?

[aaronrenner]

@aarongraham Thanks so much for looking into this! I don't see where having a pool of Chromedriver servers would hurt, and I'm happy to hear it addresses your issue!

In testing I found that chromedriver can take a very long time to reach a responsive state, sometimes over 5 seconds, by which time the test has failed. To combat this I made the init function in Chrome.ChromeDriver block until ReadinessChecker confirms ready. I feel like this implementation is preferable anyway as the long startup time happens before any tests begin, and poolboy isn't handing out chromedrivers until they are ready. Am I missing a reason not to do this?

Supervisions wait for each child's init to complete before starting the next child, so blocking in the init/1 callback means the servers would be started sequentially instead of in parallel. (So 20 instances at 5 seconds a piece would take 100 seconds before the tests could start).

To address for phantomJS, I called Server.wait_until_ready/2 in Phantom.start_session/1.

wallaby/lib/wallaby/phantom.ex

Lines 118 to 135 in 53befe0

	@doc false
	@spec start_session([start_session_opt]) :: Wallaby.Driver.on_start_session() | no_return
	def start_session(opts) do
	{:ok, server} = ServerPool.checkout()
	readiness_timeout = Keyword.get(opts, :readiness_timeout, @default_readiness_timeout)
	wait_until_ready!(server, readiness_timeout)
	Wallaby.Phantom.Driver.create(server, opts)
	end
	@spec wait_until_ready!(Server.server(), timeout) :: :ok | no_return
	defp wait_until_ready!(server, timeout) do
	case Server.wait_until_ready(server, timeout) do
	:ok -> :ok
	{:error, :timeout} -> raise "timeout waiting for phantomjs to be ready"
	end
	end

This allows everything to start up in parallel and the waiting only occurs when start_session is called before the server is ready.

After starting chromedrivers in parallel I found that HTTPoison was running out of connections. At the moment the hackney pool is hardcoded to 4, would there be any reason to not make it equal to the number of chromedriver instances that get created? Or have them both sit under a configuration option?

That sounds good to me.

Thanks for doing some research on this and I'd be happy to play around with it as well. Please keep us posted on what you find!

[aaronrenner]

@aarongraham I just pushed a basic chromedriver pool on this branch that gets wallaby_stress_test passing: https://github.com/elixir-wallaby/wallaby/tree/ar-chromedriver-pool. It definitely slows down things quite a bit, but it seems like with some tuning we might get a speed benefit.

I've also seen where chromedriver leaves chrome processes laying around after it closes. I'm not sure if its a bug in chromedriver, or if its the way our wrapper script is closing chromedriver improperly. However, the end result is it ends up slowing down the computer to a crawl if you run the test suite several times and don't kill the chrome instances afterwards.

[eahanson]

@aaronrenner it's possible that there might be some useful code in Chroxy, which "uses Elixir processes and OTP supervision to manage the chrome instances, as well as including a transparent proxy to facilitate automatic initialisation and termination of the underlying chrome page based on the upstream connection lifetime".

[AndroidOatmeal]

@aaronrenner @aarongraham Happy Friday all. :) What's the status of the Chromedriver Pool branch? I've been using ar-chromedriver-pool to enable parallel browser tests and it seems to be working.

It would be nice to merge the fix into the main wallaby branch and finally close this issue. What work remains to be done? I'm happy to help out if the branch needs a bit more polish.

[aswinmohanme]

Ran into this error today around 5 feature tests. mix test --max-cases=2 seems to make it work, even though a lot slower than running everything async.

[padjo]

had this error
" test login --core test (Devato.Features.IndexPageTest)
test/devato_web/features/index_page_test.exs:13
** (RuntimeError) Wallaby had an internal issue with HTTPoison:
%HTTPoison.Error{id: nil, reason: :timeout}
stacktrace:
(wallaby 0.23.0) lib/wallaby/httpclient.ex:37: Wallaby.HTTPClient.make_request/5
(wallaby 0.23.0) lib/wallaby/experimental/chrome.ex:96: Wallaby.Experimental.Chrome.start_session/1
(wallaby 0.23.0) lib/wallaby.ex:78: Wallaby.start_session/1
(devato 0.1.0) test/support/feature_case.ex:25: Devato.FeatureCase.__ex_unit_setup_0/1
(devato 0.1.0) test/support/feature_case.ex:1: Devato.FeatureCase.ex_unit/2
test/devato_web/features/index_page_test.exs:1: Devato.Features.IndexPageTest.ex_unit/2"

i could run the tests chrome headless but i needed it to work so i can see it running on the browser . What worked for me was configuring the timeouts

https://hexdocs.pm/wallaby/0.23.0/readme.html#config
added
config :wallaby,
hackney_options: [timeout: :infinity, recv_timeout: :infinity]

Overriding a value

config :wallaby,
hackney_options: [timeout: 5_000]

hope this helps others

[onelesd]

@mhanberg @aaronrenner @aarongraham

Sorry to @ you all, but I'm wondering if there is interest in merging ar-chromedriver-pool? The branch isn't really usable today because its deps are so far behind.

Would you accept a PR merging that branch with main?

[AndroidOatmeal]

@mhanberg @aaronrenner @aarongraham

Hi guys. Sorry to be a pain, but I'd like a bit more visibility into the status of this. This particular issue affects many many users, and has remained open for four years now, with no solution or workaround in sight.

If there are technical limitations here, I understand — that may affect our ability to resolve this issue. Can we at least get an update explaining the current status of this particular issue? Much appreciated 🙏

[mhanberg]

I'll get re acquainted with this and take another look.

[AndroidOatmeal]

@mhanberg Thank you, I appreciate it. I can readily reproduce this error when running acceptance tests in parallel. I'm happy to assist if needed.

[mhanberg]

Alright everyone, I think this can finally be closed.

#692 should be the final fix and has been released as part of v0.30.0.

Please note that the minimum Elixir version for Wallaby has been increased to v1.12. If this is a serious problem for you, please open an issue and we can discuss.

If you think that you are still seeing this problem, please open another issue. This one is too long and the codebase has shifted so much since it was created that a lot of the code examples and errors shared here are out of date/no longer relevant.