Intermittent crash/hang with TLS on Windows under poor network conditions

[enolan]

I can get a simple http-conduit program to crash or hang fairly often when the connection has occasional data corruption. I've also observed the hang when only simulating packet loss. Here's a repo with my test case. The program is tiny:

module Main where
import Network.HTTP.Conduit
import qualified Data.ByteString.Lazy.Char8 as LC8

main :: IO ()
main = do
    man <- newManager tlsManagerSettings
    req <- parseUrl "https://s3.amazonaws.com/hackage.fpcomplete.com/package/lens-4.12.3.tar.gz"
    response <- httpLbs req man
    LC8.putStrLn $ responseBody response

Using Clumsy in tamper mode set to 20%, it'll crash or hang maybe half the time.

When I instrument API calls with API Monitor I can see that the crashes happen immediately after a call to recv() fails. recv() returns SOCKET_ERROR which is a constant equal to -1. (When it's successful, recv() returns the number of bytes received.) Then memcpy() is called with the length set to -1. This causes the Windows equivalent of a segfault. Somewhere, the return value of recv() isn't getting checked.

The same thing with a cleartext HTTP connection is no problem. I also set up a simple cleartext HTTP server on a Linux machine and injected RSTs with tcpkill. This doesn't make it crash either.

I found this in the course of debugging commercialhaskell/stack#1689 (in the course of getting Idris building on AppVeyor). Stack hasn't ever crashed on me, but it seems likely the hangs and crashes in the simple http-conduit program are related to the hangs in Stack.

To reproduce:

1. Clone the testcase repo
2. stack build
3. Download Clumsy
4. Set Clumsy parameters to something like this and hit start.
5. stack exec bad-memcpy-crash

You may have to run it several times. When it crashes you should see the Windows crash dialog and if you have Visual Studio installed, an offer to open a debugger.

Let me know if there's more I can do to figure this out.

[snoyberg]

I recently heard about haskell-tls/hs-tls#124
causing trouble. Can you try upgrading to the latest tls package (by using
a nightly snapshot) and see if that fixes things?

On Thu, Apr 21, 2016, 7:11 AM Echo Nolan [email protected] wrote:

I can get a simple http-conduit program to crash or hang fairly often when
the connection has occasional data corruption. I've also observed the hang
when only simulating packet loss. Here's a repo with my test case
https://github.com/enolan/http-conduit-crash-hang. The program is tiny:

module Main whereimport Network.HTTP.Conduitimport qualified Data.ByteString.Lazy.Char8 as LC8
main :: IO ()
main = do
man <- newManager tlsManagerSettings
req <- parseUrl "https://s3.amazonaws.com/hackage.fpcomplete.com/package/lens-4.12.3.tar.gz"
response <- httpLbs req man
LC8.putStrLn $ responseBody response

Using Clumsy https://jagt.github.io/clumsy/ in tamper mode set to 20%,
it'll crash or hang maybe half the time.

When I instrument API calls with API Monitor
http://www.rohitab.com/apimonitor I can see that the crashes happen
immediately after a call to recv() fails http://i.imgur.com/rgIhYCH.png.
recv() returns SOCKET_ERROR which is a constant equal to -1. (When it's
successful, recv() returns the number of bytes received.) Then memcpy()
is called with the length set to -1. This causes the Windows equivalent of
a segfault. Somewhere, the return value of recv() isn't getting checked.

The same thing with a cleartext HTTP connection is no problem. I also set
up a simple cleartext HTTP server on a Linux machine and injected RSTs with
tcpkill. This doesn't make it crash either.

I found this in the course of debugging commercialhaskell/stack#1689
commercialhaskell/stack#1689 (in the course
of getting Idris building on AppVeyor). Stack hasn't ever crashed on me,
but it seems likely the hangs and crashes in the simple http-conduit
program are related to the hangs in Stack.

To reproduce:

1. Clone the testcase repo
   https://github.com/enolan/http-conduit-crash-hang
2. stack build
3. Download Clumsy https://jagt.github.io/clumsy/
4. Set Clumsy parameters to something like this
   http://i.imgur.com/whWZ956.png and hit start.
5. stack exec bad-memcpy-crash

You may have to run it several times. When it crashes you should see the
Windows crash dialog and if you have Visual Studio installed, an offer to
open a debugger.

Let me know if there's more I can do to figure this out.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#191

[enolan]

Still happens on nightly-2016-04-20 with tls-1.3.5 :(

[snoyberg]

I don't have ready access to a Windows machine to try this on. Can you test if this kind of problem occurs at all when using the connection package directly?

Pinging @vincenthz for any other thoughts on narrowing down the problem.

[vincenthz]

it does smells like a network bug, more than anything linked to tls. I haven't actually really look at it, but to the best of my knowledge, I'm never setting any network buffer anywhere; they are all the result of some network call.

[snoyberg]

@enolan Can you test if you can reproduce this at all with an insecure connection?

[enolan]

I tried pretty hard and couldn't. Debug printfs indicate the crash happens in hGetBuf. I'm trying to narrow it down further.

[enolan]

GHC ticket

[snoyberg]

Wow, good catch. Is there any potential workaround outside of GHC?

[enolan]

It only happens when sockets are converted to handles, so if tls used
network's API rather than hGet / hPut there wouldn't be a problem.

On Tue, May 3, 2016, 8:20 PM Michael Snoyman [email protected]
wrote:

Wow, good catch. Is there any potential workaround outside of GHC?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#191 (comment)

[snoyberg]

I don't know the details of how this works internally to tls/connection. @vincenthz is there any reasonable way to move away from Handles here? Would you consider a PR from someone to make such a change?

[vincenthz]

Good catch @enolan !

I've pushed support to connection and socks (tls was already able to handle both), I would appreciate some testing before I push a new version if possible

[snoyberg]

Are there any changes that need to be made on the http-client-tls side to take advantage of this support, or just recompiling against master?

[vincenthz]

it should just work as is

[snoyberg]

Awesome, thank you :)

[enolan]

@vincenthz Sorry! I read network's code wrong. It imports send and recv itself, with the right types for Windows, but it uses those imports on everything but Windows. Switching to using network's API directly won't actually help here. My bad.

[vincenthz]

:(

could network detect the 4294967295 (wrapped -1) value on windows 64 bits as meaning error like -1 ?

[enolan]

The crash happens before control returns from base unfortunately.

On Sun, May 8, 2016, 11:55 AM Vincent Hanquez [email protected]
wrote:

:(

could network detect the 4294967295 (wrapped -1) value on windows 64 bits
as meaning error like -1 ?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#191 (comment)

[vincenthz]

darn :\ . Could probably have network call a different foreign import maybe, one that returns the right thing, because fixing base is not going to fix any previous version, and it could take a really long time ..

[enolan]

This is the rabbit hole with no bottom. It's actually fixable in network, and I did, but now I get a different crash. haskell/network#192 is the PR. If either of you could look at it and see what I'm missing that'd be great.

@vincenthz Your change to tls to use Network.Socket over Handles actually wasn't in vain. The network workaround only works if you're using network.

[enolan]

This is fixed upstream in network-2.6.3.1 and asn1-encoding-0.9.4.

[snoyberg]

Awesome, thank you!

On Thu, Aug 4, 2016, 6:51 AM Echo Nolan [email protected] wrote:

This is fixed upstream in network-2.6.3.1 and asn1-encoding-0.9.4.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#191 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADBB-oKQP-8e-CHy3_KDK-sVm3GNjimks5qcWGjgaJpZM4IMU3S
.