Apparent deadlock with Expect: 100-continue #2994

bk2204 · 2022-09-23T18:36:13Z

Version
Hyper 0.14.20

Platform
Linux dis 5.19.0-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 5.19.6-1 (2022-09-01) x86_64 GNU/Linux
Debian amd64/sid

Description

When making a curl request with Expect: 100-continue, there is a deadlock where there is not without that.

Cargo.toml:

[package]
name = "test-100-continue"
version = "0.1.0"
edition = "2021"

[dependencies]
http = "0.2"
hyper = { version = "0.14", features = ["server", "http1", "http2", "stream"] }
tokio = { version = "1", features = [ "io-std", "macros", "net", "process", "rt-multi-thread", "signal", "time" ] }

Code (src/main.rs):

use std::net::TcpListener;
use hyper::{server::conn::Http, service::service_fn, Body};
use http::{Request, Response};
use hyper::body::HttpBody;

fn block_on_async<T>(f: T) -> T::Output
where
    T: std::future::Future + Send + 'static,
    T::Output: Send + 'static,
{
    let (tx, rx) = tokio::sync::oneshot::channel();
    tokio::task::spawn(async move {
        let r = f.await;
        let _ = tx.send(r);
    });
    tokio::task::block_in_place(|| {
        rx.blocking_recv().unwrap()
    })
}

fn process_body(body: Body) -> Result<(), hyper::Error> {
    block_on_async(async move {
    let mut body = body;
    loop {
        match body.data().await {
            Some(Ok(chunk)) => {
                eprintln!("{} bytes", chunk.len());
            }
            Some(Err(e)) => {
                eprintln!("error: {}", e);
                break;
            }
            None => break,
        }
    }
    });
    Ok(())
}

async fn serve_connection(req: Request<Body>) -> Result<Response<Body>, hyper::Error> {
    let body = req.into_body();
    process_body(body)?;
    let resp = 
        Response::builder()
            .status(200)
            .header("Content-Type", "text/plain")
            .header("Content-Security-Policy", "default-src 'none'; sandbox")
            .header("Expires", "Fri, 01 Jan 1980 00:00:00 GMT")
            .header("Pragma", "no-cache")
            .header("Cache-Control", "no-cache, max-age=0, must-revalidate")
            .header("Vary", "Accept-Encoding");
    Ok(resp.body(Body::empty()).unwrap())
}

#[tokio::main]
async fn main() -> std::io::Result<()> {
    let listener = TcpListener::bind("127.0.0.1:3000")?;
    for fp in listener.incoming() {
        let fp = fp?;
        fp.set_nonblocking(true)?;
        let fp = tokio::net::TcpStream::from_std(fp)?;
        tokio::task::spawn(async move {
            if let Err(e) = Http::new()
                .http1_keep_alive(true)
                .http1_title_case_headers(true)
                .serve_connection(
                    fp,
                    service_fn(move |req| serve_connection(req)),
                )
                .await
            {
                eprintln!("error: {}", e);
            };
        });
    }
    Ok(())
}

I ran curl -v http://github.localhost:3000/ -d0000 and curl -v http://github.localhost:3000/ -d0000 -H'Expect: 100-continue'.

I expected to see this happen:

I expected both requests to complete nearly instantly.

Instead, this happened:

The request with Expect: 100-continue hangs and the other one succeeds. A 100 Continue response is not sent. All similar requests that do not contain Expect: 100-continue succeed, regardless of method or presence or lack of a body. All requests without a body succeed.

A note that the block_on_async function seems to be relevant and I wasn't able to reproduce without it. I've instrumented my real code and it appears that code blocks awaiting in body.data().await. From the issues I've read, it seems that polling the body should cause the 100 Continue response to be sent (and it is not), and I suspect that that is related to why this triggers here. I don't believe the runtime should be unable to progress because there should be sufficient worker threads and the tokio::task::block_in_place should not block the executor. I have sufficient cores that the number of worker threads should be sufficient to progress, and gdb indicates that my real code is blocked on an epoll and the other executor threads are idle.

If there's any more information I can help provide here, please let me know, and I'll be happy to do so.

The text was updated successfully, but these errors were encountered:

seanmonstar · 2022-09-23T21:14:17Z

I suspect the problem is that internally, the handling of sending the 100 Continue header is part of the same task as the one that calls the service. So, when the service call blocks in place, while other tasks can execute, this one can't. So it will never be able to send.

seanmonstar · 2022-09-23T23:15:14Z

Having checked a bit more about that function, yes, that is the problem. It's almost never safe to use it, it seems.

bk2204 added the C-bug Category: bug. Something is wrong. This is bad! label Sep 23, 2022

seanmonstar added the B-upstream Blocked: needs a change in a dependency or the compiler. label Sep 23, 2022

seanmonstar closed this as not planned Won't fix, can't repro, duplicate, stale Sep 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Apparent deadlock with Expect: 100-continue #2994

Apparent deadlock with Expect: 100-continue #2994

bk2204 commented Sep 23, 2022 •

edited

Loading

seanmonstar commented Sep 23, 2022

seanmonstar commented Sep 23, 2022

Apparent deadlock with Expect: 100-continue #2994

Apparent deadlock with Expect: 100-continue #2994

Comments

bk2204 commented Sep 23, 2022 • edited Loading

seanmonstar commented Sep 23, 2022

seanmonstar commented Sep 23, 2022

bk2204 commented Sep 23, 2022 •

edited

Loading