Runtime Failures on TCP Stress Testing: (libuv) kqueue(): Too many open files and more

[ehsanul]

I'm on OSX 10.8.4, x86_64, latest Rust on master. And to head off any concerns about my file handle limits:

$ ulimit
unlimited

I ran into this when stress-testing my web sockets library. After just a couple hundred connections, some connections were being refused at random. The only thing I could see was some output with RUST_LOG=debug saying (libuv) kqueue(): Too many open files. I've come up with a smaller test case that brings this out with some spectacular errors:

use std::io::net::tcp::{TcpListener, TcpStream};
use std::io::net::ip::{Ipv4Addr, SocketAddr};
use std::io::{Acceptor, Listener};
use std::io::IoResult;

fn main() {
    let addr = SocketAddr { ip: Ipv4Addr(127, 0, 0, 1), port: 8989 };
    spawn(proc() {
        let mut acceptor = TcpListener::bind(addr).listen();
        loop {
            let mut stream = match acceptor.accept() {
                Ok(stream) => stream,
                Err(error) => {
                    debug!("accept failed: {:?}", error);
                    continue;
                }
            };
            println!("{:?}", stream.read_byte().unwrap());
        }
    });
    for _ in range(0, 1000) {
        spawn(proc() {
            match TcpStream::connect(addr) {
                Ok(stream) => {
                    let mut stream = stream;
                    stream.write([1]);
                    let mut buf = [0];
                    stream.read(buf);
                },
                Err(e) => debug!("{:?}", e)
            }
        });
    }
}

Fails less often with 100-200 connections. The errors are different every time I run this with RUST_LOG=debug.

Here's the end of one run:

(libuv) kqueue(): Too many open files
bug(54921,0x10881c000) malloc: *** error for object 0x7fca4b620f70: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
std::io::IoError{kind: OtherIoError, desc: "too many open files", detail: None}
task '<unnamed>' failed at 'assertion failed: `(left == right) && (right == left)` (left: `140506823655696`, right: `140506824839680`)', /Users/harish/rust/rust/src/librustuv/homing.rs:110
(libuv) kqueue(): Too many open files
std::io::IoError{kind: OtherIoError, desc: "too many open files", detail: None}
task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:199
Abort trap: 6

Another:

task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:199
task failed during unwinding (double-failure - total drag!)
rust must abort now. so sorry.
Segmentation fault: 11

Another:

task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:199
task failed during unwinding (double-failure - total drag!)
rust must abort now. so sorry.
Illegal instruction: 4

And lastly:

fatal runtime error: leaked managed memory (140518327703008 objects)
(libuv) kqueue(): Too many open files
task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:Abort trap: 6

[emberian]

cc @alexcrichton

[alexcrichton]

Oh dear that's bad, thanks for the small issue! I will definitely investigate.

[bharrisau]

What is the result of ulimit -n though? Just calling ulimit is only the maximum file size (unless OSX is different).

[ehsanul]

@bharrisau you're right!

$ ulimit -n
256

So I set it to a higher value and things seem fine now. Sorry about that. Though, It's still a surprising way for the runtime to fail, and clearly something is still wrong there.

[alexcrichton]

Despite OSX having small file descriptor limits, this is a serious bug that I'd like to address with the runtime. Something funny is going on.

[alexcrichton]

With #12818 applied, I am unable to reproduce this on OSX. I am not convinced that it fixes anything, but I'm no longer seeing any scary assertions.

[alexcrichton]

I take that back, I was using libnative, not libgreen (silly me)

[alexcrichton]

I attempted to debug this issue on linux, and I quickly ran into two problems. The first is an assertion fixed by #12818, and the other is filed as joyent/libuv#1189. Sadly none of this seems related to the assertions on OSX, so I'll continue to investigate there. If only OSX still had valgrind...