Skip to content

Commit

Permalink
Rollup merge of rust-lang#53981 - fbernier:patch-1, r=sfackler
Browse files Browse the repository at this point in the history
Implement initializer() for FileDesc

Here was my initial issue:

```rust
use std::process::{Command};

fn main() {
    let output = Command::new("curl").arg("-s").arg("http://ovh.net/files/100Mio.dat").output();
    println!("{:?}", output.unwrap().stdout.len());
}
```
```
~/stuff ❯❯❯ time ./dwl
104857600
./dwl  16.22s user 1.80s system 23% cpu 1:15.24 total
```

```rust
use std::process::{Command, Stdio};

fn main() {
    let child = Command::new("curl").arg("-s").arg("http://ovh.net/files/100Mio.dat").stdout(Stdio::piped()).spawn();
    let output = child.unwrap().wait_with_output().unwrap();
    println!("{:?}", output.stdout.len());
}
```

```
~/stuff ❯❯❯ time ./dwl2
104857600
./dwl2  0.64s user 2.18s system 5% cpu 53.072 total
```

As you can see the first version is spending much more time in userland and also uses more cpu. With the help of @programble, @talchas and @habnabit  on the rust IRC, we discovered that the slow version uses two pipes, one for `stdin` and one for `stderr` and in that case it polls when going through [this function](https://github.com/rust-lang/rust/blob/master/src/libstd/sys/unix/pipe.rs#L82). The polling calls `read_to_end` on the pipes repetitively and this results in zeroing its internal buffer each time. To avoid this zeroing, `FileDesc` needs to implement `initializer`. We see no reason why it [wouldn't work with uninitialized memory](https://doc.rust-lang.org/1.26.1/src/std/io/mod.rs.html#534) so this PR fixes that.

Here is some tracing of the slow program:
![image](https://user-images.githubusercontent.com/147585/45133180-ed8a2d80-b161-11e8-9ec7-09979ec96145.png)

versus the fast program:
![image](https://user-images.githubusercontent.com/147585/45133216-0c88bf80-b162-11e8-908e-ff81d59239fb.png)

I have not tested the change yet but will try to build it tomorrow.
  • Loading branch information
kennytm committed Sep 8, 2018
2 parents 5cc51ad + 28745a6 commit 14c21a1
Showing 1 changed file with 6 additions and 1 deletion.
7 changes: 6 additions & 1 deletion src/libstd/sys/unix/fd.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
#![unstable(reason = "not public", issue = "0", feature = "fd")]

use cmp;
use io::{self, Read};
use io::{self, Read, Initializer};
use libc::{self, c_int, c_void, ssize_t};
use mem;
use sync::atomic::{AtomicBool, Ordering};
Expand Down Expand Up @@ -270,6 +270,11 @@ impl<'a> Read for &'a FileDesc {
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
(**self).read(buf)
}

#[inline]
unsafe fn initializer(&self) -> Initializer {
Initializer::nop()
}
}

impl AsInner<c_int> for FileDesc {
Expand Down

0 comments on commit 14c21a1

Please sign in to comment.