Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Diagnostic: extend from_env function #52

Merged
merged 13 commits into from
Mar 27, 2023
3 changes: 3 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ edition = "2018"
[target.'cfg(unix)'.dependencies]
libc = "0.2.50"

[features]
do_not_check_pipe = []
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also not a good idea since features should be additive.

With this do_not_check_pipe feature,if one of the crate enables this, then every other crate is opt out of checking.

Copy link
Contributor Author

@belovdv belovdv Mar 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise if one enables check then one uses smth different from pipe will fail. I'm not sure should be this option or not and, if should, how should it behave, imho it should be decided by maintainer

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If checking or not checking the pipe-ness should be configurable, then it should probably be a runtime option passed to from_env.

Not checking is the GNU Make behavior, checking catches errors in practice.


[dev-dependencies]
futures = "0.1"
num_cpus = "1.0"
Expand Down
86 changes: 62 additions & 24 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@
//!
//! // See API documentation for why this is `unsafe`
//! let client = match unsafe { Client::from_env() } {
//! Some(client) => client,
//! None => panic!("client not configured"),
//! Ok(client) => client,
//! Err(_) => panic!("client not configured"),
//! };
//! ```
//!
Expand Down Expand Up @@ -151,6 +151,44 @@ struct HelperInner {
consumer_done: bool,
}

/// Error type for `from_env` function.
#[derive(Debug)]
pub enum ErrFromEnv {
/// There isn't env var, that describes jobserver to inherit.
IsNotConfigured,
/// Cannot connect following this process's environment.
PlatformSpecific {
/// Error.
err: io::Error,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@weihanglo
Could you try this interface in cargo before it's in a "stable" release of jobserver-rs?

It's possible that in practice we'll need to discern between different kinds of erros (e.g. "descriptors are closed" and everything else) to avoid bothering users with warnings too often.

This PR, however, merges all the errors into a single io::Error with multiple vague ErrorKinds.
I'm not sure users will be able to use it reliably without relying on "string typing" (inspecting string descriptions returned by errors) or implementation details ("we know that closed descriptors are ErrorKind::InvalidData by reading source code of jobserver-rs").
(I don't like this part and I wish I read this PR in detail before its merge.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I'll try taking a look this week.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

implementation details ("we know that closed descriptors are ErrorKind::InvalidData by reading source code of jobserver-rs").

Maybe jobserver can have the error kind documented on docs.rs?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the late response. I think the change works fine with Cargo. Cargo can just ignore the jobserver if there are bad pipes. No harm to get this patch released on Cargo side I believe :)

/// Name of gotten env var.
env: &'static str,
/// Value of gotten env var.
var: String,
},
}

belovdv marked this conversation as resolved.
Show resolved Hide resolved
impl std::fmt::Display for ErrFromEnv {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
ErrFromEnv::IsNotConfigured => {
write!(f, "couldn't find relevant environment variable")
}
ErrFromEnv::PlatformSpecific { err, env, var } => {
write!(f, "{err} ({env}={var}")
}
}
}
}

impl std::error::Error for ErrFromEnv {
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
match self {
ErrFromEnv::IsNotConfigured => None,
ErrFromEnv::PlatformSpecific { err, .. } => Some(err),
}
}
}

impl Client {
/// Creates a new jobserver initialized with the given parallelism limit.
///
Expand Down Expand Up @@ -197,8 +235,9 @@ impl Client {
/// # Return value
///
/// If a jobserver was found in the environment and it looks correct then
/// `Some` of the connected client will be returned. If no jobserver was
/// found then `None` will be returned.
/// `Ok` of the connected client will be returned. If no relevant env var
/// was found then `Err(IsNotConfigured)` will be returned. In other cases
/// `Err(PlatformSpecific)` will be returned.
///
/// Note that on Unix the `Client` returned **takes ownership of the file
/// descriptors specified in the environment**. Jobservers on Unix are
Expand All @@ -211,6 +250,9 @@ impl Client {
/// with `CLOEXEC` so they're not automatically inherited by spawned
/// children.
///
/// There is a feature to configure, will this function check if provided
/// files are actually pipes.
///
/// # Safety
///
/// This function is `unsafe` to call on Unix specifically as it
Expand All @@ -227,28 +269,24 @@ impl Client {
///
/// Note, though, that on Windows it should be safe to call this function
/// any number of times.
pub unsafe fn from_env() -> Option<Client> {
let var = match env::var("CARGO_MAKEFLAGS")
.or_else(|_| env::var("MAKEFLAGS"))
.or_else(|_| env::var("MFLAGS"))
{
Ok(s) => s,
Err(_) => return None,
};
let mut arg = "--jobserver-fds=";
let pos = match var.find(arg) {
Some(i) => i,
None => {
arg = "--jobserver-auth=";
match var.find(arg) {
Some(i) => i,
None => return None,
}
}
};
pub unsafe fn from_env() -> Result<Client, ErrFromEnv> {
belovdv marked this conversation as resolved.
Show resolved Hide resolved
let (env, var) = ["CARGO_MAKEFLAGS", "MAKEFLAGS", "MFLAGS"]
.iter()
.map(|&env| env::var(env).map(|var| (env, var)))
.find_map(|p| p.ok())
.ok_or(ErrFromEnv::IsNotConfigured)?;

let (arg, pos) = ["--jobserver-fds=", "--jobserver-auth="]
.iter()
.map(|&arg| var.find(arg).map(|pos| (arg, pos)))
.find_map(|pos| pos)
.ok_or(ErrFromEnv::IsNotConfigured)?;

let s = var[pos + arg.len()..].split(' ').next().unwrap();
imp::Client::open(s).map(|c| Client { inner: Arc::new(c) })
match imp::Client::open(s) {
Ok(c) => Ok(Client { inner: Arc::new(c) }),
Err(err) => Err(ErrFromEnv::PlatformSpecific { err, env, var }),
}
}

/// Acquires a token from this jobserver client.
Expand Down
104 changes: 66 additions & 38 deletions src/unix.rs
Original file line number Diff line number Diff line change
Expand Up @@ -81,62 +81,63 @@ impl Client {
Ok(Client::from_fds(pipes[0], pipes[1]))
}

pub unsafe fn open(s: &str) -> Option<Client> {
Client::from_fifo(s).or_else(|| Client::from_pipe(s))
pub unsafe fn open(s: &str) -> io::Result<Client> {
if let Some(client) = Self::from_fifo(s)? {
return Ok(client);
}
if let Some(client) = Self::from_pipe(s)? {
return Ok(client);
}
Err(io::Error::new(
io::ErrorKind::InvalidInput,
"unrecognized format of environment variable",
))
}

/// `--jobserver-auth=fifo:PATH`
fn from_fifo(s: &str) -> Option<Client> {
fn from_fifo(s: &str) -> io::Result<Option<Client>> {
let mut parts = s.splitn(2, ':');
if parts.next().unwrap() != "fifo" {
return None;
return Ok(None);
}
let path = match parts.next() {
Some(p) => Path::new(p),
None => return None,
};
let file = match OpenOptions::new().read(true).write(true).open(path) {
Ok(f) => f,
Err(_) => return None,
};
Some(Client::Fifo {
let path = Path::new(parts.next().ok_or_else(|| {
io::Error::new(io::ErrorKind::InvalidInput, "expected ':' after `fifo`")
})?);
let file = OpenOptions::new().read(true).write(true).open(path)?;
Ok(Some(Client::Fifo {
file,
path: path.into(),
})
}))
}

/// `--jobserver-auth=R,W`
unsafe fn from_pipe(s: &str) -> Option<Client> {
unsafe fn from_pipe(s: &str) -> io::Result<Option<Client>> {
let mut parts = s.splitn(2, ',');
let read = parts.next().unwrap();
let write = match parts.next() {
Some(s) => s,
None => return None,
};

let read = match read.parse() {
Ok(n) => n,
Err(_) => return None,
};
let write = match write.parse() {
Ok(n) => n,
Err(_) => return None,
Some(w) => w,
None => return Ok(None),
};
let read = read
.parse()
.map_err(|e| io::Error::new(io::ErrorKind::Other, e))?;
let write = write
.parse()
.map_err(|e| io::Error::new(io::ErrorKind::Other, e))?;

// Ok so we've got two integers that look like file descriptors, but
// for extra sanity checking let's see if they actually look like
// instances of a pipe before we return the client.
// instances of a pipe if feature enabled or valid files otherwise
// before we return the client.
//
// If we're called from `make` *without* the leading + on our rule
// then we'll have `MAKEFLAGS` env vars but won't actually have
// access to the file descriptors.
if is_valid_fd(read) && is_valid_fd(write) {
drop(set_cloexec(read, true));
drop(set_cloexec(write, true));
Some(Client::from_fds(read, write))
} else {
None
}
check_fd(read)?;
check_fd(write)?;
drop(set_cloexec(read, true));
drop(set_cloexec(write, true));
Ok(Some(Client::from_fds(read, write)))
}

unsafe fn from_fds(read: c_int, write: c_int) -> Client {
Expand Down Expand Up @@ -207,7 +208,7 @@ impl Client {
return Err(io::Error::new(
io::ErrorKind::Other,
"early EOF on jobserver pipe",
))
));
}
Err(e) => match e.kind() {
io::ErrorKind::WouldBlock => { /* fall through to polling */ }
Expand Down Expand Up @@ -326,7 +327,7 @@ pub(crate) fn spawn_helper(
client: client.inner.clone(),
data,
disabled: false,
}))
}));
}
Err(e) => break f(Err(e)),
Ok(None) if helper.producer_done() => break,
Expand Down Expand Up @@ -385,8 +386,35 @@ impl Helper {
}
}

fn is_valid_fd(fd: c_int) -> bool {
unsafe { libc::fcntl(fd, libc::F_GETFD) != -1 }
fn check_fd(fd: c_int) -> io::Result<()> {
#[cfg(not(feature = "do_not_check_pipe"))]
unsafe {
let mut stat = mem::zeroed();
if libc::fstat(fd, &mut stat) == -1 {
Err(io::Error::last_os_error())
} else {
// On android arm and i686 mode_t is u16 and st_mode is u32,
belovdv marked this conversation as resolved.
Show resolved Hide resolved
// this generates a type mismatch when S_IFIFO (declared as mode_t)
// is used in operations with st_mode, so we use this workaround
// to get the value of S_IFIFO with the same type of st_mode.
let mut s_ififo = stat.st_mode;
s_ififo = libc::S_IFIFO as _;
if stat.st_mode & s_ififo == s_ififo {
return Ok(());
}
Err(io::Error::last_os_error()) //
}
}
#[cfg(feature = "do_not_check_pipe")]
unsafe {
match libc::fcntl(fd, libc::F_GETFD) {
r if r == -1 => Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("{fd} is not a pipe"),
)),
_ => Ok(()),
}
}
}

fn set_cloexec(fd: c_int, set: bool) -> io::Result<()> {
Expand Down
4 changes: 2 additions & 2 deletions src/wasm.rs
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ impl Client {
})
}

pub unsafe fn open(_s: &str) -> Option<Client> {
None
pub unsafe fn open(_s: &str) -> io::Result<Client> {
Err(io::ErrorKind::Unsupported.into())
}

pub fn acquire(&self) -> io::Result<Acquired> {
Expand Down
11 changes: 4 additions & 7 deletions src/windows.rs
Original file line number Diff line number Diff line change
Expand Up @@ -127,17 +127,14 @@ impl Client {
))
}

pub unsafe fn open(s: &str) -> Option<Client> {
let name = match CString::new(s) {
Ok(s) => s,
Err(_) => return None,
};
pub unsafe fn open(s: &str) -> io::Result<Client> {
let name = CString::new(s).map_err(|e| io::Error::new(io::ErrorKind::Other, e))?;

let sem = OpenSemaphoreA(SYNCHRONIZE | SEMAPHORE_MODIFY_STATE, FALSE, name.as_ptr());
if sem.is_null() {
None
Err(io::Error::last_os_error())
} else {
Some(Client {
Ok(Client {
sem: Handle(sem),
name: s.to_string(),
})
Expand Down
6 changes: 3 additions & 3 deletions tests/client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -35,23 +35,23 @@ const TESTS: &[Test] = &[
make_args: &[],
rule: &|me| me.to_string(),
f: &|| {
assert!(unsafe { Client::from_env().is_none() });
assert!(unsafe { Client::from_env().is_err() });
},
},
Test {
name: "no j args with plus",
make_args: &[],
rule: &|me| format!("+{}", me),
f: &|| {
assert!(unsafe { Client::from_env().is_none() });
assert!(unsafe { Client::from_env().is_err() });
},
},
Test {
name: "j args with plus",
make_args: &["-j2"],
rule: &|me| format!("+{}", me),
f: &|| {
assert!(unsafe { Client::from_env().is_some() });
assert!(unsafe { Client::from_env().is_ok() });
},
},
Test {
Expand Down