Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Enable global expose with nested paths #2362

Open
wants to merge 31 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
5047d55
rebasing onto prefix main
bahugo Oct 26, 2024
576f900
Fixed formatting and executable_name call
bahugo Oct 26, 2024
5c65cbe
Removed dead code
bahugo Oct 26, 2024
b1a6b73
Added test for path stripping with nested path
bahugo Oct 26, 2024
0b38434
Removed constrain on executable directeory for global executables
bahugo Oct 26, 2024
b67e8d8
rebasing Fixing executable detection for global install
bahugo Oct 27, 2024
4870d0d
Updated format mapping for relname
bahugo Oct 27, 2024
8c8dcc4
Exposing relname to manifest
bahugo Oct 27, 2024
b1df886
Using relname in project
bahugo Oct 27, 2024
65debc8
format
bahugo Oct 27, 2024
bdcbabd
Deleting extension in Mapping new method
bahugo Oct 27, 2024
11c56f6
Using strip_executable_extension
bahugo Oct 28, 2024
f50e300
using pixi_utils::executable_from_path in global/mod.rs
bahugo Oct 28, 2024
96f2350
Attempt to test dummy package with nested exposed
bahugo Oct 28, 2024
e11105d
Added documentation for nested expose in global install
bahugo Oct 28, 2024
0d93260
Fixed testing nested_dummy
bahugo Oct 29, 2024
58fb4ce
reverted addition of rattler-build to pixi.toml
bahugo Oct 29, 2024
db1d8e7
Rebasing :Updated dummy channels
bahugo Nov 6, 2024
f9b491c
format
bahugo Oct 29, 2024
5c4ffb6
applying pre-commit
bahugo Oct 29, 2024
2302ee7
building integration tests channels after rebasing
bahugo Nov 4, 2024
69ea5ba
fixes from PR comments + format
bahugo Nov 6, 2024
ac526ed
updating dummy channels
bahugo Nov 6, 2024
4cb9ad9
Merge remote-tracking branch 'upstream/main' into bahugo/main
ruben-arts Nov 14, 2024
4c8114d
fix: nested test
ruben-arts Nov 15, 2024
ca93d62
Merge branch 'main' into main
ruben-arts Nov 15, 2024
edc84c1
format
ruben-arts Nov 15, 2024
0254a01
fix: test on windows
ruben-arts Nov 15, 2024
95f9167
fix: test for real v1
ruben-arts Nov 15, 2024
61e64dc
fix: executable path name
ruben-arts Nov 18, 2024
71f61a3
fix: test
ruben-arts Nov 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions crates/pixi_utils/src/executable_utils.rs
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ mod tests {
#[case::python3_config("python3-config", "python3-config")]
#[case::x2to3("2to3", "2to3")]
#[case::x2to3312("2to3-3.12", "2to3-3.12")]
#[case::nested_executable("subdir/executable.sh", "subdir/executable")]
fn test_strip_executable_unix(#[case] path: &str, #[case] expected: &str) {
let path = Path::new(path);
let result = strip_unix_executable_extension(path.to_string_lossy().to_string());
Expand All @@ -100,6 +101,7 @@ mod tests {
#[case::python3_config("python3-config", "python3-config")]
#[case::x2to3("2to3", "2to3")]
#[case::x2to3312("2to3-3.12", "2to3-3.12")]
#[case::nested_executable("subdir\\executable.exe", "subdir\\executable")]
fn test_strip_executable_windows(#[case] path: &str, #[case] expected: &str) {
let path = Path::new(path);
let result = strip_windows_executable_extension(path.to_string_lossy().to_string());
Expand All @@ -114,6 +116,7 @@ mod tests {
#[case::package010("package0.1.0", "package0.1.0")]
#[case::x2to3("2to3", "2to3")]
#[case::x2to3312("2to3-3.12", "2to3-3.12")]
#[case::nested_executable("subdir/executable", "subdir/executable")]
fn test_strip_executable_extension(#[case] path: &str, #[case] expected: &str) {
let result = strip_executable_extension(path.into());
assert_eq!(result, expected);
Expand Down
19 changes: 19 additions & 0 deletions docs/features/global_tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,25 @@ exposed = { ansible = "ansible" } # (1)!

1. The `ansible` binary is exposed even though it is installed by a dependency of `ansible`, the `ansible-core` package.

It's also possible to expose an executable which is located in a nested directory.
For example dotnet.exe executable is located in a dotnet folder,
to expose dotnet.exe you must specify its relative path :

```
pixi global install dotnet --expose dotnet=dotnet\dotnet.exe
```
you can also omit the extension
```
pixi global install dotnet --expose dotnet=dotnet\dotnet
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this example on Linux but that package doesn't seem to work. Do you understand why the dotnet setup is using a non bin/xx setup for it's binaries?

Copy link
Contributor Author

@bahugo bahugo Oct 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no Idea.
I had never tried to install dotnet on linux but you're right, it's installed in .pixi/envs/dotnet/lib/dotnet/dotnet
For some reason everything is painful with windows tools...
Maybe a more cross platform or generic tool would be better for the documentation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I contacted @dhirschfeld as he is a pixi user and the maintainer of dotnet maybe he has some smart things to say about this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about smart - the TL;DR is that it's just a binary repackage of upstream and that seems to be the way it's packaged by Microsoft.

The package sets a number of env vars in an activation script, including the PATH. I'd like to port setting/unsetting the env vars to the JSON format (assuming that works with pixi?) but haven't gotten around to it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes pixi supports that 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there another example package, where this would be required, that does work on linux?

```
will create the following entry in the manifest:
```toml
[envs.dotnet]
channels = ["conda-forge"]
dependencies = { dotnet = "*" }
exposed = { dotnet = 'dotnet\dotnet' }
```

### Dependencies
Dependencies are the **Conda** packages that will be installed into your environment. For example, running:
```
Expand Down
92 changes: 55 additions & 37 deletions src/global/common.rs
Original file line number Diff line number Diff line change
Expand Up @@ -975,70 +975,87 @@ mod tests {
ExposedName::from_str("test").unwrap(),
"test".to_string(),
));
exposed.insert(Mapping::new(
ExposedName::from_str("nested_test").unwrap(),
Path::new("other_dir")
.join("nested_test")
.to_str()
.unwrap()
.to_string(),
));
let (to_remove, to_add) = get_expose_scripts_sync_status(&bin_dir, &env_dir, &exposed)
.await
.unwrap();
assert!(to_remove.is_empty());
assert_eq!(to_add.len(), 1);
assert_eq!(to_add.len(), 2);

// Add a legacy script to the bin directory
// even if it's should be exposed and it's pointing to correct executable
// even if it should be exposed and it's pointing to correct executable
// it is an old script
// we need to remove it and replace with trampoline
let script_path = if cfg!(windows) {
bin_dir.path().join("test.bat")
} else {
bin_dir.path().join("test")
};
let script_names = ["test", "nested_test"];

#[cfg(windows)]
{
let script = format!(
r#"
for script_name in script_names {
let script_path = bin_dir.path().join(format!("{}.bat", script_name));
let script = format!(
r#"
@"{}" %*
"#,
env_dir
.path()
.join("bin")
.join("test.exe")
.to_string_lossy()
);
tokio_fs::write(&script_path, script).await.unwrap();
env_dir
.path()
.join("bin")
.join(format!("{}.exe", script_name))
.to_string_lossy()
);
tokio_fs::write(&script_path, script).await.unwrap();
}
}
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;

let script = format!(
r#"#!/bin/sh
for script_name in script_names {
let script_path = bin_dir.path().join(script_name);
let script = format!(
r#"#!/bin/sh
"{}" "$@"
"#,
env_dir.path().join("bin").join("test").to_string_lossy()
);
tokio_fs::write(&script_path, script).await.unwrap();
// Set the file permissions to make it executable
let metadata = tokio_fs::metadata(&script_path).await.unwrap();
let mut permissions = metadata.permissions();
permissions.set_mode(0o755); // rwxr-xr-x
tokio_fs::set_permissions(&script_path, permissions)
.await
.unwrap();
env_dir
.path()
.join("bin")
.join(script_name)
.to_string_lossy()
);
tokio_fs::write(&script_path, script).await.unwrap();
// Set the file permissions to make it executable
let metadata = tokio_fs::metadata(&script_path).await.unwrap();
let mut permissions = metadata.permissions();
permissions.set_mode(0o755); // rwxr-xr-x
tokio_fs::set_permissions(&script_path, permissions)
.await
.unwrap();
}
};

let (mut to_remove, mut to_add) =
get_expose_scripts_sync_status(&bin_dir, &env_dir, &exposed)
.await
.unwrap();
assert!(to_remove.pop().unwrap().exposed_name().to_string() == "test");
assert!(to_add.pop().unwrap().to_string() == "test");
// Test to_remove and to_add to see if the legacy scripts are removed and trampolines are added
let (to_remove, to_add) = get_expose_scripts_sync_status(&bin_dir, &env_dir, &exposed)
.await
.unwrap();
assert!(to_remove.iter().all(|bin| !bin.is_trampoline()));
assert_eq!(to_remove.len(), 2);
assert_eq!(to_add.len(), 2);

// Test to_remove when nothing should be exposed
let (mut to_remove, to_add) =
// it should remove all the legacy scripts and add nothing
let (to_remove, to_add) =
get_expose_scripts_sync_status(&bin_dir, &env_dir, &IndexSet::new())
.await
.unwrap();

assert!(to_remove.pop().unwrap().exposed_name().to_string() == "test");
assert!(to_remove.iter().all(|bin| !bin.is_trampoline()));
assert_eq!(to_remove.len(), 2);
assert!(to_add.is_empty());
}

Expand Down Expand Up @@ -1100,8 +1117,9 @@ mod tests {
get_expose_scripts_sync_status(&bin_dir, &env_dir, &IndexSet::new())
.await
.unwrap();
assert_eq!(to_remove.len(), 1);

assert!(to_remove.pop().unwrap().exposed_name().to_string() == "test");
assert_eq!(to_remove.pop().unwrap().exposed_name().to_string(), "test");
assert!(to_add.is_empty());
}
}
4 changes: 2 additions & 2 deletions src/global/list.rs
Original file line number Diff line number Diff line change
Expand Up @@ -52,13 +52,13 @@ pub fn format_exposed(exposed: &IndexSet<Mapping>, last: bool) -> Option<String>

fn format_mapping(mapping: &Mapping) -> String {
let exp = mapping.exposed_name().to_string();
if exp == mapping.executable_name() {
if exp == mapping.executable_relname() {
console::style(exp).yellow().to_string()
} else {
format!(
"{} -> {}",
console::style(exp).yellow(),
console::style(mapping.executable_name()).yellow()
console::style(mapping.executable_relname()).yellow()
)
}
}
Expand Down
46 changes: 20 additions & 26 deletions src/global/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ pub(crate) mod project;
pub(crate) mod trampoline;

pub(crate) use common::{BinDir, EnvChanges, EnvDir, EnvRoot, EnvState, StateChange, StateChanges};
use pixi_utils::executable_from_path;
pub(crate) use project::{EnvironmentName, ExposedName, Mapping, Project};

use crate::prefix::Prefix;
use crate::prefix::{Executable, Prefix};
use rattler_conda_types::PrefixRecord;
use std::path::{Path, PathBuf};

Expand All @@ -22,33 +23,26 @@ fn find_executables(prefix: &Prefix, prefix_package: &PrefixRecord) -> Vec<PathB
.collect()
}

fn is_executable(prefix: &Prefix, relative_path: &Path) -> bool {
// Check if the file is in a known executable directory.
let binary_folders = if cfg!(windows) {
&([
"",
"Library/mingw-w64/bin/",
"Library/usr/bin/",
"Library/bin/",
"Scripts/",
"bin/",
][..])
} else {
&(["bin"][..])
};

let parent_folder = match relative_path.parent() {
Some(dir) => dir,
None => return false,
};

if !binary_folders
/// Processes prefix records (that you can get by using `find_installed_packages`)
/// to filter and collect executable files.
pub fn find_executables_for_many_records(
prefix: &Prefix,
prefix_packages: &[PrefixRecord],
) -> Vec<Executable> {
let executables = prefix_packages
.iter()
.any(|bin_path| Path::new(bin_path) == parent_folder)
{
return false;
}
.flat_map(|record| {
record
.files
.iter()
.filter(|relative_path| is_executable(prefix, relative_path))
.map(|path| Executable::new(executable_from_path(path), path.clone()))
})
.collect();
executables
}

fn is_executable(prefix: &Prefix, relative_path: &Path) -> bool {
// Check if the file is executable
let absolute_path = prefix.root().join(relative_path);
is_executable::is_executable(absolute_path)
Expand Down
Loading