feat: Enable global expose with nested paths #2362

bahugo · 2024-10-27T15:09:49Z

I tried to be as explicit as I could, I modified Mapping attribute to executable_relname (relative path but with no extension), and executable_name() is now evaluated based on relname.

ruben-arts

Thanks for the work! I've left some comments but the general implementation looks good.

Could you add an integration test to tests/integration/test_global?
Could you add this case to the documentation in https://pixi.sh/latest/features/global_tools/#exposed docs/features/global_tools.md

If you need help creating a dummy package let us know! It should be pretty straight forward to create your situation in a dummy package and test its logic in there.

src/global/mod.rs

ruben-arts · 2024-10-28T07:29:19Z

src/global/project/manifest.rs

+            executable_relname: Path::new(&executable_relname)
+                .with_extension("")
+                .to_str()
+                .unwrap_or(&executable_relname)
+                .to_string(),


with extention doesn't work for python3.12 as that will create python3.

You can use pixi_utils::executable_from_path here aswell.

I've just changed new method, I've used strip_executable_extension (because executable_from_path drop nested directory)

src/global/project/manifest.rs

bahugo · 2024-10-28T20:45:58Z

@ruben-arts I tried to add an integration test.
I modified dummy-channel-1, to a add a nested executable.
I'm not sure about which directory is $PREFIX/bin is it the bin directory were pixi will store exposed executables?
Anyway, I added rattler-build dependency to pytest feature and ran pixi r update-integration-test-data but it failed:

│ ╭─ Running build script
 │ │ %SRC_DIR%>IF "" == "" (
 │ │
 │ │  call %SRC_DIR%\build_env.bat
 │ │ )
 │ │ La syntaxe de la commande n’est pas correcte.
 │ │ %SRC_DIR%>mkdir -p $PREFIX/bin
 │ │ %SRC_DIR%>echo "dummy_e on windows"  1>$PREFIX/bin/dummy_e.bat
 │ │ Le chemin d’accès spécifié est introuvable.
 │ │ × error Script failed with status 1
 │ │ × error Work directory: 'C:\Users\bahugo\dev\pixi\tests\integration\test_data\channels\dummy_channel_1\bld\rattler-build_dummy_e_1730147400\work'
 │ │ × error To debug the build, run it manually in the work directory (execute the `./conda_build.sh` or `conda_build.bat` script)
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 ╰─────────────────── (took 0 seconds)
Error:   × Script failed

Traceback (most recent call last):
  File "C:\Users\bahugo\dev\pixi\tests\integration\test_data\update-channels.py", line 37, in <module>
    main()
  File "C:\Users\bahugo\dev\pixi\tests\integration\test_data\update-channels.py", line 18, in main
    subprocess.run(
  File "C:\Users\bahugo\dev\pixi\.pixi\envs\default\Lib\subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['rattler-build', 'build', '--target-platform', 'win-64', '--output-dir', 'channels/dummy_channel_1', '--recipe', 'recipes/dummy_channel_1.yaml']' returned non-zero exit status 1.

I tried to run conda_build.bat but it failed too.
I'm not familiar with rattler-build, do you have an idea?

bahugo · 2024-10-29T21:48:42Z

@ruben-arts I tried to add an integration test. I modified dummy-channel-1, to a add a nested executable. I'm not sure about which directory is $PREFIX/bin is it the bin directory were pixi will store exposed executables? Anyway, I added rattler-build dependency to pytest feature and ran pixi r update-integration-test-data but it failed:

│ ╭─ Running build script
 │ │ %SRC_DIR%>IF "" == "" (
 │ │
 │ │  call %SRC_DIR%\build_env.bat
 │ │ )
 │ │ La syntaxe de la commande n’est pas correcte.
 │ │ %SRC_DIR%>mkdir -p $PREFIX/bin
 │ │ %SRC_DIR%>echo "dummy_e on windows"  1>$PREFIX/bin/dummy_e.bat
 │ │ Le chemin d’accès spécifié est introuvable.
 │ │ × error Script failed with status 1
 │ │ × error Work directory: 'C:\Users\bahugo\dev\pixi\tests\integration\test_data\channels\dummy_channel_1\bld\rattler-build_dummy_e_1730147400\work'
 │ │ × error To debug the build, run it manually in the work directory (execute the `./conda_build.sh` or `conda_build.bat` script)
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 ╰─────────────────── (took 0 seconds)
Error:   × Script failed

Traceback (most recent call last):
  File "C:\Users\bahugo\dev\pixi\tests\integration\test_data\update-channels.py", line 37, in <module>
    main()
  File "C:\Users\bahugo\dev\pixi\tests\integration\test_data\update-channels.py", line 18, in main
    subprocess.run(
  File "C:\Users\bahugo\dev\pixi\.pixi\envs\default\Lib\subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['rattler-build', 'build', '--target-platform', 'win-64', '--output-dir', 'channels/dummy_channel_1', '--recipe', 'recipes/dummy_channel_1.yaml']' returned non-zero exit status 1.

I tried to run conda_build.bat but it failed too. I'm not familiar with rattler-build, do you have an idea?

@ruben-arts I've managed to update integration test channel by compiling them use debian in wsl.

I ran integration tests on windows, the test I added ran successfully, but I got a fail for another test:

E           AssertionError: Return code was 106, expected 0, stderr: No pyvenv.cfg file

tests\integration\common.py:55: AssertionError

I don't get why it failed.

ruben-arts · 2024-10-30T09:01:27Z

docs/features/global_tools.md

+```
+you can also omit the extension
+```
+pixi global install dotnet --expose dotnet=dotnet\dotnet


I tried this example on Linux but that package doesn't seem to work. Do you understand why the dotnet setup is using a non bin/xx setup for it's binaries?

I have no Idea.
I had never tried to install dotnet on linux but you're right, it's installed in .pixi/envs/dotnet/lib/dotnet/dotnet
For some reason everything is painful with windows tools...
Maybe a more cross platform or generic tool would be better for the documentation.

Yeah, I contacted @dhirschfeld as he is a pixi user and the maintainer of dotnet maybe he has some smart things to say about this.

Not sure about smart - the TL;DR is that it's just a binary repackage of upstream and that seems to be the way it's packaged by Microsoft.

The package sets a number of env vars in an activation script, including the PATH. I'd like to port setting/unsetting the env vars to the JSON format (assuming that works with pixi?) but haven't gotten around to it.

Yes pixi supports that 👍

Is there another example package, where this would be required, that does work on linux?

ruben-arts · 2024-10-30T09:04:57Z

While reviewing I checked it out and did some small changes, here is the diff patch you could apply:

diff --git a/src/global/project/manifest.rs b/src/global/project/manifest.rs
index cac1fa50..ecea5f4e 100644
--- a/src/global/project/manifest.rs
+++ b/src/global/project/manifest.rs
@@ -432,6 +432,8 @@ impl Manifest {
 #[derive(Debug, Clone, Serialize, Deserialize, Hash, PartialEq, Eq)]
 pub struct Mapping {
     exposed_name: ExposedName,
+    // The executable_relname is a executable name possibly with a parts of a path in it to match on.
+    // e.g. `dotnet/dotnet` will find `$PREFIX/lib/dotnet/dotnet`
     executable_relname: String,
 }
 
@@ -450,13 +452,14 @@ impl Mapping {
     pub fn executable_relname(&self) -> &str {
         &self.executable_relname
     }
+
+    // Splitting the executable_relname by the last '/' and taking the last part
+    // e.g. 'nested/test_executable' -> 'test_executable'
     pub fn executable_name(&self) -> &str {
-        if let Some(executable_file_name) = Path::new(&self.executable_relname).file_name() {
-            return executable_file_name
-                .to_str()
-                .unwrap_or(&self.executable_relname);
-        };
-        &self.executable_relname
+        Path::new(&self.executable_relname)
+            .file_name()
+            .and_then(|name| name.to_str())
+            .unwrap_or(&self.executable_relname)
     }
 }
 
diff --git a/tests/integration/test_global.py b/tests/integration/test_global.py
index a5ab777e..93471054 100644
--- a/tests/integration/test_global.py
+++ b/tests/integration/test_global.py
@@ -308,12 +308,6 @@ def test_expose_basic(pixi: Path, tmp_path: Path, dummy_channel_1: str) -> None:
     )
     assert not dummy1.is_file()
     assert not dummy3.is_file()
-    # extension = "exe" if os.name == "nt" else "sh"
-    # # Add nested dummy1
-    # verify_cli_command(
-    #     [pixi, "global", "expose", f"nested_dummy=nested/dummy.{extension}"],
-    #     env=env,
-    # )
 
 
 def test_expose_revert_working(pixi: Path, tmp_path: Path, dummy_channel_1: str) -> None:
(END)
 
diff --git a/tests/integration/test_global.py b/tests/integration/test_global.py
index a5ab777e..93471054 100644
--- a/tests/integration/test_global.py
+++ b/tests/integration/test_global.py
@@ -308,12 +308,6 @@ def test_expose_basic(pixi: Path, tmp_path: Path, dummy_channel_1: str) -> None:
     )
     assert not dummy1.is_file()
     assert not dummy3.is_file()
-    # extension = "exe" if os.name == "nt" else "sh"
-    # # Add nested dummy1
-    # verify_cli_command(
-    #     [pixi, "global", "expose", f"nested_dummy=nested/dummy.{extension}"],
-    #     env=env,
-    # )
 
 
 def test_expose_revert_working(pixi: Path, tmp_path: Path, dummy_channel_1: str) -> None:

ruben-arts · 2024-10-30T09:05:41Z

My browser bugged out and closed this issue by accident.

bahugo · 2024-11-04T18:46:17Z

Something is not working with python activation in integration test, I don't know why...

  Windows 10.0.26100 on D049 on  main +/- is 📦 v0.34.0 🦀 v1.81.0
~\dev\pixi ➜ C:\Users\bahugo\dev\pixi\pytest-temp\popen-gw0\test_sync_dependencies0\bin\python-injected.bat --version
> No pyvenv.cfg file

  Windows 10.0.26100 on D049 on  main +/- is 📦 v0.34.0 🦀 v1.81.0
~\dev\pixi ❯ C:\Users\bahugo\dev\pixi\pytest-temp\popen-gw0\test_sync_dependencies0\envs\test\Lib\venv\scripts\nt\python.exe
> No pyvenv.cfg file

ruben-arts · 2024-11-05T10:00:46Z

I'll give this a go today! See if I can fix the issues

ruben-arts · 2024-11-06T14:28:51Z

I'm sorry you are experiencing such a crazy merge conflict train. The code works well, I hope you have another example we can use in the documentation.

bahugo · 2024-11-06T17:14:40Z

I'm sorry you are experiencing such a crazy merge conflict train. The code works well, I hope you have another example we can use in the documentation.

No problem, I'm enjoying learning a bit of the internals of pixi.
I was thinking of pyside6 tools but it's also different...
on windows:

pixi global install pyside6 --expose Library/lib/qt6/bin/qmlls

on linux:

pixi global install pyside6 --expose lib/qt6/bin/qmlls

For rebase conflicts, it might be better if dummy channels weren't in git index (they could be generated in CI).

ruben-arts · 2024-11-15T10:18:13Z

If fixed the merge issue, and tested some more.

Your implementation allows to use the following command on both platforms:

pixi global install pyside6 --expose qmlls=qt6/bin/qmlls

However it doesn't run anything on my machine. I'm fine with skipping an useful example as I'm assuming its not a big set of users hitting this. And other wise we'll find a example through that.

bahugo changed the title ~~Enable global expose with nested paths~~ feat: Enable global expose with nested paths Oct 27, 2024

bahugo marked this pull request as draft October 27, 2024 15:19

bahugo marked this pull request as ready for review October 27, 2024 16:24

ruben-arts requested changes Oct 28, 2024

View reviewed changes

ruben-arts reviewed Oct 30, 2024

View reviewed changes

ruben-arts closed this Oct 30, 2024

ruben-arts reopened this Oct 30, 2024

bahugo force-pushed the main branch 2 times, most recently from 2d24894 to b2b117d Compare November 4, 2024 08:57

bahugo force-pushed the main branch from e542a7a to 3dfcfff Compare November 6, 2024 09:25

bahugo and others added 14 commits November 6, 2024 13:53

rebasing onto prefix main

5047d55

Fixed formatting and executable_name call

576f900

Removed dead code

5c65cbe

Added test for path stripping with nested path

b1a6b73

Removed constrain on executable directeory for global executables

0b38434

rebasing Fixing executable detection for global install

b67e8d8

Updated format mapping for relname

4870d0d

Exposing relname to manifest

8c8dcc4

Using relname in project

b1df886

format

65debc8

Deleting extension in Mapping new method

bdcbabd

Using strip_executable_extension

11c56f6

using pixi_utils::executable_from_path in global/mod.rs

f50e300

Attempt to test dummy package with nested exposed

96f2350

bahugo and others added 8 commits November 6, 2024 13:53

Added documentation for nested expose in global install

e11105d

Fixed testing nested_dummy

0d93260

reverted addition of rattler-build to pixi.toml

58fb4ce

Rebasing :Updated dummy channels

db1d8e7

format

f9b491c

applying pre-commit

5c4ffb6

building integration tests channels after rebasing

2302ee7

fixes from PR comments + format

69ea5ba

bahugo force-pushed the main branch from 3dfcfff to 69ea5ba Compare November 6, 2024 13:03

updating dummy channels

ac526ed

ruben-arts added 2 commits November 14, 2024 16:31

Merge remote-tracking branch 'upstream/main' into bahugo/main

4cb9ad9

fix: nested test

4c8114d

ruben-arts added 3 commits November 15, 2024 11:18

Merge branch 'main' into main

ca93d62

format

edc84c1

fix: test on windows

0254a01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Enable global expose with nested paths #2362

feat: Enable global expose with nested paths #2362

bahugo commented Oct 27, 2024

ruben-arts left a comment

ruben-arts Oct 28, 2024

bahugo Oct 28, 2024

bahugo commented Oct 28, 2024

bahugo commented Oct 29, 2024

ruben-arts Oct 30, 2024

bahugo Oct 30, 2024 •

edited

Loading

ruben-arts Oct 30, 2024

dhirschfeld Oct 31, 2024

ruben-arts Nov 5, 2024

ruben-arts Nov 6, 2024

ruben-arts commented Oct 30, 2024 •

edited

Loading

ruben-arts commented Oct 30, 2024 •

edited

Loading

bahugo commented Nov 4, 2024

ruben-arts commented Nov 5, 2024

ruben-arts commented Nov 6, 2024

bahugo commented Nov 6, 2024

ruben-arts commented Nov 15, 2024

feat: Enable global expose with nested paths #2362

Are you sure you want to change the base?

feat: Enable global expose with nested paths #2362

Conversation

bahugo commented Oct 27, 2024

ruben-arts left a comment

Choose a reason for hiding this comment

ruben-arts Oct 28, 2024

Choose a reason for hiding this comment

bahugo Oct 28, 2024

Choose a reason for hiding this comment

bahugo commented Oct 28, 2024

bahugo commented Oct 29, 2024

ruben-arts Oct 30, 2024

Choose a reason for hiding this comment

bahugo Oct 30, 2024 • edited Loading

Choose a reason for hiding this comment

ruben-arts Oct 30, 2024

Choose a reason for hiding this comment

dhirschfeld Oct 31, 2024

Choose a reason for hiding this comment

ruben-arts Nov 5, 2024

Choose a reason for hiding this comment

ruben-arts Nov 6, 2024

Choose a reason for hiding this comment

ruben-arts commented Oct 30, 2024 • edited Loading

ruben-arts commented Oct 30, 2024 • edited Loading

bahugo commented Nov 4, 2024

ruben-arts commented Nov 5, 2024

ruben-arts commented Nov 6, 2024

bahugo commented Nov 6, 2024

ruben-arts commented Nov 15, 2024

bahugo Oct 30, 2024 •

edited

Loading

ruben-arts commented Oct 30, 2024 •

edited

Loading

ruben-arts commented Oct 30, 2024 •

edited

Loading