Local repositories and subdirectories

[nadako]

Current README states one issue with local repositories (AKA newrepo), so I thought it would be fair to create a proper issue :)

if you mistakenly run a haxelib command in a subdirectory of your project, it will be executed on the global repo (to be fixed)

That means we should look for .haxelib directory not only in CWD but also in all its parents, right? This is fairly easy to implement, one thing I'm not sure about is performance implications. But I guess if it works for Git, it should be okay.

Is there any other issues with this?

ping @ncannasse and @back2dos as the original authors of the functionality and README note

[back2dos]

I've raised the issue with Nicolas when he authored the functionality, but we didn't really agree on a satisfactory solution then and decided to leave it at that. Recursing up seems only logical, building on the status quo.

The thing is that the cost of recursing up from the cwd will affect everyone, even those not using newrepo at all. Also, you don't interact with Git as often as you call haxe --display, so what is acceptable for one, may not hold for the other. Still, in the end you will need to try it and the measure the effect.

[nadako]

I think that depends on whether we want to encourage use of local repositories, like node/npm does.
If it's a thing that encouraged and everyone is using it, the recursion should be quite short so it's probably okay (haxe --display caches the results IIRC, so it should not be a major problem with completion).

However right now, most common use is just a global repository, so it'll be an unnecessary overhead. We should indeed measure if it's significant or not.

[back2dos]

As I regularly point out, module resolution is not a feature of npm but of require. It just so happens that npm will install modules where require will look for them. I have proposed to Nicolas to similarly decouple haxe and haxelib, but he rejected such invasive changes.

Also, to further go into the differences between Haxe and node, require is reliably known to have a cache, while haxe's cache, i.e. the completion server, is as opt-in, so we probably shouldn't rely on it.

[ibilon]

If haxe used something like HAXE_LIB_FINDER env var to know which program to send its path request it'd be decoupled without invasive or breaking change.

Would be similar, instead of installing libs where haxe expect them, you would give it the information the way haxe expects it.

Pro: The package manager is free to do anything as long as path do what it's supposed to.
Con: It doesn't allow mixing two package manager, but that's not a good idea to do that imo.

[desttinghim]

I ran into an issue with this recently while using hxcpp, and I thought it might be an error with hxcpp. I opened an issue there (link), and @hughsando informed me that this was a limitation of haxelib.

There definitely should be some way for child directories to be able to use local repositories from parent directories. Maybe a flag to indicate a local repository should be used? Another possibility is to be able to specify exactly the repository you want to use.

[hughsando]

For the hxcpp issue, I might simply be able to run haxelib from the current directory and specify a full path to the hxcpp generated build.xml file. I will have to look into this.

[jeremyfa]

Hello,

This issue is personally the main reason I didn't switch to local haxelib repositories yet. Looking to parent directories is a very convenient feature of node modules resolutions and it would be great if haxelib did the same.

I am reading that it might cause a performance hit to check every parent subfolders, but is that really a concern? I would expect that any modern file system is able to check the existence of .haxelib in a dozen of directories in a fraction of a millisecond. I would also argue that it seems anyway a super cheap operation compared to most build/compilation step involved when summoning haxe or haxelib?

Is there another reason that stop us from making this change?

[nadako]

I think that everyone who cares about local/reproducible setups have moved to Lix nowadays :)

[jeremyfa]

I do care about local/reproducible builds, and it is probably not very constructive to point to a third party tool instead of addressing the problem at the source.

Lix is great, but AFAIK also relies on haxeshim with custom haxe/haxelib commands and their load of complexity, while fixing this .haxelib resolution could already improve a lot the situation with "stock" haxe toolkit.

I do hope solving this issue will be considered because that really is the last step to make newrepo very useful without any other dependency.

The main problem I am hitting with current behaviour is that I can't run a command (haxelib run ...) without cwd being at the exact same directory as .haxelib. Running hxcpp as mentioned above is a good example.

[ncannasse]

I'm fine with haxelib looking recursively for .haxelib , and any other feature that helps with local/reproducible builds.

I agree that while Lix offers additional features, it's not the standard that comes directly with Haxe, so we either need to move our whole toolchain by default to another package manager or keep improving haxelib.

[jeremyfa]

That was fast! Thanks a lot 🙏