support for git dependencies

[clemos]

It doesn't seem possible to have git dependencies currently :

"dependencies": {
        "hxnodejs": "https://github.com/HaxeFoundation/hxnodejs.git"
}

The above seems to fail. Or maybe there is another syntax ?
If not, then I think it would be nice to support git dependencies, pretty much like npm does.

[nadako]

note that it would be great to also be able to specify commit number/branch/tag (like npm does I think)

[clemos]

Yes, it's like npm install git+ssh://[email protected]:npm/npm.git#v1.0.27
see https://docs.npmjs.com/cli/install

[hexpunk]

+1 for this

[nadako]

As mentioned in #135 (comment), I think we should adopt NPM dependency format, see https://docs.npmjs.com/files/package.json#dependencies.

[ncannasse]

I'm fine with following anything that is quite standardized as NPM seems to be, as long as we keep backward compatibility with our previous format, so users can continue downloading and submitting libraries using the old format.

[boozook]

1. In this case haxelib.json must be exists in root of repo;
2. What about another (potential) supported systems such as Mercurial or Subversion? Should we detect what use by url? No :).

[porfirioribeiro]

Maybe something like Cargo from rust lang?
http://doc.crates.io/manifest.html#the-[dependencies]-section
It accepts git repo's as a depedency. The Cargo.toml is either used on Lib's or app's to specify options including dependencies.

[nadako]

That is interesting stuff, thanks!

[boozook]

I like the Cargo impl too. but this is new format and will change the our current format:
From {"libname" : "ver"} to {"libname" : {"version":"", "git/hg/svn":"", "branch/tag/rev":""} }.

But we can really preserve backward compatibility. We can easy support old(current) and new formats - detect format via Reflection and use the special provider (parser) for it (Or EitherType).

[boozook]

Another one variant - namespace-like:

"dependencies": {
        "libFromHaxelibA": "",
        "libFromHaxelibB": "10.2.4",
        "libFromGit": "git::https://github.com/HaxeFoundation/libFromGit.git",
        "libFromHg": "hg::https://bitbucket.org/HaxeFoundation/libFromHg"
}

But without tags/revs/branches.

[boozook]

I love the NPMs format but it doesn't support any other control version system.

[nadako]

The npm tool itself doesn't support anything but git, but the URL format looks extensible enough (e.g. hg+http://... similar to supported-by-npm git+http://....

[nadako]

As discussed in #105, it would be also nice to support http://... urls, which is also supported by NPM.
I propose we borrow npm's url format and use it for both "haxelib install" and dependency urls. We can implement it using some kind of "providers" architecture as it's done in @ibilon's sources related to #217

[nadako]

This raises more questions: should we / how to store multiple versions of a library checked out from some vcs.

For example, library A declares "somelib": "git://some/lib#rev1" dependency which is a rev1 commit from some/lib while library B wants "somelib" to be e.g. git://other/lib#rev2.

Technically, we can support that by generating some unique version identifier from a vcs url (like git__some_lib__rev1), but do we want to go that route?

[ibilon]

It depends if we can make the hypothesis that the user won't modify the content.

In that case when installing we only fetch, and when compiling we do a checkout of the required commit/branch/tag.

[grosmar]

Hello,

We have the same need so I've investigated the sources and this thread, and have some conclusion and some code now.

• As @ncannasse said we should be backward compatible, so even if we extend the syntax we should keep working with the existing one. I've considered it through all the points.
• I've found a strange concept at this moment. When we use a library, we can install it's dependencies two ways: if the library has it's own build.hxml we can install them from it with haxelib, or if it has a haxelib.json and we install it through haxelib install or haxelib git commands then it read up the dependencies.
  But for the haxelib.json there is no implementation to install it when is it on my local drive (like when you use npm install and it reads up the local package.json and install all the dependency from it)
  I think we should be able to follow the concepts of other package managers, and we should be able to express our dependencies with versions in the haxelib.json locally, even if this is our main project. And after we should just express the need in the build.hxml. But for keeping the backward-compatibility, i would implement also the git dependency support in the build.hxml
  So for this i've tried the following syntax:
  haxelib install <haxelib.json path>
  So the haxelib install itself looks for the haxelib.json file in the current directory. If it find it, start it's installing. If it get as a parameter the path for the haxelib.json than it will install that one.
• About multiple git dependencies: I would follow the concept that @nadako said: generate git_somebranch syntax. But only as a second step, because it raise a lot of questions. Anyway, now i would change that behavior in the code when install git dependency install it as a dev dependency. There is no point in it, it's more than enough to just set the current version to the git. It works perfectly and no need to force to that version.

To make it backward compatible, i've tried with the following syntax:
build.hxml:
-lib something:git:https://github.com/grosmar/something.git#commit-ish
This is fully compatible with the existing haxe compiler, because it splits at ":", so it will understand to look for the "git" version folder. If we later introduce support of multiple versions, than it can be git_mybranch:https://...
#commit-ish would mean anything that can be passed to the git checkout command

haxelib.json
dependencies: { "something: "git:https://github.com/grosmar/something.git#commit-ish" }

Later we can introduce maybe a structured syntax for the haxelib.json, to describe all the features of git dependency (like subfolder).

So for now i would not follow fully the npm style git+..., but i would use git:... to keep the compatibility.

According to these conclusions i've made some implementation here:
https://github.com/grosmar/haxelib/tree/library_git_dependency

This branch implement the following features that i've mentioned above:

• haxelib install <haxelib.json path>
• build.hxml describe git dependency
• haxelib.json describe git dependency

I still need to write the tests if we can agree on the things.

Also we could think about some extra syntax to describe subdir like git:https://github.com/grosmar/something.git#commit-ish@subDir
And we should be able to express the version (tags). The #commit-ish part covers it, but maybe it would be nice to directly express the need only for tags. But i have no good idea for this. Any suggetions?

What do you think guys?

[nadako]

I'm yet to look at the code, but the proposition looks nice.

I agree about expanding haxelib.json purpose so it's not only used for "libraries", but for projects in general, similar to npm's package.json. We can even add more functionality to it, like haxelib test that reads from haxelib.json that is has to run haxe test.hxml, but that's another topic.

git:https://github.com/grosmar/something.git#commit-ish@subDir

another option would be to add another colon here, e.g. git:https://github.com/grosmar/something.git#commit-ish:subDir :)

as for tags, I think commit-ish is sufficient.

[grosmar]

Let me know if you have time to check the repository.
I happily finish the devs if we can finalize the concept.

[nadako]

I'm looking at the code right now. Can't say I like it very much, but that's not about your part, but haxelib codebase in general. I think if we decide on the design here, any code will do as long as it works, then we can refactor it from there.

Also, I remembered one issue with losing dev-mode for vcs dependencies - using a subdir inside a repository for the haxelib (see #263 (comment)). I wonder though if that feature used at all and worth caring about...

[nadako]

Anyway, I can't think of any issues regarding the proposed syntax.

[grosmar]

I also cannot say that i like the code, but i didn't want to change deeply the codebase because of this implementation.

I can look into the subdir devmode problem later. But for me this subdir stuff sounds a bit nasty anyway. I prefer one repository for one lib, so why should i need a subdir?

So can we agree on the followings as a final decision:

• support haxelib.json file also for local dependency descriptions, and use the following syntax to install them:
  haxelib install <haxelib.json path> (if no path given it tries to read haxelib.json in the current directory)
• support git dependencies in haxelib.json with the following syntax:
  dependencies: { "libname: "git:https://github.com/grosmar/something.git#commit-ish:subdir" }
• support git dependencies in build.hxml with the following syntax:
  -lib something:git:https://github.com/grosmar/something.git#commit-ish:subdir

commit-ish means anything that can be passed to git checkout command
subdir is reference for the subdirectory of the git repository

And later we can improve the implementation and use versioned git folder like this:
git_commit-ish_subdir

This way we can have multiple git version checked out and used. But I don't want to mix it now.

So can we agree on this syntax? If yes, i finish the development and create the tests.

[nadako]

I can look into the subdir devmode problem later. But for me this subdir stuff sounds a bit nasty anyway. I prefer one repository for one lib, so why should i need a subdir?

No idea, I think I have never needed that.

support haxelib.json file also for local dependency descriptions, and use the following syntax to install them: haxelib install <haxelib.json path> (if no path given it tries to read haxelib.json in the current directory)

Fine with me 👍

support git dependencies in haxelib.json with the following syntax: dependencies: { "libname: "git:https://github.com/grosmar/something.git#commit-ish:subdir" }
support git dependencies in build.hxml with the following syntax: -lib git:https://github.com/grosmar/something.git#commit-ish:subdir

Not sure about the subdir syntax, but if we lose that feature, we don't need it anyway :)

I don't feel like I can decide for the HF here though, so let's summon @ncannasse @Simn @andyli. Check it out guys, this feature is quite important IMO.

Also ping @fponticelli and @andywhite37, since they are using the hmm tool they wrote, so they may have an opinion on this!

[andyli]

I really want this feature too. However, I don't think we will have enough time to make sure the design and implementation is solid enough for the upcoming haxe 3.3 release. So I will suggest looking into it after that.

[Simn]

I agree with Andy, let's focus on getting selfupdate stable so we can take our time with features like this.

[grosmar]

Ok, then I'll work on it according to "docs" that i've wrote above, and i'll let you know guys when it's finished.

[nadako]

Fine with me! We'll focus on making haxelib easily updateable and then will introduce this feature.

[nadako]

Just letting you know that we've implemented the new self-update mechanism and new version notification. So we can deliver this at any time.

[ibilon]

I've been thinking more about this and I don't think this is a good idea.

• Breaks our ability to get the full recursive list of dependencies, and therefor breaks the "no conflict" check
• Adds git/hg as a requirement besides haxe/neko (not for all libraries, so random from an user perspective)
• If the url is allowed without a specific commit/tag goodbye reproducibility (if you install it at a bad time can be broken)
• Can't do the things described in Refactoring the install mechanism #267 and Look into using mancoosi research for versioning and dependency solving #290

Also what happens when two libraries add a different git url under the same name?

"dependencies": { "theSuperLib": "https://github.com/someguy/superlib.git" }

"dependencies": { "theSuperLib": "https://github.com/someotherguy/haxe_lib_super.git" }

And since I fail to see the point of this (if you are sending your lib to haxelib with not the dependencies?) that looks like a nightmare for nothing as far as I see (could be extremely useful and I just don't realize it, but still the issues apply).

[nadako]

@ibilon These concerns are indeed valid for libraries published on lib.haxe.org, but publishing a library with git dependencies are just bad style and is headache for library's author (if they want their library to be used by someone anyway). I doubt many authors would do that. At least it's not a problem on NPM.

OTOH, as discussed above, I still think it makes sense to expand the purpose of haxelib.json file to something similar to package.json in NPM world. I.e. it'd be a manifest file file for your application, even if you're never going to publish it. It can describe dependencies, run configuration and everything available for libraries, but it'll do it for your project. In this use case, git dependencies make a lot of sense IMO.

Aside from that, it can be useful when working in a company with private repositories, when different teams create different modules and it's often much less annoying to just use git instead of doing releases on some private haxelib server.

Also what happens when two libraries add a different git url under the same name?

One option also discussed above is generating an unique version directory name like git_<repo_path>_<commit>.

[ibilon]

I doubt many authors would do that.

I think it shouldn't be allowed, if we can make the hypothesis that libraries on lib.haxe.org only have dependencies on lib.haxe.org then there are several things we can do like #290.

it'd be a manifest file file for your application

I see, I'm not sure haxelib is the best place to have this, but I don't want another file format either.

In that case the format should be relaxed, all the fields (except the name?) should be optional,
for instance having at least a contributor for your application that'll never go on lib.haxe.org doesn't make much sense.

Should it be picked by haxelib install all? And/or used with haxelib install haxelib.json?

[back2dos]

I see the advantage of having published haxelib only depend on other
published haxelibs.

At the same time, not all libraries are on haxelib for various reasons,
like lack of stability or dependency on an unstable Haxe version.

I do think that haxelib git should be able to deal with other git
dependencies.

Among other things, this makes it easier to use haxelib with an ecosystem
of private repos.

On Fri, Mar 18, 2016 at 3:54 PM, Valentin Lemière [email protected]
wrote:

I doubt many authors would do that.

I think it shouldn't be allowed, if we can make the hypothesis that
libraries on lib.haxe.org only have dependencies on lib.haxe.org then
there are several things we can do like #290
#290.

it'd be a manifest file file for your application

I see, I'm not sure haxelib is the best place to have this, but I don't
want another file format either.

In that case the format should be relaxed, all the fields (except the
name?) should be optional,
for instance having at least a contributor for your application that'll
never go on lib.haxe.org doesn't make much sense.

Should it be picked by haxelib install all? And/or used with haxelib
install haxelib.json?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#238 (comment)

[ibilon]

I do think that haxelib git should be able to deal with other git dependencies.

I agree,
as long as it's not on haxelib you should be able to use any type of dependency: haxelib, git, http/local zip or even a path.

Though about something, why is the git url used as version? That's clearly the library "name" (if you think of name as a uri it makes more sense) and commit/tag/branch the version.
Same for a zip file or path (which will always have an empty version).

That way there's a lot less name conflict between libraries, especially between published and non published. With the exception of path all name would uniquely define a library.

[back2dos]

On Sat, Mar 19, 2016 at 11:32 AM, Valentin Lemière <[email protected]

wrote:

Though about something, why is the git url used as version? That's clearly
the library "name" (if you think of name as a uri it makes more sense) and
commit/tag/branch the version.

Well, not necessarily. The same library can be in different places, for
some reasons. For example assume you use a lib from
http://github.com/joe/libx and you need to make a fix and use it right
away, then you can fork to http://github.com/ibilon/libx and use that as a
dependency while waiting for your PR to get accepted.

Same for a zip file or path (which will always have an empty version).

This also depends. The following url defines a version rather clearly:
https://github.com/HaxeFoundation/haxelib/archive/3.2.0-rc.1.zip

If we had https support (which doesn't seem worthy of inclusion in the
standard lib, because clearly it's not as fundamental as a routing
mechanism or an ORM layer or a template engine), then I would say GitHub
releases would actually be the preferable way to depend on libs from
GitHub, as it doesn't rely on git and is generally much faster.

That way there's a lot less name conflict between libraries, especially
between published and non published. With the exception of path all name
would be unique.

Well, firstly, I think -lib https://github.com/joe/libx is a bit
cumbersome to write. Second of all, being able to shadow a haxelib with a
git lib is a feature to me. If that's not what you want, you can define the
git dependency under a different name.

[nadako]

IIRC our repository server currently validates that all dependencies are present in the database. I think we could as well check for absence of git (or, rather, anything but "normal") dependencies when submitting to lib.haxe.org. That makes sense IMO.

[grosmar]

Sorry guys, i didn't disappear but became extremely overloaded in this time. As the things getting better I will get back to this issue.

[grosmar]

Hi!

Finally I'm back on the issue. I've updated with all the changes from the master since, and here is now the branch that contains the modifications:
https://github.com/grosmar/haxelib/tree/git-dependencies

As I see there are still a few open questions, also I added some.

Let's talk about them and make a decision:

• Tag support. So I've checked, it works well with just saying #tags/mytagname because these works as a branch in git, so i would not add more complexity to syntax.
• "Also what happens when two libraries add a different git url under the same name?". I still not manage different branches under different versions as we said before, like git_branchname. Is it okay to have a first version without this feature?
• Check subdependency branch changes. When I try to install the dependencies from haxelib.json, and I find a repository that has been checked out, I try to git update, and I do the same with all the subdependencies. But still not managing if a subdependency branch changed. I don't know if I should switch the branch automatically, or it should be part of the second step, when we manage multiple branches with different names?
• Subdir support. So we agreed to not support it through git dependencies, but as i see from the code now someone implemented to set devmode if subdir has been defined, so this was if some still want to use git dependency with subdir he can do it through haxelib git ...
• IMO we should not support any git dependency on lib.haxe.org I think it would be crazy as @ibilon and @back2dos said. So we should have a closed ecosystem there.
• "If the url is allowed without a specific commit/tag goodbye reproducibility". I think your right, but normally it's the responsibility of the creator to make the decision. It's the same with normal haxelib dependencies: if you do not define any version it will use the latest one. So i would not change on this concept, even because when we develop something with private repos with lot of subdependencies then we always want to have the latest one from everything.
• "Should it be picked by haxelib install all". I think no, because it would break backward compatibility. Someone who get used to use that method will see different behavior now, and even more because every package manager or build system have a very simple way to express to setup in the current directory. For example npm install, ant (without any parameter!), etc... So haxelib should also have a simple way to express setup.

Guys, please look through the problems and if we can agree what to do we should finish this long topic. I think many people would be happy with this feature!

[nadako]

Hey @grosmar, thanks for working on it!

Tag support. So I've checked, it works well with just saying #tags/mytagname because these works as a branch in git, so i would not add more complexity to syntax.

Agreed, that should work just fine.

"Also what happens when two libraries add a different git url under the same name?". I still not manage different branches under different versions as we said before, like git_branchname. Is it okay to have a first version without this feature?

Check subdependency branch changes. When I try to install the dependencies from haxelib.json, and I find a repository that has been checked out, I try to git update, and I do the same with all the subdependencies. But still not managing if a subdependency branch changed. I don't know if I should switch the branch automatically, or it should be part of the second step, when we manage multiple branches with different names?

I'm not sure about these ones, have to think some more. Right now, I feel like the feature wouldn't be very useful without supporting multiple repos/branches for a lib...

IMO we should not support any git dependency on lib.haxe.org I think it would be crazy as @ibilon and @back2dos said. So we should have a closed ecosystem there.

Yeah, just check the dependencies at upload time (both on client and server), so that it only accepts released versions.

"If the url is allowed without a specific commit/tag goodbye reproducibility". I think your right, but normally it's the responsibility of the creator to make the decision. It's the same with normal haxelib dependencies: if you do not define any version it will use the latest one. So i would not change on this concept, even because when we develop something with private repos with lot of subdependencies then we always want to have the latest one from everything.

Agreed, that's really not much different. However, this gave me an idea that we could have a command that "bakes" current active haxelib versions in a haxelib.json dependencies somehow. I think npm also has something like this.

"Should it be picked by haxelib install all". I think no, because it would break backward compatibility. Someone who get used to use that method will see different behavior now, and even more because every package manager or build system have a very simple way to express to setup in the current directory. For example npm install, ant (without any parameter!), etc... So haxelib should also have a simple way to express setup.

Sorry, I didn't understand what that was about, could you elaborate?

[grosmar]

Hey,

"Also what happens when two libraries add a different git url under the same name?". I still not manage different branches under different versions as we said before, like git_branchname. Is it okay to have a first version without this feature?

Check subdependency branch changes. When I try to install the dependencies from haxelib.json, and I find a repository that has been checked out, I try to git update, and I do the same with all the subdependencies. But still not managing if a subdependency branch changed. I don't know if I should switch the branch automatically, or it should be part of the second step, when we manage multiple branches with different names?

I'm not sure about these ones, have to think some more. Right now, I feel like the feature wouldn't be very useful without supporting multiple repos/branches for a lib...

I was thinking about it, and it's not so trivial. Let's imagine you want to work on that get dependency at the same time as you develop. Normally everyone get used to switch with git between branches when he need to switch, but now we want to override this behavior, and make clones by branches (even more maybe commits!!!).
If we want to go this way we should examine this problem deeply, because i think it's a bit of misuse of git, and i'm not sure what kind of side effects it could have.
Still need to think about it...

Agreed, that's really not much different. However, this gave me an idea that we could have a command that "bakes" current active haxelib versions in a haxelib.json dependencies somehow. I think npm also has something like this.

It would be nice, and yes npm have it. But do it as a different improvement. I think this change already getting a bit big...

"Should it be picked by haxelib install all". I think no, because it would break backward compatibility. Someone who get used to use that method will see different behavior now, and even more because every package manager or build system have a very simple way to express to setup in the current directory. For example npm install, ant (without any parameter!), etc... So haxelib should also have a simple way to express setup.

Sorry, I didn't understand what that was about, could you elaborate?

I just tried to say 2 things:

• We should not change the behavior of haxelib install all. So I would not use this command to parse ./haxelib.json, because this command already used by many people, and if it suddenly starts to install dependencies from that haxelib.json file that would be confusing and even it could override dependency versions...
• We should have a simple way to install your dependencies with a command haxelib install (without any parameter), because every package manager or build system have this simple way to express, without parameters.

Everything else that you marked as agree I consider as we finished that topic and we can go in that way that I've wrote above (including subdir support, we agreed on it before i think)

[grosmar]

Ping? :)

[nadako]

Sorry, I totally forgot about your comment, since I'm pretty busy at the moment.

Alright, let's not bother with multiple git dep versions right now. The single git checkout should be already useful for setting up local projects together with local repositories (newrepo).

What would be a nice addition is to display current git remote and branch/commit for haxelib list.

We should not change the behavior of haxelib install all. So I would not use this command to parse ./haxelib.json, because this command already used by many people, and if it suddenly starts to install dependencies from that haxelib.json file that would be confusing and even it could override dependency versions...

Yeah, haxelib install all installs from hxml files, why are we even talking about parsing haxelib.json there?

We should have a simple way to install your dependencies with a command haxelib install (without any parameter), because every package manager or build system have this simple way to express, without parameters.

I agree and that complies with my vision of haxelib.json file being similar to package.json in npm world (so it's not just for libraries, but for any project). However, this should be given much more thought and I think we should do it later as it's not directly related to this particular issue.

---

PS man, I more and more often think we should just throw away haxelib and write a brand new package manager, getting the best from npm, cargo, opam, gem...

[st3veV]

Hi guys,
just to let you know this issue didn't die. I'm taking over the changes from @grosmar and PR is coming soon.

[Efimster]

Hi,
Is this available in any haxelib version?

[RealyUniqueName]

It already works for me in haxelib bundled with Haxe 3.4.x like this:

"dependencies" : {
  "library": "git:ssh://[email protected]:7999/repo.git#e462962ee27"
}

I guess to setup a dependency on a repo from github, you need to specify it like this:

"dependencies" : {
  "library": "git:https://github.com/HaxeFoundation/hxnodejs.git"
}

[Efimster]

thank you