-
-
Notifications
You must be signed in to change notification settings - Fork 228
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
dub.selections.json woes #829
Comments
Just adding my voice to this. While I'm not exceptionally versed in dub, this is not the first time dub.selections.json has caused versioning issues for me. This time I had a small project using DSFML version " |
I agree that it is both annoying and error-prone. Just recently I was going mad trying to reason why I can't reproduce build failure from CI on my local machine (on identical commit hash) until I have finally spotted that there is a |
One thing needs to be clearly separated: Bugs are bugs and not conceptual issues. 2/3 of the text in this ticket is based on #528, which is fixed in the latest beta and doesn't have anything to do with dub.selections.json. Assuming that everything works correctly, I don't think that there is anything error prone to the approach. Any quite frankly, it's vital not only to get a reproducible build across machines, a major issue for other packaging has been that large dependency trees became virtually unmanageable. In theory this wouldn't be the case, if everyone adhered to SemVer and all dependencies were referenced using Now let's imagine that we'd do without a per-package set of version selections and we'd always use the latest available versions. There are two possibilities:
I should probably dig up the old discussions about this topic with @jacob-carlborg, or something from Ruby/Bundler. But there simply are a lot of compelling arguments for this approach, even if it may not always seem necessary. * Especially considering that maybe 80% of the packages have 0.x.x versions, which have no backwards compatibility rules. |
BTW, @Dicebot: Do you remember if you got upgrade suggestions during the local build? It would probably help to give them a different color on the console, as currently it's rather difficult so spot such information. However, this would have happened the same way without a set of selections, if you wouldn't have had the latest dependencies installed on the local machine (except if the automatic upgrade mode was the default). On the other hand, putting dub.selections.json under version control would have solved it no matter what the environment is (it really should be versioned at least in 90% of the cases). |
This is actually what I'd want, but that's because I use SemVer and As for the rest, you made a very good case for the current approach and I think this ticket can be closed. |
A couple of notes on what's necessary to have
"Implements proper optional dependency semantics, where using an optional dependency can now be controlled using dub.selections.json" In my opinion that's broken, at least according to how I think |
Why? It would of course be nice to have a CLI for selecting/deselecting optional dependencies and for upgrading specific dependencies. But I don't see any reason to disallow manual edits. |
Certainly not. Though it was rather long time ago and could be fixed in later versions.
Intuitively I would expect some user prompt whenever upgrade is available if dependency version is open. Which in relation with next statement ..
.. is probably what makes me feel most uneasy about this solution. If I wanted to pin dependencies hard I could as well just specified it using |
Here's the deal. I want Dub to somehow look down the dependency graph for a project to ensure that the same dependencies are always installed. I though that was how As soon as you start manually editing The only reason to manually edit
The solution is really simple, the optional dependencies should go in |
To me that is very unexpected from a tool, a command line tool. That Dub checks for new dependencies for each build. I would expect that from a GUI application, like a browser, but not from a command line tool.
It might work with one or two dependencies but when you have 40 dependencies (including indirect dependencies) it's a PITA. If you really want to look down the dependencies in But seeing your opinions about the dependencies of DStep (which are 3) I assume you don't have any dependencies at all in your projects 😃. |
Doesn't it do so anyway already?
How is it different from doing the same in Though I guess it is a problem of many dub packages declaring version compatibility they don't really check.
There are from 3 to 8 dependencies in our apps (if you refer to Sociomantic) which so far feels quite manageable with even something as blunt as git submodules (== always hard locking of the version) :) My opinion of dstep dependencies comes not from their count but triviality. |
Yes, that's what I think is unexpected for a CLI tool.
Dub is generating
Not sure I understand. |
A standard Rails (3.2) project comes with six direct dependencies, resulting in 40 (the above was just an arbitrary number I picked 😃) dependencies in total. That's what I have to deal with at work every day. We probably have around 30 Rails projects. Try dealing with that manually. Of course, many of these projects are using different versions of Rails, resulting in a different dependency graph. Now apply that across a couple of different versions of Ruby as well and you have dependency hell 😃. |
Leaving aside that it might get unmanageable for large dependency graphs, this would be fine for application packages. However, for library packages it would mean that the library is basically unusable with other libraries that have an overlapping dependency graph, because exact version dependencies are very likely to produce conflicts. dub.selections.json only applies to the root package and thus doesn't have that problem. |
That is how it works (well not "installed", but "included in the dependency graph"). I don't know where the disconnect happens, but I'll try to explain the idea: dub.json basically defines the topology constraints of the graph, while dub.selections.json defines a subset, which represents a single concrete dependency graph. dub.selections.json itself isn't organized as a graph, because each package can only occur as a single version within the dependency graph (everything else would usually lead to linker errors in D's case), and thus a simple list of all packages is sufficient for a unique definition.
dub.json specifies which dependencies are considered optional, while dub.selections.json specifies if an optional dependency is chosen or not. That to me appars to be a logical approach.
You'll get a message whenever you do something in dub.selections.json that can't satisfy the dependency graph constraints layed out in the package recipe hierarchy, and the tool can still continue to work with the manually edited file, so it's not an all or nothing decision. But basically, yeah, there should be a CLI-way to control optional dependencies, it just hasn't been implemented.
dub.json does specify optional dependencies. dub.selections.json just determines if they are actually chosen or not. |
It checks once per day and caches the results. I think this is a highly useful feature, because otherwise you'd have to constantly speculatively upgrade or monitor the registry to notice upgrade opportunities. But the console output really needs some color, because currently it's too easy to overlook such hints. |
I would expect that the if an optional dependency is used is specified in the Application A has a dependency, library B. Library B has an optional dependency, library C. I would expect B to specify C as an optional dependency in its Alternatively if Dub had an install command I would expect it to be to be a flag to choose if the optional dependencies should be installed as well.
I'm not sure if I agree with that. What's the reason to have two files the developer needs to edit? I would expect the developer should only edit |
Such a flag would surely be a possibility. With the small difference that it wouldn't tell it to install the optional dependencies, but to choose them in the context of the installed package (this would actually imply installation, too, but also means that the optional dependencies take part in the build process, which a simple installation doesn't imply - anymore, since the new optional-semantics have been implemented).
Sorry, but I can't follow you there. If there is not a CLI command for doing the modifications, how else would that work? A new GUI client, or some deamon process? If there is a CLI command for all necessary operations, the developer doesn't need to edit the file anymore, if there isn't, manual edits may be necessary in some cases. |
All I'm talking about is that there shouldn't be two files that a developer needs to edit. It should only be one file,
I just looked at the documentation for optional and default dependencies. Are you supposed to specify optional and default on the same dependency in the same |
Yes, specifying both will produce an optional dependency that is chosen by default. Leaving off the default part will leave it unchosen by default. "By default" means that when no dub.selections.json exists, it will create one with/without the optional dependency present. This part of the optional dependency semantics is based on dub.selections.json, i.e. it lets the root package control which optional packages are used. Separate from that, a package can force inclusion of an optional dependency of another package by depending on it in a non-optional or optional+default way, in the latter case still leaving the possibility open to control the final inclusion from dub.selections.json. |
It's quite simple. dub.json includes all meta information of a package and dub.selections.json contains the concrete set of dependency versions used when building the package as a root package. Depending on what kind of information is changed, the appropriate file needs to be changed (by tool or by hand). Do you suggest to put everything into one file? |
I suggest leaving |
And that's already the case. What you want effectively seems to be to disallow configuring optional dependencies specific to the root project, but that can actually be very useful. |
No, I'm against configure that in It's very simple, I think that there should only be a single file that user needs to edit. |
It could be configured there, but it would be redundant. dub.selections.json already naturally fills that role.
Would you also be against adding a functionality to "dub upgrade" a single dependency (which implicitly edits dub.selections.json)? It's conceptually not much different from a function to select/deselect an optional dependency. |
You probably already know that I consider whole Ruby/Rails ecosystem completely broken and being beyond salvation? :) I am aware of only one approach to avoid dependency hell problems - self-discipline combined with rigorous continuous integration. That means:
Rails approach is exactly the opposite of all those goals. I definitely don't want that and dub right now pretty much stands on crossroads. It can go the path of verifying declared dependencies and demanding strict semver from library packages, ideally with automated beta-testing of all packages being the part of future auto-tester replacement. Or it can go Ruby way with no guarantees that any library or project is actually usable unless you use exact pinned version of everything. |
Please explain who these solve dependency hell.
How do you do this with a moving target? Every time you run Dub you can get a new set of dependencies if you don't lock the dependencies. You can be running Dub on your local machine and all tests pass. Then you push to your CI system and suddenly everything breaks because you get a different set of dependencies. Semantic versioning does not help.
It's not possible to verify that a package follows semver. I would not trust anyone to use semver correctly. What do you do if a a package specifies ">= 0" as the version for one of its dependency? I guess your answer is: don't use that package 😉. Someone might change an internal data structure, array to linked list, in a patch version, which is perfectly fine since if it doesn't change the API. That can cause performance regressions which can be very difficult to find. I want Dub to be useable for anyone and not just your specific needs. |
Then you can just as well claim that having the dependencies in dub.json is redundant as well since dub.selections.json already takes care of that.
No, I would not be against that. I actually thought that was already possible. |
What? That is clearly not redundant. dub.json doesn't contain information to pin a version. But by its pure structure dub.selections.json already implies if an optional dependency is chosen or not - no additional field in dub.json is necessary for that.
So that would also manually modify dub.selections.json - I ask you where is the difference to select or deselect an optional dependency therein with a similar command? |
I think it's different. If I put it like this: your active choices (dependencies) go in I'm tried of arguing this. I think manually editing |
Upgrading a dependency is an active choice, too! The concept goes like this: dub.json contains all the meta information about a package and dub.selections.json contains the exact packages used for building it as a root project. But yes, this whole talk doesn't go anywhere. |
Minimizing/discouraging dependency pinning prevents creation of virtual version ecosystems within the registry when there is a certain set of libraries/application that only work with certain versions and are not usable with rest of registry - but it is not actually reflected in their published spec and only can be figured out by checking repos with Essentially, the idea is about making bad but easy approach harder intentionally harder.
Ideally - by making package registry your moderator, running approval tests with possible snapshots of (supposedly) declared dependencies. Of course checking all combinations is impossible by checking few random ones and marking package as bad if it is ever detected as "liar" is one possible solution to motivate developers to be more disciplined and thoughtful when defining their dependencies. And for applications and not libraries I don't see any reason to specify open dependencies in main
It is possible to detect if package does not follow source-applied SemVer though and issue some "punishment". When it comes to source libraries there isn't much point of speaking of SemVer in context of API - breaking change definition becomes "any project stops compiling non non-major version increase". Which is quite doable for hypothetical CI bound with registry. Of course perfect verification is not possible. But detecting at least some violation and marking such projects hurts people pride and thus motivated them to put more effort into something they most commonly don't care about (pedantic versioning). That is the "discipline" part I have been talking about before. Going rubygem way, the contrary, embraces the sloppy approach and tries to hide it with help of the tooling. |
How do you test that a package is following semver in practice? |
Whenever new non-major version is uploaded, you try running tests for some of other already uploaded projects which are supposed to passing compiling based on the version declaration. Success doesn't guarantee anything, of course, but failure is a clear sign that semver is not followed. |
I don't think this is going to work very well. |
There will be, as long as there are ~> version strings. For non major versions of course, but major ones are allowed yo break things anywgay.
Of course, this problem doesn't have reliable technical solution. The very point is to move into social engineering domain, motivating very people submitting packages to check how they define versions. Right now there is no reason to care at all. |
No. If I upload a new package, of course there will not be no other packages that have that newly uploaded package as a dependency because it didn't exist. |
You can also have stupid version requirements like |
??? There is existing package with dependency
I believe dub must refuse to accept such nonsense dependencies if it is agreed that all version strings are SemVer compliant. |
No, I mean when
I've said previously that "Software using Semantic Versioning MUST declare a public API" Most of the time applications does not declare a public API.
I don't see how that goes against SemVer. The specification doesn't mention much of how version requirements should be specified. |
Got burned by that damn thing again. Had this line in
And there was also old
Guess which one was chosen for build (with no warnings or errors). |
When I create a package with those lines, I get:
But unfortunately without the emphasis ;-) |
Maybe be it was added in 1.0.0? The machine in question had one of earlier releases installed and I don't see anything like that in logs. Though the fact why it even agrees to compile version that violates |
People seem confused by
dub.selections.json
on IRC.In particular it is the 3rd time people ask how to get the latest tags in dependencies (I then advise to remove dub.selections.json + dub clean-caches).
=>
dub upgrade
is supposed to help but I guess it doesn't work everytime, which is something unrelated.Honestly I somehow get why
dub.selections.json
anddub upgrade
are there. But it is a bit backwards, it is more common to want the latest available version for everything (according to version selectors of course) that to want to stay on previous versions. If you want to pin dependencies into a specific version, you can always modifydub.json
.The other case I can think of is to pin the version of a sub-dependency, but the case is pretty rare. If a program with dependency A => B => C fail, it's quite easy to reproduce it with a program B => C. And committing
dub.selections.json
breaks the nice encapsulation of sub-dependencies DUB provides (A => B => C then A shouldn't know C).Opinion: the cases where we do want dub.selections.json are a small minority.
I think it should be an opt-in choice. I remember a time where it wasn't there and never missed it.
In particular if a library pins dependencies instead of using SemVer, it creates diamond problems quickly.
The text was updated successfully, but these errors were encountered: