Advise library authors on how best to depend on child dependencies

[zachdaniel]

This seemed like valuable documentation to have. I'm happy to discuss via a mailing list as well, but since it's not a proposed change, I wasn't sure if it was appropriate.

[cheerfulstoic]

I didn't see the library guidelines docs, GTK these exist!

I want to write on the Elixir Forum thread too, but writing just a bit here 😄

The >= operator is only used once in this whole section. And I get your explanation, but in my mind >= should be the first (but not the last!) thing you should think about in a library. Of course it depends on the dependency, but if you're using the main, most stable parts of the dependency's API, I think you should be using >= (and then maybe later add a and < x.y.z if needed because of some incompatibility)

I think that absolutely sometimes library developers should be using ~> x.y(.x). I think that this documentation could use some examples as to how to make the decision in addition to the abstract description. The biggest example that I often see is just using Jason.encode / Jason.decode. I know that since JSON is moving into core that's going to become an out-of-date example, but I'm sure we could find better ones (like if you're only using Ecto.Repo.get, and a handful of really stable functions for example).

Application developers should (😅) have tests which run whenever there are updates (including in the mix.lock) and/or they should be verifying updates somehow. So I don't think it's up to the library developer to protect the app developer from future updates to sub-dependencies unless a known issue comes up (and if the library maintainer is AWOL, the app developer can limit the version of the sub-dependency without even needing to use override 😉 )

[wojtekmach]

but in my mind >= should be the first (but not the last!) thing you should think about in a library.

I agree to disagree! I believe ~> is a good default for application and library authors (RE that forum thread) as long as people follow semver which they tend to do in my experience. When hex.pm displays example version requirement it would show ~> major.minor (post 1.0) and ~> major.minor.patch (pre 1.0) exactly because of where the breakage is allowed to happen per semver.

FWIW I sometimes use >= without a corresponding < but it is very rare, I'd say exclusively for :ex_doc and :postgrex where the former is a dev-only dependency and the latter has extremely stable public API and I only ever use it in apps anyway.

[whatyouhide]

Libraries should not use >= for their child dependencies. Users might end up upgrading child_dep and taking care of breaking changes, and main_dep would allow that while not necessarily handling those breaking changes.

[sorentwo]

I wholeheartedly agree that libraries should not use >= for dependencies 👍

[cheerfulstoic]

Ok, well... 🤷 😅

I guess the main thing I've been frustrated by a number of times in when libraries authors go AWOL, and I guess that's really the problem I'm trying to solve. And really I'm specifically trying to solve the problem of maintainers going AWOL using a dependency when they're using the most stable parts of a dependency... Yes, semver is great when people use it, but updating the major version generally doesn't mean that the whole API of the library changes.

The options as I see them are a personal fork (which can go out of date), making your own fork (requiring the library name in hex to be something like better_<library> 😅), or override. Maybe there are better options?

But I take the point that an application developer isn't usually going to be able to easily understand what went wrong if a sub-dependency is updated 🤔

I also don't know that I would say libraries should never use >=. All tools are good for different things, so I try never to say never 😉

But maybe the best, main option is tooling to help library maintainers know when there are new versions of their dependencies which aren't covered yet.

Also, this is maybe harder, recommending that people have more than one maintainer...

[ericmj]

I guess the main thing I've been frustrated by a number of times in when libraries authors go AWOL, and I guess that's really the problem I'm trying to solve. And really I'm specifically trying to solve the problem of maintainers going AWOL using a dependency when they're using the most stable parts of a dependency... Yes, semver is great when people use it, but updating the major version generally doesn't mean that the whole API of the library changes.

Is incorrect version requirements really the biggest or most common problem you will face when trying to use a library from 5 years ago that is not being maintained more. I think it's more likely that there will be actual incompatibilities to fix and a bunch of other things to fix like deprecations. Are we not optimizing for something rare that the tools will tell you about and is fixed by adding override: true.

On the other hand if there were no version requirements then you would have to figure out the incompatibilities yourself and hope that you have test coverage for it all, including any transitive dependencies.

The options as I see them are a personal fork (which can go out of date), making your own fork (requiring the library name in hex to be something like better_ 😅), or override. Maybe there are better options?

What is the problem with override?

[zachdaniel]

Copying my comment from the forum here:

I’m suggesting that this is built into the mix dependency resolver. Right now the following happens all the time:

• I want to add foo to my app. I already have bar and baz.
• foo depends on a newer version of bar.
• I can’t update bar because baz depends on it.
• I check how baz and foo use bar, and confirm that its fine to just override.
• So I add {:bar, "~> ...", override: true}
• Someone else adds buzz to the app, which also depends on an old version of bar.
• Problem 1: We never find out that we just overrode bar for the sake of buzz as well.
• Next, foo releases an update that depends on more stuff from the old version of bar.
• Problem 2: mix tells us we can update foo, so we update it and have bugs.

If instead of override: true, I could say:

{:bar, "~> x.x", override: [foo: "x.x.x"]}

which would say “this override only overrides the dependency that foo at exactly version x.x.x has on bar”, then we are protected from any of those accidental changes.

Adding buzz would produce an appropriate dependency conflict warning, solving Problem 1. We can then go look at the code/docs and decide if we want to override the bar dependency for that version of buzz as well.

foo won’t appear to be automatically upgradeable, solving Problem 2.

[cheerfulstoic]

I (and Parker for that matter) said "should not", not "must not", exactly because of this.

Yeah... in my mind "should not" can mean "never" or "basically never" 🤷

It is very easy to say what was stable in hindsight, but very difficult to say what will be stable.

I would disagree 😅 I really should spend some time to come up with examples, so maybe it's not worth continuing here, but I can imagine often being confident enough that a specific library's usage specific usage of a specific depedency, especially when that dependency is large enough, can be pretty much relied upon. An example off the top of my head would be using Phoenix.PubSub.subscribe / Phoenix.PubSub.subscribe (especially if you're not using any opts)

Anyway, the PR might not be the best place. I have lots of other thoughts 😉 I'll get back to the forum thread in a bit. Need to finish up the workday!

[zachdaniel]

In those rare exceptions, it is of course up to the library author. However, as the maintainer of many small to large Elixir packages, I reserve the right to break whatever I feel like breaking in a major release, trusting that others have depended on a major version 😅 But I do see where you are coming from. I think it is better to make the guidelines fit the 99% case, even if there can be valid cases for deviation.

[josevalim]

I agree with the issue you described @zachdaniel. My suggestion is simply to say: {:bar, "~> x.x", override: [:foo, :baz]}, then we can:

1. warn if we need to override anyone else
2. warn if foo or baz are updated and no longer need the override

[josevalim]

💚 💙 💜 💛 ❤️