Dynamic Revisions with Ivy

[aecollver]

I tried using dynamic revisions (http://ant.apache.org/ivy/history/2.3.0/ivyfile/dependency.html#revision) like this:

resolvers += "myOrganization" at "s3://repo.myOrganization.s3-us-west-2.amazonaws.com/releases"

libraryDependencies ++= Seq(
"myOrganization" % "foo" % "1.0.+",
)

This is the result I see:

[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: myOrganization#foo;1.0.+: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
sbt.ResolveException: unresolved dependency: myOrganization#foo;1.0.+: not found

I'm trying to use dynamic versions because I publish artifacts with this form: majorVersion.minorVersion.buildNumber-gitCommitId

My best guess (from reading the source code) is that getObjectMetadata returns a 404 in getURLInfo when SBT/Ivy is scanning for candidates. If this isn't something you have the time or interest to support, would you be open to a pull request?

[aecollver]

I haven't figured out for certain if/where ApacheURLLister is used, but its presence suggests that Ivy is expecting an HTML directory listing to resolve dynamic revisions:

https://github.com/sbt/ivy/blob/2.3.x-sbt/src/java/org/apache/ivy/util/url/ApacheURLLister.java

[tpunder]

Yeah I was thinking the same thing after seeing that ApacheURLLister class. I'll take a look tomorrow and see if I can confirm it is expecting an Apache style directory listing. If that's the case then it seems like it would be pretty easy to add that functionality.

[tpunder]

I did some digging into this and so far don't see an obvious way to add the dynamic revisions functionality. Here is what I've found:

org.apache.ivy.util.url.ApacheURLLister is used in org.apache.ivy.plugins.repository.url.URLRepository but it is hard coded to only work with http and file URLs

There is also a org.apache.ivy.plugins.resolver.util.ApacheHttpURLLister class that implements the org.apache.ivy.plugins.resolver.util.URLLister interface which seems like what we would want to implement for S3 except that I don't see anyway to register a new URLLister for s3:// URLs.

I turned on logging of the calls in my S3URLHandler class to see what was being called if I requested a dynamic revisions and for stuff like "1.0.+" I don't even see any calls being made to the S3URLHandler class. But for stuff like "[1.0.0," I do see calls being made except it's just treating that as the version verbatim and not doing any dynamic stuff:

[info] S3URLHandler.getURLInfo(s3://maven.frugalmechanic.com/snapshots/com/frugalmechanic/fm-test_2.11/[1.0.0,/fm-test_2.11-[1.0.0,.pom, 0)
[info] S3URLHandler.getURLInfo(s3://maven.frugalmechanic.com/releases/com/frugalmechanic/fm-test_2.11/[1.0.0,/fm-test_2.11-[1.0.0,.pom, 0)

Now that might just be because there is something higher up in Ivy that should be handling the dynamic revisions but since it doesn't work for s3:// URLs it is just falling back to requesting the version verbatim.

Next Steps

Digging around the code some more I think a possible solution might be to create a new class that extends org.apache.ivy.plugins.repository.url.URLRepository and adds support for the s3:// URLs and then also create a new class that extends org.apache.ivy.plugins.resolver.RepositoryResolver and is modeled after the org.apache.ivy.plugins.resolver.URLResolver class that uses the extended URLRepository class. Then it might be possible to hook into the org.apache.ivy.core.settings.IvySettings and override the url property that it is reading from the typedef.properties file to point to our extended URLResolver.

I don't have any more time to mess around with it today (and possibly not this week) but can circle back to this issue when I have some more time. In the meanwhile if you want to play around with it I'd happily except a pull request or even just prototype or proof of concept code that I can clean up or model a solution after.

[aecollver]

Are you familiar with https://github.com/ohnosequences/sbt-s3-resolver? I chose fm-sbt-s3-resolver over sbt-s3-resolver because I liked the optional path feature and how it's configured (simple!). It turns out that sbt-s3-resolver has implemented a Ivy repository (similar to your proposed next steps).

I opened an issue against sbt-s3-resolver because it also has an issue with dynamic revisions (ohnosequences/sbt-s3-resolver#31).

[aecollver]

I ended up submitting a patch to https://github.com/ohnosequences/sbt-s3-resolver because it already implements a resolver (vs. a protocol handler). However, I ended up putting this together: https://github.com/munchii/sbt-s3-resolver (it lacks an optional path feature at the moment, but publishes an Apache-style directory listing so a static website hosted with S3 is browsable and works with dynamic revisions).

[tpunder]

I'm going to leave this open for now since I would still like to add support for dynamic revisions when I have time. I spent a little bit of time digging through the Ivy & SBT source code but haven't found the right place to hook in an updated repository/resolver in such a way that keeps the existing SBT configuration syntax.

I'm glad you found a solution that works for you. I am familiar with https://github.com/ohnosequences/sbt-s3-resolver. I started off using that plugin but had a few issues with it:

• Their AGPL v3 License was potentially problematic for me
• The SBT configuration syntax was a little weird
• Wasn't published to maven central
• Needed separate S3 buckets for releases/snapshots

[tpunder]

I spent some more time on this and got it mostly* working. I tried really hard to make it work with the existing syntax but just couldn't find a way given how IVY and SBT are setup.

Here is what I ended up with (in the current 0.5.0-SNAPSHOT):

// This brings in the implicit class with the "atS3" method
import fm.sbt.S3._

// The only change here is using "atS3" instead of just "at"
resolvers += "FrugalMechanic Snapshots" atS3 "s3://maven.frugalmechanic.com/snapshots"

// Dynamic revisions
libraryDependencies += "com.frugalmechanic" %% "fm-private" % "0.1.0-+"

• - What I haven't done yet is auto publish an index.html page with the directory listing to S3 whenever artifacts are updated so that an public S3 bucket can be published to and then used with a normal http:// resolver. But this should be pretty simple to add.

[tpunder]

Slight update on syntax. The import fm.sbt.S3._ is no longer required since I'm now using sbt.AutoPlugin:

// The only change here is using "atS3" instead of just "at"
resolvers += "FrugalMechanic Snapshots" atS3 "s3://maven.frugalmechanic.com/snapshots"

// Dynamic revisions
libraryDependencies += "com.frugalmechanic" %% "fm-private" % "0.1.0-+"

See commit cc9f1b5

[omervk]

@tpunder While you've noted that this is implemented, I'm not sure it was ever released, since it doesn't currently resolve (version 0.16.0).
Could you please give a short update on this?

Also, thank you for making this plugin - it's super useful!

[tpunder]

Hi @omervk, nothing new to report since the last update on this issue (almost 4 years ago!).

Is the atS3 syntax not working for you?

resolvers += "FrugalMechanic Snapshots" atS3 "s3://maven.frugalmechanic.com/snapshots"

I don't use this feature so its very likely that it could be broken.

[omervk]

After working on a clean repro, I found that the x.+ syntax works (yay! 🎉) but that other forms, such as latest.integration or latest.release don't (which are really what I need).

(You can see a full list of them here: https://ant.apache.org/ivy/history/latest-milestone/ivyfile/dependency.html#revision)

Do you still have enough context to try taking a look and see if there might be an easy fix for this?

Thanks again!