Should we use jarjar to repackage specific dependency versions

[ludoch]

I did a simple java main Maven project.
I see many dependencies in my project jars, for example guava 18, httpclient401,jodatime27 etc...
If I wand to use for my other application logic guava 12 or 25 for whatever reasons, we will have a version conflict...
JarJar processing? How is it done for other runtimes? Seems to be a generic Veneer issue.

[aozarov]

Guava and Joda tend to be pretty good with backward compatibility.
Some of the direct dependencies will go away when we start using Java 8 APIs and gRPC, so
the scope of this problem will, hopefully, is going to get smaller.
In general I think this is part of the "maven" spirit and newer version of libraries should be backward compatible but we could provide an option of a jar download (using gitHub "release" place - https://github.com/GoogleCloudPlatform/gcloud-java/releases) that would include all dependencies via JarJar.
I didn't check all other gcloud clients, but e.g. Python could "suffer" from the same issue.

[aozarov]

One more point which I want to mention is that there may be some libraries (guava for an instance) that would actually benefit from sharing the same classes between user-code and the library (even though library does not expose them directly). e.g. ImmutableXXX.copyOf() would avoid the copy if it detects that the input is also an Immutable type.

[jboynes]

Not managing dependencies well becomes a real pain for application developers. Maven does provide some ways to mangle dependency versions but that does not work all the time due to, often unintentional, non-compatible changes.

JarJar avoids this but with the penalty of multiple versions of the same functionality. It can also have issues with classes loaded by reflection e.g. from config files like the ServiceLoader.

Java runtimes use a variety of ClassLoader tricks to try and provide isolation. For example, Tomcat uses a class shutter to prevent an application seeing its classes, or it can be configured with split ClassLoader trees. Most full-spec JavaEE containers (e.g. JBoss, Geronimo, Glassfish) have moved to using OSGI to isolate library jars. Java9 with Jigsaw gives us modules but is not baked yet.

There are a few things we can do to simplify this problem:

• Separate API sections from each other (e.g. DataStore from PubSub)
• Separate the user-facing classes from the stub's implementation
• Limit the libraries used by stub implementations (i.e. adding a library warrants extra scrutiny)
• Use common naming patterns (e.g. -api and -impl) so that JARs that don't need to be scanned can be identified
• Provide profiles with multiple APIs bundled together (cf. JavaEE's Web and Full profile)

Our standard runtime(s) would support a specific profile, make all the user-facing API classes visible to the application from the (Java) container's ClassLoader (i.e. from Jetty or Tomcat's public "lib" directory), and hide the implementation classes and dependencies using JarJar rewriting and ClassLoader tricks. The user's web-application would reference the APIs as "provided" dependencies and not be impacted by how we implemented the stubs.

When building a custom runtime, the runtime's creator would define the profile they wanted to expose to their applications and set up the ClassLoader to match using the same techniques.

Basically, we'd provide fine-grained building blocks that allow a user to set up the class-path they want, along with pre-assembled profiles for the most common cases.

[garrettjonesgoogle]

We don't have any imminent plans for this. I'm going to close this out now.