Make used Gradle SourceSet and Configuration configurable

[remmeier]

The Jib Gradle plugin is hard-coded to make use of the "main" classpath and "runtime" configuration. It should be possible to specify those two values in the JibExtension to allow customizations:

• add further libraries to classpath for Docker only.
• remove libraries from the classpath as they are already available in a base image.
• change the source set because the projects main "purpose" is not being a docker image, but something else and have docker-related sources of the project in a non-main source set. We have that pattern quite frequently when we build plugins for other system that are then installed as initContainer. So the main artifact of the project would be the "plugin", whereas the initContainer is more a side-thing related to deployment. Currently we are forced to make setup two projects rather than just having two source sets.

Many/most other Gradle plugins allow to do the same thing to maintain flexiblity of what gets touched by plugin. And effort for allowing that is minimal.

The ticket is related to #894, but as requested created a ticket on its own. In most cases the changes requested here should be simpler and suffice for developers to allow doing customizations. The other ticket then would allow even more customization at expensve of additional complexity.

[loosebazooka]

add further libraries to the classpath for Docker only

Can you explain this point a little more? It seems like you can either use the runtime scope or the extraDirectories option.

remove libraries from the classpath as they are already available in a base image.

This can be handled by the compileOnly or provided mechanisms?

change the source set because the projects main "purpose" is not being a docker image, but something else and have docker-related sources of the project in a non-main source set. We have that pattern quite frequently when we build plugins for other system that are then installed as initContainer. So the main artifact of the project would be the "plugin", whereas the initContainer is more a side-thing related to deployment. Currently we are forced to make setup two projects rather than just having two source sets.

I'm a little confused about this point. In reality it sounds like two projects IS what you want. They are separate artifacts and that are somewhat related.

[remmeier]

I'm a little confused about this point. In reality it sounds like two projects IS what you want.
They are separate artifacts and that are somewhat related.

in this case the created images/initContainer are more of a build artifact rather than something warranting a project on its own. Examples where we have this for example are for Jenkins, Elastic and R3 Corda. We have around a dozen such projects and more coming. So what we would like to be possible to do is a Gradle plugin that takes that plugin project and jib and transparently setups an image/initContainer as part of the build. Important here are two things: the jib-related things cannot be on the runtime classpath and all logic to create the image can be put in a reusable fashion into a Gradle plugin.

add further libraries to the classpath for Docker only
Can you explain this point a little more? It seems like you can either use the runtime
scope or the extraDirectories option.

e.g. if you want to have a project both as RPM and Image. For either we end up with slightly different runtime classpaths. Goes a bit into a similar direction as above where one could have one rather than three projects.

remove libraries from the classpath as they are already available in a base image.
This can be handled by the compileOnly or provided mechanisms?

The baseImage is more of a Docker thing that should not leak into the Gradle runtime classpath. By using compileOnly we would loose the ability to make use of it outside of the Docker setting. But this is maybe more of another story, created #1436 a while ago. But being able to fiddle around with sourceSets, configurations and classpaths could provide a good entrypoint into such topics, in particular since so far there seems no consensus how to approach the problem in a generic manner in jib (there are more tickets related to layering). There are a variety of different possiblities in achieving this and nothing may fit every use case. The current lack of support is quite noticable in the layering/size of your images and registry.

But this are just a few examples where one may would like to have a bit more flexilbity. I can imagine all kind of other use cases. The Jar, JavaExec, War, Test and most of the Java classpath-related Gradle tasks allow to set the Configuration for this reason. For sure not the default use case, more part of those other 5%...

[dmurat]

I think that supporting custom source sets and configurations will be very useful. Otherwise, any Gradle project that uses these features cannot use jib. Exactly my case right now.

Please, provide these as jib's configuration options.

[cromefire]

This can be handled by the compileOnly or provided mechanisms?

A really important use case which at least sounds to be related is developmentOnly() for example (because it should put those deps into a separate configuration, namely developmentOnly). I have this case with a spring app right now, where jib just packs reactor-tools and spring-boot-devtools into the container, which clearly do not belong there and it's also pretty much impossible to extract it into a different module because it's just a conditional addition for development purposes.

It basically seems like JavaPlugin.RUNTIME_CLASSPATH_CONFIGURATION_NAME should be made configurable here (change it to runtimeElements for spring for example):

jib/jib-gradle-plugin/src/main/java/com/google/cloud/tools/jib/gradle/GradleProjectProperties.java

Line 193 in 27a949f

.getByName(JavaPlugin.RUNTIME_CLASSPATH_CONFIGURATION_NAME)

I may look into making a PR for that if it's easy to pull off.

[chanseokoh]

@cromefire see the discussion in #2336 (which also includes a potential solution to the spring-boot-devtools issue). Basically, Maven/Gradle is the source of truth that defines which dependencies are required at runtime, and Jib just asks Maven/Gradle for the information. That said, it is the user's responsibility to set up Maven/Gradle dependencies correctly.

Just FYI, Jib has a general-purpose filtering extension (Maven / Gradle) that you can use to remove arbitrary files or move into other layers.

[cromefire]

Basically, Maven is the source of truth that defines which dependencies are required at runtime

Well I'm using gradle in this instance, which has a different system, so maven profiles won't help here. If anything, configurations seem to be a similar thing to maven profiles in the use case. I think you may be confusing gradle and maven here, as there are no profiles in gradle.

That said, it is the user's responsibility to set up Maven dependencies correctly.

Well they are properly set up in a configuration so they can be properly excluded, there's just the lack of an option to tell the plugin to use the proper "non-dev" configuration

Just FYI, Jib has a general-purpose filtering extension (Maven / Gradle) that you can use to remove arbitrary files or move into other layers.

While this would probably technically be working, it would be a PITA to filter the files your self, if there's just an easy fix for it.

[chanseokoh]

Sorry, I did mean to say Gradle instead of Maven. There's no difference between Gradle and Maven in this regard. (In #2336, you'll see that we have the Spring Boot extensions for both Maven and Gradle.)

There are multiple ways to have a profile-like behavior in Gradle. Perhaps an easy way is to use Gradle properties.

if (project.hasProperty('dev')) {

While this would probably technically be working, it would be a PITA to filter the files your self, if there's just an easy fix for it.

I agree using profiles is a much simpler way. I guess using a filter extension anyways requires you do use profiles. (BTW, just in case, the filtering extension is a different extension than the Spring Boot extension mentioned in #2336.)

[chanseokoh]

There are multiple ways to have a profile-like behavior in Gradle. Perhaps an easy way is to use Gradle properties.

Actually @loosebazooka has more expertise in dependency configuration in Gradle and will have some opinions about setting up conditional dependencies in Gradle.

[cromefire]

(In #2336, you'll see that we have the Spring Boot extensions for both Maven and Gradle.)

Okay I've missed that one, that seems to work for spring

There are multiple ways to have a profile-like behavior in Gradle. Perhaps an easy way is to use Gradle properties.

I don't see a point in point in duplicating things that are already in place here.
From the gradle docs:

A Configuration represents a group of artifacts and their dependencies.

Which seems to pretty much what you'd want in this case and also allows you to be more flexible than with properties

[cromefire]

I may look into making a PR for that if it's easy to pull off.

I didn't really find the correct place where the dependencies are being resolved (yet, there's a lot of places where the project configurations and dependencies are mentioned), so sadly I wasn't able to get it working, so if somebody has a clue where that's happening, I can probably make it work

[loosebazooka]

Well gradle holds two different references to the runtime classpath. One as a Configuration and one as a property on a SourceSet. We use both of these to determine how to arrange the final artifact. You're going to have to modify both of those somehow in GradleProjectProperties.java.

The problem with this approach is that if you consider the gradle application plugin simliar to the jib plugin, they both act on the main sourceset and the default runtimeClasspath. It's not clear to me how much special handling needs to be done to handle non-standard builds.

Even configuring custom Jars seems to requires some sort of manipulation of the underlying CopySpec.

It would be helpful to show an example of how another packager handles this? And and example of how you think the configuration should be surfaced.

[cromefire]

Sure I can probably dig something up there but it'll probably take till the weekend sometime next week for me to get back to this

[loosebazooka]

What we really need is an example of someone achieving this via another packager, like an example using Jar (which probably uses the underlying archive/copyspec constructs) or an example using Application plugin.

For instance, how would you fill this out?

sourceSets {
  custom {
    // what goes here that will result in how the packager fulfills its directive?
  }
}

configurations {
  customImplementation.extendsFrom implementation
  customRuntime.extendsFrom runtime
}

I imagine for any packager to get this done correctly, either

1. it would require some sort of source merging:

sourceSets {
  custom {
      srcDirs sourceSets.main.java.srcDirs
  }
}

2. or some sort of multi sourceSet definition

// somewhere in jib config
sourceSets = ["main", "additional"]

[cromefire]

So some samples I came across are the license plugin: hierynomus/license-gradle-plugin#184
And the shadow jar plugin also seems to allow configurations: https://imperceptiblethoughts.com/shadow/configuration/dependencies/

[loosebazooka]

I guess what I'm asking for is an example of how you've set up your configurations/sourcesets that would require this.

Jib can potentially accommodate to a point, but without a user correctly configuring things, we don't want to be a position of debugging someone's gradle build.

[loosebazooka]

@remmeier @dmurat can you share some small example build with multiple source sets and how you expect things to be configured?

[cromefire]

So basically I just have spring boot, where there is the extension, but that one only filters the dev tools and not all developmentOnly. To reproduce, basically start a new project at spring initializr and add something as developmentOnly()

[chanseokoh]

We are still waiting on someone to provide us with a concrete example (#1778 (comment)), as the team doesn't have a lot of Gradle expertise.

[cromefire]

Here's a quick demo: https://gitlab.com/cromefire_/jib-exclude-demo
There are 3 ways of executing it:

• If you run it in dev mode devtools and h2 are present:

  Shell

  ./gradlew :bootRun

  Output

  Trying to open h2 connection, this should fail...
  It worked, h2 was *not* successfully excluded
  

• If you run the jar both are not present:

  Shell

  ./gradlew :bootJar
  java -jar build/libs/demo-0.0.1-SNAPSHOT.jar

  Output

  Trying to open h2 connection, this should fail...
  java.sql.SQLException: No suitable driver found for jdbc:h2:mem:
  [...]
  It failed, h2 was successfully excluded
  

• If you run it via jib with spring extension devtools are correctly excluded (explicit blacklist), but h2 is included even though it's a dev dependency:

  Shell

  ./gradlew :jibDockerBuild
  docker run demo:0.0.1-SNAPSHOT

  Output

  Trying to open h2 connection, this should fail...
  It worked, h2 was *not* successfully excluded
  

You can Inspect the dependencies (in a specific configuration) via ./gradlew :dependencies [--configuration <configuration name>].

In this spring boot app specifically the are 2 configurations of interest:

• runtimeClasspath: This seems to be used for jib and it contains both prod and dev dependencies.
• productionRuntimeClasspath: This should be used by jib (in this specific instance, it should really be configurable) as it only contains the production dependencies.

Here's a scan so you can see it right in your browser: https://scans.gradle.com/s/zrexr57liwk2s/dependencies?toggled=W1swXSxbMCwxMDldLFswLDEwOF1d

[cromefire]

Here's also a similar case of a different plugin that also needed to implement configurable configurations because android (I know that doesn't apply here, but it's a similar case) uses a different configuration: hierynomus/license-gradle-plugin#42

[cromefire]

The runtimeClasspath is hard-coded in 2 places: https://github.com/GoogleContainerTools/jib/search?l=Java&q=runtimeClasspath + RUNTIME_CLASSPATH_CONFIGURATION_NAME also contains "runtimeClasspath"
Some time ago I tried to make that configurable, but I failed to get it working.

[cromefire]

I'd also say it's more a bug than a improvement, because this makes it impossible to use for some projects because there's simply no workaround available that I know or can think of.

[chanseokoh]

Thanks for the info. But it looks like the questions that @loosebazooka asked were rather more around the application plugin, multiple sourceSets, an example using Jar, etc. (See #1778 (comment), #1778 (comment), and #1778 (comment)). However, I don't really follow what @loosebazooka had in his mind, and unfortunately, he will not be available for a few months. If you can explain to me what the thoughts of @loosebazooka could have been, I'd appreciate it. And any design proposal for supporting this?

I'd also say it's more a bug than a improvement, because this makes it impossible to use for some projects because there's simply no workaround available that I know or can think of.

Anyways, reading your old comments and the example you just provided, I think your issue is specifically about Spring Boot and the custom developmentOnly that the plugin defines. At least you can use the Jib Layer-Filter extension (Maven / Gradle) to remove developmentOnly dependencies.

[cromefire]

I think your issue is specifically about Spring Boot and the custom developmentOnly that the plugin defines

Well spring boot is just a user of this and I think this is also where @loosebazooka is coming at you can do stuff like this:

configurations {
    register("docker") {
        extendsFrom(productionRuntimeClasspath.get())
    }
}

dependencies {
    "docker"("org.apache.commons:commons-lang3:3.11")
}

Which allows you to easily customize things.

[cromefire]

As of the sourceSets: I have no idea, how to do that, it's probably useful in some cases but those are probably really rare

[cromefire]

So I created a PR, which basically solves the problem with the configurations (see the sample project)

[loosebazooka]

From your example, I'm not convinced #3034 is the best solution, it's a little too convoluted.

I, however, can see the reason that someone would want this. Perhaps we just create a new jib configuration? So users can do something like

dependencies {
  jib "my.container-only:dependency:1.2.3"
}

and if a user wants to take advantage of the extendsFrom style workflows they can just call it on the jib configuration?

[cromefire]

Well your snippet is completely working with the PR you just have to create the jib configuration and tell jib to use it.

[cromefire]

So basically:

jib {
    configurationName.set("jib")
}

configurations {
    register("jib")
}

(Syntax might differ for the groovy version, I haven't checked that yet but it should work well because of the properties)

[loosebazooka]

Right, just forcing a convention here though.

[cromefire]

The problem with your jib by default proposal is compatibility. For that you'd have to have jib extend from runtimeClasspath by default and I'm not quite sure you can reverse the extending, so it defeats the whole purpose.
In the case how I implemented it, the user has full control and there isn't any "hidden magic" happening.

[cromefire]

Right, just forcing a convention here though.

Well if you want a configuration you need to create a configuration, right? The only difference here is you explicitly opt in as opposed to something that the user might not want is happening in the background.

[cromefire]

It's basically saving 4 lines of code with your proposal while loosing all flexibility.

[loosebazooka]

So the idea wasn't to extend runtimeClasspath explicitly but instead add extra classpath options to the build.

productionRuntimeClasspath is a springboot only concept, and what wasn't clear is that spring-boot is polluting the runtimeClasspath with developmentOnly dependencies and then hacking around that when creating the production runtimeClasspath. This is not ideal and would have preferred their bootRun task to handle the developmentOnly configurations separately. Now it makes sense why you would require configuration selection rather than appending.

I'll take a look at the PR.

[cromefire]

I would imagine spring is not the only one doing this. But I've not looked at a lot of frameworks

[chanseokoh]

@remmeier @dmurat @saturnism @fhoeben @bric3 @atbentley @AndrewBentley6886 @ghilainm @cromefire

We've released Jib plugins 3.0 where you can set a custom Gradle Configuration. Check out the usage.

@cromefire thanks for your awesome contribution! Indeed, it was a highly requested feature.

[chanseokoh]

Fixed by #1776.

[bric3]

@chanseokoh Thanks !
It is useful !