How is this plugin supposed to be used together with sbtOsgi

[lefou]

I have a project that produces OSGi bundles with the sbtOsgi plugin. After adding the reproducibleBuildPlugin as described in the README, the produced packages still looks the same and e.g. contain non-reproducible manifest entries.

[raboof]

Can you share the project you're running this on?

Are you using osgiSettings from https://github.com/sbt/sbt-osgi/blob/9f7bef4019ecfcb6fab99a4fad13d6da3dc72d85/src/main/scala/com/typesafe/sbt/osgi/SbtOsgi.scala#L37 ?

I haven't looked closely yet, but indeed perhaps the 2 plugins now side-step each other.

[lefou]

We do something very similar in our build setup (https://github.com/woq-blended/blended/blob/1bb60a4c18c49e11bb61b9fc5f8f1bfcde43711c/project/BlendedBundle.scala#L70):

    Compile / packageBin := {
      packageBin.in(Compile).value
      OsgiKeys.bundle.value
    }

Maybe you can give an idea, how we should incorporate the reproducibleBuild step?

The project is located at https://github.com/woq-blended/blended, in branch sbt-reproducible (as we are currently migrating from Maven)

[raboof]

It looks like OsgiKeys.bundle creates a completely new artifact, ignoring the 'normal' packageBin.

Your replacement packageBin does add a dependency to the 'normal' packageBin, but then ignores it and returns OsgiKeys.bundle.value instead.

So sbt-reproducible-builds post-processes the 'normal' packageBin, but this is ignored and replaced by OsgiKeys.bundle.value. I guess what we'd want would be for sbt-reproducible-builds to post-process the 'new' packageBin. One way to force that would be to re-evaluate the ReproducibleBuildsPlugin.projectSettings after replacing packageBin. I tried that at https://github.com/raboof/sbt-reproducible-builds/pull/38/files#diff-dccf14823428de74bf9042c0cf115c6cR13 and that appears to work. Could you give that a try?

[lefou]

Thanks for your answer. Unfortunately, I can't get the desired behaviour after following your advice to reload rb project settings after osgiSettings got defined (https://github.com/woq-blended/blended/blob/sbt-reproducible/project/ProjectSettings.scala#L71). In the osgiSettingsI redefine packageBin to osgiBundle (https://github.com/woq-blended/blended/blob/sbt-reproducible/project/BlendedBundle.scala#L70).

As the rb plugin has no explicit task key with which I can trigger the processing/creating of the stripped package, I find it very hard to experiment further. I'm currently looking for the presence of the Bnd-LastModifiedheader, which gets properly removed by the maven rb plugin.

[raboof]

Is this issue still current? How are you testing this? I just tried this branch and at first glance it seems blended-launcher_2.12.jar is correctly stripped (no Bnd-LastModified, Jan 1 2000 timestamps)

[lefou]

Thanks for checking! Which branch/commit did you check (master or sbt-reproducible)? For me it is not working in neither branch. I'm using sbt> package.

As we've finished our migration to sbt, master is now the reference again. I'll create a new branch sbt-reproducible-2 to update the test-setup...

[raboof]

I tried sbt-reproducible, by running publishLocal and checking the file that was placed in my ~/.ivy2.

How did you check for reproducibility after running package?

But after running clean and then show package from a fresh sbt console, I'm getting:

[info] blendedAkka / Compile / package
[info]  /home/aengelen/dev/blended/blended.akka/target/scala-2.12/stripped/blended-akka_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedCamelUtils / Compile / package
[info]  /home/aengelen/dev/blended/blended.camel.utils/target/scala-2.12/stripped/blended-camel-utils_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedTestsupport / Compile / package
[info]  /home/aengelen/dev/blended/blended.testsupport/target/scala-2.12/stripped/stripped/blended-testsupport_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedSecurityJS / Compile / package
[info]  /home/aengelen/dev/blended/blended.security/js/target/scala-2.12/blendedsecurity_sjs0.6_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedJmsUtils / Compile / package
[info]  /home/aengelen/dev/blended/blended.jms.utils/target/scala-2.12/stripped/blended-jms-utils_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedAkkaHttpJmsqueue / Compile / package
[info]  /home/aengelen/dev/blended/blended.akka.http.jmsqueue/target/scala-2.12/stripped/blended-akka-http-jmsqueue_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedSecurityJVM / Compile / package
[info]  /home/aengelen/dev/blended/blended.security/jvm/target/scala-2.12/stripped/blended-security_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedActivemqBrokerstarter / Compile / package
[info]  /home/aengelen/dev/blended/blended.activemq.brokerstarter/target/scala-2.12/stripped/blended-activemq-brokerstarter_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedPersistenceH2 / Compile / package
[info]  /home/aengelen/dev/blended/blended.persistence.h2/target/scala-2.12/stripped/blended-persistence-h2_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedSecurityAkkaHttp / Compile / package
[info]  /home/aengelen/dev/blended/blended.security.akka.http/target/scala-2.12/stripped/blended-security-akka-http_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedAkkaHttpApi / Compile / package
[info]  /home/aengelen/dev/blended/blended.akka.http.api/target/scala-2.12/stripped/blended-akka-http-api_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedDomino / Compile / package
[info]  /home/aengelen/dev/blended/blended.domino/target/scala-2.12/stripped/blended-domino_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedAkkaHttpRestjms / Compile / package
[info]  /home/aengelen/dev/blended/blended.akka.http.restjms/target/scala-2.12/stripped/blended-akka-http-restjms_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedMgmtBase / Compile / package
[info]  /home/aengelen/dev/blended/blended.mgmt.base/target/scala-2.12/stripped/blended-mgmt-base_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedUpdaterTools / Compile / package
[info]  /home/aengelen/dev/blended/blended.updater.tools/target/scala-2.12/stripped/blended-updater-tools_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedMgmtRepo / Compile / package
[info]  /home/aengelen/dev/blended/blended.mgmt.repo/target/scala-2.12/stripped/blended-mgmt-repo_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedPersistence / Compile / package
[info]  /home/aengelen/dev/blended/blended.persistence/target/scala-2.12/stripped/blended-persistence_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedFile / Compile / package
[info]  /home/aengelen/dev/blended/blended.file/target/scala-2.12/stripped/blended-file_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedSecurityBoot / Compile / package
[info]  /home/aengelen/dev/blended/blended.security.boot/target/scala-2.12/stripped/blended-security-boot_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedAkkaHttp / Compile / package
[info]  /home/aengelen/dev/blended/blended.akka.http/target/scala-2.12/stripped/blended-akka-http_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedUtilLogging / Compile / package
[info]  /home/aengelen/dev/blended/blended.util.logging/target/scala-2.12/stripped/blended-util-logging_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedContainerContextApi / Compile / package
[info]  /home/aengelen/dev/blended/blended.container.context.api/target/scala-2.12/stripped/blended-container-context-api_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedUtil / Compile / package
[info]  /home/aengelen/dev/blended/blended.util/target/scala-2.12/stripped/blended-util_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedJettyBoot / Compile / package
[info]  /home/aengelen/dev/blended/blended.jetty.boot/target/scala-2.12/stripped/blended-jetty-boot_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedLauncher / Compile / package
[info]  /home/aengelen/dev/blended/blended.launcher/target/scala-2.12/stripped/blended-launcher_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedMgmtRepoRest / Compile / package
[info]  /home/aengelen/dev/blended/blended.mgmt.repo.rest/target/scala-2.12/stripped/blended-mgmt-repo-rest_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedMgmtAgent / Compile / package
[info]  /home/aengelen/dev/blended/blended.mgmt.agent/target/scala-2.12/stripped/blended-mgmt-agent_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedAkkaHttpProxy / Compile / package
[info]  /home/aengelen/dev/blended/blended.akka.http.proxy/target/scala-2.12/stripped/blended-akka-http-proxy_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedJmx / Compile / package
[info]  /home/aengelen/dev/blended/blended.jmx/target/scala-2.12/stripped/blended-jmx_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedJmsSampler / Compile / package
[info]  /home/aengelen/dev/blended/blended.jms.sampler/target/scala-2.12/stripped/blended-jms-sampler_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedTestsupportPojosr / Compile / package
[info]  /home/aengelen/dev/blended/blended.testsupport.pojosr/target/scala-2.12/stripped/stripped/blended-testsupport-pojosr_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedPrickleAkkaHttp / Compile / package
[info]  /home/aengelen/dev/blended/blended.prickle.akka.http/target/scala-2.12/stripped/blended-prickle-akka-http_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedUpdaterConfigJS / Compile / package
[info]  /home/aengelen/dev/blended/blended.updater.config/js/target/scala-2.12/blendedupdaterconfig_sjs0.6_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedContainerContextImpl / Compile / package
[info]  /home/aengelen/dev/blended/blended.container.context.impl/target/scala-2.12/stripped/blended-container-context-impl_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedUpdater / Compile / package
[info]  /home/aengelen/dev/blended/blended.updater/target/scala-2.12/stripped/blended-updater_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedUpdaterConfigJVM / Compile / package
[info]  /home/aengelen/dev/blended/blended.updater.config/jvm/target/scala-2.12/stripped/blended-updater-config_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] blendedUpdaterRemote / Compile / package
[info]  /home/aengelen/dev/blended/blended.updater.remote/target/scala-2.12/stripped/blended-updater-remote_2.12-2.5.0-SBT-SNAPSHOT.jar
[info] Compile / package
[info]  /home/aengelen/dev/blended/target/scala-2.12/blended_2.12-2.5.0-SBT-SNAPSHOT.jar

From the filenames, it looks like those should all be stripped, except for blended_2.12-2.5.0-SBT-SNAPSHOT.jar (which contains only a manifest), blendedsecurity_sjs0.6_2.12-2.5.0-SBT-SNAPSHOT.jar and blendedupdaterconfig_sjs0.6_2.12-2.5.0-SBT-SNAPSHOT.jar.

Does that give you something to go on?

[lefou]

Wow, I somehow missed that stripped directory. Now I see. Thank you very much! And sorry for my blindness. ;-)

In summary, to integrate with sbtOsgi, one must re-apply the reproducible build settings after osgi settings.

---

Some more thoughts:

It would be great to have somewhat more info in the README about this, though. There was no information where the built artifact will be located, and also that the installed artifact in not the same as the one under target/scala-2.12/. Also, as there is no dedicated task to trigger the strip explicitly, it is neither obvious how to strip nor how to use inspect or show to self-gather more details about the build.

So I suggest, to do at least one of the following

• Add some info to README, where the stripped artifact will be found and in what way it will be used by other tasks/plugins.
• Add a didicated reproducibleBuildStrip task, to explicitly trigger the build of the stripped artifact.

With the latter, everyone can easily depend on the stripped JAR, without any knowledge about your plugin. E.g. without it, I'n not exactly sure how to depend on the stripped JAR instead of the default JAR in the universal:packageBin task in same sub project blended.launcher. That task builds a ZIP file containing the ready to use launcher application. An in that ZIP in the lib directory, the unstripped blended.launcher.jar is used. What I really want is to always use the stripped JAR.

[raboof]

Added #40 for the explicit intermediary task for producing the stripped packageBin

[raboof]

Thanks for validating the solution appears to work for you, and the additional suggestions!

I agree it would be good if users could use this plugin without having to learn about how it works internally too much. Actually overriding packageBin was intended to achieve that: most other plugins and parts of SBT would use the packageBin and automatically use the stripped package.

Because of this approach, AFAICS universal:packageBin already contains stripped packages. However, it does not strip the dependencies, nor does it strip the resulting .zip itself. Might be nice to consider that, added #41.

Of course this abstraction breaks down a bit when other plugins override packageBin and ignore its 'previous' value, like sbt-osgi, so it's good to clarify/document this for users in that situation.