Support distribution of binaries

[AidanDelaney]

We have need for mvn or mvnd to be distributed on a build layer. This makes a maven binary available for subsequent buildpacks to use. In this scenario, the source project does not necessarily contain a pom.xml.

Describe the Enhancement

The enhancement allows the maven buildpack to operate as a distribution buildpack. Subsequent buildpacks can use the maven binary to perform maven operations.

Possible Solution

The maven buildpack could succeed to detect if BP_MAVEN_COMMAND is set to either mvn or mvnd. During the build phase, if BP_MAVEN_COMMAND is set to either mvn or mvnd then the corresponding binary is contributed as a build layer.

A buildpack that requires the maven buildpack might be expected to explicitly set a flag to disable running the maven command. The default operation of the buildpack, in the absence of the flag, would continue to run the build command.

Requires: []libcnb.BuildPlanRequire{
	{
		Name: maven.PlanEntryMaven,
		Metadata: map[string]interface{}{
			"RunBuild": false,
		},
	},
}

The current cache layer on which mvn/mvd are distributed would be made available as a build and cache layer.

Motivation

Our motivation is to support injection of Jars into an application image in a non-JVM source project. For example, python is a common language for data science projects and data science projects in python often use PySpark for distributed computation. Such use-cases commonly need to install a single Jar file, or a small number of Jar files, and make them available in the CLASSPATH. The set of Jars to install depend on the dependencies used in the Python project. Application developers do not want to manage both a python build system and a maven/gradle/sbt build system. Supporting a "distribution only" operation in the maven buildpack allows us to provide a subsequent buildpack to detect a PySpark project and inject appropriate Jar files.

Given this motivation, a data-science oriented build image might set BP_MAVEN_COMMAND=mvn by default. The buildpack that contributes the Jars would depend on the maven buildpack and set RunBuild: false. This moves the problem of managing dependencies in a pom.xml from the data-scientist to the platform operator.

[AidanDelaney]

I have a possible PR that could be used to start a discussion on this. I also have https://github.com/AidanDelaney/maven-dist which completely factors the distribution logic out of the maven buildpack. In my experiments I have a maven meta-buildpack that provides an order group consisting of maven-dist and maven-install.

[dmikusa]

+1 for supporting this use case. I have some thoughts on the implementation.

First, typically an env variable for config is used if that option is meant to be a.) user configurable or b.) if a buildpack needs to communicate to a downstream buildpack (i.e. A -> B -> C where A needs to tell C to do something). I'm not sensing either of those scenarios in this use case, although please correct me if that's wrong.

To me, it sounds like you have a downstream buildpack that wants to trigger a behavior in an upstream buildpack, which is the classic use case for buildplan manipulation. i.e. A -> B -> C where C wants to change A's behavior.

Right now, we have a build plan like this in Maven. Just a single entry, and this is set if a pom.xml file exists.

	return libcnb.DetectResult{
		Pass: true,
		Plans: []libcnb.BuildPlan{
			{
				Provides: []libcnb.BuildPlanProvide{
					{Name: PlanEntryJVMApplicationPackage},
					{Name: PlanEntryMaven},
				},
				Requires: []libcnb.BuildPlanRequire{
					{Name: PlanEntrySyft},
					{Name: PlanEntryJDK},
					{Name: PlanEntryMaven},
				},
			},
		},
	}, nil

If there is no pom.xml, then we set this to just fail which because this is an optional buildpack causes the buildpack to skip.

I haven't tried this, but I'm thinking that if instead of failing, we returned something like this:

	return libcnb.DetectResult{
		Pass: true,
		Plans: []libcnb.BuildPlan{
			{
				Provides: []libcnb.BuildPlanProvide{
					{Name: PlanEntryMaven},
				},
			},
		},
	}, nil

That would mean a downstream buildpack would have to say I require maven or the buildpack would fail. That could be your custom buildpack. The only thing I'm not 100% certain of off the top of my head is if this would break the case where we do not want Maven at all. That would merit some exploration.

We could then modify the build logic in the Maven buildpack to look at the buildplan. If a Maven entry is there, then it'll install Maven. This should be true for the original buildplan entry and also this new plan entry, so the buildpack would install Maven. Then if PlanEntryJVMApplicationPackage is in the buildplan, it would go on to do the actual Maven build. You could omit that in your use case and it would just contribute Maven.

The benefit here is that we a.) don't have to introduce any new env variables or require a user to set any env variables and b.) we don't need to add any logic to Maven CNB about your use case. That can all be encapsulated in your buildpack.

What do you think? Does it sound like this would give you the flexibility you require?

[AidanDelaney]

This sounds like a good approach. It's very similar to an approach were we might provide a maven-dist buildpack on which a maven buildpack depends. The maven buildpack can detect pom.xml and/or mvnw and choose to require maven-dist to install mvn or mvnd if pom.xml is present but mvnw is not. Is it cleaner to factor the distribution logic out of the maven buildpack and provide a maven-dist buildpack?

[dmikusa]

It's very similar to an approach were we might provide a maven-dist buildpack on which a maven buildpack depends. The maven buildpack can detect pom.xml and/or mvnw and choose to require maven-dist to install mvn or mvnd if pom.xml is present but mvnw is not. Is it cleaner to factor the distribution logic out of the maven buildpack and provide a maven-dist buildpack?

There isn't a right or wrong answer here. Decomposing to the smallest units possible does make smaller buildpacks, which are individually easier to understand. It can also make them easier to reuse. On the flip side, the composition of different buildpacks can be tricky to understand, you have more complicated order groups and you have to keep how they all work in your head to understand the flow. You also have to maintain a lot more individual buildpack projects (update, release, etc...). Decomposing more also increases the number of layers you'll end up with in your builder images, which is presently a problem for the Paketo builders (each buildpack is a layer).

With the Java buildpacks, we've generally taken a moderate, function-based approach to decomposing buildpacks. We're not trying to split them up into the smallest possible units, but rather group them by reasonably reusable functional units. So we have Maven, Gradle, Bellsoft (or another JVM vendor), Tomcat, Executable JAR, etc... It's easy to know what these do just based on the name & the order group is one flat order group regardless of your application type.

At any rate, I'd like to stick with that pattern at least for now.

[AidanDelaney]

Thanks, I now understand the context.