Meta buildpack and detect phases

[till]

Hi,

I've googled this for a fair amount of time and read lots of Golang, so I figured I come and ask here if I am on the right track.

The gist is: I am trying to build a meta buildpack with a custom detect which lets me set some variables and what not so users don't have to. Kind of like best practices that shouldn't need configuring by the users.

In order to get there, I started exploring pack, but too much of what I need seems to be in internal to make use of it and detect and build are heavily coupled. Then I started exploring packit for a bit and managed to build a custom detect phase for applications (kudos for that library).

What I struggle with now is how I would use the result of my own detect phase to assemble already available buildpacks and provide configuration to them. It seems the return of detect is a plan, and that's (I think) one step ahead of what I want to do.

I know that I would need a list of buildpacks I want to use, that's sort of similar to what the builder does - not a problem. But the general use of detect seems to be that my buildpack is providing everything needed to build the application, and I would like to avoid that. Instead I would like to for example say:

• this seems to be a nodejs app
• 👉🏼 let's use the node buildpack
• 👉🏼 let's inject the following config values for it

Originally, I wanted a pack build --dry-run to run a detect phase and then configure pack with --buildpack switches and settings. But as I said, I am not getting around in the code base. So I figured, maybe it's easier to use packit?

It could be that I am doing this (completely) wrong, so I figured I'd ask.

[fg-j]

Hi there! I'll try to shed some light on the situation.

1. It's totally possible (and encouraged!) to combine your buildpack with existing Nodejs buildpacks.

2. Buildpacks collaborate with each other by contributing to the Build Plan. Each buildpack contributes to the Build Plan in its detect phase, as you rightly pointed out. As an output to detect, a buildpack will say that it requires and/or provides certain things. For example. the node-engine buildpack provides the node runtime. Another buildpack can collaborate with node-engine by requiring the node runtime. The npm-install buildpack is an example, since you need node (and npm) to run npm install.

3. Once all buildpacks' detection phases have run, the lifecycle resolves the Build Plan into a Buildpack Plan (different, but similarly named!). The Build Plan is a list of requirements and provisions from all of the buildpacks whose detect phases ran. The Buildpack Plan is a deduplicated list of what things are required and which buildpack will be responsible for providing each one. During the build phase, buildpacks look at the Buildpack Plan to see what dependencies they should install, or what commands they should run.

4. So, your custom buildpack might need to use the Build Plan.
   a. If your custom buildpack relies on other tools to run successfully, it should require those.
   b. If the buildpack doesn't need any dependencies to function, it can simply pass detection when the conditions are right, like the git buildpack. The buildpack passes detection when a certain service binding is present, and neither provides nor requires anything.

5. To run a build that combines your custom buildpack with other nodejs language family ones, you can either create your own metabuildpack, or add your buildpack alongside the nodejs one at build time.
   a. The Go buildpack is a good example of what it looks like to add a buildpack that sets up config to a composite buildpack. Notice that the git buildpack is an optional member of each order group. You could fork the Nodejs buildpack and slot your custom buildpack into its existing order groups in a similar way.
   b. You can also add your custom buildpack to an individual build using the --buildpack flag. You'll probably want to include it before the Nodejs buildpack, since it sounds like your buildpack needs to configure things before the other Nodejs steps run. Something like:

   pack build --buildpack my-custom-buildpack --buildpack paketobuildpacks/nodejs:latest
   

All in all, I think the git buildpack is probably a strong example for you to follow, since it detects on the presence of a certain service binding, provides/requires nothing, and then configures the build environment during its build phase.

[sophiewigmore]

What I struggle with now is how I would use the result of my own detect phase to assemble already available buildpacks and provide configuration to them. It seems the return of detect is a plan, and that's (I think) one step ahead of what I want to do.

For clarity, I'd add on that it sounds like you want an implementation buildpack for the custom detection/configuration settings. The detect phase of your custom implementation buildpack would be responsible for figuring out if you have the right type of app for your use case, requiring anything needed from other buildpacks like @fg-j mentioned, and then in the build phase, set whatever configuration is needed. This type of behaviour would be for normal for an implementation buildpack.

Then, the meta-buildpack comes into play when you want to combine your custom buildpack with the pre-existent buildpacks. No custom detection logic would live here, the detection groups are determined by the groups set up in the buildpack.toml.

[till]

@sophiewigmore I (re-)read your blog post on the weekend. I read your blog post a couple weeks (or months) ago before coming on here and after our discussions, this makes a lot more sense! :) Much appreciated.

[till]

Re-reading this, I think we misunderstood each other. I know @fg-j mentioned build phase a few times and for lack of understanding, I also overread that or interpreted it wrong.

So what I wanted is, that my "buildpack" sets BP_-variables that enable a successful detect of another buildpack. And that is currently not possible.

To give a very concrete example: I can not create a buildpack which parses my nodejs app code and sets BP_NODE_RUN_SCRIPTS, so the node-run-script is later detected and used.

Instead I created a wrapper around pack to inject BP_NODE_RUN_SCRIPTS on that level (before any detect runs).

[fg-j]

This use-case makes me wonder if the node-run-script buildpack should always pass detection and simply no-op during the build phase if the BP_NODE_RUN_SCRIPTS env var is still not set when its build phase runs. That would allow an earlier buildpack to determine which scripts should run.

[till]

Just reading that now. It seems that was my misunderstanding. An earlier buildpack cannot set variables for others. Or would it work if it already passed?

[fg-j]

During the detect phase, all buildpacks' detect binaries run in parallel with the same inputs. They all have read-only access to the filesystem and the execution evironment. During the build phase, buildpacks' build binaries run sequentially. During the build phase, one buildpack can set environment variables for another buildpack's build phase. In its build, some Buildpack X could look for the presence of some file or attribute of the build and determine that JavaScript assets need to be built. It can set BP_NODE_RUN_SCRIPTS. If the the Node Run Scripts buildpack build runs afterwards, it will know which scripts to run.

[robdimsdale]

@till That makes sense - sorry it took a couple of attempts to get on the same page.

It sounds like you have a potential future path to solve your specific issue, but I additionally wanted to share that I recently encountered a similar situation as you which wouldn't be solved by supporting node-run scripts.

I was working on a buildpack that would take a function (e.g. a python function) as source code and wrap it inside an application framework (e.g. flask) to create an executable. In principle, this buildpack could be as simple as:

1. wrap the function in a framework
2. write out a package management file (e.g. requirements.txt)
3. write a Procfile with the specified command

And then I could rely on the existing Python buildpack to take this application and process it like any other application. I would add my buildpack to the beginning of the build order, in front of the Python buildpack.

Unfortunately, this does not work as the various Python buildpacks fail detection due to the fact that the source code isn't a valid app (e.g. it is missing a requirements.txt) because the valid application is only present during the Build phase, not during Detect. As you know, the Detect phase runs on the same source code for all buildpacks, and one buildpack cannot influence another buildpack's Detect phase.

I just wanted to share this to validate that while pre-processing / influencing downstream buildpack's detect phase is unusual - and currently unsupported - there are other use-cases for this beyond what you're looking at.

[till]

Thanks!

Your use-case sounds interesting as well. E.g. for a knative/faas thing where users provide only a function. It would be great not to have them go learn about bp right away and enable them.

TL;DR - I would really like to have this at some point.

---

The builder RFC mentioned yesterday on the call is going in another interesting direction.

The difference being, if I know what my users are running beforehand, then that makes it easier to configure the builder. But when I don't, it's more guessing.

[robdimsdale]

Yup - that's exactly the use case: https://github.com/vmware-tanzu/function-buildpacks-for-knative

[robdimsdale]

Also, although pre-processing is somewhat feasible when you're scripting pack (or kpack), it isn't feasible in the general case when you have a hosted platform like commercial offerings.

[till]

So my end goal is arbitrary apps that people wanna run on our platform.