Need a way to shorten cycle times when developing a buildpack

[dsyer]

To shorten dev cycles when developing a buildpack I have found it useful (essential) to be able to run the detect and build locally. In doing that I found that many existing buildpacks (ones built with libpack probably) make assumptions about env vars, so they barf, for example, if PACK_STACK_ID is not set. That's useful to know (and not obvious from any documentation I managed to grok), but it means I can't use PACK_STACK_ID as a signal that I'm running in pack (or some kind of platform). I have adopted PACK_PLAN_PATH as such a signal (it's not mandatory for any of the packs that I have tried, and it is always there when running in pack), but it might not be stable. So what I need, I think, is something that is "just there", and doesn't mean anything other than that we are running in pack build.

[nebhale]

One problem with running detect and build locally, without a full environment around them, is that then they're running in a non-spec complaint situation. I'm not sure that enabling those binaries to be run in a way that isn't spec-complaint should be a goal.

[dsyer]

I just want to be able to see it run quickly. The experience of waiting for pack create builder and pack build with multiple buildpacks is not efficient. The dev cycle is too long, and feedback is really painful. If you have a better "compliant" answer to that then show me.

[nebhale]

I think that you need to create your environment in a complaint way. That means setting the environment as-if you were inside the build image itself, but without being within the build image. The spec defines this entire environment, including paths and environment variables (PACK_STACK_ID for example).

[dsyer]

This is what I have been doing:

$ echo | PACK_ACTIVE=false PACK_STACK_ID=io.buildpacks.stacks.bionic \
/buildpacks/io.projectriff.native/1.0.0-BUILD-SNAPSHOT/bin/detect | \
/buildpacks/io.projectriff.native/1.0.0-BUILD-SNAPSHOT/bin/build

It works. I guess all I'm looking for here is some authoritative way of doing the env and stdin set up, e.g. a new command in pack or something?

[nebhale]

That's an interesting idea. There's an existing pattern where commands do things like direnv hook bash or rbenv init - to setup an environment. That's seems like a good requirement and pattern.

[sclevine]

The --buildpack flag that was just added to pack build should make this experience more reasonable: #25

[ekcasey]

Right now the --buildpack flag can only select buildpacks that are already packaged within the builder image. It sounds like we should prioritize a story allowing --buildpack=<dir> to point to a buildpack source dir. If you don't need to wait for pack create-builder, that should significantly shorten the feedback cycles.

[ekcasey]

We could also do some optimizations to make pack create-builder faster against the docker daemon (the same sorts of optimization we did to speed up pack build for local images).

[dsyer]

#25 says

ID of an existing buildpack supported in default builder - ie. io.buildpacks.java, io.buildpacks.ruby, io.buildpacks.go, etc

So that sounds like it's not yet what I need? I want to run my buildpack that I am trying (against all reason) to debug. Ideally I'd like to not run it in a container. But I suppose I don't mind, as long as I can control the container it runs in.

If it is running in a container, I probably want to be able to use other buildpacks - e.g. to install openjdk using one of Ben's things, and use that in my build - so I need to be able to run more than one, or simulate the result of running an earlier one, changing the buildplan or something.

UPDATE: realized that's probably what @ekcasey was saying.

[dsyer]

Re-reading #25 it looks like I can supply multiple buildpacks. It's not clear to me yet what that means. How does a buildpack ID correspond to a) my local buildpack, b) someone else's? When I have a builder.toml I guess I can resolve those things to uris or local files. If I'm building an app, where does that come from?

[ssisil]

@dsyer - you should be able to just pass in multiple --buildpack flags. Right now you map those buildpacks IDs through the builder.toml like you mention in your comment. If you want to test buildpacks without a builder.toml - you will be able to provide it in a path to your directory via the --buildpack flag - like @ekcasey mentioned above. Going to add a story for this to the backlog - will link it back to this comment

[dsyer]

Right now you map those buildpacks IDs through the builder.toml

That's the bit I didn't understand yet. How does pack know where to find a builder.toml? Unless it is running the fully-formed builder (which is what I want to avoid having to build).

[ssisil]

@dsyer - Pack points to a builder image - it uses packs/sample by default. The buildpacks in the buildpacks/sample repo on github are supported.

Until we implement #11 - using a custom builder is the only way to pass in a custom buildpack. Is there a specific reason you are looking to avoid creating your own builder - just curious?

its pretty straightforward (but undocumented :/) if you follow these commands -

pack create-builder <builder name> --builder-config <local path to builder.toml>  -s <stack ID - if not provided it uses default stack>

Sample builder.toml

[[buildpacks]]
id = "org.cloudfoundry.buildpacks.nodejs"
uri = "file:///tmp/nodejs-cnb_08c783cce0a177ab5a5c2651"
version ="0.0.1"
latest=true

[[buildpacks]]
id ="org.cloudfoundry.buildpacks.npm"
uri = "file:///tmp/npm-cnb_174f9e4604bc9aab9d49e405"
latest=true
version="0.0.1"

[[groups]]
buildpacks = [
  {id = "org.cloudfoundry.buildpacks.npm" },
  { id = "org.cloudfoundry.buildpacks.nodejs"}
]

[dsyer]

That’s what I have been doing. It’s straightforward but indirect and very slow. Short cycle times is what this is about remember.

[sclevine]

Re-reading #25 it looks like I can supply multiple buildpacks. It's not clear to me yet what that means.

Multiple buildpacks can cooperate in both the detection and build phases to support multi-stage builds or build extensions. Happy to go into more detail about this if you'd like.

That’s what I have been doing. It’s straightforward but indirect and very slow. Short cycle times is what this is about remember.

We plan to address this with #11 and additionally a way to keep the container around after a build for debugging purposes.

How does pack know where to find a builder.toml? Unless it is running the fully-formed builder (which is what I want to avoid having to build).

After #11 is implemented, passing a number of local buildpack directories in via the --buildpack flag will create a buildpack group that overrides the values from builder.toml. If you don't specify a builder, it will use pack's default builder (ubuntu:bionic with some sample buildpacks).