Add 2018 Roadmap #94

imjasonh · 2018-03-22T17:46:28Z

I'm sending this PR to get feedback from all the contributors.

If you see something you think is missing, or if you have questions, please leave a comment so we can discuss.

jchesterpivotal · 2018-03-22T19:53:34Z

Part of my totally unsecret personal agenda is to surface the things that Concourse is really good at in this particular problem domain, with an eye to adopting it or stealing from it. In particular, the Workflow, Triggering and Integrations headers play well to current strengths for Concourse or to concepts we've discussed internally.

I'm not sure how best to explore that possibility without totally derailing everyone and everything, but I'd like the opportunity to work through it openly and collaboratively with the wider Elafros community to see if it makes sense.

(Actual review of the actual document to follow)

jchesterpivotal · 2018-03-22T19:54:44Z

roadmap-2018.md

+    understandable way
+  * **Improve** the featureset of the Build CRD by standardizing support for
+    things like build caching using on-cluster resources
+  * **Integrate** Build CRD into more services, and make it more powerful by


This sentence seems redundant. Perhaps:

Integrate Build CRD into more services to make it more powerful.

@jchesterpivotal Would definitely be interested to collaborate on workflow definition (I'm PM on the Google Container Builder team). We've been thinking about workflow a lot as well and agree with the goal to develop in an open way and incorporate lessons learned, a.k.a. battle scars.

How should we compare notes? Any interest in whiteboarding or pairing to start forming a common understanding?

Oh and tagging @jama-pivotal (Concourse PM), when he's added to the org.

jchesterpivotal · 2018-03-22T19:55:14Z

roadmap-2018.md

+
+## Detection
+
+In many cases, users shouldn't have to know explicitly which version of a build


I'd drop "In many cases".

jchesterpivotal · 2018-03-22T19:57:33Z

roadmap-2018.md

+How this works in real life is a hard problem, that a lot of smart people have
+spent a lot of their smarts working on it. How that fits into the Build
+resource, or whether it's a higher-level concept "above" Builds remains to be
+seen.


My gut feeling is that it's not that hard. Which might mean this refers to experiences that I haven't had.

Or maybe I am thinking of a different meaning of "detection", given that my thinking is strongly shaped by how buildpacks approaches things.

I think there are some subtleties on attempting to do a good, language-specific detection given that some communities don't have clear, standardized semver compatibility, for example. (hi Ruby!)

This helps me understand what we're talking about in detection -- "is this Ruby?" vs "is this Ruby 2.5.1?", for example.

I was thinking of the former, which is where buildpacks start from in bin/detect. Questions about which Ruby interpreter to install are handled by reading Gemfiles, for NodeJS by reading project.json etc during the bin/supply step.

jchesterpivotal · 2018-03-22T19:59:57Z

roadmap-2018.md

+
+In either case, the heuristics used to detect what build should be run should
+be open-sourced, and whatever decision the detection logic arrives at should be
+visible to end users, and should be understandable and repeatable.


Might reword to:

In any case, these are our criteria for detection:

Detection heuristics must be open sourced

Users must have visibility into what detection decision was made and why

Detection should an understandable, repeatable process

jchesterpivotal · 2018-03-22T20:02:00Z

roadmap-2018.md

+## Community
+
+BuildTemplates are designed to be shared. We should cultivate a repo of common
+templates, and the builder images they rely on, both as working examples of how


I'd break the sentence here:

We should cultivate a repo of common templates and the build images they rely on. These can serve as working examples of how to build good templates, which will encourage users to contribute their own.

jchesterpivotal · 2018-03-22T20:03:30Z

roadmap-2018.md

+advances we've made with FTL-like layer caching techniques, [remote image
+rebasing](https://github.com/google/image-rebase), source detection, and in
+general be best-in-class in terms of fast incremental builds.
+


Perhaps add a subheader for "Work So Far"

jchesterpivotal · 2018-03-22T20:04:26Z

roadmap-2018.md

+
+## Workflow
+
+Today, a Build express a list of step containers to run in order, to


"express a list" should be "expresses a list"; or possibly reword to "a sequence of step containers" to emphasise the linearity.

jchesterpivotal · 2018-03-22T20:06:41Z

roadmap-2018.md

+workflows:
+
+  * Parallel execution (fan-out/fan-in)
+  * Conditional step execution ("only execute step X if Git branch ==


We'll need to take care to not create Turing-complete YAML. Per the Structured Programming Theorem there are three things you need to stumble on it:

A way to order instructions (this is the current state of the world for Build CRD)

A way to choose alternatives (proposed here)

A way to repeatedly execute instructions (loops, not proposed yet).

jchesterpivotal · 2018-03-22T20:07:26Z

roadmap-2018.md

+  * Runtime-conditional step execution ("only execute step Y if step X failed";
+    e.g., recover/cleanup a broken rollout)
+
+This may expand to cover generating a build from some other format, which


Do we have some example formats we'd want to convert from?

jchesterpivotal · 2018-03-22T20:08:39Z

roadmap-2018.md

+
+The Build CRD is powerful, but far from perfect.
+
+In particular, settling on a standard for expressing a secure and reliable


I think this will be the trickiest part to pull off safely.

evankanderson

I'm going to let someone from outside Google do the approve bits here, but this seems like a reasonable roadmap & charter.

evankanderson · 2018-03-22T21:01:13Z

roadmap-2018.md

+## Detection
+
+In many cases, users shouldn't have to know explicitly which version of a build
+template they're using; they have source, and they want to build and a


"they want to build a container image" (no "and")?

evankanderson · 2018-03-22T21:03:13Z

roadmap-2018.md

+How this works in real life is a hard problem, that a lot of smart people have
+spent a lot of their smarts working on it. How that fits into the Build
+resource, or whether it's a higher-level concept "above" Builds remains to be
+seen.


I think there are some subtleties on attempting to do a good, language-specific detection given that some communities don't have clear, standardized semver compatibility, for example. (hi Ruby!)

imjasonh · 2018-03-23T17:18:09Z

Thanks for the comments, I've updated some wording as suggested.

I'd like to avoid getting too deep into how we'll accomplish each thing, or what the specific challenges are. There will be plenty of time for that later. :)

smarterclayton · 2018-03-28T20:30:13Z

LGTM. One thing I might emphasize would be to tie the roadmap together with a really high mission / philosophy statement.

Make it easy to rapidly iterate on source code for use with Elafros
Make builds of images efficient and secure

Or something along those lines to focus the roadmap.

mattmoor · 2018-03-29T03:33:50Z

@smarterclayton This might make the most sense in the opening of the README.md for the repo, WDYT?

smarterclayton · 2018-03-29T03:35:25Z

That works as well

mattmoor

Thanks for putting this together, Jason.

christophersanson · 2018-03-29T16:09:00Z

Should we add a roadmap item around results handling? Using the build crd and other components as part of CICD workflows will require an interface for viewing results.

imjasonh · 2018-03-29T16:46:53Z

@christophersanson In the name of limiting the scope of the roadmap, I think I'd prefer to work on that in an issue, instead of making it a tentpole effort for 2018. It might grow into enough work that it becomes a tentpole, but for now I think it's mostly a matter of "publish more events" (#95) and "better logging story" (#9), with documented/supported examples for exporting both of those...somewhere.

Does that make sense?

christophersanson · 2018-03-30T16:46:28Z

sgtm

mattmoor · 2018-03-30T16:57:15Z

/lgtm

google-prow-robot · 2018-03-30T16:57:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ImJasonH, mattmoor

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ImJasonH,mattmoor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* Create roadmap.md * Rename to roadmap-2018.md * Incorporate some review feedback

Create roadmap.md

1b59424

google-prow-robot added the size/L label Mar 22, 2018

Rename to roadmap-2018.md

b199d8d

imjasonh requested review from jchesterpivotal and mattmoor March 22, 2018 19:11

jchesterpivotal reviewed Mar 22, 2018

View reviewed changes

evankanderson reviewed Mar 22, 2018

View reviewed changes

Incorporate some review feedback

df0290f

mattmoor approved these changes Mar 29, 2018

View reviewed changes

google-prow-robot assigned mattmoor Mar 30, 2018

google-prow-robot added the lgtm label Mar 30, 2018

google-prow-robot added the approved label Mar 30, 2018

google-prow-robot merged commit 7334d6e into master Mar 30, 2018

imjasonh deleted the roadmap branch March 30, 2018 17:34

jchesterpivotal mentioned this pull request Apr 9, 2018

Create a roadmap for 2018 #88

Closed

vdemeester pushed a commit to vdemeester/knative-build that referenced this pull request Apr 3, 2019

Add 2018 Roadmap (knative#94)

9fc6986

* Create roadmap.md * Rename to roadmap-2018.md * Incorporate some review feedback

vdemeester pushed a commit to vdemeester/knative-build that referenced this pull request Apr 3, 2019

Add 2018 Roadmap (knative#94)

261b4fd

* Create roadmap.md * Rename to roadmap-2018.md * Incorporate some review feedback

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add 2018 Roadmap #94

Add 2018 Roadmap #94

imjasonh commented Mar 22, 2018

jchesterpivotal commented Mar 22, 2018

jchesterpivotal Mar 22, 2018

christophersanson Mar 22, 2018

jchesterpivotal Mar 23, 2018

jchesterpivotal Mar 23, 2018

jchesterpivotal Mar 22, 2018

jchesterpivotal Mar 22, 2018

evankanderson Mar 22, 2018

jchesterpivotal Mar 22, 2018

jchesterpivotal Mar 22, 2018

jchesterpivotal Mar 22, 2018

jchesterpivotal Mar 22, 2018

jchesterpivotal Mar 22, 2018

jchesterpivotal Mar 22, 2018

jchesterpivotal Mar 22, 2018

jchesterpivotal Mar 22, 2018

evankanderson left a comment

evankanderson Mar 22, 2018

evankanderson Mar 22, 2018

imjasonh commented Mar 23, 2018

smarterclayton commented Mar 28, 2018

mattmoor commented Mar 29, 2018

smarterclayton commented Mar 29, 2018

mattmoor left a comment

christophersanson commented Mar 29, 2018

imjasonh commented Mar 29, 2018

christophersanson commented Mar 30, 2018

mattmoor commented Mar 30, 2018

google-prow-robot commented Mar 30, 2018


		## Detection

		In many cases, users shouldn't have to know explicitly which version of a build


		## Workflow

		Today, a Build express a list of step containers to run in order, to


		The Build CRD is powerful, but far from perfect.

		In particular, settling on a standard for expressing a secure and reliable

Add 2018 Roadmap #94

Add 2018 Roadmap #94

Conversation

imjasonh commented Mar 22, 2018

jchesterpivotal commented Mar 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evankanderson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imjasonh commented Mar 23, 2018

smarterclayton commented Mar 28, 2018

mattmoor commented Mar 29, 2018

smarterclayton commented Mar 29, 2018

mattmoor left a comment

Choose a reason for hiding this comment

christophersanson commented Mar 29, 2018

imjasonh commented Mar 29, 2018

christophersanson commented Mar 30, 2018

mattmoor commented Mar 30, 2018

google-prow-robot commented Mar 30, 2018