Update to allow dynamic branch choices during 'out'

[nat-henderson]

My use case is a code generator which needs to push to a different branch every time it runs.

Changes to check and in make them work with the branch-agnostic out.

Tests pass, although I didn't run the integration tests.

[pivotal-issuemaster]

@ndmckinley Please sign the Contributor License Agreement!

Click here to manually synchronize the status of this Pull Request.

See the FAQ for frequently asked questions.

[pivotal-issuemaster]

@ndmckinley Thank you for signing the Contributor License Agreement!

[vito]

Thanks for submitting this!

I'm not really comfortable with the changes to /in and /check necessary to support this, however. I think this actually kind of ties together a few thematically large issues, some we're still stewing and some we have in progress:

• With Dynamic build plan generation concourse#684 you wouldn't need a new param, since you could just dynamically populate it.
• With New resource interface (+ add versioning) concourse#534 you would be able to specify the param in the put, and not require it in the source, since resources would then just receive one big config object.
• With Epic: spatial resource flows concourse#1707 this might actually need to run against a different space, since that's how the git resource would represent branches. In fact, depending on what you're trying to do, spaces might be the real answer.

Could you provide an example of how this would be used? It may make more sense to focus on the aforementioned issues instead.

Thanks again!

[nat-henderson]

I see why you wouldn't be comfortable with it - git has a lot of levers and knobs, working with multiple branches is hard, and this substantially broadens the state that the repo might be in after cloning.

I need to use this because I have a code generation pipeline which needs to push intermediate state from multiple pipeline runs at once without clobbering each other. To that end I'm using a computed unique branch name which can only be known after the task generating the output has run. It's one of the use cases specified in the "spaces" issue you posted, but my particular use case would be solved by either of concourse/concourse#684 or concourse/concourse#534.

In the meantime, since this PR won't work for you, what do you suggest for this parallel pipelines use case?

[vito]

@ndmckinley Gotcha, so are you pushing the generated code to a random branch just so that you can propagate it to subsequent jobs? Could some other resource type be used, or is there some significance to using branches?

[nat-henderson]

Assuming you mean random colloquially, as in "not determinable ahead of time", yep! And unfortunately it does need to be a git repository, because a later step of the pipeline is going to open a GitHub pull request using the generated code.

[vito]

@ndmckinley Hmm, interesting. So your use case is kind of for managing un-versioned state/data and being able to pass it between jobs. I'm guessing you only need to actually push it because you want to be able to have separate jobs? (Otherwise you'd just codegen and do whatever you want with that in one giant job without pushing.)

If Concourse had the ability to produce arbitrary artifacts and pass those between jobs, would that work too?

Assuming the following limitations:

• The artifacts have zero guarantees that they'll be there forever.
• The artifact may go away if e.g. all the workers are re-created.
• The artifact may have to be streamed across the network, which may be slow if it's large.
• If the artifact goes away, you'll need to re-run the job that produced it.

Hope you don't mind my pestering, this PR kind of touches on a lot of interesting things to think about regarding concourse/concourse#534.

Alternatively, there could be a sibling interface to resources that's simply un-versioned state. You'd then need an implementation of that interface that pushes code to arbitrary branches or something. Hmm...

[nat-henderson]

Hi,

Sorry, not in this case - I see where you're going with that, and I can imagine it being useful, but my particular needs aren't met by it.

You're right that there's a good reason I'm not doing it in one giant job - we're running many acceptance tests from potentially dozens of downstream repositories, and fanning out (then back in) is the best model for the workflow. The tests would work fine with the arbitrary artifacts you describe - in this case the artifact is the generated code and it's being streamed to test workers - it's code, so it's small, and the git repo history (which is the only large part) can be left behind with no problem. But after all the tests, unfortunately, that artifact does still have to wind up on GitHub (in a branch that won't clobber any parallel runs of the pipeline), because I need to issue pull requests against the downstream (generated) repository.

Unfortunately I haven't quite understood your comment about unversioned state - my generated code is associated with a version of the codegen config repository. Plus, if it's global state, then that's trouble for me because I'm likely to have several parallel pipeline runs at a time. If it's per-run... as far as I know Concourse doesn't really have any notion of "per-run", but I suppose that would work well for me.

e: Oh, I get it now! Yes, I would write a version of git-resource which was, instead of being a resource, git-state or something, and that would have the ability to push to arbitrary branches, which would solve many of the concerns you have here. Clever!

[vito]

e: Oh, I get it now! Yes, I would write a version of git-resource which was, instead of being a resource, git-state or something, and that would have the ability to push to arbitrary branches, which would solve many of the concerns you have here. Clever!

Right! This conversation has actually spurred on a lot of thinking about spaces and the new resource interface. I'll have something visible at https://github.com/concourse/rfcs some time soon.

[vito]

@ndmckinley I've submitted concourse/rfcs#1 which includes a tentative example for branch generation. The tl;dr is that you would put to a new space with a consistent prefix (i.e. branch names like gen-*) and then have the downstream jobs depend on those spaces via pattern matching on gen-* space names.

If that sounds like a good direction, would you be OK with closing this and using a fork in the meantime?

Thanks!

[nat-henderson]

Absolutely, I love that example and that would solve my problem perfectly. What's your process / timeline / roadmap like on this sort of thing? Are we talking a month, a quarter, December? (Feel free to close with your answer. 😄) I can use the head of this PR for a while, but eventually that's going to be a maintenance burden, of course, so I just want to know what to plan for.

[vito]

It's a requirement of concourse/concourse#1707 which we're actively working on right now. I'd give it a few months at least, plus the time involved for migrating the resources to the new interface (which may go beyond simple migration in some cases).

Closing, thanks for all the discussion!