Intent-based pipelines

[justinabrahms]

Hey all.

I'd really like a system that allows me to specify a subset of information than vendors typically require to setup, validate and deploy code. Ideally, this would be a yaml-based DSL that assumes containers, but otherwise ignores implementation specifics. The result is compiled out into whatever tool you're using (e.g. tekton).

Do you know of anything like this?

[rakhbari]

@justinabrahms if I'm understanding you correctly, I'm not aware of such an abstracted interface DSL for CI/CD workflows/pipelines.

Such an abstraction was heavily discussed and debated in the early stages of the Tekton ecosystem development. However, the ultimate consensus by the group was that although such an interface might seem useful, at the end of the day targeting Kubernetes as a platform, was ultimately more useful and more worth spending development time on. The group was very careful to design the Task CRD completely open-ended so that any container image could be defined as the execution environment for the Task steps.

But if your question/query is about a completely implementation agnostic DSL that can be interpreted to produce various implementations such as Tekton, Argo Workflows, Flux CD, etc, then no, I haven't seen one out there. And the fact that there really isn't one out there is perhaps a clue that others feel the same way the Tekton development group did on the subject.

The closest I've seen to such a thing is actually Jenkins X, which has a DSL that behind the scenes uses Tekton pipelines, tasks / steps to accomplish the runtime. But it's been well over a year since I've kept up with that ecosystem so someone from that development group would have to comment on this.

[justinabrahms]

For the sake of discussion, this is the thing I was hoping that app developers would need to write. Hopefully it is clear that this would be much simpler than the corresponding code in existing CD frameworks. (and if it's not, tell me the tool you're using so I can just use that :) )

meta:
  # channel which will get updates on deploys. One message for each release and
  # information in the thread as it progresses through the pipeline.
  slackChannel: myapp-deploys
  owners: your-ldap-or-equivalent-group-here
build:
  # Explicitly not setting up a jenkins instance. Framework should do this.
  default:  # These apply to all stages below, unless they're overridden
    cores: 100m
    memory: 300Mb
    approvals:
      - canary:
          # deploy to 1% then 10% then 100% as a rollout strategy. There will be
          # a few options available.
          rolloutStrategy: ONE_TEN_ALL
          bakeTime: 5m
  dev:
    rollout:
      regions: dev-west-1a
  staging:
    rollout:
      regions: dev-west-1a
  prod:
    rollout:
      regions: prod-west-1a, prod-west-1b, prod-west-2a, prod-west-2c
      # Starting from 1, deploy to 2x the regions of the last deploy, so 1,2,4,8,16.
      strategy: GEOMETRIC
    approvals:
      - metricsValidator:
          promql: sum(increase(failure[5m])) > 5
          onResults: rollback

[ankitm123]

Jenkins X used to have it's own DSL back in v2 (v1 predates me, I am not sure what was the format of the pipelines back then), but in v3 we use vanilla tekton syntax. This allows for better integration with tekton for jx end users (just copy paste catalog and use them).
In the past Jenkins X used to support triggering Jenkins pipelines, but it's deprecated (and almost removed - the code for that has not been touched in months), maintaining logic to trigger different families of CI/CD pipelines (tekton, jenkins etc ...) quickly became hard to maintain.