How to Specify an agent (label) for entire pipeline to execute, Specify container for a build Step in JTE

[trqlcgcg]

Your checklist for this issue

🚨 Please review the Support Policy

• The question was asked in the project's Gitter chat
• Jenkins version
• Plugin version
• OS
• Reproduction steps

Description

Please describe your issue here, and explain what you have already tried to resolve it.

[trqlcgcg]

In our pipeline, we specify an agent (build node - multiple containers in a single K8S pod) to run the pipeline. In the build step of the build stage, we would specify the actual container (such as maven, graddle, python) to do specific build. Would that be possible in JTE?

QUESTION 1: Is there a way to specify the agent that would be executing the pipeline in JTE? It is not clear to me other the image reference in the Step in JTE documentation. see example below

pipeline {
agent (label params.MULTISTAGE_BUILD_NODE}
....
}

QUESTION 2: Is there a way to specify the container (we are using multiple containers in a single pod), to run the step, in addition to question 1; See below example

stage('build') {
steps {
container ( params.MULTISTAGE_BUILD_CONTAINER ){
...
}
}
}

[steven-terrana]

Hello @trqlcgcg - support for Jenkins declarative syntax is pending.

Implemented in the following PRs:
[1]: #109
[2]: jenkinsci/pipeline-model-definition-plugin#403

---

However, you can accomplish what you're asking using Scripted Pipeline Syntax.

Question 1:

specifying the agent is done via the node block:

node(env.MULTISTAGE_BUILD_NODE){
  ...
}

docs: https://www.jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#node-allocate-node

Question 2:

The kubernetes plugin has examples using scripted pipeline syntax.

for example (straight from their documentation):

podTemplate(containers: [
    containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),
    containerTemplate(name: 'golang', image: 'golang:1.8.0', ttyEnabled: true, command: 'cat')
  ]) {

    node(POD_LABEL) {
        stage('Get a Maven project') {
            git 'https://github.com/jenkinsci/kubernetes-plugin.git'
            container('maven') {
                stage('Build a Maven project') {
                    sh 'mvn -B clean install'
                }
            }
        }

        stage('Get a Golang project') {
            git url: 'https://github.com/hashicorp/terraform.git'
            container('golang') {
                stage('Build a Go project') {
                    sh """
                    mkdir -p /go/src/github.com/hashicorp
                    ln -s `pwd` /go/src/github.com/hashicorp/terraform
                    cd /go/src/github.com/hashicorp/terraform && make core-dev
                    """
                }
            }
        }

    }
}

[trqlcgcg]

Thanks for your reply @steven-terrana, in term of JTE, would the node (POD_LABEL) be specified at the Jenkinsfile level since I need all subsequent library calls to run on the same agent? The JTE Jenkinsfile sample looks like
node(POD_LABEL) { unit_test() build() static_code_analysis() }

Also would each library call need to have the container syntax so that I can differentiate to run maven build on a maven container and a gradle build on a gradle container?

void call(){ container ('gradle') { stage("Gradle: Build"){ println "build from the gradle library" } } }

[trqlcgcg]

Also, regarding the JTE, is there an equivalent to the declarative when {} expression?

[steven-terrana]

i believe the POD_LABEL variable is injected within podTemplate closure argument by the kubernetes plugin. it does not come from JTE.

if you need every step from JTE to be executed within the same agent then you would define that in your pipeline template:

All JTE does is inject things into the runtime environment of the pipeline template, which is then executed just like any other Jenkinsfile.

so how you choose to organize things is more a software design question than what JTE can or can't do.

---

that being said, if it were me, i would likely create a utility library that provides helper steps and i would hide all the kubernetes node logic inside that step -- this keeps the template as easy to read as possible.

template:

agent{
  // invoke whatever steps you will 
  unit_test()
  build()
  scan() 
}

and then a library configuration that looks something like:

libraries{
  utility // or whatever you call your helper library
}

and then finally the definition of the agent step:

utility/agent.groovy

/*
  the body parameter is the closure being passed from the pipeline template

  so:
    agent{ println "hey" }

  is equivalent to:
    def c = { println "hey" }
    agent(c)

  so this step takes a closure, creates an agent, and then runs the entire closure inside the build agent 
*/
void call(Closure body){
  podTemplate(containers: [
    containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),
  ]) {
    node(POD_LABEL, body)
}

[steven-terrana]

JTE is not a custom pipeline syntax like scripted or declarative syntax.

it's just a fancy way to dynamically inject objects into the pipeline runtime.

so the when functionality would be handled in Scripted pipeline syntax via if statements.

or using the same pattern above, hiding the if statements inside easier to read helper methods

---

i would probably just recommend writing your template in the declarative pipeline syntax as you're used to an utilizing JTE 2.0-alpha, currently released in the experimental update center

[trqlcgcg]

@steven-terrana , thanks. I would like to explore JTE-2.0 since most of our pipeline templates were implemented with declarative pipeline with scripted stages that uses common functions from a utility module. Hopefully we can utilize it without much rewrite. Do you have any links to JTE-2.0 plugin and docs?
The most complex construct we use is the multi-containers in our stages most of the stages would use the default jnlp base containter , with our build stage uses different flavor (maven, gradle, python, npm, etc..)

def call(closure) {
	build = [:]
	closure.resolveStrategy = Closure.DELEGATE_FIRST
	closure.delegate = build
	
pipeline {

	environment {
			GIT_SSL_NO_VERIFY = 'true'	
		}
	}
	agent { label  params.MULTISTAGE_BUILD_NODE}
	stage('validate-code') {
				when {
					expression {  deployExistingImageId == null  }
				}

				steps {
					script {
						echo "Checkout Code"
						cgutil.checkoutTag(params.GIT_REPOSITORY, currBuildTag)
						echo "Verify Code"
					}
				}
			}
               }
       stage('build') {
				
				when {
					expression {  deployExistingImageId == null  }
				}

				steps {
					container ( params.MULTISTAGE_BUILD_CONTAINER ){
						//--------------------------------------------------------------------------------		
						// Checkout code
						//--------------------------------------------------------------------------------		
						script {
							echo "Checkout Code"
							cgutil.checkoutTag(params.GIT_REPOSITORY, currBuildTag)
						}

						//--------------------------------------------------------------------------------		
						// Run build script
						//--------------------------------------------------------------------------------		
						script {
						
...}

`

[etoews]

@trqlcgcg I'm in a very similar position.

most of our pipeline templates were implemented with declarative pipeline with scripted stages that uses common functions from a utility module. Hopefully we can utilize it without much rewrite.

Did you wind up using the new Declarative Syntax Support? Do you have any examples you can share?

@steven-terrana Declarative Syntax Support is great. Do you have or are aware of any more extensive examples than what's on that page?

In particular, I'm looking for an example of a Declarative Pipeline where a different agent is used per stage but defined in the library step. Something like this

pipeline-templates/Jenkinsfile

pipeline {
  agent none
  stages {
    setup()
  }
}

pipeline-libraries/setup/steps/setup.groovy

void call() {
    stage("Setup") {
        agent {label 'base-worker'}
        steps {
            echo "run the setup stage"
        }
    }
}

But (expectedly) I get this error when trying to do so

org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 4: Expected a stage @ line 4, column 9.
           setup()
           ^

WorkflowScript: 3: No stages specified @ line 3, column 3.
     stages {
     ^

2 errors

[sdave2]

Hey, interested in a declarative syntax example as well. If it works fro you, please share an example, thank you!