Trying to download git repo within withVault block breaks #311

rscottwatson · 2023-08-09T20:07:30Z

Jenkins and plugins versions report

Environment

Jenkins: 2.401.3
OS: Linux - 5.4.0-1094-azure
Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
ace-editor:1.1
ansicolor:1.0.3
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5
authentication-tokens:1.53.v1c90fd9191a_b_
authorize-project:1.7.1
azure-commons:1.1.3
blueocean:1.27.5
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.5
blueocean-commons:1.27.5
blueocean-config:1.27.5
blueocean-core-js:1.27.5
blueocean-dashboard:1.27.5
blueocean-display-url:2.4.2
blueocean-events:1.27.5
blueocean-executor-info:1.27.5
blueocean-git-pipeline:1.27.5
blueocean-github-pipeline:1.27.5
blueocean-i18n:1.27.5
blueocean-jira:1.27.5
blueocean-jwt:1.27.5
blueocean-personalization:1.27.5
blueocean-pipeline-api-impl:1.27.5
blueocean-pipeline-editor:1.27.5
blueocean-pipeline-scm-api:1.27.5
blueocean-rest:1.27.5
blueocean-rest-impl:1.27.5
blueocean-web:1.27.5
bootstrap4-api:4.6.0-6
bootstrap5-api:5.3.0-1
bouncycastle-api:2.29
branch-api:2.1122.v09cb_8ea_8a_724
build-timeout:1.31
caffeine-api:3.1.6-115.vb_8b_b_328e59d8
checks-api:2.0.0
cloudbees-bitbucket-branch-source:825.va_6a_dc46a_f97d
cloudbees-folder:6.815.v0dd5a_cb_40e0e
command-launcher:106.vb_a_b_8f751309c
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.10.0-68.v0d0b_c439292b_
credentials:1271.v54b_1c2c6388a_
credentials-binding:631.v861c06d062b_4
data-tables-api:1.13.5-1
display-url-api:2.3.8
docker-commons:439.va_3cb_0a_6a_fb_29
docker-workflow:563.vd5d2e5c4007f
durable-task:513.vc48a_a_075a_d93
echarts-api:5.4.0-5
email-ext:2.100
favorite:2.4.3
font-awesome-api:6.4.0-2
git:5.2.0
git-client:4.4.0
git-server:99.va_0826a_b_cdfa_d
github:1.37.2
github-api:1.314-431.v78d72a_3fe4c3
github-branch-source:1732.v3f1889a_c475b_
github-oauth:588.vf696a_350572a_
gradle:2.8.2
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
hashicorp-vault-pipeline:1.4
hashicorp-vault-plugin:360.v0a_1c04cf807d
htmlpublisher:1.32
instance-identity:173.va_37c494ec4e5
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.15.2-350.v0c2f3f8fc595
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.5
jersey2-api:2.40-1
jira:3.10
jjwt-api:0.11.5-77.v646c772fddb_0
jquery-detached:1.2.1
jquery3-api:3.7.0-1
jsch:0.2.8-65.v052c39de79b_2
junit:1217.v4297208a_a_b_ce
kubernetes:3952.v88e3b_0cf300b_
kubernetes-cd:2.3.1
kubernetes-cli:1.12.0
kubernetes-client-api:6.4.1-215.v2ed17097a_8e9
kubernetes-credentials:0.10.0
kubernetes-credentials-provider:1.225.v14f9e6b_28f53
kubernetes-pipeline-devops-steps:1.6
ldap:694.vc02a_69c9787f
lockable-resources:1185.v0c528656ce04
mailer:463.vedf8358e006b_
matrix-auth:3.1.10
matrix-project:802.v8013b_40c7edc
mercurial:1260.vdfb_723cdcc81
metrics:4.2.18-442.v02e107157925
mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
momentjs:1.1.1
okhttp-api:4.11.0-157.v6852a_a_fa_ec11
pam-auth:1.10
pipeline-build-step:505.v5f0844d8d126
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:671.v07c339c842e8
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2144.v077a_d1928a_40
pipeline-model-definition:2.2144.v077a_d1928a_40
pipeline-model-extensions:2.2144.v077a_d1928a_40
pipeline-rest-api:2.33
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40
pipeline-stage-view:2.33
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.3.0
popper-api:1.16.1-3
popper2-api:2.11.6-2
pubsub-light:1.17
resource-disposer:0.23
scm-api:676.v886669a_199a_a_
script-security:1264.vecf66020eb_7d
simple-theme-plugin:160.vb_76454b_67900
slack:664.vc9a_90f8b_c24a_
snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4
sse-gateway:1.26
ssh-credentials:308.ve4497b_ccd8f4
ssh-slaves:2.916.vd17b_43357ce4
sshd:3.312.v1c601b_c83b_0e
structs:324.va_f5d6774f3a_d
timestamper:1.26
token-macro:383.v36161104b_002
trilead-api:2.84.v72119de229b_7
variant:59.vf075fe829ccb
workflow-aggregator:596.v8c21c963d92d
workflow-api:1251.vd4889a_b_0a_065
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3731.ve4b_5b_857b_a_d3
workflow-cps-global-lib:609.vd95673f149b_b
workflow-durable-task-step:1278.v94b_dc2b_50c6f
workflow-job:1316.vd2290d3341a_f
workflow-multibranch:756.v891d88f2cd46
workflow-scm-step:415.v434365564324
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:848.v5a_383b_d14921
ws-cleanup:0.45

What Operating System are you using (both controller, and any agents involved in the problem)?

Running on kubernetes 1.24.6 on AKS

Reproduction steps

This is producing the error for me.

def now = new Date()
def label = 'acr-sync-' + now.format("yyMMdd.HHmm", TimeZone.getTimeZone('America/Los_Angeles'))

def download_repo() {
    dir(targetDir){
        withVault(configuration: [engineVersion: 2, timeout: 60, vaultCredentialId: VAULT_BOT, vaultNamespace: VAULT_NAMESPACE, vaultUrl: 'https://vault.url'], vaultSecrets: [[path: VAULT_GBAAS_PATH, secretValues: [[vaultKey: 'git_bot_username'], [vaultKey: 'git_bot_token']]]]){
                sh """ echo 'My working directory is '; pwd; env; echo ${git_bot_username} """
                 git 'https://github.com/rtyley/small-test-repo'
        }
    }
}


podTemplate(label: label, yaml: """apiVersion: v1
kind: Pod
metadata:
  namespace: terraform
  labels:
    ${label}: yes
spec:
  imagePullSecrets:
    - name: acr-sp-secret
  containers:
    - name: terraform-base
      imagePullPolicy: Always
      image: "${docker_registry}/base-images/terraform:v4.8"
      command:
        - cat
      tty: true
""")


{
    node(label) {
        stage ('download repo') {
             download_repo()
        }
    }
}

Expected Results

The repo should be downloaded

Actual Results

The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
[Pipeline] }
[Pipeline] // withVault
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.244.3.221/10.244.3.221:58770
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
		at hudson.remoting.Channel.call(Channel.java:1000)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
		at com.sun.proxy.$Proxy186.execute(Unknown Source)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1226)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1309)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
		at java.base/java.lang.Thread.run(Thread.java:829)
Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: e60ebf5b-c68e-46f3-8392-d8fb07742821
java.lang.IllegalStateException: Not running on the Jenkins controller JVM
	at jenkins.util.JenkinsJVM.checkJenkinsJVM(JenkinsJVM.java:46)
	at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns.getAggregateSecretPattern(SecretPatterns.java:57)
	at com.datapipe.jenkins.vault.log.MaskingConsoleLogFilter.lambda$decorateLogger$0(MaskingConsoleLogFilter.java:43)
	at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns$MaskingOutputStream.eol(SecretPatterns.java:93)
	at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:61)
	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:57)
	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:75)
	at java.base/java.io.PrintStream.write(Unknown Source)
	at java.base/sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
	at java.base/sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source)
	at java.base/sun.nio.cs.StreamEncoder.flushBuffer(Unknown Source)
	at java.base/java.io.OutputStreamWriter.flushBuffer(Unknown Source)
	at java.base/java.io.PrintStream.newLine(Unknown Source)
	at java.base/java.io.PrintStream.println(Unknown Source)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:807)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:377)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
	at java.base/java.lang.Thread.run(Unknown Source)

Could not update commit status, please check if your scan credentials belong to a member of the organization or a collaborator of the repository and repo:status scope is selected


GitHub has been notified of this commit’s build result

Anything else?

No response

The text was updated successfully, but these errors were encountered:

ansig · 2023-08-15T11:40:46Z

We also have this problem. It happens whenever we call anything that requires a node within a withVault block.

For example, we found it when using "tool":

pipeline {
    agent any
    stages {
        stage('Foo') {
            steps {
                withVault(
                    configuration: [timeout: 60, vaultCredentialId: 'jenkins-vault-app-role', vaultUrl: "https://vault.example.com:8200"],
                    vaultSecrets: [[path: "secret/apps/jenkins", secretValues: [
                        [envVar: 'FOO', vaultKey: 'BAR']]]]) {
                    echo "${tool 'govc-v0.24.0'}"
                }
            }
        }
    }
    post {
        cleanup {
            deleteDir()
        }
    }
}

Output:

java.lang.IllegalStateException: Not running on the Jenkins controller JVM
	at jenkins.util.JenkinsJVM.checkJenkinsJVM(JenkinsJVM.java:46)
	at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns.getAggregateSecretPattern(SecretPatterns.java:57)
	at com.datapipe.jenkins.vault.log.MaskingConsoleLogFilter.lambda$decorateLogger$0(MaskingConsoleLogFilter.java:43)
	at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns$MaskingOutputStream.eol(SecretPatterns.java:93)
	at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:61)
	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:57)
	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:75)
	at java.base/java.io.PrintStream.write(PrintStream.java:568)
	at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234)
	at java.base/sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:313)
	at java.base/sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:111)
	at java.base/java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:178)
	at java.base/java.io.PrintStream.writeln(PrintStream.java:723)
	at java.base/java.io.PrintStream.println(PrintStream.java:1051)
	at hudson.Launcher.printCommandLine(Launcher.java:817)
	at hudson.Launcher.maskedPrintCommandLine(Launcher.java:832)
	at hudson.Launcher$LocalLauncher.launch(Launcher.java:977)
	at hudson.Launcher$ProcStarter.start(Launcher.java:509)
	at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1398)
	at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1340)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:377)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

But if we just do, say, "echo" it works fine:

pipeline {
    agent any
    stages {
        stage('Foo') {
            steps {
                withVault(
                    configuration: [timeout: 60, vaultCredentialId: 'jenkins-vault-app-role', vaultUrl: "https://vault.example.com:8200"],
                    vaultSecrets: [[path: "secret/apps/jenkins", secretValues: [
                        [envVar: 'FOO', vaultKey: 'BAR']]]]) {
                    echo "${FOO}"
                }
            }
        }
    }
    post {
        cleanup {
            deleteDir()
        }
    }
}

Output:

[Pipeline] stage
[Pipeline] { (Foo)
[Pipeline] echo
Warning: A secret was passed to "echo" using Groovy String interpolation, which is insecure.
		 Affected argument(s) used the following variable(s): [FOO]
		 See https://jenkins.io/redirect/groovy-string-interpolation for details.
****
[Pipeline] }

It does not matter if we use withVault in the Pipeline options, even if that is run with "agent none" and the agent is assigned in the stage instead:

pipeline {
    agent none
    options {
        withVault(
            configuration: [timeout: 60, vaultCredentialId: 'jenkins-vault-app-role', vaultUrl: "https://vault.example.com:8200"],
            vaultSecrets: [[path: "secret/apps/jenkins", secretValues: [
                [envVar: 'FOO', vaultKey: 'BAR']]]])
    }
    stages {
        stage('Foo') {
            agent any
            steps {
                echo "${tool 'govc-v0.24.0'}"
            }
            post {
                cleanup {
                    deleteDir()
                }
            }
        }
    }
}

So it seems related specifically to printing masked values. I believe it is related to a new version of the credentials-binding plugin (631.v861c06d062b_4) that includes the following PR: jenkinsci/credentials-binding-plugin#260

The PR is related to: https://www.jenkins.io/security/advisory/2023-04-12/#SECURITY-3075

It might be related to: https://www.jenkins.io/security/advisory/2023-05-16/#SECURITY-3077

alviss7 · 2023-08-15T12:36:19Z

I confirm that it's linked to this PR, I've downgraded the version of this plugin to make my pipelines work again.

jglick · 2023-08-23T14:54:58Z

Tracked as https://issues.jenkins.io/browse/JENKINS-71788 but unfortunately this plugin seems to have kept both GitHub Issues and Jira open, so it is unclear which is authoritative.

rscottwatson added the bug label Aug 9, 2023

basil mentioned this issue Aug 22, 2023

[SECURITY-3075] getAggregateSecretPattern to fail when run inside agent JVM jenkinsci/credentials-binding-plugin#260

Merged

jglick mentioned this issue Aug 23, 2023

fix: proper way of loading pattern secrets [SECURITY-3077] #309

Closed

This was referenced Sep 5, 2023

[JENKINS-71788] Fix timing of SecretPatterns.getAggregateSecretPattern #314

Merged

Can't use checkout step inside withVault: "Not running on the Jenkins controller JVM" #313

Closed

jetersen closed this as completed in #314 Sep 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trying to download git repo within withVault block breaks #311

Trying to download git repo within withVault block breaks #311

rscottwatson commented Aug 9, 2023

ansig commented Aug 15, 2023

alviss7 commented Aug 15, 2023

jglick commented Aug 23, 2023

Trying to download git repo within withVault block breaks #311

Trying to download git repo within withVault block breaks #311

Comments

rscottwatson commented Aug 9, 2023

Jenkins and plugins versions report

What Operating System are you using (both controller, and any agents involved in the problem)?

Reproduction steps

Expected Results

Actual Results

Anything else?

ansig commented Aug 15, 2023

alviss7 commented Aug 15, 2023

jglick commented Aug 23, 2023