Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Interrupted jenkins builds may retry and continue #7549

Open
pshipton opened this issue Oct 22, 2019 · 3 comments
Open

Interrupted jenkins builds may retry and continue #7549

pshipton opened this issue Oct 22, 2019 · 3 comments
Labels
comp:build prio:high

Comments

@pshipton
Copy link
Member

After cancelling a build, the jobs get interrupted, however they may just retry and continue, wasting valuable machine time.

Cloning repository https://github.com/eclipse/openj9.git
 > git init /var/jenkins_home/workspace/Build_JDKnext_x86-64_windows_OpenJDK@script # timeout=10
Using reference repository: /var/jenkins_home/openjdk_cache
Fetching upstream changes from https://github.com/eclipse/openj9.git
 > git --version # timeout=10
 > git fetch --tags --force --progress -- https://github.com/eclipse/openj9.git +refs/heads/*:refs/remotes/origin/*
ERROR: Checkout failed
java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Unknown Source)
	at java.lang.UNIXProcess.waitFor(Unknown Source)
	at hudson.Proc$LocalProc.join(Proc.java:325)
	at hudson.Proc.joinWithTimeout(Proc.java:171)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2168)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1864)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:78)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:545)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:758)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1152)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:149)
	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:299)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Retrying after 10 seconds
No credentials specified
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository https://github.com/eclipse/openj9.git
 > git init /var/jenkins_home/workspace/Build_JDKnext_x86-64_windows_OpenJDK@script # timeout=10
Using reference repository: /var/jenkins_home/openjdk_cache
@pshipton
Copy link
Member Author

@AdamBrousseau @jdekonin

@AdamBrousseau
Copy link
Contributor

Yes, I've known about this for a while and had it in the back of my mind I would fix these some day. Thanks for opening the issue. I never actually dug into one of these before now but the one you have quoted was killed before it got to our script. I.e. It was doing the initial SCM checkout on Jenkins master to load the pipeline script.

https://ci.eclipse.org/openj9/job/Build_JDKnext_x86-64_windows_OpenJDK/76/console

Started by upstream project "Pipeline_Build_Test_JDKnext_x86-64_windows" build number 274
originally caused by:
 Started by upstream project "Pipeline-OpenJDK-Acceptance" build number 86
 originally caused by:
  Started by user Peter Shipton
Running as Peter Shipton
Checking out git https://github.com/eclipse/openj9.git into /var/jenkins_home/workspace/Build_JDKnext_x86-64_windows_OpenJDK@script to read buildenv/jenkins/jobs/pipelines/Build-Any-Platform.groovy
No credentials specified
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository https://github.com/eclipse/openj9.git
 > git init /var/jenkins_home/workspace/Build_JDKnext_x86-64_windows_OpenJDK@script # timeout=10
Using reference repository: /var/jenkins_home/openjdk_cache
Fetching upstream changes from https://github.com/eclipse/openj9.git
 > git --version # timeout=10
 > git fetch --tags --force --progress -- https://github.com/eclipse/openj9.git +refs/heads/*:refs/remotes/origin/*
ERROR: Checkout failed
java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Unknown Source)
	at java.lang.UNIXProcess.waitFor(Unknown Source)
	at hudson.Proc$LocalProc.join(Proc.java:325)
	at hudson.Proc.joinWithTimeout(Proc.java:171)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2168)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1864)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:78)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:545)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:758)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1152)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:149)
	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:299)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Retrying after 10 seconds
No credentials specified
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository https://github.com/eclipse/openj9.git
 > git init /var/jenkins_home/workspace/Build_JDKnext_x86-64_windows_OpenJDK@script # timeout=10
Using reference repository: /var/jenkins_home/openjdk_cache
Fetching upstream changes from https://github.com/eclipse/openj9.git
 > git --version # timeout=10
 > git fetch --tags --force --progress -- https://github.com/eclipse/openj9.git +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/eclipse/openj9.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/eclipse/openj9.git # timeout=10
Fetching upstream changes from https://github.com/eclipse/openj9.git
 > git fetch --tags --force --progress -- https://github.com/eclipse/openj9.git 
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/refs/heads/master^{commit} # timeout=10
Checking out Revision 0ed23240dcbe304891252efee4914266f487a963 (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0ed23240dcbe304891252efee4914266f487a963
Commit message: "Use getROMMethodFromRAMMethod for JITServer (#7537)"
 > git rev-list --no-walk 2a6f784ad24596c98ad6fb2443f1cb7ad4b294f5 # timeout=10
Running in Durability level: PERFORMANCE_OPTIMIZED
[Pipeline] Start of Pipeline
[Pipeline] timeout
Timeout set to expire in 10 hr
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] node
00:01:38.118  Running on worker-1 in /home/jenkins/jenkins-agent/workspace/Build_JDKnext_x86-64_windows_OpenJDK
[Pipeline] {
[Pipeline] checkout
00:01:38.377  No credentials specified
00:01:39.106  Cloning the remote Git repository
00:01:39.145  Cloning repository https://github.com/eclipse/openj9.git
00:01:39.145   > git init /home/jenkins/jenkins-agent/workspace/Build_JDKnext_x86-64_windows_OpenJDK # timeout=10
00:01:39.155  Using reference repository: /home/jenkins/openjdk_cache
00:01:39.156  Fetching upstream changes from https://github.com/eclipse/openj9.git
00:01:39.156   > git --version # timeout=10
00:01:39.158   > git fetch --tags --progress -- https://github.com/eclipse/openj9.git +refs/heads/*:refs/remotes/origin/*
00:01:41.146   > git config remote.origin.url https://github.com/eclipse/openj9.git # timeout=10
00:01:41.150   > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
00:01:41.223   > git config remote.origin.url https://github.com/eclipse/openj9.git # timeout=10
00:01:41.299  Fetching upstream changes from https://github.com/eclipse/openj9.git
00:01:41.299   > git fetch --tags --progress -- https://github.com/eclipse/openj9.git +refs/heads/*:refs/remotes/origin/*
00:01:42.422  Checking out Revision 0ed23240dcbe304891252efee4914266f487a963 (refs/remotes/origin/master)
00:01:42.322   > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
00:01:42.391   > git rev-parse refs/remotes/origin/refs/heads/master^{commit} # timeout=10
00:01:42.461   > git config core.sparsecheckout # timeout=10
00:01:42.465   > git checkout -f 0ed23240dcbe304891252efee4914266f487a963
00:02:04.207  Commit message: "Use getROMMethodFromRAMMethod for JITServer (#7537)"
[Pipeline] load
[Pipeline] { (buildenv/jenkins/common/variables-functions.groovy)
[Pipeline] load
[Pipeline] { (buildenv/jenkins/common/pipeline-functions.groovy)
[Pipeline] }
[Pipeline] // load
[Pipeline] }
[Pipeline] // load
[Pipeline] echo
00:02:05.558  VARIABLE_FILE:'buildenv/jenkins/variables/defaults.yml'
[Pipeline] echo
00:02:05.712  VENDOR_REPO:''
[Pipeline] echo
00:02:05.714  VENDOR_BRANCH:''
[Pipeline] echo
00:02:05.716  VENDOR_CREDENTIALS_ID:''
[Pipeline] fileExists
[Pipeline] echo
00:02:06.066  Using variables file: buildenv/jenkins/variables/defaults.yml
[Pipeline] readYaml
[Pipeline] echo
00:02:06.375  Using OPENJDK_REPO = https://github.com/ibmruntimes/openj9-openjdk-jdk.git OPENJDK_BRANCH = openj9-staging OPENJDK_SHA = 2740f02de0ae17e022bfc53a77e7c3d5058e32b6
[Pipeline] echo
00:02:06.390  Using OPENJ9_REPO = https://github.com/eclipse/openj9.git OPENJ9_BRANCH = master OPENJ9_SHA = 0ed23240dcbe304891252efee4914266f487a963
[Pipeline] echo
00:02:06.393  Using OMR_REPO = https://github.com/eclipse/openj9-omr.git OMR_BRANCH = openj9 OMR_SHA = a12acf1cb23d40c45c9375342f2bc14406172fdf
[Pipeline] echo
00:02:06.447  Using BUILD_ENV_CMD = , BUILD_ENV_VARS_LIST = [PATH+TOOLS=/cygdrive/c/openjdk/LLVM64/bin:/cygdrive/c/openjdk/nasm-2.13.03]
[Pipeline] sh
00:02:07.444  + date +%Y%m%d-%H%M%S
[Pipeline] echo
00:02:07.651  Using SDK_FILENAME = OpenJ9-JDKnext-x86-64_windows-20191022-141518.tar.gz
[Pipeline] echo
00:02:07.653  Using TEST_FILENAME = native-test-libs.tar.gz
[Pipeline] echo
00:02:07.697  Using artifactory server/repo: ci-eclipse-openj9 / ci-eclipse-openj9
[Pipeline] echo
00:02:07.700  Keeping '30' artifacts
[Pipeline] echo
00:02:07.703  Artifactory Manual Cleanup: true
[Pipeline] load
[Pipeline] { (buildenv/jenkins/common/build.groovy)
[Pipeline] load
[Pipeline] { (buildenv/jenkins/common/pipeline-functions.groovy)
[Pipeline] }
[Pipeline] // load
[Pipeline] }
[Pipeline] // load
[Pipeline] cleanWs
00:02:08.422  [WS-CLEANUP] Deleting project workspace...
00:02:08.422  [WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
00:02:15.688  [WS-CLEANUP] done
[Pipeline] }
[Pipeline] // node
[Pipeline] stage
[Pipeline] { (Queue)
[Pipeline] node
00:02:30.727  Still waiting to schedule task
00:02:30.728  Waiting for next available executor on ‘ci.role.build&&hw.arch.x86&&sw.os.windows’
Calling Pipeline was cancelled
Aborted by Peter Shipton
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] End of Pipeline
Finished: ABORTED

Found this
https://issues.jenkins-ci.org/browse/JENKINS-39194
and
jenkinsci/workflow-cps-plugin#147

Tested internally. There is a global setting SCM checkout retry count which will retry scm checkouts when lightweight checkout is not selected (different type of clone used). This retry does not obey interrupted exception and will do a retry even when the build is cancelled. A regular pipeline retry(X) will stop when a build is cancelled (https://issues.jenkins-ci.org/browse/JENKINS-44379).

Opened https://issues.jenkins-ci.org/browse/JENKINS-59895

@AdamBrousseau
Copy link
Contributor

A workaround would be to set the value to zero so there are no retries on SCM checkout.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
comp:build prio:high
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@AdamBrousseau @pshipton