[JENKINS-39194] Implement SCM retry count #147

mmitche · 2017-06-20T22:23:05Z

Honors the retry count of the Jenkins server, and also allows for a custom retry if desired. Only has effect in non-lightweight cases.

mmitche · 2017-06-20T22:24:35Z

@jglick Fixes https://issues.jenkins-ci.org/browse/JENKINS-39194

Honors the retry count of the Jenkins server. Only has effect in non-lightweight cases.

mmitche · 2017-06-21T18:38:52Z

@jglick I was having trouble with the custom retry. The job config save doesn't go through the descriptor's configure, so ensuring that the jenkins backend doesn't always set the retry count to 0 for a job was impossible.

To unblock for now I just removed the advanced custom retry setting.

jglick · 2017-06-21T18:45:32Z

The job config save doesn't go through the descriptor's configure

Not sure what you mean by that.

the jenkins backend doesn't always set the retry count to 0

Sorry, what?

mmitche · 2017-06-21T18:55:18Z

@jglick So if you look at the way that the config-retry.jelly is implemented, it wants to call hasCustomScmCheckoutRetryCount() on the plugin to determine the checkbox, then binds the value of the text box to scmCheckoutRetryCount. scmCheckoutRetryCount is an Integer which is supposed to be null if the box isn't checked. This is accomplished for AbstractProject by overriding submit() on AbstractBuild and determining whether the checkbox was checked, otherwise it resets scmCheckoutRetryCount to null.

In our case, if you don't implement similar logic, then when you save the job, scmCheckoutRetryCount gets assigned a default value, which ends up being 0, rather than null. The right way to fix this I think would be to override Descriptor's configure method, parse the json like AbstractBuild does, and then set the appropriate values. Saving the job doesn't appear to go through configure() in this case.

Anyways, I could be totally wrong on how to implement the custom retry count UI. I played around with it for a while but wasn't able to get it to work. If you have some suggestions, let me know.

jglick · 2017-06-21T18:46:15Z

src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java

@@ -127,15 +134,38 @@ public boolean isLightweight() {
        delegate.setPoll(true);
        delegate.setChangelog(true);
        try (WorkspaceList.Lease lease = computer.getWorkspaceList().acquire(dir)) {
-            delegate.checkout(build, dir, listener, node.createLauncher(listener));


diff for review

jglick · 2017-06-21T18:46:41Z

src/main/resources/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition/config.jelly

@@ -24,7 +24,7 @@ THE SOFTWARE.
 -->

 <?jelly escape-by-default='true'?>
-<j:jelly xmlns:j="jelly:core" xmlns:f="/lib/form" xmlns:st="jelly:stapler">
+<j:jelly xmlns:j="jelly:core" xmlns:f="/lib/form" xmlns:st="jelly:stapler" xmlns:p="/lib/hudson/project">


Why? Revert if unused.

jglick · 2017-06-21T18:46:57Z

src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java

@@ -79,6 +82,10 @@ public String getScriptPath() {
    public boolean isLightweight() {
        return lightweight;
    }
+
+    public int getScmCheckoutRetryCount() {


Currently used only internally, so better inlined.

jglick · 2017-06-21T18:51:27Z

src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java

+                }
+
+                if (retryCount == 0)   // all attempts failed
+                    throw new RunnerAbortedException();


Mentally I had to rearrange this into something like

for (int x = 0; x <= getScmCheckoutRetryCount(); x++) { // … script = scriptFile.readToString(); break; // … listener.getLogger().println("Retrying after 10 seconds"); Thread.sleep(10000); } if (script == null { throw …; }

to follow the logic, though I think it is right as written.

I followed the same form as AbstractBuild: https://github.com/jenkinsci/jenkins/blob/d13b1361e1650494528a7b46f82fe25d7fb5e3c3/core/src/main/java/hudson/model/AbstractBuild.java#L589

I can change it, just want to make sure there is a good reason to do so.

jglick · 2017-06-21T18:51:41Z

src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java

+                }
+
+                if (retryCount == 0)   // all attempts failed
+                    throw new RunnerAbortedException();


Use AbortException not RunnerAbortedException. Can probably rearrange the logic a bit so that if we are on the last iteration (the normal case), the original AbortException or whatever is rethrown unmodified.

jglick · 2017-06-21T18:52:25Z

src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java

+                        listener.error(e.getMessage());
+                    }
+                } catch (InterruptedIOException e) {
+                    throw (InterruptedException)new InterruptedException().initCause(e);


Why wrap it?

I followed the same form as AbstractBuild: https://github.com/jenkinsci/jenkins/blob/d13b1361e1650494528a7b46f82fe25d7fb5e3c3/core/src/main/java/hudson/model/AbstractBuild.java#L589

I think this is because InterruptedIOException is IOException derived, and InterruptedException is Exception derived. Maybe plain InterruptedException

But this method throws Exception, so there is no need for a wrapper.

Okay sounds good, will remove.

jglick · 2017-06-21T18:56:19Z

src/test/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinitionTest.java

+        p.setDefinition(def);
+        // Initially check for the global Jenkins retry count
+        assertEquals(r.jenkins.getScmCheckoutRetryCount(), def.getScmCheckoutRetryCount());
+        // Set the jenkins global retry count to 3 and ensure that they are still equal


jglick · 2017-06-21T18:56:52Z

src/test/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinitionTest.java

+        assertEquals(r.jenkins.getScmCheckoutRetryCount(), def.getScmCheckoutRetryCount());
+        WorkflowRun b = r.assertBuildStatus(Result.FAILURE, p.scheduleBuild2(0));
+        r.assertLogContains("Could not read from remote repository", b);
+        r.assertLogContains("Retrying after 10 seconds", b);


Could I suppose use waitForMessage, initialize the repository, then assert then the build completes.

SpencerMalone · 2017-07-25T16:39:05Z

So what's missing for this to get a merge, besides resolving conflicts?

mmitche · 2017-09-01T16:59:51Z

Can we get a merge here? It's a big issue in certain cases (like GitHub being flaky)

mmitche · 2017-09-18T16:48:29Z

@jglick Anything else?

svanoort · 2017-09-27T15:06:56Z

Close/re-open to try to retrigger CI

jglick

Minor issues but basically looks right.

jglick · 2017-09-27T15:19:05Z

src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java

@@ -83,7 +86,7 @@ public boolean isLightweight() {
    @DataBoundSetter public void setLightweight(boolean lightweight) {
        this.lightweight = lightweight;
    }
-
+    


BTW please configure your text editor to not make gratuitous whitespace changes to otherwise unmodified lines.

jglick · 2017-09-27T15:19:58Z

src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java

+                        listener.error(e.getMessage());
+                    }
+                } catch (InterruptedIOException e) {
+                    throw (InterruptedException)new InterruptedException().initCause(e);


But this method throws Exception, so there is no need for a wrapper.

jglick · 2017-09-27T15:22:12Z

src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java

+                    throw (InterruptedException)new InterruptedException().initCause(e);
+                } catch (IOException e) {
+                    // checkout error not yet reported
+                    listener.error(e.toString());


Should be something like

listener.error("Checkout failed").println(Functions.printThrowable(e).trim()); // TODO 2.43+ use Functions.printStackTrace

jglick · 2017-09-27T15:22:42Z

src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java

+                }
+
+                if (retryCount == 0)   // all attempts failed
+                    throw new AbortException();


Should specify a message.

jglick · 2017-09-27T15:24:39Z

src/test/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinitionTest.java

+        def.setLightweight(false);
+        p.setDefinition(def);
+        r.jenkins.setScmCheckoutRetryCount(1);
+        WorkflowRun b = r.assertBuildStatus(Result.FAILURE, p.scheduleBuild2(0));


Would be nice to extend this test to run a second build, this time using r.waitForMessage("Retrying after 10 seconds", b) and then initializing the repo and then asserting that the build does succeed with the second checkout.

svanoort · 2017-09-27T21:19:16Z

Looks like CI issues?

Installing /home/jenkins/tools/hudson.model.JDK/jdk8/jdk.sh
Can not call launch on a dummy launcher.

kofwang · 2017-12-01T08:08:06Z

Why is this PR not merged?

abayer · 2018-01-12T15:18:37Z

@mmitche Can you resolve the merge conflict, please?

mmitche · 2018-01-12T15:49:31Z

Done

svanoort

Looks fine to me, and thank you for the contribution -- once CI shows green, I'll merge and it'll get shipped with the next release (should be in the next week or two).

mmitche · 2018-01-25T15:33:26Z

Anything else @svanoort?

svanoort · 2018-01-25T19:16:02Z

Nope, merging now, thanks for the contribution @mmitche (and the reminder now that it is passing)

Inspired by jenkinsci/workflow-cps-plugin#147

Implement SCM retry count

d52b4b5

Honors the retry count of the Jenkins server. Only has effect in non-lightweight cases.

mmitche force-pushed the add-retry branch from c3ac9e5 to d52b4b5 Compare June 21, 2017 18:37

jglick reviewed Jun 21, 2017

View reviewed changes

mmitche added 4 commits June 23, 2017 11:10

PR feedback

7510f76

Reset config.jelly

efdcf9f

Update comment

a3f8b53

Remove inined calls

e9a314e

mmitche added 3 commits August 16, 2017 10:12

Merge branch 'master' into add-retry

a0b97c1

Workflow script might be null

71307f4

Merge remote-tracking branch 'upstream/master' into add-retry

6baed11

abayer approved these changes Sep 18, 2017

View reviewed changes

svanoort closed this Sep 27, 2017

svanoort reopened this Sep 27, 2017

jglick approved these changes Sep 27, 2017

View reviewed changes

jglick changed the title ~~Implement SCM retry count~~ [JENKINS-39194] Implement SCM retry count Sep 27, 2017

svanoort closed this Sep 27, 2017

svanoort reopened this Sep 27, 2017

Merge remote-tracking branch 'upstream/master' into add-retry

b9b5e38

review feedback

774e4f7

Merge branch 'master' into add-retry

e8fb2c8

svanoort approved these changes Jan 12, 2018

View reviewed changes

svanoort merged commit d91d14d into jenkinsci:master Jan 25, 2018

olivierdagenais mentioned this pull request Jul 12, 2018

feat: retry checkout of global pipeline library jenkinsci/workflow-cps-global-lib-plugin#47

Merged

AdamBrousseau mentioned this pull request Oct 22, 2019

Interrupted jenkins builds may retry and continue eclipse-openj9/openj9#7549

Open

topikachu pushed a commit to topikachu/workflow-scm-step-plugin that referenced this pull request Dec 9, 2022

Honor SCM checkout retry count

991f04f

Inspired by jenkinsci/workflow-cps-plugin#147

topikachu mentioned this pull request Dec 9, 2022

Honor SCM checkout retry count jenkinsci/workflow-scm-step-plugin#110

Merged

6 tasks

dwnusbaum mentioned this pull request Mar 27, 2024

Avoid retrying if the build is aborted jenkinsci/workflow-scm-step-plugin#134

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JENKINS-39194] Implement SCM retry count #147

[JENKINS-39194] Implement SCM retry count #147

mmitche commented Jun 20, 2017

mmitche commented Jun 20, 2017

mmitche commented Jun 21, 2017

jglick commented Jun 21, 2017

mmitche commented Jun 21, 2017

jglick Jun 21, 2017

jglick Jun 21, 2017

jglick Jun 21, 2017

jglick Jun 21, 2017

mmitche Jun 21, 2017

jglick Jun 21, 2017

mmitche Jun 23, 2017

jglick Jun 21, 2017

mmitche Jun 21, 2017

jglick Sep 27, 2017

mmitche Sep 27, 2017

jglick Jun 21, 2017

jglick Jun 21, 2017

SpencerMalone commented Jul 25, 2017

mmitche commented Sep 1, 2017

mmitche commented Sep 18, 2017

svanoort commented Sep 27, 2017

jglick left a comment

jglick Sep 27, 2017

jglick Sep 27, 2017

jglick Sep 27, 2017

jglick Sep 27, 2017

jglick Sep 27, 2017

svanoort commented Sep 27, 2017

kofwang commented Dec 1, 2017

abayer commented Jan 12, 2018

mmitche commented Jan 12, 2018

svanoort left a comment

mmitche commented Jan 25, 2018

svanoort commented Jan 25, 2018

[JENKINS-39194] Implement SCM retry count #147

[JENKINS-39194] Implement SCM retry count #147

Conversation

mmitche commented Jun 20, 2017

mmitche commented Jun 20, 2017

mmitche commented Jun 21, 2017

jglick commented Jun 21, 2017

mmitche commented Jun 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SpencerMalone commented Jul 25, 2017

mmitche commented Sep 1, 2017

mmitche commented Sep 18, 2017

svanoort commented Sep 27, 2017

jglick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

svanoort commented Sep 27, 2017

kofwang commented Dec 1, 2017

abayer commented Jan 12, 2018

mmitche commented Jan 12, 2018

svanoort left a comment

Choose a reason for hiding this comment

mmitche commented Jan 25, 2018

svanoort commented Jan 25, 2018