Add integration tests for both Freestyle and Pipeline jobs #81

basil · 2019-02-20T00:01:44Z

See JENKINS-36013. The Swarm Plugin doesn't have any integration tests that automatically run the Swarm Client against a real Jenkins master. Such tests would be highly desirable to prevent future regressions.

This change adds two new test classes: SwarmClientIntegrationTest (which tests the Swarm Client using a Freestyle job) and PipelineTest (which tests the Swarm Client using a Pipeline job). The tests work by downloading the Swarm Client from the Jenkins master and using it to connect to Jenkins.

basil · 2019-02-20T00:04:31Z

plugin/pom.xml

+          <configuration>
+            <warSourceDirectory>../client/target</warSourceDirectory>
+          </configuration>
+        </plugin>


This change is necessary to allow the Swarm Client JAR to be downloaded from integration test context. Normally, this JAR is made available through the .hpi file via the maven-dependency-plugin (see lines 25–51 above). However, in integration test context the .hpi file is ignored and the test uses the .hpl version of the plugin. As such we need to customize the .hpl metadata to include the Swarm Client JAR on its classpath.

It is not safe, because it also injects the client's classes to the plugin. And not all this libs are guaranteed to be compatible.

It is not safe, because it also injects the client's classes to the plugin. And not all this libs are guaranteed to be compatible.

It is safe, for the reasons explained in my top-level reply.

basil · 2019-02-20T00:06:54Z

plugin/src/test/java/hudson/plugins/swarm/PipelineJobTest.java

+    @Test
+    public void buildShellScriptAfterRestart() throws Exception {
+        Assume.assumeNotNull(
+                System.getProperty("port"), "This test requires a fixed port to be available.");


This tests that a Swarm Client launched with -deleteExistingClients can reconnect to a Jenkins master after the master has been restarted. The -deleteExistingClients only works in this use case when the Jenkins master has the same port before and after the restart; however, RestartableJenkinsRule chooses a random port each time. Therefore, we are forced to skip this test unless the tests are being run with -Dport=<port-number> to fix the port before and after the restart. Unfortunately, this means this test cannot run on ci.jenkins.io since AFAIK there is no isolation between builds there (so tests cannot assume that a particular port will be free).

RestartableJenkinsRule chooses a random port each time

That ought to be fixable, just like it already keeps the same $JENKINS_HOME.

Good point. I'll work on that in a downstream PR so as to keep this PR from developing a dependency on a Jenkins test harness change.

Sure. If you want CI on that,

mvn -f jenkins-test-harness clean install source:jar-no-fork deploy:deploy -DaltDeploymentRepository=maven.jenkins-ci.org::default::https://repo.jenkins-ci.org/snapshots/ -DskipTests

and specify the timestamped snapshot for jenkins-test-harness.version. Though I should probably just incrementalify it (JEP-305) to make this easier. Give me a moment.

Sure. If you want CI on that,

mvn -f jenkins-test-harness clean install source:jar-no-fork deploy:deploy -DaltDeploymentRepository=maven.jenkins-ci.org::default::https://repo.jenkins-ci.org/snapshots/ -DskipTests

and specify the timestamped snapshot for jenkins-test-harness.version.

Thanks! I did as you suggested in #83.

oleg-nenashev

Thanks @basil ! It would be great to have such tests, but I do not think they justify inclusion of the client code into the plugin. It may cause various issues, e.g. remoting lib conflicts for newer Jenkins versions.

I highly recommend starting an external process or a Docker fixture if you want to test the connection logic

oleg-nenashev · 2019-03-21T07:47:06Z

plugin/pom.xml

+          <configuration>
+            <warSourceDirectory>../client/target</warSourceDirectory>
+          </configuration>
+        </plugin>


It is not safe, because it also injects the client's classes to the plugin. And not all this libs are guaranteed to be compatible.

basil · 2019-03-21T16:22:15Z

Thanks @basil ! It would be great to have such tests, but I do not think they justify inclusion of the client code into the plugin.

Thanks for the review, @oleg-nenashev. The client JAR is already included in the plugin as of #62, which made the client JAR available from the master via HTTP at $JENKINS_URL/swarm/swarm-client.jar. #62 did this by putting the client JAR in the base directory of the .hpi, which made it available through HTTP. Note that this did not place the JAR on the classpath, which would have required putting it in the classes or WEB-INF/lib directory of the .hpi.

#62 successfully made the client JAR available via HTTP from the .hpi in production. Tests, however, use the .hpl rather than the .hpi, so #62 did not successfully make the client JAR available via HTTP from tests. My change merely brings the .hpl into parity with the .hpi, thereby completing the implementation of #62. Downloading the client JAR via HTTP now works in both production context and test context.

It may cause various issues, e.g. remoting lib conflicts for newer Jenkins versions.

No, because warSourceDirectory is a "single directory for extra files to include in the WAR" whose default value is ${basedir}/src/main/webapp, which is not on the production classpath. These extra files are only available through HTTP requests to $JENKINS_URL/swarm/, just as in production context.

I verified my understanding experimentally both in production context (where the existing logic from #61 applies) and in test context (where my new logic applies) as follows:

From the Jenkins script console, I imported hudson.plugins.swarm.DownloadClientAction. This class is only available in the plugin and not the client, so it should be on the master's classpath. The import succeeded, as expected.
From the Jenkins script console, I imported hudson.plugins.swarm.Candidate. This class is only available in the client and not the plugin, so it should not be on the master's classpath. The import failed, as expected.

Based on the above reasoning and experimental verification, I am confident this is not an issue.

I highly recommend starting an external process or a Docker fixture if you want to test the connection logic

I am already doing the former (starting an external process) in this change. See the new TestUtils#createSwarmClient method I added.

jglick

Good to see this! Left some minor comments.

jglick · 2019-04-30T12:07:30Z

plugin/src/test/java/hudson/plugins/swarm/PipelineJobTest.java

+    @Test
+    public void buildShellScriptAfterRestart() throws Exception {
+        Assume.assumeNotNull(
+                System.getProperty("port"), "This test requires a fixed port to be available.");


RestartableJenkinsRule chooses a random port each time

That ought to be fixable, just like it already keeps the same $JENKINS_HOME.

jglick · 2019-04-30T12:08:59Z

plugin/pom.xml

+        <dependency>
+            <groupId>org.jenkins-ci.plugins.workflow</groupId>
+            <artifactId>workflow-support</artifactId>
+            <version>2.13</version>


Introduce a POM property to ensure that the unclassified artifact is also in the same version.

BTW the GH feature of marking a conversation as resolved works very nicely in my experience to hide obsolete discussions.

jglick · 2019-04-30T12:09:20Z

plugin/src/test/java/hudson/plugins/swarm/PipelineJobTest.java

+    @ClassRule public static BuildWatcher buildWatcher = new BuildWatcher();
+
+    private final ProcessDestroyer processDestroyer = new ProcessDestroyer();
+    private final TemporaryDirectoryAllocator tmpDir = new TemporaryDirectoryAllocator();


Better to use org.junit.TemporaryFolder.

jglick · 2019-04-30T12:09:38Z

plugin/src/test/java/hudson/plugins/swarm/PipelineJobTest.java

+    /** Executes a shell script build on a Swarm Client agent. */
+    @Test
+    public void buildShellScript() throws Exception {
+        story.addStep(


If you are in Java 8, it is nicer to use .then.

jglick · 2019-04-30T12:10:09Z

plugin/src/test/java/hudson/plugins/swarm/PipelineJobTest.java

+                        project.setDefinition(new CpsFlowDefinition(getFlow(node, 0), true));
+
+                        WorkflowRun build = project.scheduleBuild2(0).waitForStart();
+                        story.j.waitForCompletion(build);


WorkflowRun build = story.j.buildAndAssertSuccess(project);

jglick · 2019-04-30T12:15:28Z

plugin/src/test/java/hudson/plugins/swarm/test/TestUtils.java

+        File output = new File(tmpDir, "swarm-client.jar");
+        try (InputStream inputStream = input.openStream();
+                FileOutputStream outputStream = new FileOutputStream(output)) {
+            ByteStreams.copy(inputStream, outputStream);


I think there i a FileUtils.copy or similar that makes that simpler.

jglick · 2019-04-30T12:15:43Z

plugin/src/test/java/hudson/plugins/swarm/test/TestUtils.java

+        return output;
+    }
+
+    /** Wait for the agent with the given name to come online against the given Jenkins instance. */


There is already a method for this in JenkinsRule.

I tried using JenkinsRule#waitOnline, but I wasn't able to make it work. That method takes as input a Slave, and I don't have a Slave to pass into it because the Slave is created by the Swarm plugin and not my test code.

jglick · 2019-04-30T12:16:06Z

plugin/src/test/java/hudson/plugins/swarm/test/TestUtils.java

+        StringBuilder sb = new StringBuilder();
+        sb.append("\"" + System.getProperty("java.home") + "/bin/java\"");
+        sb.append(" -Djava.awt.headless=true");
+        sb.append(" -jar \"" + swarmClientJar.getAbsolutePath() + "\"");


Suggested change

sb.append(" -jar \"" + swarmClientJar.getAbsolutePath() + "\"");

sb.append(" -jar \"" + swarmClientJar + "\"");

(toString already gets the path)

jglick · 2019-04-30T12:17:02Z

plugin/src/test/java/hudson/plugins/swarm/test/TestUtils.java

+            sb.append(" " + arg);
+        }
+
+        ProcessBuilder pb = new ProcessBuilder(Util.tokenize(sb.toString()));


So, simpler to construct a List<String> argv to begin with, and dispense with the quoting.

jglick · 2019-04-30T12:17:59Z

plugin/pom.xml

+        <dependency>
+            <groupId>org.jenkins-ci.plugins</groupId>
+            <artifactId>scm-api</artifactId>
+            <version>1.3</version>


Things like this you probably want in dependencyManagement, since they are not direct dependencies, pending JENKINS-47498.

basil · 2019-04-30T15:47:27Z

Good to see this! Left some minor comments.

Thanks for the review!

jglick · 2019-04-30T17:45:10Z

plugin/src/test/java/hudson/plugins/swarm/PipelineJobTest.java

+                    project.setDefinition(new CpsFlowDefinition(getFlow(node, 1), true));
+
+                    WorkflowRun build = project.scheduleBuild2(0).waitForStart();
+                    SemaphoreStep.waitForStart("wait-0/1", build);


Note that since this is waiting in node but not sh it only exercises part of jenkinsci/workflow-durable-task-step-plugin#104 (when jenkinsci/workflow-durable-task-step-plugin@58b127f was added).

Note that since this is waiting in node but not sh it only exercises part of jenkinsci/workflow-durable-task-step-plugin#104

Good point. I added a new test, PipelineJobTest#buildShellScriptAcrossDisconnect, which waits in sh (essentially a port of ExecutorStepTest#buildShellScriptAcrossDisconnect).

basil · 2019-05-21T15:14:33Z

@oleg-nenashev I think I addressed all of your feedback above. Do you have any objection to proceeding with this change?

oleg-nenashev

Thanks for the explanation in #81 (comment) . I think we can go forward with this PR

basil commented Feb 20, 2019

View reviewed changes

oleg-nenashev self-requested a review February 20, 2019 07:30

oleg-nenashev requested changes Mar 21, 2019

View reviewed changes

basil mentioned this pull request Apr 30, 2019

[JENKINS-49707] (part): Fail the build if an agent has been removed jenkinsci/workflow-durable-task-step-plugin#104

Merged

4 tasks

jglick reviewed Apr 30, 2019

View reviewed changes

jglick mentioned this pull request Apr 30, 2019

Enable deployment of Incremental releases jenkinsci/jenkins-test-harness#131

Merged

basil force-pushed the testing branch from bb86844 to 5a87f33 Compare April 30, 2019 15:51

This was referenced Apr 30, 2019

Simplify node replacement code #82

Merged

Allow port reuse in RestartableJenkinsRule jenkinsci/jenkins-test-harness#132

Merged

Use a fixed port when running integration tests that restart Jenkins #83

Merged

jglick reviewed Apr 30, 2019

View reviewed changes

basil force-pushed the testing branch 3 times, most recently from 6a96558 to 8b4fa2e Compare April 30, 2019 20:57

basil mentioned this pull request May 15, 2019

New Docker approach to test the client #64

Closed

basil requested a review from oleg-nenashev May 21, 2019 15:13

Add integration tests for both Freestyle and Pipeline jobs

7934fbd

basil force-pushed the testing branch from 8b4fa2e to 7934fbd Compare May 21, 2019 15:25

oleg-nenashev approved these changes May 21, 2019

View reviewed changes

basil merged commit c0e44ac into jenkinsci:master May 21, 2019

basil mentioned this pull request May 21, 2019

Add environment variables support #80

Merged

basil deleted the testing branch June 2, 2019 14:59

basil mentioned this pull request Jun 2, 2019

EOL Docker support #111

Merged

	sb.append(" -jar \"" + swarmClientJar.getAbsolutePath() + "\"");
	sb.append(" -jar \"" + swarmClientJar + "\"");

Add integration tests for both Freestyle and Pipeline jobs #81

Add integration tests for both Freestyle and Pipeline jobs #81

Conversation

basil commented Feb 20, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleg-nenashev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

basil commented Mar 21, 2019

jglick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

basil commented Apr 30, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

basil commented May 21, 2019

oleg-nenashev left a comment

Choose a reason for hiding this comment