WIP: Julia CI build

[iblislin]

(Ref: #7741 (comment))
I can help for adding Julia CI build. Any tips for how to start?

[szha]

Thanks for offering to help. To my best knowledge, the source code is in https://github.com/dmlc/MXNet.jl, and it has travis-ci and appvayor integration, though the builds might be failing.

[iblislin]

Hi @szha,

I want to add CI build into upstream Jenkins.
MXNet.jl is quite passive. We do not get any notification about api breaking changes or new feature added.
Currently, we are using Travis nightly build to discover api changes. :p

e.g.

• cblas: import all func prototype into header dmlc/MXNet.jl#267
• random: upstream api change dmlc/MXNet.jl#288

[szha]

My understanding is that you can achieve this by adding job to the Jenkinsfile.
Adding @mbaijal in case I missed something.

[iblislin]

https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/PR-8175/5/pipeline

The graph of pipeline does not include Julia. Do I miss something?

[iblislin]

Figured out this: https://support.cloudbees.com/hc/en-us/articles/227246367-Why-Jenkinsfile-changes-Are-Not-Reflected-in-PR-Build
I do not have permission to do that...

[szha]

@srochel could you advise who would be the best contact to help with this PR's setup?

[goswamig]

@iblis17 Thanks for doing this.
As you might have noticed since your PR was on master branch. The build https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/PR-8175/5/pipeline didn't use your Jenkins Change. So even if the build has succeeded your change has not been verified . The very ideal way to verify the Jenkins change is by publishing your change to a branch other than master hence that branch can use this Jenkins (with your change) file. I know its a bit confusing but thats how the Jenkins is :)

@indhub has recently created a new branch "ci_test" your change should be verified using that branch.

[indhub]

@gautamkmr: Chris was able to modify Jenkinsfile in this PR: #7854.

Why can't this PR change Jenkinsfile similarly?

Changes made to Jenkinsfile in that PR:
e6cad58 (this broke the build)
99083f1 (this fixed it)

[goswamig]

The reason I suspect that https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/PR-8175/5/pipeline doesn't have any build entry for Julia.

[indhub]

@iblis17 Could you please do this change in ci-test branch? We can test it there and then move the changes to master. Please let me know once you have a PR ready for ci-test branch and I can merge it. Let me know if you have any question. Thanks!

[iblislin]

@indhub the reason is here: https://support.cloudbees.com/hc/en-us/articles/227246367-Why-Jenkinsfile-changes-Are-Not-Reflected-in-PR-Build

I do not have write permission to this repo, and unable to commit to ci-test

[szha]

@iblis17 you can make a pull request to that branch, and @indhub will be able to help test out the change there without affecting the master branch. It will then be merged to master branch if everything works fine.

[iblislin]

ok, switched

[iblislin]

https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/ci-test/2/pipeline

my bad, there is a typo:

diff --git a/Jenkinsfile b/Jenkinsfile
index fcbf5840..b7639bb9 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -413,7 +413,7 @@ try {
             init_git()
             unpack_lib('cpu')
             withEnv(['MXNET_HOME=/workspace/ut-julia-cpu']) {
-              timeout(time: max_timem unit: 'MINUTES') {
+              timeout(time: max_time, unit: 'MINUTES') {
                 sh "${docker_run} cpu julia -e 'Pkg.clone(\"julia-package\", \"MXNet\")'"
                 sh "${docker_run} cpu julia -e 'Pkg.build(\"MXNet\")'"
                 sh "${docker_run} cpu julia -e 'Pkg.test(\"MXNet\")'"

[iblislin]

https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/ci-test/3/pipeline

Running 'julia -e Pkg.clone("julia-package", "MXNet")' inside mx-ci.cpu...
Adding group `jenkins' (GID 1001) ...
Done.
ERROR: syntax: incomplete: premature end of input
script returned exit code 1

no idea why there is a syntax error. perhaps the problem is single quote?

---

patch

diff --git a/Jenkinsfile b/Jenkinsfile
index fcbf5840..eff1edb6 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -413,10 +413,8 @@ try {
             init_git()
             unpack_lib('cpu')
             withEnv(['MXNET_HOME=/workspace/ut-julia-cpu']) {
-              timeout(time: max_timem unit: 'MINUTES') {
-                sh "${docker_run} cpu julia -e 'Pkg.clone(\"julia-package\", \"MXNet\")'"
-                sh "${docker_run} cpu julia -e 'Pkg.build(\"MXNet\")'"
-                sh "${docker_run} cpu julia -e 'Pkg.test(\"MXNet\")'"
+              timeout(time: max_time, unit: 'MINUTES') {
+                sh "${docker_run} cpu tests/jenkins/run_julia_test.sh"
               }
             }
           }
diff --git a/tests/jenkins/run_julia_test.sh b/tests/jenkins/run_julia_test.sh
new file mode 100755
index 00000000..0d7a73ff
--- /dev/null
+++ b/tests/jenkins/run_julia_test.sh
@@ -0,0 +1,28 @@
+#!/bin/sh
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+# Exit script with error if any errors occur
+set -e
+
+julia -e 'versioninfo()'
+
+julia -e 'Pkg.clone("julia-package", "MXNet")'
+julia -e 'Pkg.build("MXNet")'
+julia -e 'Pkg.test("MXNet")'

[iblislin]

bump?

[mbaijal]

Hi @iblis17 I should be able to help you with this.
You can ping me on slack at mbaijal.

[iblislin]

bump again.

[marcoabreu]

Sorry, is there anything I can do for you?

[iblislin]

Seem there aren't any response on the mailing list?
I will try to make a PR this weekend, to import julia code via git subtree.

[marcoabreu]

I heard quite a lot of enthusiasm from customer teams in our office about the usage of MXNet with Julia, so I'm sure they'll be happy to see it!