Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add Java 1.8 easyconfig that can be updated in-place to more recent JDK 1.8.x #6712

Merged
merged 13 commits into from
Sep 21, 2018

Conversation

boegel
Copy link
Member

@boegel boegel commented Aug 16, 2018

cfr. discussion in #5203

I feel we should use this approach going forward (starting with the *2018b* toolchain generation), but we shouldn't change older easyconfigs to also adapt to this policy since that will lead to too much confusion.
We can easily switch to this policy for the 2018b toolchain generation since currently only 1 easyconfig already merged in develop is affected (Bazel), and it wasn't included in an EasyBuild release yet.

cc @JackPerdue

@verdurin, @vanzod: Since you were critical about following this approach, I'd like to hear back from you before this gets merged...

edit: now requires easybuilders/easybuild-easyblocks#1503

@@ -12,6 +12,6 @@ sources = ['%(namelower)s-%(version)s-dist.zip']
checksums = ['c730593916ef0ba62f3d113cc3a268e45f7e8039daf7b767c8641b6999bd49b1']

builddependencies = [('binutils', '2.30')]
dependencies = [('Java', '1.8.0_162', '', True)]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this might not merge if the currently pending TensorFlow 1.10 gets merged first, since it uses _172

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm aware of that, I'm hoping #6677 gets merged first, and then I'll resolve the conflict here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@boegel Doesn't look like #6677 will merge before this one.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it doesn't, that's fine, then @akesandgren just need to update #6677 accordingly, the change will be trivial.

akesandgren
akesandgren previously approved these changes Aug 17, 2018
Copy link
Contributor

@akesandgren akesandgren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me and it does reduce the Java problem somewhat.

@vanzod
Copy link
Member

vanzod commented Aug 17, 2018

@boegel I have changed my mind about the wrapping. I was young back then...

@boegel
Copy link
Member Author

boegel commented Aug 17, 2018

@vanzod So, to be clear, no objections from your side on this?

@boegel boegel requested a review from verdurin August 18, 2018 08:03
easyblock = 'Bundle'

name = 'Java'
version = '1.8'
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, would 8 make more sense here? Whatever the source tarball may be named, the actual version is actually Java 8, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really care....
[j-perdue@tlogin-0502 (08:50) ~]$ ml Java/1.8.0
[j-perdue@tlogin-0502 (08:50) ~]$ java -version
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)

I just went with the naming standard that EB already used. 1.8 seems to make some sense.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might be a good idea for 9

[j-perdue@login2 (08:59) ~]$ ml Java/9.0
[j-perdue@login2 (09:00) ~]$ java -version
java version "9.0.4"
Java(TM) SE Runtime Environment (build 9.0.4+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.4+11, mixed mode)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This thought came up mostly due to the new Java 10, jumping from 1.8 to 10 would be sort of strange.

Also, I think using ('Java', '8') would make it more clear that we're using a wrapper, since you'd expect a major.minor...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went with 'java -version'... makes sense for 9 and 10 but Oracle was still calling it 1.8 even if every one else called it Java 8 (like they've have through at least five other past major releases). I'd stick to upstream versioning.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At www.oracle.com/technetwork/java/javase/downloads they refer to it as Java SE 8 and Java 8u181.

What java -version prints is probably mostly a concern with backwards compatibility, which they broke in Java 9.x

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

your call... I don't care either way

@verdurin
Copy link
Member

@boegel I'm a bit confused - why are all these other changes in here now?

@boegel
Copy link
Member Author

boegel commented Aug 19, 2018

@verdurin Hmm, not sure what went wrong there... The other commits were all stuff that's already in develop. I did a git rebase develop + push --force for this branch, which seems to have fixed it.

@@ -12,6 +12,6 @@ sources = ['%(namelower)s-%(version)s-dist.zip']
checksums = ['c730593916ef0ba62f3d113cc3a268e45f7e8039daf7b767c8641b6999bd49b1']

builddependencies = [('binutils', '2.30')]
dependencies = [('Java', '1.8.0_162', '', True)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@boegel Doesn't look like #6677 will merge before this one.

easyblock = 'Bundle'

name = 'Java'
version = '8'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes quite a lot of changes all in one go - I feel this should be made more prominent somehow.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you clarify? All we're changing here is using a wrapper for Java...

This change in policy will be clearly mentioned in the EasyBuild release notes, and maybe we should also highlight it with a separate post on the mailing list...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just meant that we're going to change the way the Java version is reported at the same time. I would agree with the mailing list post.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I just think it's good to start using this versioning scheme now with a switch to the wrapper... I've always found the 1.8.x versioning a bit awkward since it was Java (SE) 8.

Also, the JDK will still be versioned according to how the sources are versioned.

That said, I'm happy to roll this back to Java/1.8 if people think that makes more sense.

In any case, for Java 10.x it'll likely be Java/10...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't disagree, just noting that we're making lots of simultaneous changes, so we should communicate this as widely as possible.

verdurin
verdurin previously approved these changes Aug 19, 2018
Copy link
Member

@verdurin verdurin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine now.

@vanzod
Copy link
Member

vanzod commented Aug 19, 2018

@boegel No, I am ok with the change. The only thing that may cause issues on our side is the impact on module collections as discussed in #5203.

@boegel
Copy link
Member Author

boegel commented Aug 22, 2018

As highlighted by @vanzod, the current approach using using a Java/8 (or Java/1.8) wrapper around a fully versioned JDK module causes problems with Lmod module collections (see #5203 (comment) for more details).

So, if we want to go through with this we need to do it without a wrapper: a Java-8.eb easyconfig that installs a specific version (e.g. 1.8.0_181), and which gets updated in place over time.

The downside of this approach is that it will take a little bit longer to do an in-place update using eb Java-8.eb -df, since it involves an actual re-installation rather than simply replacing a wrapper module, so changes are a bit higher of causing troubling by briefly removing the Java/8 module.

I see no other option, so we should discuss whether or not to go through with this.
We still have a window of opportunity here w.r.t. the upcoming EasyBuild release (3.7.0) and the 2018b toolchain generation, so now's a good time to make a decision.

@JackPerdue
Copy link
Contributor

I don't like it. I specifically wrote my version for quick rebuild of the module while in live use. If "module save" has problems then lets talk to Robert about it.

@JackPerdue
Copy link
Contributor

And I can update the damn versions... its only taken 10 months for this issue to come to some (wrong) resolution.

@boegel
Copy link
Member Author

boegel commented Aug 23, 2018

@JackPerdue We can't change the policy w.r.t. how we deal with Java as a dependency at any random moment, part of the reason why this took so long to be picked up is because we have a window of opportunity now to take the approach you suggested for the 2018b toolchain generation & beyond.
That, and the fact that we're all struggling to find time to work on things like this unfortunately led to where we are now w.r.t. time...

I consider the problem that @vanzod brought up w.r.t. module collections a blocker for going through with the wrapper approach: this would hit any user relying on module collections as soon as the Java wrapper is updated in-place, which is not acceptable imho. Especially because a user could restore a collections weeks or months after the Java wrapper was updated in place, so there's no correlation time-wise between the update & the ml restore failure...

We can let @rtmclay pitch in on this, but I'm not sure it matters too much: the check that Lmod does makes sense to me, since the user can't restore the module collection anymore exactly as it was saved...

As already mentioned, the other option we have is to simply change the versioning scheme of the Java easyconfigs to only use the major version (e.g. 8 or 10 in the near future) in version, but use a fully specified version in sources and update that over time, and then let sites do a forced reinstall of Java whenever they see fit.

Your suggested approach is better since it only involves rewriting a module file for the wrapper, so it's fast, while now updating Java in place involves an actual re-installation. But (re-)installing Java is fast enough that this is mostly a non-argument imho...

Come to think of it, another downside of going forward without the wrapper is that we'll effectively be removing an existing Java installation, so the impact is potentially more dramatic compared to just changing a wrapper (where no actual software installations are removed), in the sense that it may affect running jobs that all of a sudden can't find java anymore for a couple of seconds.

That may just be an academic concern though, reinstalling Java takes about 3 seconds on an (fairly old) Intel Sandy Bridge system, I just timed that...

@boegel
Copy link
Member Author

boegel commented Sep 7, 2018

Thanks to a suggestion by @mboisson, it looks like we do have a better option after all: effectively making Java/1.8 a wrapper for Java/1.8.0_181 via module-version in a .modulerc file (see also easybuilders/easybuild-easyblocks#1503)...

@boegel
Copy link
Member Author

boegel commented Sep 19, 2018

Test report by @boegel
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in this PR)
node3140.skitty.os - Linux centos linux 7.5.1804, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz, Python 2.7.5
See https://gist.github.com/0cfcb80fde3a73c66b4e348f621622e9 for a full test report.

@boegel
Copy link
Member Author

boegel commented Sep 19, 2018

Test report by @boegel
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in this PR)
node2404.golett.os - Linux centos linux 7.5.1804, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, Python 2.7.5
See https://gist.github.com/b030f77ed23d1a9d86043596d70b39a0 for a full test report.

Copy link
Member

@migueldiascosta migueldiascosta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@migueldiascosta
Copy link
Member

test report coming soon, and also manually tested other easyconfigs (e.g. octave), looks good

@bartoldeman did you mean to review this yourself?

@migueldiascosta
Copy link
Member

Test report by @migueldiascosta
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in this PR)
grc-cluster1 - Linux centos 6.9, Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz, Python 2.7.14
See https://gist.github.com/173a52f3c8220195485142a40ff2cb92 for a full test report.

@bartoldeman
Copy link
Contributor

bartoldeman commented Sep 21, 2018 via email

@bartoldeman
Copy link
Contributor

Test report by @bartoldeman
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in this PR)
aw-4r13-n22 - Linux centos 6.6, Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz, Python 2.6.6
See https://gist.github.com/f761aa5d98339ce5ff90038ee6d2a74c for a full test report.

@bartoldeman
Copy link
Contributor

going in, thanks @boegel !

@bartoldeman bartoldeman merged commit 3f29f07 into easybuilders:develop Sep 21, 2018
@boegel boegel deleted the java18 branch September 21, 2018 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants