{lang}[dummy/,dummy/dummy] Java v1.7.0, Java v1.8.0, JDK v1.7.0_80, ...

[JackPerdue]

(created using eb --new-pr)

[JackPerdue]

Intended to provide a more timely way to update packages built with Oracle's Java. I realize it is antithetical to the one true EasyBuild philosophy (which I agree with more than I disagree with) since it subtracts the exact revision of, say, Java/1.8.0_15x. But the present EB implementation for Java makes it very hard to maintain time/secure updates for Oracle's Java.

The idea here is that the sysadm can install a new JDK when released and then bump the Java bundle version (which takes very little time).

[boegel]

Hmm, I'd like to get some feedback from @easybuilders/easybuild-easyconfigs-maintainers on this approach...

[JackPerdue]

me too (hint hint)

[verdurin]

@JackPerdue - could you spell out the benefits more explicitly? In principle I don't like the idea of a module that hides the sub-version value like this.

[JackPerdue]

eb --search Java | grep 1.[78].0_

How many of those would you expect to have a secure Java? I believe Oracle would say that only 1.7.0_80 and 1.8.0_144 are supported at the moment. If I built all of those previously should I have to rebuild them all again (and renumber them) for the latest Java?

It's like OpenSSH/OpenSSL in a way.... we use the system (os) version to get timely updates. At present, the only way to keep up to date with Java updates with EB is rebuild everything that needs it.

Look... I'm totally down with reproducible builds and all that (and it should still be supported through the saved configs/logs for the JDK used) but at present EB's Java nonmenclature is causing problems for me (my boss only wants the latest 1.7 and 1.8 on the system)

[boegel]

Strictly speaking, this is not specific to Java.

It would be nice to be a bit more liberal about versions and support some type of wildcards/ranges, but it's also a can of worms when it's being used too frequently...

See also easybuilders/easybuild-framework#746 and easybuilders/easybuild-framework#219.

[JackPerdue]

Tis true that EB could utilize some of the features Lmod offers in terms of versions. Tis also true that its a can of writhing slimy worms that won't just wiggle onto the hook so you can catch a fish for dinner. This is just a stop gap measure to experiment with to keep everyone's Java up to date.

Oracle's Java (and I'll point out its Oracle and not some open source or Sun anymore) has very clear delimiters. Not like whether GCC 6.3.0 will work the same just like 6.3.9... they pretty much say "toss the old stuff, only the new version is supported"..... nailing down the version numbers ends up with a module collection using whatever Java was current at the time.

Like I said, its kinda like OpenSSL. EB likes consistency but at the end of the day it doesn't know what version of OpenSSL was used in the underlying build so to some extent it isn't reproducible anyway.

Anyway... I manually edited all the modules here to use 1.8.0 instead of 1.8.0_xx in when loading Java. There has been no screaming (I did this on Halloween too). Until we open that can of worms (it can wait), I thought it was a good idea.

[mboisson]

For Java, and for most software in general, I think that EasyBuild is much too strict on version numbers. Using the hierarchy, for compilers and MPI, we are using the two-digits rule within our stack, i.e. the same modules will be visible with OpenMPI 2.1.1 as with 2.1.0, or with GCC 5.4.0 as with GCC 5.4.1

Even two digits seems to be too much for some things. For example, I'm not convinced that modules compiled with Intel 2017.1 aren't compatible with those compiled with Intel 2017.5...

Now, with non-toolchain dependencies, we can't really do that as EasyBuild outputs load for specific versions in the modules.

[JackPerdue]

ping I still think this would simplify things quite a bit. I'm going to have to go through all my easyconfigs that use it before submitting PRs that use Java. I don't much care for it hiding the subversion either but the present EB implementation for Java ends up with a lot of different packages using a lot of different versions.

And then there is the security issue (which overrides the version hiding in my book). Sun/Oracle always recommend the latest release for security issues. At present, if I want to secure my system I have to bump the version number in every package that uses and rebuild which causes disruptions on our systems. I should be able to just bump the version number in this and all that use say 1.8.0 would automatically be updated to the latest.

Adding Lmod features would be ideal in the end but for now I think this is a reasonable approach in that:

1. it doesn't break old easyconfigs
2. it allows new easyconfigs not to be bound to a particular (perhaps old) version of Java

[boegel]

@JackPerdue Maybe use 1.7.0 here, and change it below to use

dependencies = [
    ('JDK', version + '_80'),
]

[boegel]

@JackPerdue I like this approach, but I wonder how we can transition to it... Once this is merged, do we just update all easyconfigs that now depend on a specific version of Java to 1.7.0 or 1.8.0 instead?

How do you handle updating the Java/1.80 module, do you keep a close eye on Java updates?

How do you effectively do the update when there's a new version? Do you manually change the module, or replace with eb --force?

[JackPerdue]

@boegel To answer your last questions.

Yes, I would update all easyconfigs to use the shortened Java version.

At present, I manually check for updates "on occasion" (perhaps not often enough). I don't have any tools to notify me when there are new ones.

For now, to update, I --force a rebuild of the Java module using the latest JDK (instead manually updating the module) because I want a record in ebfiles_repos (and the installation directory) indicating updates. It only takes a few seconds (as a Bundle) so doesn't cause "too much" disruption (haven't had an issue yet [knock on wood])

[boegel]

To get back to this: I'm wondering whether we should go forward with this approach for the 2018b generation of easyconfigs onwards...

@vanzod Can you discuss this during the upcoming EasyBuild conf call, to collect some more feedback?

The concrete proposal would be:

• Java-1.8.eb easyconfig which just wraps around latest JDK 1.8.0_x, for example JDK-1.8.0_172.eb
• use ('Java', '1.8') in dependencies where relevant

This allows to update the Java/1.8 module in place with JDK updates, without having to reinstalling everything that depends on Java/1.8.

Although this is an improvement over the current approach (hardcoding to a specific subversion of Java, e.g. 1.8.0_162), I'm not terrible happy about the wrapping aspect, as @verdurin already mentioned, but I can't think of a better approach either...
Except maybe for supporting wildcards in dependency versions, but that will cause other problem (different Java versions being used as dependencies making installed modules incompatible with each other over time since they will be using different Java 1.8.0_x versions...).

[mboisson]

At least for Lmod, you can use Lmod features to state a dependency over a minimum version instead of a fix3d version, so this should not be an issue for Lmod.

…

On juill. 6, 2018 at 4:56 a.m., <Kenneth Hoste ***@***.***)> wrote: To get back to this: I'm wondering whether we should go forward with this approach for the 2018b generation of easyconfigs onwards... @vanzod (https://github.com/vanzod) Can you discuss this during the upcoming EasyBuild conf call, to collect some more feedback? The concrete proposal would be: Java-1.8.eb easyconfig which just wraps around latest JDK 1.8.0_x, for example JDK-1.8.0_172.eb use ('Java', '1.8') in dependencies where relevant This allows to update the Java/1.8 module in place with JDK updates, without having to reinstalling everything that depends on Java/1.8. Although this is an improvement over the current approach (hardcoding to a specific subversion of Java, e.g. 1.8.0_162), I'm not terrible happy about the wrapping aspect, as @verdurin (https://github.com/verdurin) already mentioned, but I can't think of a better approach either... Except maybe for supporting wildcards in dependency versions, but that will cause other problem (different Java versions being used as dependencies making installed modules incompatible with each other over time since they will be using different Java 1.8.0_x versions...). — You are receiving this because you commented. Reply to this email directly, view it on GitHub (#5203 (comment)), or mute the thread (https://github.com/notifications/unsubscribe-auth/AHKsam1wm8TTDKHXtvc4OmbcLgRIjPcnks5uDyZNgaJpZM4P0nIg).

[vanzod]

I like the general idea and at the same time I join @verdurin as not being a supporter of the wrapping. We should definitely discuss what Lmod could do to help us out here.

[mstud]

@mboisson I believe the Lmod version modifier functions (atleast, between, latest) do not work together with the depends_on feature yet though. I'd prefer to have that over the ability to specify minimum versions, so in my eyes, this wouldn't be of much use.

[boegel]

@verdurin @vanzod What's your big concern with the wrapping? I see no better (general) way to allow for updating Java to the latest bugfix/security release "in place" without having to regenerate a whole bunch of modules.

Relying on Lmod-specific features is not a good solution imho, we need something general that'll also work for sites that don't use Lmod (yet).

[verdurin]

@boegel I don't like the fact that users lose explicit track of which version they're running. In general that feels like a cause of possible problems later. I do see the benefits as well, so it's a trade-off. Certainly the Lmod features mitigate that, with the downside that you mention.

[boegel]

I propose we follow this approach going forward, starting with the 2018b toolchain generation, but keep things as is for older toolchain, to avoid confusion, see #6712

[boegel]

@verdurin Users won't lose track imho... java -version will still print out the exact version they're using, and module list will also show it. Or did you mean something else?

[verdurin]

@boegel I'm primarily thinking of their job submission scripts. I've seen cases where people have been caught out because they've kept in a non-versioned module load when versions change. Even though we ask them always to use a specific version.
Granted that is lees likely to happen with minor Java version updates.

[boegel]

@verdurin Not only that, it's also a way more general problem, I don't see how making Java a wrapper for JDK makes that worse...

BTW, you can configure Lmod to only allow full-versioned loads:

$ module avail JDK/

------ /prefix/modules/all -------
   JDK/1.8.0_181


$ export LMOD_EXACT_MATCH=1
$ module load JDK
Lmod has detected the following error:  These module(s) exist but cannot be loaded as requested: "JDK"
   Try: "module spider JDK" to see how to load the module(s).

The error message could be better, but you can tell Lmod to only allow fully specified module load statements...

[vanzod]

@boegel May this approach cause issues with Lmod named collections?
If I create a module collection it will contain both the wrapper module and the actual Java module. Then if I point the wrapper to a new Java version, the collection may not be consistent anymore.
Thoughts?

[boegel]

@vanzod I think it depends on whether you've configured Lmod to pin the versions of the modules in the collection. And even then, the previous JDK module will still be around.

Maybe @rtmclay can pitch in here? If module A loaded B/1.0 yesterday, and a collection involving A was saved, will there be any problems after A is changed to load B/2.0 instead when the collection is restored?

[boegel]

@vanzod Hmm, seems like your concern is justified...

$ ml --version

Modules based on Lua: Version 7.7.26  2018-04-03 23:18 -05:00
    by Robert McLay [email protected]

# A/1.2.3 loads B/1, which currently loads C/1.0
$ ml

Currently Loaded Modules:
  1) C/1.0   2) B/1   3) A/1.2.3

$ ml save A_today
Saved current collection of modules to: "A_today"

# update B/1 to load C/1.1
$ echo 'load("C/1.1")' > /tmp/$USER/modules/all/B/1.lua

# try to restore saved collection...
$ ml purge
$ ml restore A_today
Restoring modules from user's A_today
Lmod Warning:  One or more modules in your A_today collection have changed: "B".
To see the contents of this collection execute:
  $ module describe A_today
To rebuild the collection, load the modules you wish, then execute:
  $ module save A_today
If you no longer want this module collection execute:
  $ rm ~/.lmod.d/A_today

Not sure if this Lmod config setting has something to do with it...

$ ml --config 2>&1 | grep Pin
Pin Versions in restore          yes

[akesandgren]

yes it does, because with that option enabled Lmod writes the actual version numbers in the saved collection.

[boegel]

This will never get merged as is, since i) it's outdated now, and ii) the problems with Lmod collections are a blocker for using a Java wrapper, so if we want to go through with this, we'll need another approach.

Let's follow up in this at #6712

[mboisson]

Maybe I am late to the party here, but instead of doing a wrapper, how about using modulerc and default versions ?

For example :
depends_on("java/1.8")

with modulerc
module-version java 1.8 1.8.0_181

[boegel]

@mboisson Ugh, I hate it when people come up with good suggestions... ;-)

I played around with this a bit, and it seems like this can actually work, even with Lmod collections:

$ cat Java/.modulerc
#%Module
module-version Java/1.8.0_181 8
$ ml av Java/
---- /tmp/test_java ----
   Java/1.8.0_181 (8)

(the (8) looks a bit weird here, but fine I guess)

$ ml Java/8
$ ml
Currently Loaded Modules:
  1) Java/1.8.0_181

Note how Java/8 isn't really there, it's literally just an alias for Java/1.8.0_181.

This helps with saved collections, as long as the Java/1.8.0_181 module stays in place:

$ ml save javatest
Saved current collection of modules to: "javatest"
$ ml purge

$ cat Java/.modulerc
#%Module
module-version Java/1.8.0_213 8

$ ml av Java/
---------------- /tmp/test_java ----------------
   Java/1.8.0_181    Java/1.8.0_213 (D:8)

Restoring the saved collection works fine, since the module version alias is irrelevant:

$  ml restore javatest
Restoring modules from user's javatest

$ ml
Currently Loaded Modules:
  1) Java/1.8.0_181

And the new version alias is indeed active, as long as we load Java/8 rather than a specific version (which is what EB would do):

$ ml Java/8

The following have been reloaded with a version change:
  1) Java/1.8.0_181 => Java/1.8.0_213

So this is effectively a clean wrapper as originally intended by JackPerdue, which can be updated in a heartbeat/atomically, no need to reinstall anything.

The only problem is: EasyBuild will need to be made aware of these module version aliases, since when it tries to resolve the Java/8 dependency, it won't be able to currently, even if the .modulerc is in place.

EasyBuild relies on the output of ml --terse av:

$ ml --terse av Java/
/tmp/test_java:
Java/1.8.0_181
Java/8(@Java/1.8.0_213)
Java/1.8.0_213

(but this format is most likely specific to Lmod...)

And then the other question is: should the .modulerc be under the control of EasyBuild?
It probably should be, since it affects the dependency resolution mechanism, etc.

So, we're looking at adding support for using a new ModuleRC easyblock that can be used in the Java-8.eb easyconfig, and somehow making the check for available modules aware of module version aliases (which I doubt can be done in a modules tool agnostic way...).

Thoughts on this @JackPerdue, @vanzod?

[mboisson]

modulerc is hierarchical, i.e. there's a global one and there's a user one. We make a rather heavy use of it on Compute Canada (see below for our current one).

If EasyBuild goes this way, there should probably be a mechanism to have both some EasyBuild- managed part and some system-managed part. Since recent versions of Lmod, the modulerc can be written in .lua, so presumably we can "import" a modulerc from another modulerc. @rtmclay would know if this could work ?

#%Module

# default versions
module-version boost/1.60.0 default
module-version boost-mpi/1.60.0 default
module-version imkl/11.3.4.258 default
module-version intel/2016.4 default
module-version openmpi/2.1.1 default
module-version gcc/5.4.0 default
module-version cuda/8.0.44 default
module-version python/3.5.4 default
module-version singularity/2.5 default
module-version matlab/2017a default

# shortcut versions
module-version intel/2017.1 2017 17
module-version intel/2016.4 2016 16
module-version python/2.7.14 2.7 2
module-version python/3.5.4 3.5
module-version python/3.6.3 3.6
module-version python/3.7.0 3.7 3
module-version r/3.3.3 3.3 3
module-version imkl/11.3.4.258 11
module-version imkl/2017.1.132 2017
module-version cuda/8.0.44 8.0 8
module-version cuda/9.0.176 9.0 9

# module aliases
module-alias allinea-cpu ddt-cpu
module-alias allinea-gpu ddt-gpu
module-alias arm-forge-cpu ddt-cpu
module-alias arm-forge-gpu ddt-gpu

# hidden versions
hide-version parallel/20170622
hide-version parallel/20160722
hide-version boost/1.63.0
hide-version flex/2.5.39
hide-version python/3.5.2
hide-version python/2.7.13
hide-version python27-scipy-stack/2017a
hide-version python35-scipy-stack/2017a
hide-version python27-mpi4py/2.0.0
hide-version python35-mpi4py/2.0.0
hide-version caffe2/0.8.1
hide-version librt-compat/centos7
hide-version singularity/2.3
hide-version singularity/2.4
# default versions for hidden deprecated modules,
# where there is only one version per directory
module-version python27-scipy-stack/2017a default
module-version python35-scipy-stack/2017a default
module-version python27-mpi4py/2.0.0 default
module-version python35-mpi4py/2.0.0 default
module-version caffe2/0.8.1 default

[boegel]

@mboisson My idea here would be to make EasyBuild control Java/.modulerc via Java-8.eb which uses the (to be implemented) ModuleRC generic easyblock.