Add version number to library names

[javagl]

The main library is always called lwjgl.jar, and the native library is always called lwjgl.dll (on Win64, as an example).

The name should be extended with the version number of the library. For example, it should be called lwjgl-3.1.1.jar and lwjgl-3.1.1.dll to make it possible to disambiguate between different versions, and make sure that the JAR and the native library match.

[sriharshachilakapati]

For this purpose, I recommend to use either maven or gradle, where you can say the version like this:

compile "org.lwjgl:lwjgl-core:${lwjglVersion}"

That would be the better solution than changing the naming of the files.

[Spasi]

You can find the version in the jar manifest (META-INF/MANIFEST.MF).

The native jar files have no manifest, but there's a .git file (e.g. lwjgl.dll.git), which contains the revision used to build the corresponding binary. Both class jar and native jar files contain .sha1 files that can be used to verify their build matches. This is actually done automatically every time LWJGL loads a native library and a warning will be printed when a mismatch is detected.

The version number is also part of the name of the zip file that bundles everything in downloaded releases (from the site build configurator or from github releases). Bundles also contain a build.txt file with the exact build number.

I don't think it's worth doing anything more complex than the above. Adding the version number to jar and native files is only going to make the life of users harder (for those that do not use Maven/Gradle/etc), without adding much value.

[javagl]

I think that appending a version number to the library name would make sense, and this is a maven convention. For the JARs, it is done automatically in Maven Central. For the download package, it may not be so important. But the fact that the natives cannot be distinguished via their name is a bit irritating.

---

Some context, or the things that I'm currently struggling with:

As far as I see, a one-click-runnable JAR involving LWJGL cannot be created with Maven. There are some Wiki pages mentioning plugins that can unpack natives to a target/natives folder or so. But creating a single, runnable JAR that can directly be deployed to the "end user" is not possible. One always has to juggle with the native libraries and the java.library.path. Is this right? The workarounds for this are either "manual" ones, like http://stackoverflow.com/q/12036607 , or the "JarSplicer" that was mentioned on some LWJGL Wiki site. The JarSplicer works fine for me, but will only allow one file with a certain name. Name collisions and version conflicts are becoming more likely when the number of native libraries increases...

---

(Admittedly, my actual use-case is not a realistic one: I wanted to offer an application (as a one-click-runnable JAR) with LWJGL2 and LWJGL3 as a backends. When the DLL in both cases is named lwjgl.dll, this turns out to be difficult. But the fact that the package names are the same in both versions makes it impossible right from the beginning...)

[httpdigest]

But creating a single, runnable JAR that can directly be deployed to the "end user" is not possible

It is possible. See for example the lwjgl3-demos pom.xml.

[javagl]

Although this may now turn out to be off-topic regarding the original issue:

• At which point (in time, and in the code) are the natives unpacked from the JAR?
• Did this also work with LWJGL2? (I think LWJGL2 did not yet have the option to place the natives into the JAR, right?)

[Spasi]

But creating a single, runnable JAR that can directly be deployed to the "end user" is not possible

It is possible. See for example the lwjgl3-demos pom.xml.

There's also LWJGL/lwjgl3-www#16.

There are some Wiki pages mentioning plugins that can unpack natives to a target/natives folder or so. But creating a single, runnable JAR that can directly be deployed to the "end user" is not possible. One always has to juggle with the native libraries and the java.library.path. Is this right?

No. Using -Djava.library.path and/or -Dorg.lwjgl.librarypath is supported in LWJGL 3 (like it was in LWJGL 2), but not required. LWJGL 3 automatically discovers natives in the classpath, extracts and loads them.

I wanted to offer an application (as a one-click-runnable JAR) with LWJGL2 and LWJGL3 as a backends.

I can feel your pain, but LWJGL 2 interop was never a goal for LWJGL 3. Also, one-click-runnable JARs... is that even a thing? Afaik the most common deployment strategy these days is platform-specific installers/executables with application-private JREs.

With that said, if that's what you really want to do, there is an easy workaround. Add the version number to LWJGL 3 natives and use -Dorg.lwjgl.libname=<libname> or Configuration.LIBRARY_NAME.set("<libname>") to override the default.

[javagl]

LWJGL 3 automatically discovers natives in the classpath, extracts and loads them.

Until now, I had only used LWJGL2, where this was not possible AFAIK. I wasn't aware of the fact that LWJGL3 finally supported this.

(I went through the same issues, and will probably have a look at the current solution in LWJGL. My https://github.com/jcuda/jcuda/blob/master/JCudaJava/src/main/java/jcuda/LibUtils.java may need a cleanup+refactoring, and I considered using https://github.com/scijava/native-lib-loader instead, but maybe the LWJGL approach is superior to both. I wonder how you handle the temp files: There's always this issue that DLLs on windows cannot be deleted as long as they are loaded in the JVM, and they can only be "unloaded" with ugly reflection hacks. I'm curious how you solved this).

I can feel your pain, but LWJGL 2 interop was never a goal for LWJGL 3.

Sure, that this is a very unusual intention. Do not take this too serious - it was rather an experiment. As I said, the fact that the package names are the same makes it nearly impossible to combine both versions in one application anyhow. (One could have called the package org.lwjgl3 just for the "semantic versioning" sake, but considering that there usually will be no reason to combine the versions, this may not be so important).

In general, I still think that version numbers in the native library names could add some clarity and help to disambiguate, but of course: When the fiddling of unpacking natives and messing with the java.library.path is no longer necessary in LWJGL3, then the natives will usually never be visible to the user, so this is far less critical.

EDIT:

BTW, the main reason of why I did not switch to LWJGL3 earlier was the lack of the AWTGLCanvas, but https://github.com/httpdigest/lwjgl3-awt basically seems to work, at least during my first tests.

[httpdigest]

... but https://github.com/httpdigest/lwjgl3-awt basically seems to work

If you plan to release your app for Windows only, then yes. :)

[javagl]

Um... there is https://github.com/httpdigest/lwjgl3-awt/blob/master/src/org/lwjgl/opengl/awt/PlatformLinuxGLCanvas.java , but your statement suggests that it does not work in some way... (I couldn't try it out). I'll probably still stick to LWGL2 for a while. The attempt to mix both versions also was intended to allow a "smooth" transition between versions, but I think this is not yet really an issue.

[Spasi]

At which point (in time, and in the code) are the natives unpacked from the JAR?

They're unpacked and loaded lazily, whenever Library.loadSystem or Library.loadNative are called. See the calls to SharedLibraryLoader.load in Library.java.

I wonder how you handle the temp files: There's always this issue that DLLs on windows cannot be deleted as long as they are loaded in the JVM, and they can only be "unloaded" with ugly reflection hacks. I'm curious how you solved this).

The extracted libraries are never deleted by LWJGL. By default they're extracted to java.io.tmpdir (and some other fallbacks if that fails). The default location can be overridden with -Dorg.lwjgl.system.SharedLibraryExtractPath or Configuration.SHARED_LIBRARY_EXTRACT_PATH.

[javagl]

Thanks, I already started browsing into the relevant classes here.

The fact that the native libraries are never deleted makes me wonder, now, again related to the original issue here: What happens when you start a LWJGL 3.1.1 application and keep it running, and then addtionally start a LWJGL 3.1.2 application - won't that mess up the temp files? (Yes, I know, once more an unlikely scenario, but these are the things that don't let me sleep at night ;-) - maybe you solved this somehow, I just started examining the code....)

(I remember that I once naively wrote the libraries into a file created with File#createTempFile. They could not be automatically deleted on Windows, and then people reported that their temp-directory was messed up with hundreds of copies of the native library :-/ Lesson learned....)

[Spasi]

The relevant code is here. The final path includes the user name and LWJGL version.

[Spasi]

then people reported that their temp-directory was messed up with hundreds of copies of the native library :-/ Lesson learned....

In case it's not clear: The SharedLibraryLoader is not supposed to be a solution you use in production. It's there so that it's super simple for developers to setup and use LWJGL (just add a bunch of jars to the classpath).

I repeat that the best way to deploy an LWJGL-based application is with platform-specific installers. This usually involves pre-extracting all natives to a sub-directory of the application and using good-old java.library.path.

[javagl]

I don't have experience with installers, but imagine that it may be a considerable overhead regarding the implementation effort. Of course, this is not about sophisticated Rich Client Applications, but rather about people who want to publish some GL rendering demo, or maybe a small game. The'd likely prefer to have a single JAR that others can try out easily - since it also refers to the users: It may be a matter of attitude (or convenience, aka laziness), but I think that many people prefer the "über-JAR" solution: Download. Doubleclick. Runs.
("Write once, run anywhere" - I have not given it up yet ;-)).

And if this can be solved with a (versioned) temp-file, then why should an installer be the "best" way? I agree that writing a resource to a temp-file feels like an odd workaround, but I think that it is reasonable.

[Spasi]

Of course, this is not about sophisticated Rich Client Applications, but rather about people who want to publish some GL rendering demo, or maybe a small game.

See capsule.io for an alternative solution.

Download. Doubleclick. Runs.

The problem is that this requires a public JRE installed globally in the system. You'll find that many users do not have that, or they're on an obsolete version. With Java's security reputation on the client and applets, many users avoid Java like the plague. I personally don't remember the last time I installed the JRE on my systems.

then why should an installer be the "best" way? I agree that writing a resource to a temp-file feels like an odd workaround, but I think that it is reasonable.

Because, as you've found out, writing to temp or user folders won't be appreciated by users. Certainly many applications do it (especially on Windows), but that doesn't mean it's a good practice.

[httpdigest]

As far as the question of simple deployment (with an assumed JRE being installed on the end-users system) is concerned: You already have a solution for this, right? If you work with Maven you can just use the Shade plugin. Gradle has this also (Shadow plugin). And every IDE provides a way to export a run configuration as a standalone jar, too.

[javagl]

Again, my disclaimer: I'm not really familiar with the (many!) options of deployment, and will certainly have to read mode about it. Also, I don't know about possible preferences by end-users in practice - where "end-users" may refer to geeks and other programmers who want to have a look at what someone else has built, and real end users who might ask: "What is a JVM?"

The latter touches this point:

The problem is that this requires a public JRE installed globally in the system. You'll find that many users do not have that, or they're on an obsolete version. With Java's security reputation on the client and applets, many users avoid Java like the plague. I personally don't remember the last time I installed the JRE on my systems.

Admittedly, an installed JVM is what I usually assume. When deploying a Java application, this is as a requirement, period. And deploying a 30MB JVM just to run your 50KB JAR with some OpenGL demo is simply not reasonable. But of course, I see that there may be the "real end-users" (as described above) who don't want a JAR, but essentially "something like a .EXE". And for this, there are different solutions on various levels. I didn't know capsule.io, but for this purpose, it may be worth a look.

Because, as you've found out, writing to temp or user folders won't be appreciated by users.

I also don't really like this solution. But without an installer, it's the only option. And it is by far the most convenient option. We all went through the trauma of the UnsatisfiedLinkError for much too long. And at least for JCuda and JOCL, since I started packing the natives into the JAR, these complaints have essentially disappeared.

The best thing would be if Java started allowing to load natives from memory, as in System.load(byte libraryData[]) or even from the JAR, as in System.load(URI resourcePath), but I assume that there are reasons of why this is not possible (security issues, probably). Even though such a solution does not exist yet, I'm pretty sure that something like this (only better) will be available in the near future.

As far as the question of simple deployment (with an assumed JRE being installed on the end-users system) is concerned: You already have a solution for this, right? If you work with Maven you can just use the Shade plugin. Gradle has this also (Shadow plugin).

When I looked at the shade plugin, I thought it was mainly intended to resolve ambiguities (i.e. overlaps) in the package names. At least, the über-JAR-creation is something that I've done with the assembly plugin. (Followed by applying JarSplice to the result, because ... native libraries, yes)

And every IDE provides a way to export a run configuration as a standalone jar, too.

From my experience, this also causes some glitches when natives are involved (but again, my experience is very limited here). And for easy deployment, and automated, command-line based way to do this may be preferable.

[httpdigest]

The best thing would be if Java started allowing to load natives from memory, as in System.load(byte libraryData[]) or even from the JAR, as in System.load(URI resourcePath), but I assume that there are reasons of why this is not possible (security issues, probably).

Yes, that would be nice. But fyi, the reason why this is not supported is that the underlying platform (Windows, Linux, ...) does not support loading and linking shared libraries from memory. It always needs to be a file on the file system, accessed via file system operations. At least this is the case for Windows and Linux.
While it always is possible to load executable code at runtime from memory, you cannot load and most importantly link a shared library (PE or ELF format) from memory.
If Java were to support it, the JVM would also need to go the way through a temporary file much like the SharedLibraryLoader.

Even though such a solution does not exist yet, I'm pretty sure that something like this (only better) will be available in the near future.

Since it is not a problem with Java, but with the OS, don't hold your breath for it.

[sormuras]

I wonder, if someday, the linker will support native libraries.

[javagl]

OK, when these are OS constraints, then things may become more difficult. There are so many legacy issues that have to be coped with (or: "What you never wanted to know about -fPIC, but had to dig into nevertheless, just to get your JNI library running").

I've even heard rumours that newer MacOS versions introduce some constraints: It was said that they do not allow a native library B to be loaded as a dependency of library A when A is contained in the temp-folder. (This has not been confirmed for me until now, but sounds concerning - did anybody hear something similar, or can dispel my concerns?)

I wonder, if someday, the linker will support native libraries.

My hopes that Java will receive an extended functionality here was related to this project, and the general efforts for a better support and integration of native libraries in other projects.

If Java were to support it, the JVM would also need to go the way through a temporary file

This would at least be a first step. Moving the responsibility for this "upwards" into the JVM would tremendously simplify things for all JNI developers and it would feel more reliable than manual workarounds (like the SharedLibraryLoader, LibUtils or https://github.com/scijava/native-lib-loader ). All the OS-specific information about the architecture and OS name and bitness that the JNI developer has to assemble manually and tediously are inherent for the JVM.

And, most importantly: It would make the process transparent. If there was a method like System.load(URI jarUri), then it could be implemented based on a temp-file, and later be changed to load from the JAR/memory directly (maybe after they implemented the required functionality in the linker, or somehow coped with the OS constraints), without affecting the JNI developers' code.

[Spasi]

JNI is dying. If you're looking for new JVM features in this area, keep an eye on Project Panama. Among other things, it includes "native library management APIs".

[javagl]

That's one of the projects that I referred to above, but admittedly, I'm not up to date about the current state and planning of the "native library management APIs" that they refer to.

Of course, right now there is a plethora of options for interfacing Java and native code: https://github.com/java-native-access , https://github.com/jnr , https://github.com/nativelibs4java/BridJ , https://github.com/bytedeco/javacpp (the latter summarizing more than a dozen more ""failed attempts""). Many of them are basically JNI code generators. Some of them go one level deeper and try to create a "generic" function call interface.

But I think that these will always be kinds of workarounds, as long as the support for interfacing native libraries is not intrinsically supported by the JVM. This goal is ambituous, of course, and may even be considered as a step away from the idea of "Write once, run anywhere". But experience shows that there is a need for this.

In any case, I'm curious how things will turn out here.

[JustGregory-zz]

... anyhow. Back to the issue mentioned in the issue title. Apologies in advance, but I'm opinionated on this subject.

I'm not, repeat NOT, in favor of enforcing adding version numbers to any jar, EXCEPT for a "main jar" in a multi-jar program installation. Class-libraries which may be replaced by an updated jar file in situ by an end-user, should not be versioned by name. Doing so allows the possibility of different versions of the same class-library to be found in the library path. I've experienced this before, and I would assume I'm not the only one who has.

I would instead recommend that each class-library have some utility class or metadata file which provides "updater info" that the main jar will use to seek out updated libraries. I have this feeling that developers should be getting into the habit of coding in a way that allows a multi-jar program to effectively handle dependency management. My favorite IDE does this, many non-Java applications do this too, even first-tier Java applications and games should do this.

Again, apologies for being so vocal on the subject, ... but I just had to say something.

[Spasi]

I would instead recommend that each class-library have some utility class or metadata file which provides "updater info" that the main jar will use to seek out updated libraries.

I'm not sure what you're asking for exactly. There's no "main jar" in LWJGL and, as I said in my first reply, there's version information in each jar, in META-INF/MANIFEST.MF.

[httpdigest]

There are other problems to be solved for this to work, of course:

• defining what "compatibility" actually means: mere binary/signature compatibility or semantic compatibility (methods still behave in the same way), both of which can be defined as an API break
• either actually following a semantic versioning scheme (by which today LWJGL would probably called LWJGL 74)
• or: maintaining and deploying a compatibility matrix (declaring whether it is actually okay to just "swap out" one library for another)

[javagl]

(Just a reminder: The issue is closed, so strictly speaking, it may not be "necessary" to "convince" anybody - but of course, these points may be relevant for the discussion nevertheless)

@JustGregory I'm not entirely sure what you referred to with the "in situ update by an end-user". If this refers to sophisticated things like RCP with underlying OSGi or some Spring magic where JARs can be replaced at runtime, then I can't say anything here. But can imagine that having two different versions of the same JAR in the classpath can cause arbitrary problems. (I think that these infrastructures are aware of this, and try to cope with this, using some ClassLoader magic, but I'm not familiar with that - or how well it works).

But to emphasize this again:

This issue originally mainly referred to the native libraries. And it was mainly based on the fact that I did not know that LWJGL3 now also allows keeping the natives in the JAR. For LWJGL2, it was still necessary to unpack them so "some" directory, and set the java.library.path accordingly, and there, the problem of two different lwjgl.dll files overwriting each other is obvious.

As for the JAR libraries: They are versioned automatically by Maven. So there's not much freedom here anyhow. (This freedom only exists for the downloadable release packages, and there, one can say that it's up to the user to rename the library file or not).

[JustGregory-zz]

Ah, okay. I somehow missed the "native" part of the issue. Mea culpa.

@Spasi
I wasn't asking for anything to be done on the part of the lwjgl libraries (or natives, it seems). My misunderstanding came from overlooking the discussion of "native" libraries and latching onto the "versioning by library name" aspect.

@javagl
Yes, I did mean mostly about an RCP updating things at runtime, although a post-update is often mandatory (and recommended).

In my direct case, in a somewhat ironic example at that, is of NetBeans IDE with the 3rd-party NbGradle plugin (which is itself, at least in part, a "dependency managament" infrastructure over maven), I have found multiple copies of gradle support libraries (not the gradle wrapper, something else) in the NetBeans IDE path, same base archive name but incremental version strings before the ".jar" extension. It is apparent that the NbGradle plugin must have been retaining dependencies instead of releasing/deleting them from the path on "upgrade".

[javagl]

(It's not really "your fault". I was not so clear here: The issue originally did refer to the JARs and natives. But the JARs part is not so important and handled by Maven. The natives part was what most of the actual discussion eventually referred to).

The RCP scenario that your described sounds like a source of headaches even if it only refers to JARs. But I think that this also becomes really ugly when natives are involved. As mentioned above: There is hardly a way to "unload" a native library at all. Hopefully, the upcoming projects (Jigsaw/Panama/...) will offer some nice and clean solutions here, to justify the "D" in "DLL"....