[XWayland] libX11 SIGSEV

[mkroening]

Tinkering around with https://github.com/johanvos/javafx11samples I found that ./gradlew run fails when run in the GNOME Wayland session on Arch Linux. I do not experience any problems when running the command in the X11 session or using the OracleJDK 10 with integrated JavaFX.
I tested it with the OpenJDK 10 and the OracleJDK 11 in combination with all openjfx versions from The Central Repository (11-ea+19, 11-ea+23 and 11-ea+24). All fail with the following message:

[...]
> Task :run FAILED

(java:3817): Gdk-CRITICAL **: 23:30:54.099: gdk_x11_display_set_window_scale: assertion 'GDK_IS_X11_DISPLAY (display)' failed
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fdd058db7db, pid=3817, tid=3849
#
# JRE version: OpenJDK Runtime Environment (10.0.2+13) (build 10.0.2+13)
# Java VM: OpenJDK 64-Bit Server VM (10.0.2+13, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libX11.so.6+0x2b7db]
[...]

Here is the full error report: hs_err_pid3817.log

[mkroening]

I am able to reproduce this behavior on Ubuntu 18.04 (openjdk 10.0.2) and Fedora 28 (openjdk 11-ea+22) in virtual machines.
The step to reproduce is just running ./gradlew run in https://github.com/johanvos/javafx11samples/tree/master/topics/javafx3d/hello3d .

[brcolow]

This is probably the expected behavior, as JavaFX only supports X11. In order to add native support for running directly on Wayland to JavaFX (and not use some sort of X11 compatibility which I believe Wayland does offer - but take that with a very large grain of salt because I know next to nothing about Wayland) all of the X11-based native code would need a new Wayland equivalent. Take, for example, modules/javafx.graphics/src/main/native-glass/gtk/glass_screen.cpp. This is the native GDK screen that is created for JavaFX to draw on (and listen for all types of events on) when running on Linux. You can see it includes X11/Xatom.h, and uses the gdk_x11_display_get_xdisplay method to get the X display (the screen). A Wayland equivalent (ideally provided by GDK itself so it would require less work, as JavaFX already uses GDK) would need to be written. There are many such examples in the native-glass library where it assumes it is running on an X11 display/device, and not a Wayland one. These would all have to be accounted for and then one could perform some sort of detection to decide whether to use X11 or Wayland (and/or use a system property as is currently done, at least in part, by jdk.gtk.version.

[mkroening]

Ah, yes. I should have been clearer in my wording.
The X11 compatibility you mean is XWayland.
It allows any X11 client to run under Wayland and so should JavaFX.
The thing is that my JavaFX application runs (although not correctly scaled for HiDPI, but that is another issue) on Wayland with XWayland when using the OracleJDK 10 with integrated JavaFX. My experimentation for future migration to JavaFX 11 implies, that there has been some kind of regression, as I am not able to start JavaFX 11 applications under Wayland.
Native Wayland support would be awesome, but that would be another issue.

[kevinrushforth]

You may have discovered a regression in OpenJFX 11. We don't do any testing on Wayland, so it wouldn't be too surprising if something we fixed in 11 caused this regression. We can take a look at it.

[kevinrushforth]

We tested it locally and can reproduce a crash (at a different place than you reported), and it looks like it's related to running using GTK3. The default backend was GTK2 for FX in JDK 10 and is GTK3 in FX 11. As a workaround, you can force GTK 2 by running:

java -Djdk.gtk.version=2 ...

Also, you can pass -Djdk.gtk.verbose=true to see what is being used. I also note that we can make JDK 10 crash as well on Wayland by forcing GTK 3.

[prrace]

FYI, OpenJDK apps using the Swing stack and loading the GTK L&F for Swing do NOT crash in the same way. However OpenJDK loads GTK much later and only when running the Swing GTK L&F. So there is no direct comparison, it is just a data point. Also although it does not crash, OpenJDK - at least the one from Oracle - explicitly declares wayland mode as unsupported because of test failures in that mode.

[mkroening]

Thanks for your investigation!
Will there be support for XWayland or even native Wayland in the future?
I think, in the meantime I will incorporate the fallback to GTK 2 into my start script.

[brcolow]

My guess is that it will have to be contributed by a volunteer.

[kevinrushforth]

I will open a JBS bug against JavaFX for the GTK3 crash on XWayland (pointing to this GitHub issue) -- given that FX runs with GTK2 and that AWT runs with both GTK2 and GTK3, it shouldn't be too hard to reolve.

As @brcolow suggested, a native Wayland port would need to be contributed by an interested volunteer.

[johanvos]

IIUC, Wayland is default in Ubuntu 18.02, which means there is a large userbase that will run into this. We can probably detect if the system is using wayland and in that case by default use gtk2?

[johanvos]

It seems System.getenv("XDG_SESSION_TYPE") will return "wayland" in case XWayland is used.
A quick fix might be a smal patch to GtkApplication. In the constructor of that class, we check what version to use based on the jdk.gtk.version (which defaults to 3). We might add a check for that XDG_SESSION_TYPE property, and fallback to 2 in case it is "wayland"

[johanvos]

While probably not the best workaround (removed final from a variable), PR #199 seems to fix the issue on Ubuntu 18.04 and still works on Ubuntu 16.04

[kevinrushforth]

Wayland is not the default for 18.04 (it was going to be, but they backed away from that plan).

I would prefer to fix the issue, if possible, rather than masking it. Note also that changing the default will create a mismatch with Swing, so it won't work with SwingInterop.

[johanvos]

I stand corrected. I was worried about the impact if Wayland is the default, but that indeed doesn't seem to be the case. I updated to 18.04 and am still using Xorg.
Hence, I agree it is better to fix it rather than to bypass it -- or to hope XWayland will fix it.

[kevinrushforth]

Tracked in JBS as JDK-8210411

[kevinrushforth]

https://bugs.openjdk.java.net/browse/JDK-8210411 is now fixed.

[walec51]

I have the same problem on OpenJDK 13 on Fedora 30:

java:17103): Gdk-CRITICAL **: 12:07:19.910: gdk_x11_display_set_window_scale: assertion 'GDK_IS_X11_DISPLAY (display)' failed
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fd3ae3cd90f, pid=17103, tid=17137
#
# JRE version: OpenJDK Runtime Environment (13.0+33) (build 13+33)
# Java VM: OpenJDK 64-Bit Server VM (13+33, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libX11.so.6+0x3190f]  XInternAtom+0x3f
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to /mnt/reszta/Projekty/WALCZAK IT/scr/pdf-decorator/build/image/bin/core.17103)
#
# An error report file with more information is saved as:
# /mnt/reszta/Projekty/WALCZAK IT/scr/pdf-decorator/build/image/bin/hs_err_pid17103.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/AdoptOpenJDK/openjdk-build/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

please reopen this bug

[kevinrushforth]

This GitHub issue tracker is not actively tracked or managed. If you believe you have found a reoccurrence of this bug, please double-check the version of JavaFX that you are running to make sure it is a new enough version that it contains the fix for JDK-8210411. If so, please file a new bug at bugreport.java.com with a reproducible test case, the crash log, and the details of your system.