Bug Report: [5.2021.9] @Stateless @Remote Bean causes java.lang.NoSuchMethodError during deployment /FISH-5933

[ctabin]

Description

---

Just by upgrading from version 5.2021.8 to 5.2021.9 of payara-embedded-all, the deployment of our project crashes with the following stacktrace:

SEVERE: Exception while loading the app : EJB Container initialization error
java.lang.NoSuchMethodError: 'java.lang.Class sun.misc.Unsafe.defineClass(java.lang.String, byte[], int, int, java.lang.ClassLoader, java.security.ProtectionDomain)'
        at org.glassfish.pfl.basic.reflection.BridgeBase.defineClass(BridgeBase.java:241)
        at org.glassfish.pfl.dynamic.codegen.impl.CodeGeneratorUtil.makeClass(CodeGeneratorUtil.java:46)
        at org.glassfish.pfl.dynamic.codegen.spi.Wrapper._generate(Wrapper.java:1026)
        at org.glassfish.pfl.dynamic.codegen.spi.Wrapper._generate(Wrapper.java:1010)
        at com.sun.ejb.EJBUtils.generateAndLoad(EJBUtils.java:608)
        at com.sun.ejb.EJBUtils.loadGeneratedGenericEJBHomeClass(EJBUtils.java:546)
        at com.sun.ejb.containers.BaseContainer.<init>(BaseContainer.java:677)
        at com.sun.ejb.containers.StatelessSessionContainer.<init>(StatelessSessionContainer.java:141)
        at com.sun.ejb.containers.StatelessSessionContainer.<init>(StatelessSessionContainer.java:136)
        at com.sun.ejb.containers.StatelessContainerFactory.createContainer(StatelessContainerFactory.java:61)
        at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:225)
        at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:286)
        at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:104)
        at org.glassfish.internal.data.ModuleInfo.load(ModuleInfo.java:218)
        at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:334)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.prepare(ApplicationLifecycle.java:570)
        at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:588)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:361)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:361)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1300(CommandRunnerImpl.java:120)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755)
        at com.sun.enterprise.admin.cli.embeddable.DeployerImpl.deploy(DeployerImpl.java:131)

Here is the full output of the server. Nothing else changed and all is working fine under the exact same conditions in version 5.2021.8.

EDIT: We were able to track down the source of the failure: actually it fails because of the following Bean:

@Stateless
@LocalBean
@Remote(EntryPointBeanRemote.class)
@RolesAllowed({"ADMIN", "SUPERADMIN", "USER"})
@TransactionManagement(TransactionManagementType.BEAN)
public class EntryPointBean implements EntryPointBeanRemote {
  //...
}

If we remove either the @Remote OR the @Stateless annotation, then it works fine (of course, that causes our application to crash further but that's expected because those annotations are needed).

Hence it seems that the version 5.2021.9 doesn't support stateless beans that have to be exposed remotely. Note that if @LocalBean is commented out, still the same crash occurs.

May be related to #5482 ?

Environment

• Distribution: embedded-all
• JDK Version: OpenJDK 11
• Operating System: Linux Debian

[ctabin]

I was able to reproduce a minimal test case that you can find here:

• The deployment works with 5.2021.8 but fails with 5.2021.9.
• Either if @Stateless or @Remote is removed from LocalStatelessServiceBean, then it works with 5.2021.9.

[shub8968]

Hi @ctabin,

I am experiencing BUILD FAILURE upon building the app. Also, can you provide a simple-to-follow scenario on how to reproduce this on the latest version? A reproducer should ideally follow the SSCCE rules: http://www.sscce.org/. It will greatly help us to find the cause and fix it. If you are looking for application support then please contact Payara Enterprise.

[ctabin]

Hi @shub8968,

Thanks for your reply. Actually it fails on javadoc because it has to know your JAVA_HOME. You can find it for instance with the command ls -lh /etc/alternative/java.

Here is the command I use to compile and run the test:

JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64" mvn clean package

By default, the test fails because the version of Payara is 5.2021.9. You can see the Exception relative to the ClassLoader in the console. If you change the version to 5.2021.8 in the main pom.xml and launch the same command, then the test succeeds.

[ctabin]

@shub8968 Do you need any more information ? Maybe it's linked to #5510 🤔 However we are using JDK11 instead.

[shub8968]

Hi @ctabin,

Please allow me some time. I'll update you soon. However, If you think you need to get it fixed to the earliest or need immediate action, I recommend you to consider Payara Enterprise.

[ctabin]

Hi @shub8968,

Thanks for coming back. I just wanted to be sure that you could have a reproducer with the project since there wasn't any feedback.
We are looking to migrate to JDK17 and this is a blocker for us, but there is no need for urgent action for now.

Thanks for the work and I'm looking for your update 👍

[fturizo]

Greetings, @ctabin. After a detailed analysis of your reproducer, we have found that indeed the Payara Embedded API's deployment mechanisms trigger the error that you reported. We have escalated this issue to our Platform Development team in the form of the internal JIRA issue FISH-5933.

Since only the Payara Embedded distribution under JDK 11 is affected at the moment, this issue has been tagged with a low priority, so it will take some time until a proper fix is implemented.

[ctabin]

Hi @fturizo and thanks for your reply.
I can also confirm that the same problem arise with JDK 17 unfortunately.
If the information can help, I tried to migrate back to GlassFish 6.3.2, but unfortunately I hit the same problem.

Is there a possible workaround that we could apply (other than removing the @Remote annotation) until the issue is properly fixed ?

[smillidge]

I've raised a related upstream issue to replace sun.misc.Unsafe usage in the PFL library see eclipse-ee4j/orb-gmbal-pfl#43

[fturizo]

@ctabin, as Steve clarified, the issue's cause is located upstream in the Gmbal PFL library, which needs to be patched out first, so at the moment there are no other workarounds. If this issue is critical on your end, I encourage you to get a Payara Enterprise subscription for your organisation so you can get a solution with celerity.

[ctabin]

Hi, thanks @smillidge to address the issue at the gmbal-pfl level.
I was considering opening the same issue in eclipse-ee4j/glassfish but since the gmbal-pfl library is also used I guess it is't relevant ?
@fturizo What it strange is that all was perfectly working up to payara-embedded-all 5.2021.8 (maybe prior #5482) 🤔 Just FYI, it still crash in 5.2021.10 with the same error. 😿
It is not a critical issue on our end, but this is clearly a blocker because it prevents us to update to the latest payara versions and thus also prevents us to move to JDK17.
At this point, the only way for us would be to remove the @Remote annotation and migrate our code to the same EAR (but then we'll face a whole bunch of classloading issues). It would be really great if we could avoid this and see a fix in eclipse-ee4j/orb-gmbal-pfl#43 in a foreseeable future 😺
Since the @Remote and @Stateless are pretty common annotations (are they ?), I guess this kind of problem may impact significantly other users. 🙏 Let me know if I can provide any further help.

[ctabin]

@fturizo FYI we were able to find a workaround, see this answer. And don't miss the system property org.glassfish.gmbal.no.multipleUpperBoundsException=true in order to be able to deploy @Remote @Stateless EJB.

[Blavo]

G'day,

I also had and independently solved this one with the same basic solution. The reason I am commenting is that I don't use Maven so I had to include pfl-basic-4.1.2 by hand in the library JAR list of my IDE project. The point is that pfl-basic-4.1.2 has to come before the embedded payara JAR or it has no effect.