Bug Report: Payara Server doesn't apply jvm parameters from pre-boot commands during first start

[kuhtini]

Brief Summary

For some reason Payara Server doesn't apply jvm parameters from pre-boot commands during first start. When container is restarted, JVM options from pre-boot are added.

pre-boot-commands.asadmin:

## Add custom JVM options
create-jvm-options --target=server-config -- "--add-exports=java.base/sun.security.pkcs=ALL-UNNAMED"
create-jvm-options --target=server-config -- "--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED"

Command used to start Payara:

 docker run -p 8080:8080 -v ./config:/config -e PREBOOT_COMMANDS=/config/pre-boot-commands.asadmin payara/server-full:6.2024.11-jdk17

First run JVM args:

Executing Payara Server with the following command line:
/usr/lib/jvm/zulu17/bin/java
-cp
/opt/payara/appserver/glassfish/domains/domain1/lib/ext/*:/opt/payara/appserver/glassfish/modules/glassfish.jar
-XX:+UnlockDiagnosticVMOptions
--add-exports=java.base/sun.net.www=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/sun.net.www.protocol.jrt=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.management/javax.management=ALL-UNNAMED
--add-opens=java.desktop/java.beans=ALL-UNNAMED
--add-opens=java.management/sun.management=ALL-UNNAMED
--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens=java.naming/javax.naming.spi=ALL-UNNAMED
--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.logging/java.util.logging=ALL-UNNAMED
--add-exports=jdk.naming.dns/com.sun.jndi.dns=ALL-UNNAMED
--add-opens=java.base/sun.net.www.protocol.jar=ALL-UNNAMED
--add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED
--add-opens=java.base/jdk.internal.loader=ALL-UNNAMED
--add-exports=java.base/sun.security.util=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.management/javax.management.openmbean=ALL-UNNAMED
-XX:NewRatio=2
-XX:+UseContainerSupport
-XX:MaxRAMPercentage=70.0
-Xbootclasspath/a:/opt/payara/appserver/glassfish/lib/grizzly-npn-api.jar
-Xss512k
-server
-javaagent:/opt/payara/appserver/glassfish/lib/monitor/flashlight-agent.jar
-Djavax.xml.accessExternalSchema=all
-Djavax.net.ssl.trustStore=/opt/payara/appserver/glassfish/domains/domain1/config/cacerts.p12
-Djdk.tls.rejectClientInitiatedRenegotiation=true
-Djdk.util.zip.disableZip64ExtraFieldValidation=true
-Djdk.corba.allowOutputStreamSubclass=true
-Dfelix.fileinstall.dir=/opt/payara/appserver/glassfish/modules/autostart/
-Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command,org.apache.felix.shell.remote,org.apache.felix.fileinstall
-Dcom.sun.aas.installRoot=/opt/payara/appserver/glassfish
-Dfelix.fileinstall.poll=5000
-Djava.security.policy=/opt/payara/appserver/glassfish/domains/domain1/config/server.policy
-Dosgi.shell.telnet.maxconn=1
-Dfelix.fileinstall.bundles.startTransient=true
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dfelix.fileinstall.log.level=2
-Djavax.net.ssl.keyStore=/opt/payara/appserver/glassfish/domains/domain1/config/keystore.p12
-Djava.security.auth.login.config=/opt/payara/appserver/glassfish/domains/domain1/config/login.conf
-Dfelix.fileinstall.disableConfigSave=false
-Dorg.glassfish.grizzly.DEFAULT_MEMORY_MANAGER=org.glassfish.grizzly.memory.HeapMemoryManager
-Dfelix.fileinstall.bundles.new.start=true
-Dcom.sun.aas.instanceRoot=/opt/payara/appserver/glassfish/domains/domain1
-Dosgi.shell.telnet.port=6666
-Dgosh.args=--nointeractive
-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
-Dorg.jboss.weld.serialization.beanIdentifierIndexOptimization=false
-Dosgi.shell.telnet.ip=127.0.0.1
-Djdk.attach.allowAttachSelf=true
-DANTLR_USE_DIRECT_CLASS_LOADING=true
-Djava.awt.headless=true
-Dcom.ctc.wstx.returnNullForDefaultNamespace=true
-Djdbc.drivers=org.h2.Driver
-Dorg.glassfish.grizzly.nio.DefaultSelectorHandler.force-selector-spin-detection=true
-Djava.library.path=/opt/payara/appserver/glassfish/lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
com.sun.enterprise.glassfish.bootstrap.ASMain
-prebootcommandfile
/opt/payara/config/pre-boot-commands-final.asadmin
-upgrade
false
-read-stdin
true
-postbootcommandfile
/opt/payara/config/post-boot-commands-final.asadmin
-domainname
domain1
-domaindir
/opt/payara/appserver/glassfish/domains/domain1
-asadmin-args
--host,,,localhost,,,--port,,,4848,,,--user,,,admin,,,--passwordfile,,,/opt/payara/passwordFile,,,--secure=false,,,--terse=false,,,--extraterse=false,,,--echo=false,,,--interactive=false,,,--autoname=false,,,start-domain,,,--verbose=false,,,--watchdog=false,,,--debug=false,,,--warmup=false,,,--domaindir,,,/opt/payara/appserver/glassfish/domains,,,domain1
-instancename
server
-type
DAS
-verbose
false
-warmup
false
-asadmin-classpath
/opt/payara/appserver/glassfish/lib/client/appserver-cli.jar
-debug
false
-asadmin-classname
com.sun.enterprise.admin.cli.AdminMain
-watchdog
false

JVM args after restart:

Executing Payara Server with the following command line:
/usr/lib/jvm/zulu17/bin/java
-cp
/opt/payara/appserver/glassfish/domains/domain1/lib/ext/*:/opt/payara/appserver/glassfish/modules/glassfish.jar
-XX:+UnlockDiagnosticVMOptions
--add-exports=java.base/sun.net.www=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/sun.net.www.protocol.jrt=ALL-UNNAMED
--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
--add-exports=java.base/sun.security.pkcs=ALL-UNNAMED
--add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED
--add-exports=java.base/sun.security.util=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.management/javax.management=ALL-UNNAMED
--add-opens=java.desktop/java.beans=ALL-UNNAMED
--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
--add-opens=java.management/sun.management=ALL-UNNAMED
--add-opens=java.naming/javax.naming.spi=ALL-UNNAMED
--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.logging/java.util.logging=ALL-UNNAMED
--add-exports=jdk.naming.dns/com.sun.jndi.dns=ALL-UNNAMED
--add-opens=java.base/sun.net.www.protocol.jar=ALL-UNNAMED
--add-opens=java.base/jdk.internal.loader=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.management/javax.management.openmbean=ALL-UNNAMED
-XX:NewRatio=2
-XX:+UseContainerSupport
-XX:MaxRAMPercentage=70.0
-Xbootclasspath/a:/opt/payara/appserver/glassfish/lib/grizzly-npn-api.jar
-Xss512k
-server
-javaagent:/opt/payara/appserver/glassfish/lib/monitor/flashlight-agent.jar
-Djavax.xml.accessExternalSchema=all
-Djavax.net.ssl.trustStore=/opt/payara/appserver/glassfish/domains/domain1/config/cacerts.p12
-Djdk.tls.rejectClientInitiatedRenegotiation=true
-Djdk.util.zip.disableZip64ExtraFieldValidation=true
-Djdk.corba.allowOutputStreamSubclass=true
-Dfelix.fileinstall.dir=/opt/payara/appserver/glassfish/modules/autostart/
-Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command,org.apache.felix.shell.remote,org.apache.felix.fileinstall
-Dcom.sun.aas.installRoot=/opt/payara/appserver/glassfish
-Dfelix.fileinstall.poll=5000
-Djava.security.policy=/opt/payara/appserver/glassfish/domains/domain1/config/server.policy
-Dosgi.shell.telnet.maxconn=1
-Dfelix.fileinstall.bundles.startTransient=true
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dfelix.fileinstall.log.level=2
-Djavax.net.ssl.keyStore=/opt/payara/appserver/glassfish/domains/domain1/config/keystore.p12
-Djava.security.auth.login.config=/opt/payara/appserver/glassfish/domains/domain1/config/login.conf
-Dfelix.fileinstall.disableConfigSave=false
-Dorg.glassfish.grizzly.DEFAULT_MEMORY_MANAGER=org.glassfish.grizzly.memory.HeapMemoryManager
-Dfelix.fileinstall.bundles.new.start=true
-Dcom.sun.aas.instanceRoot=/opt/payara/appserver/glassfish/domains/domain1
-Dosgi.shell.telnet.port=6666
-Dgosh.args=--nointeractive
-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
-Dorg.jboss.weld.serialization.beanIdentifierIndexOptimization=false
-Dosgi.shell.telnet.ip=127.0.0.1
-Djdk.attach.allowAttachSelf=true
-DANTLR_USE_DIRECT_CLASS_LOADING=true
-Djava.awt.headless=true
-Dcom.ctc.wstx.returnNullForDefaultNamespace=true
-Djdbc.drivers=org.h2.Driver
-Dorg.glassfish.grizzly.nio.DefaultSelectorHandler.force-selector-spin-detection=true
-Djava.library.path=/opt/payara/appserver/glassfish/lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
com.sun.enterprise.glassfish.bootstrap.ASMain
-prebootcommandfile
/opt/payara/config/pre-boot-commands-final.asadmin
-upgrade
false
-read-stdin
true
-postbootcommandfile
/opt/payara/config/post-boot-commands-final.asadmin
-domainname
domain1
-domaindir
/opt/payara/appserver/glassfish/domains/domain1
-asadmin-args
--host,,,localhost,,,--port,,,4848,,,--user,,,admin,,,--passwordfile,,,/opt/payara/passwordFile,,,--secure=false,,,--terse=false,,,--extraterse=false,,,--echo=false,,,--interactive=false,,,--autoname=false,,,start-domain,,,--verbose=false,,,--watchdog=false,,,--debug=false,,,--warmup=false,,,--domaindir,,,/opt/payara/appserver/glassfish/domains,,,domain1
-instancename
server
-type
DAS
-verbose
false
-warmup
false
-asadmin-classpath
/opt/payara/appserver/glassfish/lib/client/appserver-cli.jar
-debug
false
-asadmin-classname
com.sun.enterprise.admin.cli.AdminMain
-watchdog
false

Parameters were added only after the restart

--add-exports=java.base/sun.security.pkcs=ALL-UNNAMED"
--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED"

Expected Outcome

JVM options from pre-boot-commands.asadmin are added during first start.

Current Outcome

JVM options from pre-boot-commands.asadmin are added only after container restart.

Reproducer

1. Create pre-boot-commands.asadmin file with create-jvm-options
2. Start the server with mounted pre-boot-commands.asadmin

• e.g. docker run -p 8080:8080 -v ./config:/config -e PREBOOT_COMMANDS=/config/pre-boot-commands.asadmin payara/server-full:6.2024.11-jdk17

3. Check the Payara (glassfish.jar) JVM parameters
4. Restart the container
5. Check the Payara (glassfish.jar) JVM parameters

Operating System

Ubuntu 22

JDK Version

17

Payara Distribution

Payara Server Full Profile

[Elifzeynepedman]

Hi @kuhtini,

JVM options specified in pre-boot commands are applied after the initial startup because the JVM reads its options during the boot process, before executing pre-boot commands. Consequently, these options take effect only upon restarting the server. This behavior is consistent with the Payara Server's startup sequence.

This behavior is not a bug but a result of the Payara Server's startup process, where JVM options are read before pre-boot commands are applied. Since this is expected, I will proceed to close this issue.

Best Regards,
Elif

[kuhtini]

Since the script startInForeground.sh tries to start with a dry-run option and copy parameters for "real" start, I would expect that this dry-run prepared need parameters. So the current behavior is strange.

Payara/appserver/extras/docker-images/server-full/src/main/docker/bin/startInForeground.sh

Line 89 in bbae3a6

OUTPUT=`${PAYARA_DIR}/bin/asadmin --user=${ADMIN_USER} --passwordfile=${PASSWORD_FILE} start-domain --dry-run --prebootcommandfile=${PREBOOT_COMMANDS_FINAL} --postbootcommandfile=${POSTBOOT_COMMANDS_FINAL} $@ $DOMAIN_NAME`