Jetty start.jar fails with NullPointerException when referencing a non existent module and using JVM args

[zaynetro]

Jetty version(s) 10.0.11

Java version/vendor openjdk 11.0.15+10

Description

Running webapp with Jetty distribution fails with a null pointer exception:

> java -jar /home/chat/jetty/start.jar
java.lang.NullPointerException
	at org.eclipse.jetty.start.Main.lambda$start$1(Main.java:476)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:176)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at org.eclipse.jetty.start.Main.start(Main.java:478)
	at org.eclipse.jetty.start.Main.main(Main.java:75)

Usage: java -jar $JETTY_HOME/start.jar [options] [properties] [configs]
       java -jar $JETTY_HOME/start.jar --help  # for more information

We are upgrading from Jetty 9.4.45.v20220203 and reusing the same .ini files. It seems that continuation module has been removed.

Reference: https://github.com/eclipse/jetty.project/blob/jetty-10.0.x/jetty-start/src/main/java/org/eclipse/jetty/start/Main.java#L474:L476

My suspicion is that line 474 returns null because module is not found. It would be nice to throw an Exception when module is not found to help figuring out the problem.

How to reproduce?

1. Add continuation module to start.ini:

# ---------------------------------------
# Module: continuation
--module=continuation

2. Run Jetty distribution java -jar /jetty/start.jar

[joakime]

Correct, the module continuation does not exist in Jetty 10+, it was removed in commit 79cafb3 back in Jul 2017.

The standard Servlet AsyncContext is the replacement.
The jetty-continuations featureset was deprecated back in March 2013 when Jetty 9.0.0 was released, as the standard Servlet AsyncContext was introduced in Jetty 7.

This still shouldn't result in a NPE, that's quite surprising. (will attempt to replicate)

[joakime]

Cannot replicate

[old-continuations-base]$ tree
.
├── start.d
│   ├── continuations.ini
│   ├── deploy.ini
│   └── http.ini
└── webapps

2 directories, 3 files
[old-continuations-base]$ cat start.d/continuations.ini 
--module=continuation
[old-continuations-base]$ cat start.d/deploy.ini 
--module=deploy
[old-continuations-base]$ cat start.d/http.ini 
--module=http
jetty.http.port=8080

then attempt to run ...

[old-continuations-base]$ java -jar /home/joakim/code/jetty/distros/jetty-home-10.0.11/start.jar
2022-06-29 07:18:43.251:INFO :oejs.Server:main: jetty-10.0.11; built: 2022-06-21T21:12:44.640Z; git: d988aa016e0bb2de6fba84c1659049c72eae3e32; jvm 11.0.15+10
2022-06-29 07:18:43.288:INFO :oejdp.ScanningAppProvider:main: Deployment monitor [file:///home/joakim/code/jetty/distros/bases/old-continuations-base/webapps/]
2022-06-29 07:18:43.309:INFO :oejs.AbstractConnector:main: Started ServerConnector@48b67364{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2022-06-29 07:18:43.322:INFO :oejs.Server:main: Started Server@4722ef0c{STARTING}[10.0.11,sto=5000] @753ms

As expected, invalidate module entries (ones that don't exist) are ignored.

[joakime]

Lets try a few things (in order)

First, ensure you are following the recommendations and are using a split ${jetty-home} and ${jetty-base} directory setup.

Next, run start.jar --list-config against your ${jetty.base} directory, and copy/paste the output to this issue.

Finally, if that fails with an NPE, run with start.jar --list-config --debug and copy/paste that output to this issue.

[zaynetro]

Hmm, strange. The problem disappeared after we removed that module from our start.ini file.
I will try to replicate and post the results.

[zaynetro]

OK what I did to reproduce:

1. Download and unarchive Jetty home: https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-home/10.0.11/jetty-home-10.0.11.tar.gz to jetty-home
2. mkdir -p jetty-home/chat/start.d
3. mkdir -p jetty-home/chat/webapps
4. Save to jetty-home/chat/start.d/start.ini:

--module=continuation

--exec
-Xmx1024m

5. From jetty-home/chat run: java -jar ../start.jar (Used Java 17 on my Linux machine)

java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.start.Module.getJvmArgs()" because "module" is null
	at org.eclipse.jetty.start.Main.lambda$start$1(Main.java:476)

[joakime]

The jetty-home artifact is not to be modified. (this is important!)

Once you unpack it, it should be treated as read-only.
Don't add/change/remove/delete/edit any content in it.

The ${jetty.base} directory should be somewhere that isn't nested inside (or out) of the jetty-home directory.

To properly setup that your example in a similar and proper way.

[joakim@hyperion bases]$ mkdir example-continuations-base
[joakim@hyperion bases]$ cd example-continuations-base/
[joakim@hyperion example-continuations-base]$ java -jar ../../jetty-home-10.0.11/start.jar --add-modules=http,deploy,jvm
INFO  : mkdir ${jetty.base}/start.d
INFO  : webapp          transitively enabled, ini template available with --add-module=webapp
INFO  : jvm             initialized in ${jetty.base}/start.d/jvm.ini
INFO  : server          transitively enabled, ini template available with --add-module=server
INFO  : logging-jetty   transitively enabled
INFO  : security        transitively enabled
INFO  : servlet         transitively enabled
INFO  : http            initialized in ${jetty.base}/start.d/http.ini
INFO  : resources       transitively enabled
INFO  : threadpool      transitively enabled, ini template available with --add-module=threadpool
INFO  : logging/slf4j   transitive provider of logging/slf4j for logging-jetty
INFO  : logging/slf4j   dynamic dependency of logging-jetty
INFO  : bytebufferpool  transitively enabled, ini template available with --add-module=bytebufferpool
INFO  : deploy          initialized in ${jetty.base}/start.d/deploy.ini
INFO  : mkdir ${jetty.base}/resources
INFO  : copy ${jetty.home}/modules/logging/jetty/resources/jetty-logging.properties to ${jetty.base}/resources/jetty-logging.properties
INFO  : mkdir ${jetty.base}/webapps
INFO  : Base directory was modified
[joakim@hyperion example-continuations-base]$ tree
.
├── resources
│   └── jetty-logging.properties
├── start.d
│   ├── deploy.ini
│   ├── http.ini
│   └── jvm.ini
└── webapps

3 directories, 4 files
[joakim@hyperion example-continuations-base]$ echo "--module=continuation" >> start.d/continuation.ini
[joakim@hyperion example-continuations-base]$ echo "--exec" >> start.d/jvm.ini 
[joakim@hyperion example-continuations-base]$ echo "-Xmx1g" >> start.d/jvm.ini 

You can then look at the configuration ...

[joakim@hyperion example-continuations-base]$ java -jar ../../jetty-home-10.0.11/start.jar --list-config

Enabled Modules:
----------------
  0) bytebufferpool            transitive provider of bytebufferpool for server
                               ini template available with --add-module=bytebufferpool
  1) jvm                       ${jetty.base}/start.d/jvm.ini
  2) resources                 transitive provider of resources for logging-jetty
  3) logging/slf4j             transitive provider of logging/slf4j for logging-jetty
                               dynamic dependency of logging-jetty
  4) logging-jetty             transitive provider of logging for threadpool
                               transitive provider of logging for server
  5) threadpool                transitive provider of threadpool for server
                               ini template available with --add-module=threadpool
  6) server                    transitive provider of server for http
                               transitive provider of server for servlet
                               ini template available with --add-module=server
  7) security                  transitive provider of security for webapp
  8) servlet                   transitive provider of servlet for webapp
  9) webapp                    transitive provider of webapp for deploy
                               ini template available with --add-module=webapp
 10) deploy                    ${jetty.base}/start.d/deploy.ini
 11) http                      ${jetty.base}/start.d/http.ini

Java Environment:
-----------------
 java.home = /home/joakim/java/jvm/jdk-11.0.15+10
 java.vm.vendor = Eclipse Adoptium
 java.vm.version = 11.0.15+10
 java.vm.name = OpenJDK 64-Bit Server VM
 java.vm.info = mixed mode
 java.runtime.name = OpenJDK Runtime Environment
 java.runtime.version = 11.0.15+10
 java.io.tmpdir = /tmp
 user.dir = /home/joakim/code/jetty/distros/bases/example-continuations-base
 user.language = en
 user.country = US

Jetty Environment:
------------------
 jetty.version = 10.0.11
 jetty.tag.version = jetty-10.0.11
 jetty.build = d988aa016e0bb2de6fba84c1659049c72eae3e32
 jetty.home = /home/joakim/code/jetty/distros/jetty-home-10.0.11
 jetty.base = /home/joakim/code/jetty/distros/bases/example-continuations-base

Config Search Order:
--------------------
 <command-line>
 ${jetty.base} -> /home/joakim/code/jetty/distros/bases/example-continuations-base
 ${jetty.home} -> /home/joakim/code/jetty/distros/jetty-home-10.0.11

Forked JVM Arguments:
---------------------
 -Xmx1g 

System Properties:
------------------
 (no system properties specified)

Properties:
-----------
 java.version = 11.0.15
 java.version.major = 11
 java.version.micro = 15
 java.version.minor = 0
 java.version.platform = 11
 jetty.base = /home/joakim/code/jetty/distros/bases/example-continuations-base
 jetty.base.uri = file:///home/joakim/code/jetty/distros/bases/example-continuations-base
 jetty.home = /home/joakim/code/jetty/distros/jetty-home-10.0.11
 jetty.home.uri = file:///home/joakim/code/jetty/distros/jetty-home-10.0.11
 jetty.webapp.addServerClasses = org.eclipse.jetty.logging.,${jetty.home.uri}/lib/logging/,org.slf4j.
 runtime.feature.alpn = true
 slf4j.version = 2.0.0-alpha6

Jetty Server Classpath:
-----------------------
Version Information on 13 entries in the classpath.
Note: order presented here is how they would appear on the classpath.
      changes to the --module=name command line options will be reflected here.
 0:                    (dir) | ${jetty.base}/resources
 1:             2.0.0-alpha6 | ${jetty.home}/lib/logging/slf4j-api-2.0.0-alpha6.jar
 2:                  10.0.11 | ${jetty.home}/lib/logging/jetty-slf4j-impl-10.0.11.jar
 3:                    4.0.6 | ${jetty.home}/lib/jetty-servlet-api-4.0.6.jar
 4:                  10.0.11 | ${jetty.home}/lib/jetty-http-10.0.11.jar
 5:                  10.0.11 | ${jetty.home}/lib/jetty-server-10.0.11.jar
 6:                  10.0.11 | ${jetty.home}/lib/jetty-xml-10.0.11.jar
 7:                  10.0.11 | ${jetty.home}/lib/jetty-util-10.0.11.jar
 8:                  10.0.11 | ${jetty.home}/lib/jetty-io-10.0.11.jar
 9:                  10.0.11 | ${jetty.home}/lib/jetty-security-10.0.11.jar
10:                  10.0.11 | ${jetty.home}/lib/jetty-servlet-10.0.11.jar
11:                  10.0.11 | ${jetty.home}/lib/jetty-webapp-10.0.11.jar
12:                  10.0.11 | ${jetty.home}/lib/jetty-deploy-10.0.11.jar

Jetty Active XMLs:
------------------
 ${jetty.home}/etc/jetty-bytebufferpool.xml
 ${jetty.home}/etc/jetty-threadpool.xml
 ${jetty.home}/etc/jetty.xml
 ${jetty.home}/etc/jetty-webapp.xml
 ${jetty.home}/etc/jetty-deploy.xml
 ${jetty.home}/etc/jetty-http.xml

Now THIS configuration triggers the NPE.

[joakim@hyperion example-continuations-base]$ java -jar ../../jetty-home-10.0.11/start.jar
java.lang.NullPointerException
	at org.eclipse.jetty.start.Main.lambda$start$1(Main.java:476)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:176)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at org.eclipse.jetty.start.Main.start(Main.java:478)
	at org.eclipse.jetty.start.Main.main(Main.java:75)

Usage: java -jar $JETTY_HOME/start.jar [options] [properties] [configs]
       java -jar $JETTY_HOME/start.jar --help  # for more information

Investigating...

[joakime]

The NPE is from the combination of --module=does-not-exist and use of --exec.

The configuration is fine, StartArgs is fine, the modules (discovered and enabled) is fine.
The command line is calculated fine.
Interesting indeed.

[joakime]

Opened PR #8223