Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not an issue - JVM exit code and gcr.io/distroless/java image #464

Closed
mabrarov opened this issue Feb 17, 2020 · 8 comments
Closed

Not an issue - JVM exit code and gcr.io/distroless/java image #464

mabrarov opened this issue Feb 17, 2020 · 8 comments

Comments

@mabrarov
Copy link

Hi.

I'm trying gcr.io/distroless/java image and I have a question about container exit code. I didn't find answer myself, so looking for possibility to get answer here. I'm sorry if this is against of rules of this project and if someone could point me a better place for such questions I appreciate that.

I'm curios how one solves issue with (for example) Spring Boot and JVM exit code - refer to spring-boot issue #10697 for details. In short, JVM returns non zero exit code when stopped with SIGTERM and there is no easy way to override exit code for this case.

It looks like java.lang.Runtime#halt (when called in shutdown hook) is the only way to override exit code for this case (Java application is stopped with SIGTERM). Unfortunately, Runtime#halt method doesn't call shutdown hooks and order of shutdown hooks is undefined, so calling Runtime#halt in shutdown hook may skip invocation of other shutdown hooks which is not acceptable, because it can break cleanup which may be setup via shutdown hooks by 3rdparty code (like Spring Boot).

Usually (my experience) JVM (java executable) is never called as entrypoint of docker image but instead some shell script wrapper is used. This script just checks JVM exit code and set it to zero before returning from script (i.e. from container entrypoint) if it matches exit code(s) returned by JVM when it is stopped by SIGTERM.

Thank you.

@mabrarov
Copy link
Author

mabrarov commented Feb 17, 2020

Here is one of possble solutions I found recently:

  1. Use tini (tini-static) as docker ENTRYPOINT
  2. Use tini "remapping exit codes" feature.

Refer to example at mabrarov/java-exit-code/spring-boot/docker/src/main/resources/Dockerfile.

@chanseokoh
Copy link
Member

In short, JVM returns non zero exit code when stopped with SIGTERM and there is no easy way to override exit code for this case.

It's not only JVM. I believe this is customary. Sending SIGTERM to sleep returns the same exit code 143 as JVM does. For example, the following script prints 143 on my Linux.

#!/bin/sh
sleep 1234 &
pid=$!
kill -TERM $pid
wait $pid
echo $?

I don't really understand why you want to override an exit code of a process terminated with a certain signal. At least, I think your issue is unrelated to Java or JVM. I'd like to understand more about your environment: who sends SIGTERM for what purpose, why does it want to send SIGTERM, what is your runtime platform, and what do you want the platform to do when your application (or container) gets the signal? For a web application, I think you most likely want your platform to revive/restart/respawn an application or container when it goes down. If you want the platform to stop spawning a new container once it goes down with an exit code of 0, there must be the right way to configure this on the platform? And does it have anything to do with Java or Spring Boot?

@mabrarov
Copy link
Author

mabrarov commented Feb 19, 2020

who sends SIGTERM for what purpose, why does it want to send SIGTERM, what is your runtime platform

Docker, maybe with Docker Compose or K8s or OpenShift

what do you want the platform to do when your application (or container) gets the signal?

I expect that if I (or Spring Boot) handle SIGTERM in Java code (via shutdown hook) and perform graceful shutdown, then there should be possibility to return zero exit code from JVM even if it's stopped by SIGTERM. Unfortunately, it's not so.

In C/C++ I use signal handler, catch SIGTERM, handle it (graceful shutdown) and return zero exit code from application. It looks like there are some internal / undocumented classes for signal handling in Java - like sun.misc.SignalHandler - but this approach doesn't look like the right one.

If you want the platform to stop spawning a new container once it goes down with an exit code of 0, there must be the right way to configure this on the platform?

Maybe I can configure K8s (maybe it doesn't require additional configuration) to not restart container when it's stopped by K8s sending SIGTERM and when container exit code is not zero (like Docker restart policy), but I'd prefer to handle this at application or container level for consistency with other applications / containers (refer to example with C/C++ above), because containers are some sort of encapsulation and I'd prefer to not think about what's inside - C/C++, Go or Java - and just assume only zero exit code as graceful container stop. The idea is simple - if application / service is gracefully stopped then it's exit code should be zero - without need of detecting if this stop was requested by runtime platform or not.

And does it have anything to do with Java or Spring Boot?

Because it looks to be specific to Java where SIGTERM is handled internally (i.e. signal is caught) and there is possibility to react on SIGTERM (shutdown hook), but there is no way to specify exit code for such case.

I believe for the script you provided there is possibility to catch and handle SIGTERM via trap and to return zero exit code (if SIGTERM is fully handled and graceful shutdown is performed).

SIGTERM is just a signal to my application. It should not imply exit code of my application. Exit code of my application is defined by application logic.

For example, application can catch SIGTERM, start graceful shutdown and then (either or):

  • succeed with graceful shutdown and return zero exit code
  • fail - like timeout of graceful shutdown period - perform emergency shutdown (forcibly terminating client connections) and return non zero exit code to indicate that.

@chanseokoh
Copy link
Member

chanseokoh commented Feb 19, 2020

Fair enough! I will just assume that, whatever it is, you have your own legitimate business reason to override an exit code from JVM–I just wanted to understand your situation a bit more out of curiosity.

I believe for the script you provided there is possibility to catch and handle SIGTERM via trap and to return zero exit code (if SIGTERM is fully handled and graceful shutdown is performed).

I think you are a bit confused here. sleep is still very much like a JVM process; you don't have any control over sleep to change its behavior on how it handles SIGTERM and its exit code 143, unless you modify the source code of sleep to replace its SIGTERM handler and build your own sleep program, for example. The same goes for JVM, generally speaking. Unless you modify the JVM source and build it yourself, you don't really have control over how JVM interprets signals. What you are taking about "catching and handling SIGTERM via trap" is only by spawning a child process (sleep) from another process (bash). And of course, you can do the same for JVM by spawning it as a child process. No difference between sleep and JVM in this regard.

To my understanding, there is no standard interface for Java applications running inside a JVM to handle POSIX signals and modify how the JVM reacts to the signals, unfortunately. This is understandable, knowing that Java applications run on top of a virtual machine. The process that runs your application is the JVM, not a native binary you build with C/C++ where you can do whatever you want with your own program. OTOH, JVM rightfully installs its own signal handlers for its operations, and it is not your program. In any case, I would avoid using sun.misc.SignalHandler, as you said.

To me, it seems like the only option is to write a wrapper program to spawn JVM as a child process and handle the exit code in the parent process–pretty easy and straightforward to do when you write such a wrapper in bash. However, I think tini is a lot better choice; you need to understand what it means to fork a child process in a container environment. The wrapper process will run as PID 1 and have a lot of responsibilities including reaping zombie processes and handling signals properly. And why write a wrapper program when you can simply prepend tini to your entrypoint, which also provides the additional feature you want (i.e,. overriding exit codes)?

(BTW, I just learned what tini is. Thanks for the info. Very nice tool.)

@chanseokoh
Copy link
Member

@mabrarov I saw your thumbs-up, so I'm going to close this. Hope it helped.

@mabrarov
Copy link
Author

mabrarov commented Mar 15, 2020

@chanseokoh, I just realized that Debian 10 has tini package. Do you find it reasonable to add tini into Debian 10 Java images? I understand that it will introduce difference b/w Debian 9 and Debian 10 images, but the pros of adding tini via package instead of just copying binary (like in my example - in java-exit-code) is that installed Debian packages can be scanned by security analysis tools like Aqua Security Trivy, while I doubt that copied file will be detected / matched by such tools (at least by free version).

@chanseokoh
Copy link
Member

chanseokoh commented Mar 15, 2020

The way Distroless puts packages is simply copying the contents of (i.e, exploding) a .deb archive. It doesn't use any sort of package managers, track or maintain any package history(EDIT: I'm not 100% sure about this), or run any pre- or post-installation scripts. So copying the binary might actually work for security analysis tools, if they worked for other packages in Distroless. If it doesn't, I think at least copying all the contents of tini should work. It has only 4 files, so shouldn't be a big deal.

/usr/bin/tini
/usr/bin/tini-static
/usr/share/doc/tini/changelog.Debian.gz
/usr/share/doc/tini/copyright

I don't think Distroless will officially incorporate this binary into the base. I could list a few reasons, but even before that, the official Debian image doesn't come with it by default.

@mabrarov
Copy link
Author

Thank you for detailed answer, which I completely agree.

Just FYI, regarding

So copying the binary might actually work for security analysis tools, if they worked for other packages in Distroless.

I tried Aqua Security Trivy and it looks able to handle installed packages correctly

$ docker run --rm aquasec/trivy --no-progress -q gcr.io/distroless/java-debian10:latest

gcr.io/distroless/java-debian10:latest (debian 10.3)
========================
Total: 34 (UNKNOWN: 0, LOW: 2, MEDIUM: 29, HIGH: 2, CRITICAL: 1)

+-----------------+------------------+----------+-------------------+---------------+----------------------------------------+
|     LIBRARY     | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |                 TITLE                  |
+-----------------+------------------+----------+-------------------+---------------+----------------------------------------+
| libc6           | ...              | ...      | ...               | ...           | ...                                    |
+-----------------+------------------+----------+-------------------+---------------+----------------------------------------+
| libexpat1       | ...              | ...      | ...               | ...           | ...                                    |
+-----------------+------------------+----------+-------------------+---------------+----------------------------------------+
| libgcc1         | ...              | ...      | ...               | ...           | ...                                    |
+-----------------+------------------+----------+-------------------+---------------+----------------------------------------+
| libgomp1        | ...              | ...      | ...               | ...           | ...                                    |
+-----------------+------------------+----------+-------------------+---------------+----------------------------------------+
| libjpeg62-turbo | ...              | ...      | ...               | ...           | ...                                    |
+-----------------+------------------+----------+-------------------+---------------+----------------------------------------+
| libpng16-16     | ...              | ...      | ...               | ...           | ...                                    |
+-----------------+------------------+----------+-------------------+---------------+----------------------------------------+
| libssl1.1       | ...              | ...      | ...               | ...           | ...                                    |
+-----------------+------------------+----------+-------------------+---------------+----------------------------------------+
| libstdc++6      | ...              | ...      | ...               | ...           | ...                                    |
+-----------------+------------------+----------+-------------------+---------------+----------------------------------------+
| openssl         | ...              | ...      | ...               | ...           | ...                                    |
+-----------------+------------------+----------+-------------------+---------------+----------------------------------------+

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants