Added documentation for stackdriver logging

[gregw]

Added documentation to enable stackdriver logging locally, on GCP and other platforms (see #69).

Note that the GAE prefix is used in several places, when perhaps it should have been GCP? However it is now part of the stackdriver client library and also I believe the resourceType of gae_app may be significant.

[meltsufin]

I think we should instead show them how to do it in code. On App Engine Flex, they don't have control over the docker run command. We can also show how to load logging.properties from within the app.

[meltsufin]

@gregw WDYT?

[gregw]

@meltsufin If these images are to be usable generally on any platform, then I think we need to show them both how to setup logging with configuration and with code. So I'll update with some example code.

[gregw]

@meltsufin I've been trying to come up with some simple programmatic examples for configuring JUL andto put it simply it is just dire! It is a mess of private static variables initialised from system properties, some of which can be reset, but others that cannot. When the configuration happens is really crucial and to have full control over that you need to set environment variables anyway.

I think full programmatic configuration of JUL is for super users (perhaps super-human users).

So instead, I'll work on an alternative simple programmatic example that shows how a logging.properties file can be loaded from the classpath using programmatic API and system variables.

[gregw]

@meltsufin is this sufficient to show how JUL can be configured if there is no custom image and no access to the docker run command ?

[meltsufin]

@gregw I'm trying to follow the instructions for a Spring Boot app and it keeps failing with a java.lang.ClassNotFoundException: com.google.cloud.logging.LoggingHandler. Apparently, it's because JUL is always using the system class loader rather than the context class loader. See Spring Boot docs. Any way around this, because Spring Boot is one of the main frameworks targeted by this runtime?

Another concern, and I haven't been able validate it yet, is that without the GAE Enhancer, the logs will not have enough metadata, like the monitored resource, to appear in the right category in the the Cloud Console Logging UI.

[gregw]

I assume you are adding the jars as nested jars within the springboot jar, which as they say is not going to work as JUL will not see them.

Their recommendation is to use something other than JUL, which is very tempting as JUL is so poor. But the problem is that there are so many other choices: SLF4J would be my preference, but there is also Log4j, logback etc. JUL has the benefit of just being there.

So if we want to make JUL work, then we need to get our classes onto the system c lasspath. Two approaches come to mind:

• Give instructions how to install stackdriver logging into the JVM's library - ugly and not a convenient way to bundle.
• Write a JUL LogHandler that we can pre-install into the JVM's lib that is just a delegate handler, which uses SpringBoot loader to find the real handler. Thus we can bundle arbitrary JUL handlers in the springboot jar and delegate to them. However, I'm not sure if the Thread context classloader will be set in time, so we'd need to experiment to see if this is possible.

[gregw]

With regards to the lack of enhancer, my experience is that the logs are still accessible, but will not be linked to the nginx request log and may be difficult to find as they may not be automatically accessible from a link from the GCP instance screen. Will check exactly what info is available.

[meltsufin]

For Spring Boot, maybe it's worth looking into how to control the packaging of the fat jar to make it work with JUL. The Spring Boot docs I linked to mention Maven Shade Plugin. Maybe that will work, and we can document it? I haven't explored it yet.

Otherwise, I'm tempted to hold off on this and wait for support for more logging frameworks to come in the logging lib -- planned for Q2.

[gregw]

So are you saying that rather than use spring boot nested jars, we create a shaded fat jar with a flat structure. That could possibly work, but would not allow arbitrary spring boot apps to easily be deployed.

I think that JUL will need something on the JVM classpath to work, so I'd like to experiment to see how minimal that could be.

[meltsufin]

Right, we would have to provide instructions in the logging section on how to configure the Spring Boot app -- not ideal.

I don't mind just dropping the JAR into the JVM as long as it won't interfere with user application dependencies. There's even an advantage to that because the user wouldn't have to include the logging jar in the app itself.

[gregw]

@meltsufin
I have tested if it is possible to put a simple delegating JUL into $JAVA_HOME/lib/ext, which uses the Thread context classpath. This appears to work surprisingly well! By the time the JUL handler is instantiated. the context classloader is already a org.springframework.boot.loader.LaunchedURLClassLoader and is able to load classes that are bundled in the executable spring boot jar/war including log handlers.

However, by default, spring boot is still instantiating it's own logging default setup, including a JUL configuration. So it is blasting over anything I setup in logging.properties anyway. So the general approach of put-the-jars-in-your-app-and-set-a-logging-dot-properties is just not going to work in general with spring boot as regardless we are going to have to come up with instructions on how to configure spring boots logging setup away from it's default setup.

Note that we could document how to make spring boot use the existing JUL handler, but would require moderately extensive reconfiguration of spring boots logging, which by default uses:

logback-classic-1.1.11.jar
logback-core-1.1.11.jar
jcl-over-slf4j-1.7.24.jar
jul-to-slf4j-1.7.24.jar
log4j-over-slf4j-1.7.24.jar
slf4j-api-1.7.24.jar
jboss-logging-3.3.0.Final.jar

This will collect logs from JCL,JUL,Log4J and JBoss logging, route them over slf4j and use logback to actually produce the logs. To use the existing stackdriver log handler, the jul-to-slf4j bridge and logback implementation would need to be replaced with a slf4j-to-jdk14 jar and then a configuration applied.

Thus for spring boot, it may be simpler to wait until a stack driver client is available either for SLF4j or logback.

However, given all that, I don't necessarily think we need to delay merging in the instructions in this PR, as regardless they will be a valid way for an application to use JUL and strackdriver. I think we just need to perhaps structure the documentation a little differently:

• Firstly we should say that any logging system can be used and that stderr is captured and available on the console.
• Then give the examples of how JUL logging can be added and configured in the image generally, noting that the approach will not work for frameworks like springboot that configure their own logging.
• The give the example of how to configure stackdriver logging over JUL. Noting that stackdriver handlers will soon be provided for other logging frameworks.

Shall I update this PR along those lines?