No INFO level logs are appearing in AppInsight

[abhikt48]

We started using applicationinsights-agent:3.4.1 in the existing application. We can see REQUEST entry logs in App Insight but there are no application level INFO logs.

Looks like, we are missing some configuration and we haven't found any sample Log4j2.xml configuration example for codeless agent.

Could you please suggest solution to fix issue or provide sample configuration with Log4j2.xml ?

Note - we don't have any AI /agent dependency in log4j2.xml, no AI-agent.xml file in resources folder. Only we are passing command line argument - -javaagent:applicationinsights-agent-3.4.1.jar

Please find below sample Log4j2.xml

<?xml version="1.0" encoding="utf-8"?>
<Configuration>

   <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%-5p %d [%t] %c: %m%n"/>
        </Console>
    </Appenders>
    
    <Loggers>
    	
    	
		<!-- Http Logger shows wire traffic on DEBUG -->
		<AsyncLogger name="org.mule.module.http.internal.HttpMessageLogger" level="WARN"/>
	
		<!-- JDBC Logger shows queries and parameters values on DEBUG -->
		<AsyncLogger name="com.mulesoft.mule.transport.jdbc" level="WARN"/>
    
        <!-- CXF is used heavily by Mule for web services -->
        <AsyncLogger name="org.apache.cxf" level="WARN"/>

        <!-- Apache Commons tend to make a lot of noise which can clutter the log-->
        <AsyncLogger name="org.apache" level="WARN"/>

        <!-- Reduce startup noise -->
        <AsyncLogger name="org.springframework.beans.factory" level="WARN"/>

        <!-- Mule classes -->
        <AsyncLogger name="org.mule" level="INFO"/>
        <AsyncLogger name="com.mulesoft" level="INFO"/>
        <AsyncLogger name="com.mulesoft.module.batch.engine" level="ERROR"/>
        <AsyncLogger name="com.mulesoft.weave.mule.utils" level="ERROR"/>

        <!-- Reduce DM verbosity -->
        <AsyncLogger name="org.jetel" level="WARN"/>
        <AsyncLogger name="Tracking" level="WARN"/>
        
        <AsyncRoot level="INFO">
            <AppenderRef ref="Console"/>
        </AsyncRoot>
       
    </Loggers>
</Configuration>

Sample Agent Configuration

applicationinsights.json: |
    {
    "preview":{
       "captureControllerSpans":true
       },
       "sampling": {
         "overrides": [{
           "attributes": [{
              "key": "http.url",
              "value": "https://127.0.0.1:1111/test",
              "matchType": "strict"
            }],
            "percentage": 0
             },
              {
            "attributes": [{
              "key": "http.url",
              "value": "https://127.0.0.1:1111/test",
              "matchType": "strict"
            }],
            "percentage": 0
        }]
       }
    },
    "instrumentation":{
       "mongo":{
          "enabled":true
       },
       "jdbc":{
          "enabled":true
       },
       "logging":{ 
           "level": "INFO" 
       }
    },
    "sampling": {
         "limitPerSecond": 5
    }
    }

[trask]

hi @abhikt48! check out https://learn.microsoft.com/en-us/troubleshoot/azure/azure-monitor/app-insights/java-standalone-troubleshoot#some-logging-is-not-auto-collected

if those tips don't help, can you post a small repro that exhibits the same issue that we can use to troubleshoot? thx!

[abhikt48]

Hi @trask , Thank you so much for the quick feedback. I found one root cause of this issue and the root cause is jcl-over-slf4j library.

Context - We are using Mule 3 logger component for logging which internally uses jcl-over-slf4j library and we can't get rid of this library due to heavy usage and high risk.

I created a sample application to replicate this issue.
Source Code - https://github.com/abhikt48/test-applicationinsights-agent/tree/main

Please refer Readme.MD to replicate the issue - https://github.com/abhikt48/test-applicationinsights-agent/blob/main/README.md

You should find the issue while running AiAgentWithJclOverSlf4jTest.java.

Could you please suggest any opinion to publish logs to App Insight with jcl-over-slf4j + applicationinsights-agent-3.4.7.jar ?

NOTE - At this moment, We are using applicationinsights-core:2.6.4 and applicationinsights-logging-log4j2:2.6.4 to publish logs to App Insight which is working fine, but we want to migrate to codeless AI agent applicationinsights-agent-3.4.7.jar but having above issue.

Please feel free to let me know for any query.

[abhikt48]

HI @trask or Team,

Can anyone please take a look on this issue?

Regards,
Abhishek

[abhikt48]

Hi @trask & Team,

We did further testing and found that old jar applicationinsights-agent-3.2.11.jar is working fine but having issue in latest applicationinsights-agent-3.4.7.jar . Looks like there is a potential defect in the latest 3.4.7 version.

Please find below the test result with a different combinations,

[abhikt48]

Hi Team,

Could you please take a look at this issue?

Regards,
Abhishek

[trask]

@abhikt48 thanks for the repro! i'm able to reproduce and looking into it...

[trask]

this should be fixed upstream by open-telemetry/opentelemetry-java-instrumentation#7656

my guess is that this fix should make it into the next upstream release (approx Feb 15), and into the next Application Insights Java release (approx Feb 22)

[abhikt48]

@trask - I did testing with 3.4.10 and I can see logs are appearing in App Insight. I hope you have applied fixes in the latest 3.4.10 jar, if yes then please feel free to close this issue.

[trask]

great, yes, the fix was included in 3.4.10, thx for confirming!