splunk.md

Overview

• Splunk Saas product for metrics, traces and logs
• 14 day free trial
• Servers are only in united states or europe
• Really just signalFx? which was acquired by splunk in aug 2019
• https://app.us1.signalfx.com/#/signin

Infra monitoring

• Easy to setup just install agent

• Uses the otel collector agent which has lots of settings i believe

• More general ways of integration for each cloud provider

• One host dashboard

• Hosts dashboard

• Docker host dashboard need extra settings and permissionsto be added

• Docker container dashboard

Distributed Tracing

• Instrumentation

 # Splunk observability
  export OTEL_RESOURCE_ATTRIBUTES="service.name=${APP_NAME},deployment.environment=dev"
  export OTEL_EXPORTER_OTLP_ENDPOINT='http://splunk-otel-collector:4317'

  JAVA_OPTS="${JAVA_OPTS} \
    -javaagent:${APP_HOME}/splunk-otel-javaagent.jar -Dsplunk.metrics.enabled=true"

• Service map

• Services overview

• List of traces - can see ui getting very messy with lots of services

• An example trace

• Trace drilldown details

Dashboards

• Difficult to use, no SPL
• Granuality can be increased by changing collector settings
• Didn't see a way to display a table of metrics e.g. what are the processing running on host with their cpu/memory usage

• As you onboard more data, dashboards are automatically added for you

Logs

• Not available in trial account
• If it is anything like splunk, it is good

Alerting

• Alert condition

• Alert settings