Supported Metrics

The following metric types are supported currently:

• GC logs
• Generic metrics in CSV format
• All singly printed SAR logs
• INNOTOP logs
• Jmeter JTL/XML Files
• /proc/vmstat
• /proc/meminfo
• /proc/zoneinfo

Section naming convention

Metric type is currently inferred from the name of the section. So the section name should follow this convention:

• Section name should begin with the metric name. Currently supported metric names are: GC, SAR-cpuusage, SAR-device, SAR-cpuhz, SAR-memory, SAR-memutil, SAR-paging, SAR-etcp, SAR-tcp, SAR-dev, SAR-edev, SAR-sock and JMETER.
• Rest of the section name should be separate from metric name by a dash (-)

E.g., GC-1, GC-a-1, SAR-cpuusge-host1 etc are valid section names, while GC_1, GC 1, SAR-cpuusage:host1 are not valid.

Generic metric in CSV format

You can also define your own metric, say e.g., MYAPP-PERF and write a section with that as long as the infile option points to a csv file. But you need to define the separator character (usually comma) using the option sep. You also need to define all the columns in the csv file with option columns. First column is always assumed to be timestamp, so columns option should specify all options from 2nd column onwards. An example of MYAPP-PERF section could be:

[MYAPP-PERF]
infile=/app/logs/myapplog.csv
columns=throughput latency
sep=,

[GRAPH]
graphs=MYAPP-PERF.throughput,MYAPP-PERF.latency
outdir=/home/ritesh/naarad-out

Garbage Collection (GC)

GC logs are supported using an awk parsing engine called PrintGCStats. Supported GC logs should have DateStamps (JVM option -XX:+PrintGCDateStamps).

The following sub-metrics can be specified for GC:

subMetric Name	Description
alloc	object allocation in MB (approximate***)
promo	object promotion in MB (approximate***)
used0	young gen used memory size (before gc)
used1	old gen used memory size (before gc)
used	heap space used memory size (before gc) (excludes perm gen)
commit0	young gen committed memory size (after gc)
commit1	old gen committed memory size (after gc)
commit	heap committed memory size (after gc) (excludes perm gen)
gen0	young gen collection time, excluding gc_prologue & gc_epilogue
gen0t	young gen collection time, including gc_prologue & gc_epilogue
gen0usr	young gen collection time in cpu user secs
gen0sys	young gen collection time in cpu sys secs
gen0real	young gen collection time in elapsed time secs
cmsIM	CMS initial mark pause
cmsRM	CMS remark pause
cmsRS	CMS resize pause
GCPause	all stop-the-world GC pauses
cmsCM	CMS concurrent mark phase
cmsCP	CMS concurrent preclean phase
cmsCS	CMS concurrent sweep phase
cmsCR	CMS concurrent reset phase
apptime	amount of time application threads were running
used0AfterGC	young gen used memory size (after gc)
used1AfterGC	old gen used memory size (after gc)
usedAfterGC	heap space used memory size (after gc)
gen1t	old generation collection/full GC
g1-pause-young	G1 Young GC Pause (seconds)
g1-pause-mixed	G1 Mixed GC Pause (seconds)
g1-pause-remark	G1 Remark Pause (seconds)
g1-pause-cleanup	G1 Cleanup Pause (seconds)
g1-pause-remark.ref-proc	G1 Remark: Reference Processing (seconds)
g1-pause-young.parallel	G1 Young GC Pause: Parallel Operations (ms)
g1-pause-young.parallel.gcworkers	G1 Young GC Pause: Number of Parallel GC Workers
g1-pause-young.parallel.gc-worker-start.avg	G1 Young GC Pause : Parallel : Avg Time spent in GC worker start (ms)
g1-pause-young.parallel.gc-worker-start.max	G1 Young GC Pause : Parallel : Max Time spent in GC worker start (ms)
g1-pause-young.parallel.ext-root-scanning.avg	G1 Young GC Pause: Avg Time spent in ext-root-scanning
g1-pause-young.parallel.ext-root-scanning.max	G1 Young GC Pause: Max Time spent in ext-root-scanning
g1-pause-young.parallel.update-rs.avg	G1 Young GC Pause: Parallel : Avg Time spent in updating Rsets
g1-pause-young.parallel.update-rs.max	G1 Young GC Pause: Parallel : Max Time spent in updating Rsets
g1-pause-young.parallel.update-rs.processed-buffers.avg	G1 Young GC Pause : Parallel : Update Rset : Avg number of processed buffers
g1-pause-young.parallel.update-rs.processed-buffers.max	G1 Young GC Pause : Parallel : Update Rset : Max number of processed buffers
g1-pause-young.parallel.scan-rs.avg	G1 Young GC Pause: Parallel : Avg Time spent in scanning Rsets
g1-pause-young.parallel.scan-rs.max	G1 Young GC Pause: Parallel : Max Time spent in scannning Rsets
g1-pause-young.parallel.object-copy-rs.avg	G1 Young GC Pause : Parallel : Avg Time spent in Object Copy
g1-pause-young.parallel.object-copy-rs.max	G1 Young GC Pause : Parallel : Max Time spent in Object Copy
g1-pause-young.parallel.termination.avg	G1 Young GC Pause : Parallel : Avg Time spent in termination
g1-pause-young.parallel.termination.max	G1 Young GC Pause : Parallel : Max Time spent in termination
g1-pause-young.parallel.gc-worker-other.avg	G1 Young GC Pause : Parallel : Avg Time spent in other
g1-pause-young.parallel.gc-worker-other.max	G1 Young GC Pause : Parallel : Max Time spent in other
g1-pause-young.parallel.gc-worker-total.avg	G1 Young GC Pause : Parallel : Avg Total time for GC worker
g1-pause-young.parallel.gc-worker-total.max	G1 Young GC Pause : Parallel : Max Total time for GC worker
g1-pause-young.parallel.gc-worker-end.avg	G1 Young GC Pause : Parallel : Avg Time for GC worker end
g1-pause-young.parallel.gc-worker-end.max	G1 Young GC Pause : Parallel : Max Time for GC worker end
g1-pause-young.code-root-fixup	G1 Young GC Pause : Time spent in code root fixup (ms)
g1-pause-young.clear-ct	G1 Young GC Pause: Time spent in clear ct (ms)
g1-pause-young.other	G1 Young GC Pause: Time spent in other (ms)
g1-pause-young.other.choose-cset	G1 Young GC Pause : Other : Time spent in choosing CSet (ms)
g1-pause-young.other.ref-proc	G1 Young GC Pause : Other : Time spent in reference processing (ms)
g1-pause-young.other.reg-enq	G1 Young GC Pause : Other : Time spent in reg-enq(ms)
g1-pause-young.other.free-cset	G1 Young GC Pause : Other : Time spent in processing free Cset(ms)
g1-pause-mixed.parallel	G1 Mixed GC Pause: Parallel Operations (ms)
g1-pause-mixed.parallel.gcworkers	G1 Mixed GC Pause: Number of Parallel GC Workers
g1-pause-mixed.parallel.gc-worker-start.avg	G1 Mixed GC Pause : Parallel : Avg Time spent in GC worker start (ms)
g1-pause-mixed.parallel.gc-worker-start.max	G1 Mixed GC Pause : Parallel : Max Time spent in GC worker start (ms)
g1-pause-mixed.parallel.ext-root-scanning.avg	G1 Mixed GC Pause: Avg Time spent in ext-root-scanning
g1-pause-mixed.parallel.ext-root-scanning.max	G1 Mixed GC Pause: Max Time spent in ext-root-scanning
g1-pause-mixed.parallel.update-rs.avg	G1 Mixed GC Pause: Parallel : Avg Time spent in updating Rsets
g1-pause-mixed.parallel.update-rs.max	G1 Mixed GC Pause: Parallel : Max Time spent in updating Rsets
g1-pause-mixed.parallel.update-rs.processed-buffers.avg	G1 Mixed GC Pause : Parallel : Update Rset : Avg number of processed buffers
g1-pause-mixed.parallel.update-rs.processed-buffers.max	G1 Mixed GC Pause : Parallel : Update Rset : Max number of processed buffers
g1-pause-mixed.parallel.scan-rs.avg	G1 Mixed GC Pause: Parallel : Avg Time spent in scanning Rsets
g1-pause-mixed.parallel.scan-rs.max	G1 Mixed GC Pause: Parallel : Max Time spent in scannning Rsets
g1-pause-mixed.parallel.object-copy-rs.avg	G1 Mixed GC Pause : Parallel : Avg Time spent in Object Copy
g1-pause-mixed.parallel.object-copy-rs.max	G1 Mixed GC Pause : Parallel : Max Time spent in Object Copy
g1-pause-mixed.parallel.termination.avg	G1 Mixed GC Pause : Parallel : Avg Time spent in termination
g1-pause-mixed.parallel.termination.max	G1 Mixed GC Pause : Parallel : Max Time spent in termination
g1-pause-mixed.parallel.gc-worker-other.avg	G1 Mixed GC Pause : Parallel : Avg Time spent in other
g1-pause-mixed.parallel.gc-worker-other.max	G1 Mixed GC Pause : Parallel : Max Time spent in other
g1-pause-mixed.parallel.gc-worker-total.avg	G1 Mixed GC Pause : Parallel : Avg Total time for GC worker
g1-pause-mixed.parallel.gc-worker-total.max	G1 Mixed GC Pause : Parallel : Max Total time for GC worker
g1-pause-mixed.parallel.gc-worker-end.avg	G1 Mixed GC Pause : Parallel : Avg Time for GC worker end
g1-pause-mixed.parallel.gc-worker-end.max	G1 Mixed GC Pause : Parallel : Max Time for GC worker end
g1-pause-mixed.code-root-fixup	G1 Mixed GC Pause : Time spent in code root fixup (ms)
g1-pause-mixed.clear-ct	G1 Mixed GC Pause: Time spent in clear ct (ms)
g1-pause-mixed.other	G1 Mixed GC Pause: Time spent in other (ms)
g1-pause-mixed.other.choose-cset	G1 Mixed GC Pause : Other : Time spent in choosing CSet (ms)
g1-pause-mixed.other.ref-proc	G1 Mixed GC Pause : Other : Time spent in reference processing (ms)
g1-pause-mixed.other.reg-enq	G1 Mixed GC Pause : Other : Time spent in reg-enq(ms)
g1-pause-mixed.other.free-cset	G1 Mixed GC Pause : Other : Time spent in processing free Cset(ms)
g1-eden-occupancy-before-gc	G1 Eden Occupancy (MB) (Before GC)
g1-eden-capacity-before-gc	G1 Eden Capacity (MB) (Before GC)
g1-eden-occupancy-after-gc	G1 Eden Occupancy (MB) (After GC)
g1-eden-capacity-after-gc	G1 Efen Capacity (MB) (After GC)
g1-survivor-before-gc	G1 Survivor Size (MB) (Before GC)
g1-survivor-after-gc	G1 Survivor Size (MB) (After GC)
g1-heap-occupancy-before-gc	G1 Heap Occupancy (MB) (Before GC)
g1-heap-capacity-before-gc	G1 Heap Capacity (MB) (Before GC)
g1-heap-occupancy-after-gc	G1 Heap Occupancy (MB) (After GC)
g1-heap-capacity-after-gc	G1 Heap Capacity (MB) (After GC)
g1-young-cpu.sys	G1 Young GC : sys cpu time (seconds)
g1-young-cpu.usr	G1 Young GC : usr cpu time (seconds)
g1-young-cpu.real	G1 Young GC : elapsed time (seconds)
g1-mixed-cpu.usr	G1 Mixed GC : usr cpu time (seconds)
g1-mixed-cpu.sys	G1 Mixed GC : sys cpu time (seconds)
g1-mixed-cpu.real	G1 Mixed GC : elapsed time (seconds)

So, e.g., you can specify all GC options using this section:

[GC]
infile=/home/ritesh/logs/gc.log
gc-options=GC alloc promo used0 used1 used commit0 commit1 commit gen0 gen0t gen0usr gen0sys cmsIM cmsRM cmsRS cmsCM cmsCS csmCR apptime safept gen1t gen1i

System metrics using sar

Naarad supports various sar metrics, each one as its own metric. Supported metrics and the corresponding command needed to collect them are listed here (note that last argument controls how frequently metrics are printed in the log and can be updated as suitable):

• SAR-cpuusage using command: sar -u ALL 1
• SAR-device using command: sar -d 1
• SAR-memory using command: sar -R 1
• SAR-memutil using command: sar -r 1
• SAR-paging using command: sar -B 1
• SAR-swapping using command: sar -W 1
• SAR-dev using command: sar -n DEV 1
• SAR-edev using command: sar -n EDEV 1
• SAR-tcp using command: sar -n TCP 1
• SAR-etcp using command: sar -n ETCP 1
• SAR-sock using command: sar -n SOCK 1