Logfile format not parseable by default by AWS Cloudwatch logs

[HakimCassimallyBBC]

The consul_server.log file prefixes lines with spaces, causing them to be interpreted as multiline logs by AWS Cloudwatch logs.

The odd logfile format has been mentioned previously in #1446 for other logging systems. I suspect it may annoy users of other brands of log-parser too.

When filing a bug, please include the following:

consul version for both Client and Server

Client: n/a
Server: v0.6.4

consul info for both Client and Server

For server only:

$ consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 1
build:
prerelease =
revision = 26a0ef8
version = 0.6.4
consul:
bootstrap = false
known_datacenters = 1
leader = true
server = true
raft:
applied_index = 4212917
commit_index = 4212917
fsm_pending = 0
last_contact = 0
last_log_index = 4212917
last_log_term = 58406
last_snapshot_index = 4206051
last_snapshot_term = 58063
num_peers = 2
state = Leader
term = 58406
runtime:
arch = amd64
cpu_count = 1
goroutines = 210
max_procs = 1
os = linux
version = go1.6
serf_lan:
encrypted = false
event_queue = 0
event_time = 226
failed = 4
intent_queue = 0
left = 0
member_time = 2485
members = 42
query_queue = 0
query_time = 1
serf_wan:
encrypted = false
event_queue = 0
event_time = 1
failed = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1

Operating system and Environment details

Linux ip-....eu-west-1.compute.internal 2.6.32-642.13.1.el6.x86_64 #1 SMP Wed Jan 11 20:56:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Description of the Issue (and unexpected/desired result)

When uploading the logs to AWS cloudwatch logs (see http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html ) multiple consecutive lines are treated as a single log entry.

Reproduction steps

• requires AWS instance and the awslogs daemon. If you do want to triage this to reproduce, I can document this.

Log Fragments

e.g.

==> Log data will now stream in as it occurs:

    2017/02/16 15:23:25 [INFO] raft: Restored from snapshot 58063-4206051-1487237032808
    2017/02/16 15:23:25 [INFO] serf: EventMemberJoin: my-host-name-10.55.118.200 10.55.118.200
    2017/02/16 15:23:25 [INFO] serf: EventMemberJoin: imy-host-name-10.55.118.200.dc1 10.55.118.200
 ...

Workaround

From initial testing, it looks like adding multi_line_start_pattern = ^ to the awslogs.conf file makes it parseable.

[mterron]

Any reason why logging is indented?

[HakimCassimallyBBC]

If I remember, from the previous ticket linked above, the reason cited by the devs was a human-readability concern.

(Which I think is valid, up to a point, but the fact that it drives the actual, useful, machine-clients of the output crazy seems more important to me, hence the bug report :D)

[slackpad]

Closing this per #1446.