Encoding issues with UTF-8 characters

[cschug]

I am just about to migrate from logstash-forwarder to log-courier but experiencing encoding issues with UTF-8 characters which are not part of the ASCII subset, to be more specific with German umlauts.
In a very simple setup I am running both shippers (logstash-forwarder 0.3.1 and log-courier 0.11) on the very same machine as logstash 1.4.2. While the same test string "Pufferüberlauf" (German for "buffer overflow") is processed correctly by the logstash-forwarder, with log-courer the umlaut "ü" gets replaced by "??".

rubydebug output written by Logstash for the logstash-forwarder input:

{"message":"Pufferüberlauf","@version":"1","@timestamp":"2014-07-20T21:39:31.419Z","type":"forwarder","file":"/var/tmp/forwarder","host":"XX","offset":"78"}

rubydebug output written by Logstash for the log-courier input:

{"file":"/var/tmp/courier","host":"XX","message":"Puffer??berlauf","offset":72,"type":"courier","@version":"1","@timestamp":"2014-07-20T21:39:34.885Z"}

[driskell]

Hi,

I hadn't yet tested with UTF-8 or invalid characters.

I've just done so now and it works OK. Only invalid UTF-8 characters will be stripped. I will push the changes up shortly.

Thanks for the report!

Jason

[cschug]

Excellent, seems to fix the issue of my simple test case, will put it under more stress tomorrow ... thanks a lot! Keep up the good work :-) 👍

[driskell]

Thanks :)