Logstash init.d script should source default locale set.

[abcfy2]

First, let's look at a snippet of tomcat7 init.d script in Ubuntu 14.04:

# Make sure tomcat is started with system locale
if [ -r /etc/default/locale ]; then
        . /etc/default/locale
        export LANG
fi

Ubuntu store its locale set into /etc/default/locale, while RHEL/CentOS 6 store it into /etc/sysconfig/i18n.

This makes sense, as that if logstash isn't starting from a login shell, would display UTF-8 characters as ????. Some program will consider LC_ALL=C if run from a non-login environment. So this will cause the log cannot output the UTF8 characters.

For example, I use saltstack to manage logstash service, which doesn't use login shell to manage service, and I find the email output will display ????.

My config is like:

output {
    email {
        subject => "服务器 %{host} %{type} 日志发现慢查询, query耗时: %{duration} ms"
    }
}

But I receive the email subject is ??? shifudao mongodb ???????, query??: 20233 ms. When I restart the logstash service via login bash, it's OK.

[suyograo]

@abcfy2 would you be able to contribute a pull request to fix this? That will be super helpful!

[abcfy2]

Sure, and it's very simple. But which branch should I commit to? And which file should I modify?

[zxy12]

我遇到了同样的问题，请问下怎么解决呢，谢谢

[abcfy2]

You can check your locale setting via locale:

$ locale
LANG=zh_CN.UTF-8
LANGUAGE=
LC_CTYPE=zh_CN.UTF-8
LC_NUMERIC=zh_CN.UTF-8
LC_TIME=zh_CN.UTF-8
LC_COLLATE="zh_CN.UTF-8"
LC_MONETARY=zh_CN.UTF-8
LC_MESSAGES="zh_CN.UTF-8"
LC_PAPER=zh_CN.UTF-8
LC_NAME=zh_CN.UTF-8
LC_ADDRESS=zh_CN.UTF-8
LC_TELEPHONE=zh_CN.UTF-8
LC_MEASUREMENT=zh_CN.UTF-8
LC_IDENTIFICATION=zh_CN.UTF-8
LC_ALL=

If your locale contains UTF8 setting, then you can disaplay any UTF-8 characters.

I find that you submit an issue #5272, which is different from this.

I followed your type, and it works very well.

$ /opt/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'                                                                                                                                 127 ↵
Settings: Default pipeline workers: 8
Logstash startup completed
中
2016-05-11T03:37:32.623Z fengyu-Vostro-3900 中
Logstash shutdown completed

I think you should check your locale setting first.

[zxy12]

thank you,but i change the file /etc/sysconfig/i18n with content LANG="zh_CN.UTF-8".
the locale command output is:
locale
LANG=zh_CN.UTF-8
LC_CTYPE="zh_CN.UTF-8"
LC_NUMERIC="zh_CN.UTF-8"
LC_TIME="zh_CN.UTF-8"
LC_COLLATE="zh_CN.UTF-8"
LC_MONETARY="zh_CN.UTF-8"
LC_MESSAGES="zh_CN.UTF-8"
LC_PAPER="zh_CN.UTF-8"
LC_NAME="zh_CN.UTF-8"
LC_ADDRESS="zh_CN.UTF-8"
LC_TELEPHONE="zh_CN.UTF-8"
LC_MEASUREMENT="zh_CN.UTF-8"
LC_IDENTIFICATION="zh_CN.UTF-8"
LC_ALL=

---

bin/logstash -e 'input { stdin { } } output { stdout {} }'
Settings: Default pipeline workers: 4
Pipeline main started
中
2016-05-11T04:22:34.461Z cp01-msg-mcp-odp-dev001.epc.baidu.com ?

but the logstash 's output is still question-mark,i think there is some other place need to check, but i dont know where is it..

[abcfy2]

Strange. Check if your locale if support zh_CN.UTF-8:

locale -a
# check if it contains "zh_CN.utf8"

If so. Try:

$ export LC_ALL=zh_CN.UTF-8
$ export LANG=zh_CN.UTF-8
$ export LANGUAGE=zh_CN.UTF-8
$ locale
# check the locale output, now all of the locale settings will be zh_CN.UTF-8

Then start logstash and try again.

[zxy12]

Very strange.. i give up, Thx very much
sh-3.00$ locale -a | grep zh_CN
zh_CN
zh_CN.gb18030
zh_CN.gb2312
zh_CN.gbk
zh_CN.utf8
sh-3.00$ locale
LANG=zh_CN.UTF-8
LC_CTYPE="zh_CN.UTF-8"
LC_NUMERIC="zh_CN.UTF-8"
LC_TIME="zh_CN.UTF-8"
LC_COLLATE="zh_CN.UTF-8"
LC_MONETARY="zh_CN.UTF-8"
LC_MESSAGES="zh_CN.UTF-8"
LC_PAPER="zh_CN.UTF-8"
LC_NAME="zh_CN.UTF-8"
LC_ADDRESS="zh_CN.UTF-8"
LC_TELEPHONE="zh_CN.UTF-8"
LC_MEASUREMENT="zh_CN.UTF-8"
LC_IDENTIFICATION="zh_CN.UTF-8"
LC_ALL=zh_CN.UTF-8
sh-3.00$ bin/logstash -e 'input { stdin { } } output { stdout {} }'
Settings: Default pipeline workers: 4
Pipeline main started
是
2016-05-11T05:24:44.389Z cp01-msg-mcp-odp-dev001.epc.baidu.com ?
搜索
2016-05-11T05:24:45.694Z cp01-msg-mcp-odp-dev001.epc.baidu.com ??

[suyograo]

@untergeek is making much needed changes to our release packages, so he'll incorporate locale settings.

[untergeek]

We're going to bypass this altogether by using a java options file called config/jvm.options. You can put any legitimate java flags into this file, including:

-Duser.language=en
-Duser.country=US
-Duser.variant=

The reason for not trying to solve the locale problem with environment variables is that this approach is completely cross-platform. It should work the same on any machine or environment.

[untergeek]

This is addressed in #5341 by having Java locale options set in jvm.options.