Main process crashes on Json::Encode (third-party/yajl) #6684

miso231 · 2018-10-15T08:39:42Z

Icinga2 crashes from time to time (SIGSEGV). It happens approximately when log rotation is happening so I guess it might be related. I'm using distributed setup with HA master and this issue so far occurs only on client nodes.

Current Behavior

Client nodes in 3 level environment crash from time to time. All nodes run version 2.9.1.

Steps to Reproduce (for bugs)

Hard to tell since this issue happens occasionally.

Context

backtrace

{   "signal": 11
,   "executable": "/usr/lib64/icinga2/sbin/icinga2"
,   "stacktrace":
      [ {   "crash_thread": true
        ,   "frames":
              [ {   "address": 47784319202749
                ,   "build_id": "c3f28802314af4ee866bf8d2e1b506b7bbf34cf6"
                ,   "build_id_offset": 505277
                ,   "function_name": "malloc_consolidate"
                ,   "file_name": "/usr/lib64/libc-2.17.so"
                }
              , {   "address": 47784319210405
                ,   "build_id": "c3f28802314af4ee866bf8d2e1b506b7bbf34cf6"
                ,   "build_id_offset": 512933
                ,   "function_name": "_int_malloc"
                ,   "file_name": "/usr/lib64/libc-2.17.so"
                }
              , {   "address": 47784319222028
                ,   "build_id": "c3f28802314af4ee866bf8d2e1b506b7bbf34cf6"
                ,   "build_id_offset": 524556
                ,   "function_name": "malloc"
                ,   "file_name": "/usr/lib64/libc-2.17.so"
                }
              , {   "address": 6076934
                ,   "build_id": "3b0346f9d2b617e2c3c31c6e330319889c1da958"
                ,   "build_id_offset": 1882630
                ,   "function_name": "yajl_buf_append"
                ,   "file_name": "/usr/lib64/icinga2/sbin/icinga2"
                }
              , {   "address": 9652871
                ,   "build_id": "3b0346f9d2b617e2c3c31c6e330319889c1da958"
                ,   "build_id_offset": 5458567
                ,   "function_name": "yajl_gen_map_open"
                ,   "file_name": "/usr/lib64/icinga2/sbin/icinga2"
                }
              , {   "address": 9759919
                ,   "build_id": "3b0346f9d2b617e2c3c31c6e330319889c1da958"
                ,   "build_id_offset": 5565615
                ,   "function_name": "Encode(yajl_gen_t*, icinga::Value const&) [clone .77529.22147]"
                ,   "file_name": "/usr/lib64/icinga2/sbin/icinga2"
                }
              , {   "address": 9760701
                ,   "build_id": "3b0346f9d2b617e2c3c31c6e330319889c1da958"
                ,   "build_id_offset": 5566397
                ,   "function_name": "icinga::JsonEncode(icinga::Value const&, bool)"
                ,   "file_name": "/usr/lib64/icinga2/sbin/icinga2"
                }
              , {   "address": 9774577
                ,   "build_id": "3b0346f9d2b617e2c3c31c6e330319889c1da958"
                ,   "build_id_offset": 5580273
                ,   "function_name": "icinga::Process::Run(std::function<void (icinga::ProcessResult const&)> const&)"
                ,   "file_name": "/usr/lib64/icinga2/sbin/icinga2"
                }
              , {   "address": 9779982
                ,   "build_id": "3b0346f9d2b617e2c3c31c6e330319889c1da958"
                ,   "build_id_offset": 5585678
                ,   "function_name": "icinga::Application::StartReloadProcess()"
                ,   "file_name": "/usr/lib64/icinga2/sbin/icinga2"
                }
              , {   "address": 9780675
                ,   "build_id": "3b0346f9d2b617e2c3c31c6e330319889c1da958"
                ,   "build_id_offset": 5586371
                ,   "function_name": "icinga::Application::RunEventLoop()"
                ,   "file_name": "/usr/lib64/icinga2/sbin/icinga2"
                }
              , {   "address": 9785220
                ,   "build_id": "3b0346f9d2b617e2c3c31c6e330319889c1da958"
                ,   "build_id_offset": 5590916
                ,   "function_name": "icinga::IcingaApplication::Main()"
                ,   "file_name": "/usr/lib64/icinga2/sbin/icinga2"
                }
              , {   "address": 8354465
                ,   "build_id": "3b0346f9d2b617e2c3c31c6e330319889c1da958"
                ,   "build_id_offset": 4160161
                ,   "function_name": "icinga::Application::Run()"
                ,   "file_name": "/usr/lib64/icinga2/sbin/icinga2"
                }
              , {   "address": 9432456
                ,   "build_id": "3b0346f9d2b617e2c3c31c6e330319889c1da958"
                ,   "build_id_offset": 5238152
                ,   "function_name": "icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::string, std::allocator<std::string> > const&) const"
                ,   "file_name": "/usr/lib64/icinga2/sbin/icinga2"
                }
              , {   "address": 9371914
                ,   "build_id": "3b0346f9d2b617e2c3c31c6e330319889c1da958"
                ,   "build_id_offset": 5177610
                ,   "function_name": "Main() [clone .14744.18446]"
                ,   "file_name": "/usr/lib64/icinga2/sbin/icinga2"
                }
              , {   "address": 5942231
                ,   "build_id": "3b0346f9d2b617e2c3c31c6e330319889c1da958"
                ,   "build_id_offset": 1747927
                ,   "function_name": "main"
                ,   "file_name": "/usr/lib64/icinga2/sbin/icinga2"
                } ]
        } ]
}

/var/log/messages

Oct 11 07:01:11 client-host kernel: traps: icinga2[2933469] general protection ip:2b75a75f35bd sp:7fff47bd1ed0 error:0 in libc-2.17.so[2b75a7578000+1b8000]
Oct 11 07:01:11 client-host abrt-hook-ccpp[3175889]: Process 2933469 (icinga2) of user 481 killed by SIGSEGV - dumping core
Oct 11 07:01:13 client-host systemd[1]: icinga2.service: main process exited, code=dumped, status=11/SEGV

Let me know if you need more information.

Your Environment

Version used (icinga2 --version):

icinga2 - The Icinga 2 network monitoring daemon (version: r2.9.1-1)

Copyright (c) 2012-2018 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-693.5.2.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: unknown

Operating System and version: CentOS Linux 7 (Core)
Enabled features (icinga2 feature list): api mainlog
Icinga Web 2 version and modules (System - About):
Config validation (icinga2 daemon -C) - I removed apply rule warnings:

[2018-10-15 07:57:57 +0000] information/cli: Icinga application loader (version: r2.9.1-1)
[2018-10-15 07:57:57 +0000] information/cli: Loading configuration file(s).
[2018-10-15 07:57:58 +0000] information/ConfigItem: Committing config item(s).
[2018-10-15 07:57:58 +0000] information/ApiListener: My API identity: pattern-feed-prod-prg-002.s.jumpshot.com
[2018-10-15 07:57:58 +0000] information/ConfigItem: Instantiated 9 ServiceGroups.
[2018-10-15 07:57:58 +0000] information/ConfigItem: Instantiated 143 HostGroups.
[2018-10-15 07:57:58 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2018-10-15 07:57:58 +0000] information/ConfigItem: Instantiated 6 NotificationCommands.
[2018-10-15 07:57:58 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2018-10-15 07:57:58 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2018-10-15 07:57:58 +0000] information/ConfigItem: Instantiated 3 Zones.
[2018-10-15 07:57:58 +0000] information/ConfigItem: Instantiated 2 Endpoints.
[2018-10-15 07:57:58 +0000] information/ConfigItem: Instantiated 3 UserGroups.
[2018-10-15 07:57:58 +0000] information/ConfigItem: Instantiated 250 CheckCommands.
[2018-10-15 07:57:58 +0000] information/ConfigItem: Instantiated 5 TimePeriods.
[2018-10-15 07:57:58 +0000] information/ConfigItem: Instantiated 10 Users.
[2018-10-15 07:57:58 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2018-10-15 07:57:58 +0000] information/cli: Finished validating the configuration file(s).

If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

object Zone "client-host" {
    endpoints = [ "client-host" ]
    parent = "zone1"
}

object Endpoint "client-host" {
}

object Zone "zone1" {
    endpoints = [ "satelite-host" ]
}

object Endpoint "satelite-host" {
}

object Zone "global-templates"{
    global = true
}

The text was updated successfully, but these errors were encountered:

dnsmichi · 2018-10-17T07:30:59Z

This would indicate a bug in the Json library we're using, yajl. I was just wondering which library we're using here, either from CentOS/EPEL or from our third-party repository. Looking into the package builds, I can see that's the content from third-party.

-I/home/jenkins/workspace/icinga2-snapshot/rpm-centos-7-1binary/arch/x86_64/rpmbuild/BUILD/icinga2-2.10.0.18.gc0398ed/third-party/yajl/include

yail is relatively "old", we're already using the latest release 2.1.0. I am reading lloyd/yajl#168 which indicates that this project is not actively maintained anymore.

Considering looking into an alternative JSON implementation, granted that we can now use C++11 features in compilers we couldn't for many years.

https://github.com/miloyip/nativejson-benchmark#parsing-time

pimperator · 2018-11-27T09:24:54Z

Hi Guys,
I am also suffering actively on this problem.

File core_backtrace:

/var/spool/abrt/ccpp-2018-11-27-09:28:39-29228 { "signal": 11 , "executable": "/usr/lib64/icinga2/sbin/icinga2" , "stacktrace": [ { "crash_thread": true , "frames": [ { "address": 47618400720317 , "build_id": "95ff02a4bebabc573c7827a66d447f7babddaa44" , "build_id_offset": 505277 , "function_name": "malloc_consolidate" , "file_name": "/usr/lib64/libc-2.17.so" } , { "address": 47618400727973 , "build_id": "95ff02a4bebabc573c7827a66d447f7babddaa44" , "build_id_offset": 512933 , "function_name": "_int_malloc" , "file_name": "/usr/lib64/libc-2.17.so" } , { "address": 47618400739596 , "build_id": "95ff02a4bebabc573c7827a66d447f7babddaa44" , "build_id_offset": 524556 , "function_name": "malloc" , "file_name": "/usr/lib64/libc-2.17.so" } , { "address": 6142038 , "build_id": "ebf33136d469a4ef2f58e7d6a76714ff46133c99" , "build_id_offset": 1947734 , "function_name": "yajl_buf_append" , "file_name": "/usr/lib64/icinga2/sbin/icinga2" } , { "address": 8566471 , "build_id": "ebf33136d469a4ef2f58e7d6a76714ff46133c99" , "build_id_offset": 4372167 , "function_name": "yajl_gen_map_open" , "file_name": "/usr/lib64/icinga2/sbin/icinga2" } , { "address": 8684231 , "build_id": "ebf33136d469a4ef2f58e7d6a76714ff46133c99" , "build_id_offset": 4489927 , "function_name": "Encode(yajl_gen_t*, icinga::Value const&) [clone .82478.21492]" , "file_name": "/usr/lib64/icinga2/sbin/icinga2" } , { "address": 8684925 , "build_id": "ebf33136d469a4ef2f58e7d6a76714ff46133c99" , "build_id_offset": 4490621 , "function_name": "icinga::JsonEncode(icinga::Value const&, bool)" , "file_name": "/usr/lib64/icinga2/sbin/icinga2" } , { "address": 8698801 , "build_id": "ebf33136d469a4ef2f58e7d6a76714ff46133c99" , "build_id_offset": 4504497 , "function_name": "icinga::Process::Run(std::function<void (icinga::ProcessResult const&)> const&)" , "file_name": "/usr/lib64/icinga2/sbin/icinga2" } , { "address": 8704206 , "build_id": "ebf33136d469a4ef2f58e7d6a76714ff46133c99" , "build_id_offset": 4509902 , "function_name": "icinga::Application::StartReloadProcess()" , "file_name": "/usr/lib64/icinga2/sbin/icinga2" } , { "address": 8704899 , "build_id": "ebf33136d469a4ef2f58e7d6a76714ff46133c99" , "build_id_offset": 4510595 , "function_name": "icinga::Application::RunEventLoop()" , "file_name": "/usr/lib64/icinga2/sbin/icinga2" } , { "address": 8709444 , "build_id": "ebf33136d469a4ef2f58e7d6a76714ff46133c99" , "build_id_offset": 4515140 , "function_name": "icinga::IcingaApplication::Main()" , "file_name": "/usr/lib64/icinga2/sbin/icinga2" } , { "address": 10554228 , "build_id": "ebf33136d469a4ef2f58e7d6a76714ff46133c99" , "build_id_offset": 6359924 , "function_name": "icinga::Application::Run()" , "file_name": "/usr/lib64/icinga2/sbin/icinga2" } , { "address": 9863879 , "build_id": "ebf33136d469a4ef2f58e7d6a76714ff46133c99" , "build_id_offset": 5669575 , "function_name": "icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::string, std::allocator<std::string> > const&) const" , "file_name": "/usr/lib64/icinga2/sbin/icinga2" } , { "address": 9902499 , "build_id": "ebf33136d469a4ef2f58e7d6a76714ff46133c99" , "build_id_offset": 5708195 , "function_name": "Main() [clone .15176.16432]" , "file_name": "/usr/lib64/icinga2/sbin/icinga2" } , { "address": 5957687 , "build_id": "ebf33136d469a4ef2f58e7d6a76714ff46133c99" , "build_id_offset": 1763383 , "function_name": "main" , "file_name": "/usr/lib64/icinga2/sbin/icinga2" } ] } ] }

/var/log/messages looks similar:
Nov 27 09:28:39 hostname kernel: traps: icinga2[29228] general protection ip:2b4f05dc15bd sp:7ffcb08e4800 error:0 in libc-2.17.so[2b4f05d46000+1b8000] Nov 27 09:28:39 hostname abrt-hook-ccpp[17802]: Process 29228 (icinga2) of user 1107800094 killed by SIGSEGV - dumping core

`icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.2-1)

Copyright (c) 2012-2018 Icinga Development Team (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
Platform: CentOS Linux
Platform version: 7 (Core)
Kernel: Linux
Kernel version: 3.10.0-693.5.2.el7.x86_64
Architecture: x86_64

Build information:
Compiler: GNU 4.8.5
Build host: unknown`

yajl is installed in the following version:
rpm -qa | grep yajl yajl-2.0.4-4.el7.x86_64

`yum info yajl

Installed Packages
Name : yajl
Arch : x86_64
Version : 2.0.4
Release : 4.el7
Size : 114 k
Repo : installed
From repo : base
Summary : Yet Another JSON Library (YAJL)
URL : http://lloyd.github.com/yajl/
License : ISC
Description : Yet Another JSON Library. YAJL is a small event-driven
: (SAX-style) JSON parser written in ANSI C, and a small
: validating JSON generator.`

Unfortunately there are no up-to-date packages for yajl provided by centos.

If you need furhter information: please feel free to ask.

Best wishes from Unterföhring

EL

pimperator · 2018-11-27T09:51:16Z

a colleague gave me a hint that there might be a connection with #6737

dnsmichi · 2019-03-12T16:02:42Z

https://github.com/Icinga/icinga2/tree/feature/replace-json-library holds a first attempt, but feel free to do it differently with the header only library @Al2Klimov - thanks 👍

Al2Klimov · 2019-03-12T17:36:47Z

@dnsmichi Just FYI, that lib very dislikes non-UTF8, so I'll have to do #5014 for en- and decoding.

dnsmichi · 2019-03-13T08:19:36Z

I feared that and hoped it would provide a mechanism for doing that in its own. I don't like our algorithm, I think there are better ones for that. Still, go for it.

Al2Klimov · 2019-03-13T09:04:00Z

Thanks for the green light. As this will implicitly solve all of our already known UTF8-problems, please assign me the respective issues.

dnsmichi · 2019-03-18T16:26:36Z

@miso231 @pimperator 2.11 will drop YAJL and use a header-only JSON library compiled into the binary.

The main reasons for picking another library here is that YAJL isn't maintained anymore, last commit in 2015.

The linked logrotate issue is different from this unfortunately.

dnsmichi changed the title ~~Icinga2 main process crashes (SIGSEGV)~~ Main process crashes on Json::Encode (third-party/yajl) Oct 17, 2018

dnsmichi added the core/crash Shouldn't happen, requires attention label Oct 17, 2018

dnsmichi self-assigned this Oct 17, 2018

dnsmichi added the bug Something isn't working label Oct 19, 2018

This was referenced Nov 30, 2018

WIP: Replace JSON library #6812

Closed

Malformed characters - invalid continuation byte #5395

Closed

lippserd added this to the 2.11.0 milestone Feb 6, 2019

dnsmichi mentioned this issue Feb 11, 2019

HttpUtility::SendJsonBody(): auto-sanitize non-UTF-8 JSON #6809

Closed

dnsmichi assigned Al2Klimov Mar 12, 2019

Al2Klimov mentioned this issue Mar 15, 2019

Replace YAJL with nlohmann::json #7019

Merged

dnsmichi closed this as completed in #7019 Mar 18, 2019

Elias481 mentioned this issue Apr 10, 2019

Logrotate issues since 2.10.1-1 #6737

Closed

Elias481 mentioned this issue Apr 25, 2019

SIGBUS crash at "Restarting after configuration change." #7132

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Main process crashes on Json::Encode (third-party/yajl) #6684

Main process crashes on Json::Encode (third-party/yajl) #6684

miso231 commented Oct 15, 2018

dnsmichi commented Oct 17, 2018

pimperator commented Nov 27, 2018 •

edited

Loading

pimperator commented Nov 27, 2018

dnsmichi commented Mar 12, 2019

Al2Klimov commented Mar 12, 2019

dnsmichi commented Mar 13, 2019

Al2Klimov commented Mar 13, 2019

dnsmichi commented Mar 18, 2019

Main process crashes on Json::Encode (third-party/yajl) #6684

Main process crashes on Json::Encode (third-party/yajl) #6684

Comments

miso231 commented Oct 15, 2018

Current Behavior

Steps to Reproduce (for bugs)

Context

Your Environment

dnsmichi commented Oct 17, 2018

pimperator commented Nov 27, 2018 • edited Loading

pimperator commented Nov 27, 2018

dnsmichi commented Mar 12, 2019

Al2Klimov commented Mar 12, 2019

dnsmichi commented Mar 13, 2019

Al2Klimov commented Mar 13, 2019

dnsmichi commented Mar 18, 2019

pimperator commented Nov 27, 2018 •

edited

Loading