Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

use_grpc throws "Segmentation fault" error after upgrade from 0.12.4 to 0.12.5 #488

Open
jagdish-176 opened this issue Apr 13, 2022 · 2 comments
Open

use_grpc throws "Segmentation fault" error after upgrade from 0.12.4 to 0.12.5 #488

jagdish-176 opened this issue Apr 13, 2022 · 2 comments
Assignees
@igorpeshansky

Comments

@jagdish-176
Copy link

jagdish-176 commented Apr 13, 2022 •
edited by igorpeshansky
Loading

I'm using the below configuration for plugin.
this is working with 0.12.4 version but not the latest 0.12.5

    @type google_cloud
    ...
    use_metadata_service false
    buffer_type file
    buffer_path 'C:\xxx\other_logs'
    buffer_chunk_limit 5120KB
    flush_interval 5s
    disable_retry_limit false
    retry_limit 3
    retry_wait 10
    max_retry_wait 300
    num_threads 8
    use_grpc true
    partial_success true

Errors as below:

C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-google-cloud-0.12.5/lib/fluent/plugin/out_google_cloud.rb:1789: [BUG] Segmentation fault
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x64-mingw32]

-- Control frame information -----------------------------------------------
c:0023 p:---- s:0145 e:000144 CFUNC  :new
c:0022 p:0155 s:0141 e:000140 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-google-cloud-0.12.5/lib/fluent/plugin/out_google_cloud.rb:1789
c:0021 p:0008 s:0127 e:000126 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-google-cloud-0.12.5/lib/fluent/plugin/out_google_cloud.rb:638
c:0020 p:0004 s:0123 e:000122 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/compat/call_super_mixin.rb:42
c:0019 p:0011 s:0119 e:000118 BLOCK  C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:200
c:0018 p:0007 s:0115 e:000114 BLOCK  C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:189 [FINISH]
c:0017 p:---- s:0111 e:000110 CFUNC  :each
c:0016 p:0069 s:0107 e:000106 BLOCK  C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:188 [FINISH]
c:0015 p:---- s:0100 e:000099 CFUNC  :each
c:0014 p:0049 s:0096 e:000095 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:175
c:0013 p:0006 s:0088 e:000087 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:199
c:0012 p:0005 s:0084 e:000083 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/engine.rb:248
c:0011 p:0035 s:0080 e:000079 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/engine.rb:147
c:0010 p:0120 s:0075 e:000074 BLOCK  C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/supervisor.rb:715
c:0009 p:0109 s:0072 e:000071 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/supervisor.rb:966
c:0008 p:0159 s:0065 e:000064 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/supervisor.rb:706
c:0007 p:1910 s:0061 E:000460 TOP    C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/command/fluentd.rb:364 [FINISH]
c:0006 p:---- s:0038 e:000037 CFUNC  :require
c:0005 p:0111 s:0033 e:000032 METHOD C:/opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72
c:0004 p:0096 s:0017 e:000016 TOP    C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/bin/fluentd:15 [FINISH]
c:0003 p:---- s:0013 e:000012 CFUNC  :load
c:0002 p:0112 s:0008 E:001020 EVAL   C:/opt/td-agent/bin/fluentd:23 [FINISH]
c:0001 p:0000 s:0003 E:000df0 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
C:/opt/td-agent/bin/fluentd:23:in `<main>'
C:/opt/td-agent/bin/fluentd:23:in `load'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/bin/fluentd:15:in `<top (required)>'
C:/opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
C:/opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/command/fluentd.rb:364:in `<top (required)>'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/supervisor.rb:706:in `run_worker'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/supervisor.rb:966:in `main_process'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/supervisor.rb:715:in `block in run_worker'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/engine.rb:147:in `run'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/engine.rb:248:in `start'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:199:in `start'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:175:in `lifecycle'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:175:in `each'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:188:in `block in lifecycle'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:188:in `each'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:200:in `block in start'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/compat/call_super_mixin.rb:42:in `start'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-google-cloud-0.12.5/lib/fluent/plugin/out_google_cloud.rb:638:in `start'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-google-cloud-0.12.5/lib/fluent/plugin/out_google_cloud.rb:1789:in `init_api_client'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-google-cloud-0.12.5/lib/fluent/plugin/out_google_cloud.rb:1789:in `new'

-- C level backtrace information -------------------------------------------
C:\Windows\SYSTEM32\ntdll.dll(NtWaitForSingleObject+0x14) [0x00007fff67b8fa74]
C:\Windows\System32\KERNELBASE.dll(WaitForSingleObjectEx+0x93) [0x00007fff63f385c3]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_bugreport+0x37b) [0x00000000653c96ab]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_bug_for_fatal_signal+0x7f) [0x00000000651e432f]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(ruby_sysinit+0x790) [0x0000000065322030]
 [0x0000000000402436]
C:\Windows\System32\msvcrt.dll(_C_specific_handler+0x98) [0x00007fff675e8068]
C:\Windows\SYSTEM32\ntdll.dll(_chkstk+0x11f) [0x00007fff67b94a2f]
C:\Windows\SYSTEM32\ntdll.dll(RtlWalkFrameChain+0x14bf) [0x00007fff67af4cef]
C:\Windows\SYSTEM32\ntdll.dll(KiUserExceptionDispatcher+0x2e) [0x00007fff67b9379e]
 [0xffffffffa1470000]
C:\opt\td-agent\lib\ruby\gems\2.7.0\gems\grpc-1.45.0-x64-mingw32\grpc_c.64.ruby(Z15grpc_iomgr_initv+0x6a) [0x0000000009ac8fda]
C:\opt\td-agent\lib\ruby\gems\2.7.0\gems\grpc-1.45.0-x64-mingw32\grpc_c.64.ruby(grpc_init+0x86) [0x0000000009b54066]
 [0x000000007104835c]
 [0x000000007104664f]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_undefined_alloc+0xc0) [0x0000000065294800]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_error_arity+0x338) [0x00000000653a1bb8]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_invoke_bmethod+0x131b) [0x00000000653adb0b]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_check_funcall_default+0x8f6) [0x00000000653ba126]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_exec+0x23c) [0x00000000653abf2c]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_yield+0x307) [0x00000000653b03b7]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_ary_each+0x3d) [0x0000000065145afd]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_error_arity+0x338) [0x00000000653a1bb8]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_check_funcall_default+0x9ee) [0x00000000653ba21e]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_exec+0x23c) [0x00000000653abf2c]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_yield+0x307) [0x00000000653b03b7]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_ary_each+0x3d) [0x0000000065145afd]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_error_arity+0x338) [0x00000000653a1bb8]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_invoke_bmethod+0x131b) [0x00000000653adb0b]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_check_funcall_default+0x9ee) [0x00000000653ba21e]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_exec+0x9e5) [0x00000000653ac6d5]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_get_expanded_load_path+0x13f1) [0x0000000065250251]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_f_require+0x22) [0x0000000065250412]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_error_arity+0x338) [0x00000000653a1bb8]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_invoke_bmethod+0x131b) [0x00000000653adb0b]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_check_funcall_default+0x8f6) [0x00000000653ba126]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_exec+0x9e5) [0x00000000653ac6d5]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_resolve_feature_path+0x80a) [0x000000006524ecda]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_error_arity+0x338) [0x00000000653a1bb8]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_invoke_bmethod+0x131b) [0x00000000653adb0b]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_check_funcall_default+0x8f6) [0x00000000653ba126]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_exec+0x9e5) [0x00000000653ac6d5]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_call_end_proc+0x152) [0x00000000651e88f2]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(ruby_run_node+0x95) [0x00000000651ed815]
 [0x0000000000402d49]
 [0x00000000004013b4]
 [0x000000000040150b]
C:\Windows\System32\KERNEL32.DLL(BaseThreadInitThunk+0x14) [0x00007fff65c87974]

-- Other runtime information -----------------------------------------------
@jagdish-176
Copy link
Author

Can anyone help into this, currently we have downgraded to lower version.

@igorpeshansky igorpeshansky self-assigned this May 26, 2022
@igorpeshansky
Copy link
Member

Between 0.12.4 and 0.12.5, the grpc gem was upgraded from 1.31.1 to 1.45.0, which likely introduced this instability. Since we regularly test on Windows with use_grpc enabled, there must be some other factor in your setup that's triggering this. Any information you can provide that would help us reproduce the issue would be helpful.

Line 1789 of out_google_cloud.rb creates a new GRPC::Core::ChannelCredentials object. However, the C stack trace indicates that it crashes in grpc_init (more specifically, in grpc_iomgr_init()), so the credentials thing may be a red herring.

Nonetheless, what kind of credentials are you using to authorize the plugin? What version of Windows do you observe this on? In what environment/platform? Do you see this on Linux as well with the same credentials? Also, from your paths, it looks like you're adding this plugin into an upstream td-agent installation. This plugin is not really supported outside of the Google Cloud logging agent package — do you observe the same crash with that package?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@igorpeshansky igorpeshansky

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jagdish-176 @igorpeshansky