segfault in TLS #1696

EricTheOne · 2015-05-13T17:48:23Z

Reproduces 100% of the time within a minute of running the server. Server opens multiple TLS client connections (as a client).

I'll gladly help with testing a fix or sharing some of the backtrace data, but cannot share server code or specific scenario.

Please see nodejs/node-v0.x-archive#8780, I'm not 100% sure but I think it may be the same issue. I'm unable to reproduce it in node 0.12.2 (server is running fine, the segfault doesn't happen).

Backtrace (commit 7693705):

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000069eed0 in SSL_read ()
(gdb) backtrace
#0  0x000000000069eed0 in SSL_read ()
#1  0x0000000000d40a39 in node::TLSWrap::ClearOut() ()
#2  0x0000000000d4117e in node::TLSWrap::OnReadImpl(long, uv_buf_t const*, uv_handle_type, void*) ()
#3  0x0000000000ce109b in node::JSStream::ReadBuffer(v8::FunctionCallbackInfo<v8::Value> const&) ()
#4  0x0000000000850002 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#5  0x000000000087721b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
#6  0x00002307b4e060bb in ?? ()
#7  0x00003c26fce690f9 in ?? ()
#8  0x00002307b4e06001 in ?? ()
#9  0x00007ffe6b7e47f0 in ?? ()
#10 0x00007ffe6b7e4850 in ?? ()
#11 0x00002307b4e99739 in ?? ()
#12 0x00002bb54efbee61 in ?? ()
#13 0x00003c26fce6b669 in ?? ()
...

dmesg:

segfault at 30 ip 000000000069eed0 sp 00007ffe6b7e41a8 error 4 in iojs[400000+d0e000]

Although nodejs/node-v0.x-archive#8780 contains a suggested fix, I don't see that it was applied in node.js 0.12.2, so either something else fixed it or the situation is different.

Naively applying the fix suggested in 8780 on io.js 2.0.1 (18d457b) doesn't seem to help. Instead I'm failing on an assertion.

"Fix" (tls_wrap.cc):

if (wrap->enc_out_) {
    NodeBIO::FromBIO(wrap->enc_out_)->Read(nullptr, wrap->write_size_);
    // Ensure that the progress will be made and `InvokeQueued` will be called.
    wrap->ClearIn();
}

Backtrace with fix applied:

Program terminated with signal SIGABRT, Aborted.
#0  0x00007fc17f768cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) backtrace
#0  0x00007fc17f768cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007fc17f76c0d8 in __GI_abort () at abort.c:89
#2  0x00007fc17f761b86 in __assert_fail_base (fmt=0x7fc17f8b2830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xf7d1f9 "(bio->ptr) != (nullptr)", file=file@entry=0xf7d1e0 "../src/node_crypto_bio.h", line=line@entry=77, 
    function=function@entry=0xf7e080 <_ZZN4node7NodeBIO7FromBIOEP6bio_stE19__PRETTY_FUNCTION__> "static node::NodeBIO* node::NodeBIO::FromBIO(BIO*)") at assert.c:92
#3  0x00007fc17f761c32 in __GI___assert_fail (assertion=0xf7d1f9 "(bio->ptr) != (nullptr)", file=0xf7d1e0 "../src/node_crypto_bio.h", line=77, 
    function=0xf7e080 <_ZZN4node7NodeBIO7FromBIOEP6bio_stE19__PRETTY_FUNCTION__> "static node::NodeBIO* node::NodeBIO::FromBIO(BIO*)") at assert.c:101
#4  0x0000000000673989 in node::NodeBIO::FromBIO(bio_st*) [clone .isra.22] [clone .part.23] ()
#5  0x0000000000d409bf in node::TLSWrap::EncOutCb(node::WriteWrap*, int) ()
#6  0x0000000000850002 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#7  0x000000000087721b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
#8  0x00003993517060bb in ?? ()
#9  0x00007fff1da78730 in ?? ()
#10 0x0000399351706001 in ?? ()
#11 0x00007fff1da78710 in ?? ()
#12 0x00007fff1da78778 in ?? ()
#13 0x0000399351a8292c in ?? ()
#14 0x00001507279beb79 in ?? ()
#15 0x0000000000000000 in ?? ()

The text was updated successfully, but these errors were encountered:

EricTheOne · 2015-05-13T17:49:03Z

@indutny

When loading session, OCSP response, SNI, always check that the `self._handle` is present. If it is not - the socket was closed - and we should emit the error instead of throwing an uncaught exception. Fix: nodejs/node-v0.x-archive#8780 Fix: nodejs#1696

indutny · 2015-05-14T10:46:13Z

May I ask you to give a try to the following patch #1702 ?

EricTheOne · 2015-05-14T12:29:33Z

@indutny gladly and thanks for the quick response. Applied the patch on io.js master (4e2f999), still getting segfault:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000069eed0 in SSL_read ()
(gdb) backtrace
#0  0x000000000069eed0 in SSL_read ()
#1  0x0000000000d40ae9 in node::TLSWrap::ClearOut() ()
#2  0x0000000000d4122e in node::TLSWrap::OnReadImpl(long, uv_buf_t const*, uv_handle_type, void*) ()
#3  0x0000000000ce109b in node::JSStream::ReadBuffer(v8::FunctionCallbackInfo<v8::Value> const&) ()
#4  0x0000000000850002 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#5  0x000000000087721b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
#6  0x00000abcd52060bb in ?? ()
#7  0x00000abcd5206001 in ?? ()
#8  0x00007fffb979c580 in ?? ()
#9  0x00007fffb979c5d0 in ?? ()
#10 0x00000abcd55a6f1a in ?? ()
#11 0x000030f334efeea9 in ?? ()
#12 0x000015aab64b2489 in ?? ()
#13 0x00001bd0488c0841 in ?? ()
#14 0x000030f334efeea9 in ?? ()
#15 0x00000b493ed87e89 in ?? ()
#16 0x00001bd04889a081 in ?? ()
#17 0x00007fffb979c6a8 in ?? ()
#18 0x00000abcd5296468 in ?? ()
#19 0x000015aab64b2489 in ?? ()
#20 0x00001bd0488addf1 in ?? ()
#21 0x00000abcd54c8824 in ?? ()
#22 0x00000abcd54c8824 in ?? ()
#23 0x00003d23e40be3c1 in ?? ()
#24 0x00002f4a2ebf7a29 in ?? ()
#25 0x00000a9d72d66921 in ?? ()
#26 0x000015aab64b2489 in ?? ()
#27 0x000011d9d60bb679 in ?? ()
#28 0x00000a9d72dab0a9 in ?? ()
#29 0x00000a9d72d8dcf1 in ?? ()
#30 0x00007fff00000001 in ?? ()
#31 0x00000abc00000002 in ?? ()
#32 0x00007fffb979c700 in ?? ()
#33 0x000011d9d60bb631 in ?? ()
#34 0x00000a9d72d04141 in ?? ()
#35 0x00000a9d72d04161 in ?? ()
#36 0x00000b493ed87e89 in ?? ()
#37 0x00000a9d72d04101 in ?? ()
#38 0x00001bd0488c06f9 in ?? ()
#39 0x00000a9d72d04161 in ?? ()
#40 0x00000a9d72dc9181 in ?? ()
#41 0x000011d9d607ccf9 in ?? ()
#42 0x00000a9d72dc9181 in ?? ()
#43 0x000011d9d607ccf9 in ?? ()
#44 0x00007fffb979c6e0 in ?? ()
#45 0x00000abcd5214b35 in ?? ()
#46 0x000012917282dcf1 in ?? ()
#47 0x00001bd0488addf1 in ?? ()
#48 0x0000000200000000 in ?? ()
#49 0x00000a9d72dc9181 in ?? ()
#50 0x0000000a00000000 in ?? ()
#51 0x00007fffb979c778 in ?? ()
#52 0x00000abcd5621a9f in ?? ()
#53 0x000015aab64b2489 in ?? ()
#54 0x000012917282dcf1 in ?? ()
#55 0x00001bd0488addf1 in ?? ()
#56 0x00000a9d72d04101 in ?? ()
#57 0x00007fffb979c758 in ?? ()
#58 0x000030f334e1c389 in ?? ()
#59 0x00007fffb979c800 in ?? ()
#60 0x0000000000ac2c48 in v8::internal::Isolate::RunMicrotasks() ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

EricTheOne · 2015-05-14T12:40:11Z

By the way, as I was testing the patch npm seemed to fail installing four different modules.
I'm running configure && make && make install on the io.js repo, anything I'm missing to get npm to work?

> [email protected] install /.../node_modules/heapdump
> node-gyp rebuild

gyp WARN install got an error, rolling back install
gyp ERR! configure error 
gyp ERR! stack Error: 404 status code downloading tarball
gyp ERR! stack     at Request.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/install.js:246:14)
gyp ERR! stack     at emitOne (events.js:82:20)
gyp ERR! stack     at Request.emit (events.js:169:7)
gyp ERR! stack     at Request.onRequestResponse (/usr/local/lib/node_modules/npm/node_modules/request/request.js:1156:10)
gyp ERR! stack     at emitOne (events.js:77:13)
gyp ERR! stack     at ClientRequest.emit (events.js:169:7)
gyp ERR! stack     at HTTPParser.parserOnIncomingClient (_http_client.js:414:21)
gyp ERR! stack     at HTTPParser.parserOnHeadersComplete (_http_common.js:88:23)
gyp ERR! stack     at TLSSocket.socketOnData (_http_client.js:304:20)
gyp ERR! stack     at emitOne (events.js:77:13)
gyp ERR! System Linux 3.xx.x-xx-generic
gyp ERR! command "/usr/local/bin/iojs" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /.../node_modules/heapdump
gyp ERR! node -v v2.0.2
gyp ERR! node-gyp -v v1.0.3
gyp ERR! not ok

* Destroy `SSL*` and friends on a next tick to make sure that we are not doing it in one of the OpenSSL callbacks * Add more checks to the C++ methods that might be invoked during destructor's pending queue cleanup Fix: nodejs/node-v0.x-archive#8780 Fix: nodejs#1696

EricTheOne · 2015-05-26T10:46:11Z

@indutny I've tested the changes on 2.1.0 as well, still segfaults:

#0  0x000000000069f060 in SSL_read ()
#1  0x0000000000d31499 in node::TLSWrap::ClearOut() ()
#2  0x0000000000d31bce in node::TLSWrap::OnReadImpl(long, uv_buf_t const*, uv_handle_type, void*) ()
#3  0x0000000000cd285b in node::JSStream::ReadBuffer(v8::FunctionCallbackInfo<v8::Value> const&) ()
#4  0x000000000084d342 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#5  0x000000000087457b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()

Any way I can help with more info?

indutny · 2015-05-26T10:47:22Z

@EricTheOne I commented about it on other issue. I really need a test case that reproduces the problem. It doesn't look like I fully understand the problem atm.

When loading session, OCSP response, SNI, always check that the `self._handle` is present. If it is not - the socket was closed - and we should emit the error instead of throwing an uncaught exception. Fix: nodejs/node-v0.x-archive#8780 Fix: nodejs#1696

* Destroy `SSL*` and friends on a next tick to make sure that we are not doing it in one of the OpenSSL callbacks * Add more checks to the C++ methods that might be invoked during destructor's pending queue cleanup Fix: nodejs/node-v0.x-archive#8780 Fix: nodejs#1696

When loading session, OCSP response, SNI, always check that the `self._handle` is present. If it is not - the socket was closed - and we should emit the error instead of throwing an uncaught exception. Fix: nodejs/node-v0.x-archive#8780 Fix: nodejs#1696

* Destroy `SSL*` and friends on a next tick to make sure that we are not doing it in one of the OpenSSL callbacks * Add more checks to the C++ methods that might be invoked during destructor's pending queue cleanup Fix: nodejs/node-v0.x-archive#8780 Fix: nodejs#1696

When loading session, OCSP response, SNI, always check that the `self._handle` is present. If it is not - the socket was closed - and we should emit the error instead of throwing an uncaught exception. Fix: nodejs/node-v0.x-archive#8780 Fix: #1696 PR-URL: #1702 Reviewed-By: Trevor Norris <[email protected]>

* Destroy `SSL*` and friends on a next tick to make sure that we are not doing it in one of the OpenSSL callbacks * Add more checks to the C++ methods that might be invoked during destructor's pending queue cleanup Fix: nodejs/node-v0.x-archive#8780 Fix: #1696 PR-URL: #1702 Reviewed-By: Trevor Norris <[email protected]>

indutny · 2015-06-06T22:39:48Z

@EricTheOne PTAL at #1910, I think it should fix the issue for you ;)

Queued write requests should be invoked on handle close, otherwise the "consumer" might be already destroyed when the write callbacks of the "consumed" handle will be invoked. Fix: nodejs#1696

Queued write requests should be invoked on handle close, otherwise the "consumer" might be already destroyed when the write callbacks of the "consumed" handle will be invoked. Same applies to the shutdown requests. Make sure to "move" away socket from server to not break the `connections` counter in `net.js`. Otherwise it might not call `close` callback, or call it too early. Fix: #1696 PR-URL: #1910 Reviewed-By: Trevor Norris <[email protected]>

Fishrock123 · 2015-07-07T19:44:47Z

Ala #1910 (comment), @EricTheOne did that patch fix this issue? :)

Queued write requests should be invoked on handle close, otherwise the "consumer" might be already destroyed when the write callbacks of the "consumed" handle will be invoked. Same applies to the shutdown requests. Make sure to "move" away socket from server to not break the `connections` counter in `net.js`. Otherwise it might not call `close` callback, or call it too early. Fix: nodejs#1696 PR-URL: nodejs#1910 Reviewed-By: Trevor Norris <[email protected]>

Fishrock123 · 2015-07-24T00:23:53Z

I'm going to assume it did. Please re-open if it didn't.

mscdex added the tls Issues and PRs related to the tls subsystem. label May 13, 2015

indutny mentioned this issue May 14, 2015

tls: emit errors on close whilst async action #1702

Closed

indutny mentioned this issue Jun 6, 2015

_stream_wrap: prevent use after free in TLS #1910

Closed

Fishrock123 closed this as completed Jul 24, 2015

snyk-bot mentioned this issue Apr 24, 2023

[Snyk] Fix for 12 vulnerabilities aliscco/alisco-node#93

Open

This was referenced Apr 25, 2023

[Snyk] Fix for 11 vulnerabilities aliscco/alisco-node#144

Open

[Snyk] Fix for 12 vulnerabilities aliscco/alisco-node#162

Open

This was referenced Apr 26, 2023

[Snyk] Fix for 12 vulnerabilities aliscco/alisco-node#211

Open

[Snyk] Fix for 12 vulnerabilities aliscco/alisco-node#241

Open

aliscco mentioned this issue Jun 26, 2023

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 aliscco/alisco-node#619

Open

leonardoadame mentioned this issue Jul 7, 2023

[Snyk] Fix for 13 vulnerabilities leonardoadame/node#69

Open

This was referenced Nov 28, 2023

[Snyk] Fix for 13 vulnerabilities aliscco/alisco-node#686

Open

[Snyk] Fix for 15 vulnerabilities aliscco/alisco-node#711

Open

This was referenced Nov 29, 2023

[Snyk] Fix for 15 vulnerabilities leonardoadame/node#225

Open

[Snyk] Fix for 13 vulnerabilities leonardoadame/node#227

Open

[Snyk] Fix for 13 vulnerabilities leonardoadame/node#232

Open

This was referenced Nov 29, 2023

[Snyk] Fix for 13 vulnerabilities aliscco/alisco-node#756

Open

[Snyk] Fix for 13 vulnerabilities aliscco/alisco-node#897

Open

[Snyk] Fix for 16 vulnerabilities aliscco/alisco-node#1125

Open

This was referenced Apr 15, 2024

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 aliscco/alisco-node#1380

Open

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 aliscco/alisco-node#1381

Open

This was referenced Apr 15, 2024

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 leonardoadame/node#775

Open

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 leonardoadame/node#776

Open

aliscco mentioned this issue Apr 16, 2024

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 aliscco/alisco-node#1387

Open

leonardoadame mentioned this issue Apr 16, 2024

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 leonardoadame/node#782

Open

This was referenced May 13, 2024

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 aliscco/alisco-node#1421

Open

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 aliscco/alisco-node#1433

Open

leonardoadame mentioned this issue May 14, 2024

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 leonardoadame/node#861

Open

This was referenced May 14, 2024

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 aliscco/alisco-node#1475

Open

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 aliscco/alisco-node#1502

Open

leonardoadame mentioned this issue May 15, 2024

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 leonardoadame/node#916

Open

aliscco mentioned this issue May 16, 2024

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 aliscco/alisco-node#1570

Open

leonardoadame mentioned this issue May 16, 2024

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 leonardoadame/node#971

Open

This was referenced May 17, 2024

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 aliscco/alisco-node#1607

Open

[Snyk] Security upgrade gulp from 3.9.1 to 4.0.0 aliscco/alisco-node#1611

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

segfault in TLS #1696

segfault in TLS #1696

EricTheOne commented May 13, 2015

EricTheOne commented May 13, 2015

indutny commented May 14, 2015

EricTheOne commented May 14, 2015

EricTheOne commented May 14, 2015

EricTheOne commented May 26, 2015

indutny commented May 26, 2015

indutny commented Jun 6, 2015

Fishrock123 commented Jul 7, 2015

Fishrock123 commented Jul 24, 2015

segfault in TLS #1696

segfault in TLS #1696

Comments

EricTheOne commented May 13, 2015

EricTheOne commented May 13, 2015

indutny commented May 14, 2015

EricTheOne commented May 14, 2015

EricTheOne commented May 14, 2015

EricTheOne commented May 26, 2015

indutny commented May 26, 2015

indutny commented Jun 6, 2015

Fishrock123 commented Jul 7, 2015

Fishrock123 commented Jul 24, 2015