bug: daemon dies unexpectedly - Error: read ECONNRESET (Re-Opened)

[unplugged216]

Duplicate of #3398

Describe the bug

While running Haraka in daemon mode, after some time, hours, the daemon dies with a very short message.

Expected behavior

The expected behavior would be to not die... However, more clearly would be to log the issue and move along as in a daemon environment no human eyes are babysitting it to ensure it does stop.

Observed behavior

While the daemon is running, the daemon will die and post to the log the following output.

[CRIT] [-] [core] Error: read ECONNRESET
[CRIT] [-] [core]     at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
[NOTICE] [-] [core] Shutting down
[NOTICE] [-] [core] Shutting down

Steps To Reproduce

Frankly, I enabled demonize in smtp.ini and ran it. I let it run over night and sometime in the evening it dies. This happens during the day as well. So far, every 6 hours or so.

System Info

Haraka	Haraka.js — Version: 3.0.4
Node	v20.17.0
OS	Linux haraka 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
openssl	OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

Additional context

I would not be trying to run this as a daemon indefinitely if I could get the service to work. There is a whole lot of documentation missing from hark.service to drop it in and go lol. I am trying to work through it but omg this one is rough. Im likely missing something obvious but yeah.

[unplugged216]

For clarification, this issue was marked resolved before while I mentioned I would test the proposed solution. The proposed solution did not workout as anticipated.

We are now collecting debug logs to try and identify the issue.

[unplugged216]

Well, 0 new insights lol. With debugging on it produce no new information surrounding the crash. In this cash all three workers crashed or rather the 2 workers and the supervisor.

2024-09-04T01:38:58.755Z [NOTICE] [FBA932CF-6BF2-47E9-96CF-6A9363714517.1] [core] disconnect ip=200.105.166.13 rdns=mail.diremar.gob.bo helo=mail.diremar.gob.bo relay=N early=N esmtp=Y tls=Y pipe=Y errors=2 txns=1 rcpts=0/0/0 msgs=0/0/0 bytes=0 lr="503 MAIL required first" time=2.159
2024-09-04T01:39:47.057Z [CRIT] [-] [core] Error: read ECONNRESET
2024-09-04T01:39:47.058Z [CRIT] [-] [core]     at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
2024-09-04T01:39:47.058Z [NOTICE] [-] [core] Shutting down
2024-09-04T01:39:47.068Z [NOTICE] [-] [core] Shutting down
2024-09-04T01:39:47.068Z [NOTICE] [-] [core] Shutting down

Any insight would be help. I am having a rough time troubleshooting this.

[lnedry]

I am also seeing this issue a few times a month. I'll be happy to troubleshoot; I just don't know where to start. I've looked at the connections preceding the crash but nothing stands out.

I guess my first question is how can I determine which connection caused this?

Sep 16 05:07:51 daisy haraka[2360302]: [CRIT] [-] [core] Error: read ECONNRESET
Sep 16 05:07:51 daisy haraka[2360302]: [CRIT] [-] [core]     at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
Sep 16 05:07:51 daisy haraka[2360302]: [NOTICE] [-] [core] Shutting down
Sep 16 05:07:51 daisy haraka[2266791]: [NOTICE] [-] [server] worker 5 exited with error code: 1
Sep 16 05:07:51 daisy haraka[2266791]: [NOTICE] [-] [server] worker started worker=8 pid=2559968

Haraka	Haraka.js — Version: 3.0.4
Node	v20.15.0
OS	Linux daisy.bluestreak.net 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux
openssl	OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)

[msimerson]

I recently ran into a similar (maybe the same) issue. I didn't have time to figure out what's going on and fix it, but I have temporarily resolved it by disabling OCSP.

diff --git a/tls_socket.js b/tls_socket.js
index 9fcbb6e8..1b29acb1 100644
--- a/tls_socket.js
+++ b/tls_socket.js
@@ -538,6 +538,7 @@ exports.ensureDhparams = done => {
 }
 
 exports.addOCSP = server => {
+    return;
     if (!ocsp) {
         log.debug(`addOCSP: 'ocsp' not available`);
         return;

# node -v
v20.17.0
# uname -a
FreeBSD haraka 14.1-RELEASE-p5