Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

It stops quite often with Error 502 nginx #53

Closed
tio-trom opened this issue Aug 22, 2021 · 19 comments
Closed

It stops quite often with Error 502 nginx #53

tio-trom opened this issue Aug 22, 2021 · 19 comments

Comments

@tio-trom
Copy link

tio-trom commented Aug 22, 2021

  • Hardware: VPS bought online
  • YunoHost version: 4.2.8 (stable).
  • I have access to my server: Through SSH | through the webadmin | direct access via keyboard / screen | ...
  • Are you in a special context or did you perform some particular tweaking on your YunoHost instance?: *no *

The application stops quite often and randomly - I can see this error:

Aug 21 21:47:15 invidious[331]: Invalid memory access (signal 11) at address 0xe9
Aug 21 21:47:15 invidious[331]: [0x55a508e61036] print_backtrace at /usr/share/crystal/src/exception/call_stack.cr:129:5
Aug 21 21:47:15 invidious[331]: [0x55a508ab5ce2] __crystal_sigfault_handler at /usr/share/crystal/src/signal.cr:348:3
Aug 21 21:47:15 invidious[331]: [0x7f7a1ffa4730] ???
Aug 21 21:47:15 invidious[331]: [0x55a508ec23a4] lsquic_conn_set_ctx +4
Aug 21 21:47:15 invidious[331]: [0x55a508b7e4fe] -> at /var/www/invidious/lib/lsquic/src/lsquic/client.cr:210:11
Aug 21 21:47:15 invidious[331]: [0x55a508ab06bd] run at /usr/share/crystal/src/primitives.cr:255:3
Aug 21 21:47:15 invidious[331]: [0x0] ???
Aug 21 21:47:15 systemd[1]: invidious.service: Main process exited, code=exited, status=11/n/a
Aug 21 21:47:15 systemd[1]: invidious.service: Failed with result 'exit-code'.
Aug 21 21:47:17 systemd[1]: invidious.service: Service RestartSec=2s expired, scheduling restart.
Aug 21 21:47:17 systemd[1]: invidious.service: Scheduled restart job, restart counter is at 43.
Aug 21 21:47:17 systemd[1]: Stopped Invidious (An alternative YouTube front-end).
Aug 21 21:47:17 systemd[1]: Started Invidious (An alternative YouTube front-end).


@tio-trom tio-trom changed the title It restarts quite often It restarts quite often with Error 502 nginx Aug 31, 2021
@tio-trom tio-trom changed the title It restarts quite often with Error 502 nginx It stops quite often with Error 502 nginx Sep 2, 2021
@tio-trom
Copy link
Author

tio-trom commented Sep 2, 2021

Here are more errors I could find:

Sep 02 14:47:44 invidious[23468]: Unhandled exception in spawn: Error while writing data to the client (HTTP::Server::ClientError)
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/http/server/response.cr:206:9 in 'unbuffered_write'
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/compress/deflate/writer.cr:94:7 in 'consume_output'
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/compress/deflate/writer.cr:73:11 in 'close'
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/compress/gzip/writer.cr:104:5 in 'close'
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/http/server/request_processor.cr:63:11 in 'process'
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/http/server.cr:498:5 in '->'
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
Sep 02 14:47:44 invidious[23468]:   from ???
Sep 02 14:47:44 invidious[23468]: Caused by: Error writing to socket: Broken pipe (IO::Error)
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/io/evented.cr:156:7 in 'unbuffered_write'
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/io/buffered.cr:219:5 in 'write'
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/http/server/response.cr:220:11 in 'unbuffered_write'
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/compress/deflate/writer.cr:94:7 in 'consume_output'
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/compress/deflate/writer.cr:73:11 in 'close'
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/compress/gzip/writer.cr:104:5 in 'close'
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/http/server/request_processor.cr:63:11 in 'process'
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/http/server.cr:498:5 in '->'
Sep 02 14:47:44 invidious[23468]:   from /usr/share/crystal/src/primitives.cr:255:3 in 'run'

@ericgaspar
Copy link
Member

Can you test this branch? sudo yunohost app install https://github.com/YunoHost-Apps/invidious_ynh/tree/patch

@tio-trom
Copy link
Author

tio-trom commented Sep 7, 2021

For sure! Doing it now!

@tio-trom
Copy link
Author

tio-trom commented Sep 7, 2021

https://ytb.trom.tf/ I got no errors but the instance does not work. Logs here https://paste.yunohost.org/raw/xuhafifudo

@ericgaspar
Copy link
Member

ericgaspar commented Sep 7, 2021

Strange... I am able to install and use it. Have you restart NGINX and Invidious services?

@tio-trom
Copy link
Author

tio-trom commented Sep 7, 2021

Ah ok I had to manually restart those. Works now. Let me test to see if it still keeps on restarting. Thanks!

@tio-trom
Copy link
Author

tio-trom commented Sep 7, 2021

So far works really well. Didn't interrupt any streams for the past hour, however I still see invidious restarting:

admin@tromland:~$ sudo journalctl -u invidious -n 100 --no-pager --no-hostname
-- Logs begin at Sun 2021-09-05 13:19:44 CEST, end at Tue 2021-09-07 17:19:23 CEST. --
Sep 07 15:45:40 invidious[18927]: [0x7ff273e3d730] ???
Sep 07 15:45:40 invidious[18927]: [0x563115c18018] ???
Sep 07 15:45:40 systemd[1]: invidious.service: Main process exited, code=exited, status=11/n/a
Sep 07 15:45:40 systemd[1]: invidious.service: Failed with result 'exit-code'.
Sep 07 15:45:43 systemd[1]: invidious.service: Service RestartSec=2s expired, scheduling restart.
Sep 07 15:45:43 systemd[1]: invidious.service: Scheduled restart job, restart counter is at 3.
Sep 07 15:45:43 systemd[1]: Stopped Invidious (An alternative YouTube front-end).
Sep 07 15:45:43 systemd[1]: Started Invidious (An alternative YouTube front-end).
Sep 07 15:50:59 invidious[23865]: Unhandled exception in spawn: Error while writing data to the client (HTTP::Server::ClientError)
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/http/server/response.cr:206:9 in 'unbuffered_write'
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/compress/deflate/writer.cr:94:7 in 'consume_output'
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/compress/deflate/writer.cr:73:11 in 'close'
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/compress/gzip/writer.cr:104:5 in 'close'
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/http/server/request_processor.cr:63:11 in 'process'
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/http/server.cr:498:5 in '->'
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
Sep 07 15:50:59 invidious[23865]:   from ???
Sep 07 15:50:59 invidious[23865]: Caused by: Error writing to socket: Broken pipe (IO::Error)
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/io/evented.cr:156:7 in 'unbuffered_write'
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/io/buffered.cr:219:5 in 'write'
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/http/server/response.cr:220:11 in 'unbuffered_write'
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/compress/deflate/writer.cr:94:7 in 'consume_output'
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/compress/deflate/writer.cr:73:11 in 'close'
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/compress/gzip/writer.cr:104:5 in 'close'
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/http/server/request_processor.cr:63:11 in 'process'
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/http/server.cr:498:5 in '->'
Sep 07 15:50:59 invidious[23865]:   from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
Sep 07 15:50:59 invidious[23865]:   from ???
Sep 07 16:00:11 systemd[1]: Stopping Invidious (An alternative YouTube front-end)...
Sep 07 16:00:11 systemd[1]: invidious.service: Main process exited, code=killed, status=15/TERM
Sep 07 16:00:11 systemd[1]: invidious.service: Succeeded.
Sep 07 16:00:11 systemd[1]: Stopped Invidious (An alternative YouTube front-end).
Sep 07 16:03:16 systemd[1]: Started Invidious (An alternative YouTube front-end).
Sep 07 16:21:34 systemd[1]: Stopping Invidious (An alternative YouTube front-end)...
Sep 07 16:21:34 systemd[1]: invidious.service: Main process exited, code=killed, status=15/TERM
Sep 07 16:21:34 systemd[1]: invidious.service: Succeeded.
Sep 07 16:21:34 systemd[1]: Stopped Invidious (An alternative YouTube front-end).
Sep 07 16:21:34 systemd[1]: Started Invidious (An alternative YouTube front-end).
Sep 07 16:24:54 invidious[24578]: Unhandled exception in spawn: Error while writing data to the client (HTTP::Server::ClientError)
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/http/server/response.cr:227:9 in 'unbuffered_write'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/io/buffered.cr:152:14 in 'write'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/compress/deflate/writer.cr:94:7 in 'consume_output'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/compress/deflate/writer.cr:71:5 in 'close'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/compress/gzip/writer.cr:104:5 in 'close'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/http/server/request_processor.cr:63:11 in 'process'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/http/server.cr:513:5 in 'handle_client'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/http/server.cr:468:13 in '->'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/fiber.cr:92:34 in '->'
Sep 07 16:24:54 invidious[24578]:   from ???
Sep 07 16:24:54 invidious[24578]: Caused by: Error writing to socket: Broken pipe (IO::Error)
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/io/evented.cr:82:13 in 'unbuffered_write'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/io/buffered.cr:217:5 in 'flush'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/io/buffered.cr:151:7 in 'write'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/http/server/response.cr:220:11 in 'unbuffered_write'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/io/buffered.cr:152:14 in 'write'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/compress/deflate/writer.cr:94:7 in 'consume_output'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/compress/deflate/writer.cr:71:5 in 'close'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/compress/gzip/writer.cr:104:5 in 'close'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/http/server/request_processor.cr:63:11 in 'process'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/http/server.cr:513:5 in 'handle_client'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/http/server.cr:468:13 in '->'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
Sep 07 16:24:54 invidious[24578]:   from /usr/share/crystal/src/fiber.cr:92:34 in '->'
Sep 07 16:24:54 invidious[24578]:   from ???
Sep 07 16:43:36 invidious[24578]: Invalid memory access (signal 11) at address 0x109
Sep 07 16:43:36 invidious[24578]: [0x563373e37916] print_backtrace at /usr/share/crystal/src/exception/call_stack.cr:129:5
Sep 07 16:43:36 invidious[24578]: [0x563373b2fdac] __crystal_sigfault_handler at /usr/share/crystal/src/signal.cr:347:3
Sep 07 16:43:36 invidious[24578]: [0x7fd6a5b5f730] ???
Sep 07 16:43:36 invidious[24578]: [0x563374613764] lsquic_conn_set_ctx +4
Sep 07 16:43:36 invidious[24578]: [0x563373b97d60] -> at /var/www/invidious/lib/lsquic/src/lsquic/client.cr:210:11
Sep 07 16:43:36 invidious[24578]: [0x563373f17c19] run at /usr/share/crystal/src/primitives.cr:255:3
Sep 07 16:43:36 invidious[24578]: [0x563373b2f2b6] -> at /usr/share/crystal/src/fiber.cr:92:34
Sep 07 16:43:36 invidious[24578]: [0x0] ???
Sep 07 16:43:36 systemd[1]: invidious.service: Main process exited, code=exited, status=11/n/a
Sep 07 16:43:36 systemd[1]: invidious.service: Failed with result 'exit-code'.
Sep 07 16:43:38 systemd[1]: invidious.service: Service RestartSec=2s expired, scheduling restart.
Sep 07 16:43:38 systemd[1]: invidious.service: Scheduled restart job, restart counter is at 1.
Sep 07 16:43:38 systemd[1]: Stopped Invidious (An alternative YouTube front-end).
Sep 07 16:43:38 systemd[1]: Started Invidious (An alternative YouTube front-end).
Sep 07 16:46:01 systemd[1]: Stopping Invidious (An alternative YouTube front-end)...
Sep 07 16:46:01 systemd[1]: invidious.service: Main process exited, code=killed, status=15/TERM
Sep 07 16:46:01 systemd[1]: invidious.service: Succeeded.
Sep 07 16:46:01 systemd[1]: Stopped Invidious (An alternative YouTube front-end).
Sep 07 16:46:01 systemd[1]: Started Invidious (An alternative YouTube front-end).
Sep 07 17:13:58 invidious[13263]: Invalid memory access (signal 11) at address 0x80
Sep 07 17:13:58 invidious[13263]: [0x55e8e7669916] print_backtrace at /usr/share/crystal/src/exception/call_stack.cr:129:5
Sep 07 17:13:58 invidious[13263]: [0x55e8e7361dac] __crystal_sigfault_handler at /usr/share/crystal/src/signal.cr:347:3
Sep 07 17:13:58 invidious[13263]: [0x7ff18fb9f730] ???
Sep 07 17:13:58 invidious[13263]: [0x55e8e7e5315c] ???
Sep 07 17:13:58 invidious[13263]: [0x55e8e73c9d6d] -> at /var/www/invidious/lib/lsquic/src/lsquic/client.cr:211:11
Sep 07 17:13:58 invidious[13263]: [0x55e8e7749c19] run at /usr/share/crystal/src/primitives.cr:255:3
Sep 07 17:13:58 invidious[13263]: [0x55e8e73612b6] -> at /usr/share/crystal/src/fiber.cr:92:34
Sep 07 17:13:58 invidious[13263]: [0x0] ???
Sep 07 17:13:58 systemd[1]: invidious.service: Main process exited, code=exited, status=11/n/a
Sep 07 17:13:58 systemd[1]: invidious.service: Failed with result 'exit-code'.
Sep 07 17:14:00 systemd[1]: invidious.service: Service RestartSec=2s expired, scheduling restart.
Sep 07 17:14:00 systemd[1]: invidious.service: Scheduled restart job, restart counter is at 1.
Sep 07 17:14:00 systemd[1]: Stopped Invidious (An alternative YouTube front-end).
Sep 07 17:14:00 systemd[1]: Started Invidious (An alternative YouTube front-end).

But as said no streams stopped for the past hour. It is also that this is a new install of invidious. Could be that once the database is larger and larger, and more users using it, it will start to interrupt the streams again?

@tio-trom
Copy link
Author

tio-trom commented Sep 7, 2021

Another thing, my system asks to update to Crystal version 1.1.1-1+4.1. Maybe that can break it? I will try not to update it. And, maybe the restart is a scheduled restart for invidious that you may have added in the configs? Maybe that's why it rebooted?

@ericgaspar
Copy link
Member

Crystal 1.0.0 was released as a .deb and this is the one we are using in the package. The newer crystall version only provides the .tar.gz package... : /
My system is not showing an update for crystal (yet), it should be worth trying to update to the latest version... I will have a go when Crystal version 1.1.1-1+4.1. pop out in the package to update liste.
As for the restart timer so far, there isn't one in the app script...

@tio-trom
Copy link
Author

tio-trom commented Sep 7, 2021

Ok then. I am going to test with the current crystal version today and tomorrow update to the new one. Just to see if we notice any difference. And oh, ok so there is no scheduled restart process for invidious. Good to know. Wonder why it rebooted then...

@tio-trom
Copy link
Author

tio-trom commented Sep 7, 2021

I saw again the 502 nginx error:

sudo journalctl -u invidious -n 100 --no-pager --no-hostname
-- Logs begin at Sun 2021-09-05 13:19:44 CEST, end at Tue 2021-09-07 18:27:43 CEST. --
Sep 07 18:12:16 invidious[28107]:   from ???
Sep 07 18:18:26 invidious[28107]: Invalid memory access (signal 11) at address 0xa8
Sep 07 18:18:26 invidious[28107]: [0x563a27e15916] print_backtrace at /usr/share/crystal/src/exception/call_stack.cr:129:5
Sep 07 18:18:26 invidious[28107]: [0x563a27b0ddac] __crystal_sigfault_handler at /usr/share/crystal/src/signal.cr:347:3
Sep 07 18:18:26 invidious[28107]: [0x7fd34a285730] ???
Sep 07 18:18:26 invidious[28107]: [0x563a285f1764] lsquic_conn_set_ctx +4
Sep 07 18:18:26 invidious[28107]: [0x563a27b75d60] -> at /var/www/invidious/lib/lsquic/src/lsquic/client.cr:210:11
Sep 07 18:18:26 invidious[28107]: [0x563a27ef5c19] run at /usr/share/crystal/src/primitives.cr:255:3
Sep 07 18:18:26 invidious[28107]: [0x563a27b0d2b6] -> at /usr/share/crystal/src/fiber.cr:92:34
Sep 07 18:18:26 invidious[28107]: [0x0] ???
Sep 07 18:18:26 systemd[1]: invidious.service: Main process exited, code=exited, status=11/n/a
Sep 07 18:18:26 systemd[1]: invidious.service: Failed with result 'exit-code'.
Sep 07 18:18:28 systemd[1]: invidious.service: Service RestartSec=2s expired, scheduling restart.
Sep 07 18:18:28 systemd[1]: invidious.service: Scheduled restart job, restart counter is at 4.
Sep 07 18:18:28 systemd[1]: Stopped Invidious (An alternative YouTube front-end).
Sep 07 18:18:28 systemd[1]: Started Invidious (An alternative YouTube front-end).
Sep 07 18:20:08 invidious[1428]: Invalid memory access (signal 11) at address 0xa8
Sep 07 18:20:08 invidious[1428]: [0x558a47957916] print_backtrace at /usr/share/crystal/src/exception/call_stack.cr:129:5
Sep 07 18:20:08 invidious[1428]: [0x558a4764fdac] __crystal_sigfault_handler at /usr/share/crystal/src/signal.cr:347:3
Sep 07 18:20:08 invidious[1428]: [0x7fc68a41b730] ???
Sep 07 18:20:08 invidious[1428]: [0x558a48133764] lsquic_conn_set_ctx +4
Sep 07 18:20:08 invidious[1428]: [0x558a476b7d60] -> at /var/www/invidious/lib/lsquic/src/lsquic/client.cr:210:11
Sep 07 18:20:08 invidious[1428]: [0x558a47a37c19] run at /usr/share/crystal/src/primitives.cr:255:3
Sep 07 18:20:08 invidious[1428]: [0x558a4764f2b6] -> at /usr/share/crystal/src/fiber.cr:92:34
Sep 07 18:20:08 invidious[1428]: [0x0] ???
Sep 07 18:20:08 systemd[1]: invidious.service: Main process exited, code=exited, status=11/n/a
Sep 07 18:20:08 systemd[1]: invidious.service: Failed with result 'exit-code'.
Sep 07 18:20:10 systemd[1]: invidious.service: Service RestartSec=2s expired, scheduling restart.
Sep 07 18:20:10 systemd[1]: invidious.service: Scheduled restart job, restart counter is at 5.
Sep 07 18:20:10 systemd[1]: Stopped Invidious (An alternative YouTube front-end).
Sep 07 18:20:10 systemd[1]: Started Invidious (An alternative YouTube front-end).
Sep 07 18:20:26 invidious[3030]: Unhandled exception in spawn: Error while writing data to the client (HTTP::Server::ClientError)
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/http/server/response.cr:227:9 in 'unbuffered_write'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/io/buffered.cr:152:14 in 'write'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/compress/deflate/writer.cr:94:7 in 'consume_output'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/compress/deflate/writer.cr:71:5 in 'close'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/compress/gzip/writer.cr:104:5 in 'close'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/http/server/request_processor.cr:63:11 in 'process'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/http/server.cr:513:5 in 'handle_client'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/http/server.cr:468:13 in '->'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/fiber.cr:92:34 in '->'
Sep 07 18:20:26 invidious[3030]:   from ???
Sep 07 18:20:26 invidious[3030]: Caused by: Error writing to socket: Broken pipe (IO::Error)
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/io/evented.cr:82:13 in 'unbuffered_write'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/io/buffered.cr:217:5 in 'flush'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/io/buffered.cr:151:7 in 'write'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/http/server/response.cr:220:11 in 'unbuffered_write'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/io/buffered.cr:152:14 in 'write'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/compress/deflate/writer.cr:94:7 in 'consume_output'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/compress/deflate/writer.cr:71:5 in 'close'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/compress/gzip/writer.cr:104:5 in 'close'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/http/server/request_processor.cr:63:11 in 'process'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/http/server.cr:513:5 in 'handle_client'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/http/server.cr:468:13 in '->'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
Sep 07 18:20:26 invidious[3030]:   from /usr/share/crystal/src/fiber.cr:92:34 in '->'
Sep 07 18:20:26 invidious[3030]:   from ???
Sep 07 18:27:14 invidious[3030]: Unhandled exception in spawn: Error while writing data to the client (HTTP::Server::ClientError)
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/http/server/response.cr:227:9 in 'unbuffered_write'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/io/buffered.cr:152:14 in 'write'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/compress/deflate/writer.cr:94:7 in 'consume_output'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/compress/deflate/writer.cr:71:5 in 'close'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/compress/gzip/writer.cr:104:5 in 'close'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/http/server/request_processor.cr:63:11 in 'process'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/http/server.cr:513:5 in 'handle_client'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/http/server.cr:468:13 in '->'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/fiber.cr:92:34 in '->'
Sep 07 18:27:14 invidious[3030]:   from ???
Sep 07 18:27:14 invidious[3030]: Caused by: Error writing to socket: Broken pipe (IO::Error)
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/io/evented.cr:82:13 in 'unbuffered_write'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/io/buffered.cr:217:5 in 'flush'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/io/buffered.cr:151:7 in 'write'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/http/server/response.cr:220:11 in 'unbuffered_write'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/io/buffered.cr:152:14 in 'write'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/compress/deflate/writer.cr:94:7 in 'consume_output'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/compress/deflate/writer.cr:71:5 in 'close'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/compress/gzip/writer.cr:104:5 in 'close'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/http/server/request_processor.cr:63:11 in 'process'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/http/server.cr:513:5 in 'handle_client'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/http/server.cr:468:13 in '->'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
Sep 07 18:27:14 invidious[3030]:   from /usr/share/crystal/src/fiber.cr:92:34 in '->'
Sep 07 18:27:14 invidious[3030]:   from ???
Sep 07 18:27:25 invidious[3030]: Invalid memory access (signal 11) at address 0xa8
Sep 07 18:27:26 invidious[3030]: [0x55f3c1a53916] print_backtrace at /usr/share/crystal/src/exception/call_stack.cr:129:5
Sep 07 18:27:26 invidious[3030]: [0x55f3c174bdac] __crystal_sigfault_handler at /usr/share/crystal/src/signal.cr:347:3
Sep 07 18:27:26 invidious[3030]: [0x7f3404f83730] ???
Sep 07 18:27:26 invidious[3030]: [0x55f3c222f764] lsquic_conn_set_ctx +4
Sep 07 18:27:26 invidious[3030]: [0x55f3c17b3d60] -> at /var/www/invidious/lib/lsquic/src/lsquic/client.cr:210:11
Sep 07 18:27:26 invidious[3030]: [0x55f3c1b33c19] run at /usr/share/crystal/src/primitives.cr:255:3
Sep 07 18:27:26 invidious[3030]: [0x55f3c174b2b6] -> at /usr/share/crystal/src/fiber.cr:92:34
Sep 07 18:27:26 invidious[3030]: [0x0] ???
Sep 07 18:27:26 systemd[1]: invidious.service: Main process exited, code=exited, status=11/n/a
Sep 07 18:27:26 systemd[1]: invidious.service: Failed with result 'exit-code'.
Sep 07 18:27:28 systemd[1]: invidious.service: Service RestartSec=2s expired, scheduling restart.
Sep 07 18:27:28 systemd[1]: invidious.service: Scheduled restart job, restart counter is at 6.
Sep 07 18:27:28 systemd[1]: Stopped Invidious (An alternative YouTube front-end).
Sep 07 18:27:28 systemd[1]: Started Invidious (An alternative YouTube front-end).

But the instance worked so much better for the past hours since I changed to the branch you recommended.

@tio-trom
Copy link
Author

tio-trom commented Sep 8, 2021

With the new crystal and all that, I do not think it works well still. It is this dilemma for one it restarts very often again, which is not good, and second if it doesn't restart often then it completely stalls. It will stop working until I restart it manually.

@tio-trom
Copy link
Author

An invidious dev said "the restarts are due to memory leaks or invalid memory accesses (generally, blame QUIC)" - if that helps.

@tio-trom
Copy link
Author

So what I noticed is that it starts to restart more and more often after I did the installation. I suspect the bigger the database, and user base, the more often the restarts. Almost as if our server does not have enough resources...is there a way to increase whatever resources it needs? Because we have plenty of those. At this point our instance restarts ever other minute. This is unusable. And it may also paint a bad picture on yunohost from what I've been talking to the invidious developers who keep on saying it is a yunohost issue and go as far as saying yunohost is only meant for testing stuff. Which I disagreed with, and I do not see this as painting a bad reputation for ynh, but these are the feedback I keep on getting form the invidious devs, at least some of them.

I am here to test whatever needs to be tested and even to be a bridge between yunohost invidious and main invidious dev. I you think it may be an invidious issue detail it please so I can bring it to the invidious devs.

I hope we can fix this. Thanks!

@alexAubin
Copy link
Member

We discussed this a while ago and the issue seems to lie in Crystal itself, c.f. crystal-lang/crystal#9065 (comment)

@tio-trom
Copy link
Author

tio-trom commented Oct 1, 2021

We discussed this a while ago and the issue seems to lie in Crystal itself, c.f. crystal-lang/crystal#9065 (comment)

This is from the one who made the comment you are referring to, and she is an invidious dev:

" Hello! The goal is to silence the exception raised when the client unexpectedly closes the connection
And fyi, instead of commenting the line in the Crystal source code, we re-implemented the same function in invidious (in order to override the stdlib implementation) because it's simpler and cleaner.

Also, this is not a crash. It's simply an unhandled exception being printed to the logs

We (the invidious team) think that a client closing the connection unexpectedly is a normal thing and should be managed by the standard library instead of raising an exception such thing happens e.g when a user has a slow connection or a small drop, and they press the "X" button to stop the page loading.

The restarts are due to memory leaks or invalid memory accesses (generally, blame QUIC)
however, 10 minutes is much lower than what we see on other instances "

I brought to them the issue at hand so they told me this is not the cause.

@tio-trom
Copy link
Author

tio-trom commented Oct 1, 2021

So it seems invidious restarts quite often, but our yunohost installation restarts so often that it is difficult to use it.

@tio-trom
Copy link
Author

tio-trom commented Nov 2, 2021

Ok what seems to have fixed it for now is to add

#Execute and serve PHP files
    location ~* \.php$ {
	       fastcgi_buffers 8 16k;
        fastcgi_buffer_size 32k;
        fastcgi_connect_timeout 60;
        fastcgi_send_timeout 300;
        fastcgi_read_timeout 300;
        fastcgi_split_path_info ^(.+?\.php)(/.*)$;
        try_files $uri =404;
        fastcgi_index index.php;	
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    }

to the nginx file for invidious.

So far works ok.

@ericgaspar
Copy link
Member

please reopen if needed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants