DDOS crash

[MGlolenstine]

I wrote a public website using Rocket.
That application gets targetted by people trying to find vulnerabilities (from SQLInjection to XSS and other stuff).
The app is hosted within a Docker environment and once in a while, the container ends with "Error(255)", which means that Rocket terminated with error code 255.
When I check the logs, there's nothing out of the ordinary (except for all those requests); no errors, just end of the log.
I was wondering if there's a way to switch to a different logging or somehow disable parsing of the paths that don't exist.
Whenever they're trying to access /, and they append tons of GET parameters, it just passes normally, but I'm unsure how that weighs the system down.

[jebrosen]

The app is hosted within a Docker environment and once in a while, the container ends with "Error(255)", which means that Rocket terminated with error code 255.

Yeah, that's pretty unexpected; Rust programs and Rocket web servers are typically robust. I don't recognize 255 as an error code that either Rocket (or a typical Rust application) would normally produce, either. This could potentially be caused by something outside of Rocket's direct control, such as an out-of-memory error, application or library code calling exit(), or a bug in some unsafe or C native code.

When I check the logs, there's nothing out of the ordinary (except for all those requests); no errors, just end of the log.

That's even more unfortunate. Do you have other information you could correlate with Rocket's log, such as logs of incoming requests before they reach Rocket? You could also try using a more detailed log level to try and narrow down the issue.

I was wondering if there's a way to switch to a different logging or somehow disable parsing of the paths that don't exist.

I don't understand this part; is there some reason you think the logging is causing the crashes? What you are asking for here seems impossible on its face: the path pretty much must be parsed first, in order to do anything with it - including determining whether not it exists.

[MGlolenstine]

I don't understand this part; is there some reason you think the logging is causing the crashes?

I was referring to a different way of logging (worded it awkwardly, but meant different log level).

What you are asking for here seems impossible on its face: the path pretty much must be parsed first, in order to do anything with it - including determining whether not it exists.

That does make sense... I guess I should set up a fail2ban onto the server.

You could also try using a more detailed log level to try and narrow down the issue.

I will try that on Monday to try and catch it.

Do you have other information you could correlate with Rocket's log, such as logs of incoming requests before they reach Rocket?

I do. I'm currently redirecting data through Nginx and storing the requests in a log.

This could potentially be caused by something outside of Rocket's direct control, such as an out-of-memory error, application or library code calling exit(), or a bug in some unsafe or C native code.

There was no other error in the docker container. If it was OOM, some other programs could've gone offline as well, but haven't seen any other outages. If there was a crash within the program, wouldn't I be getting Stack dumped?

[MikailBag]

If it was OOM,

If a process is terminated by kernel OOM killer, its exit code is set to 137 (128 + SIGKILL), not 255. However, it is also possible that 255 is not an exit code of your application, but the exit code of e.g. the container entrypoint.

some other programs could've gone offline as well

Not necessarily: OOM killer identifies one process and kills it. It will only kill other processes if memory pressure continues.

AFAIK, the most robust way to check if your app was OOM killed is to check the kernel logs. You can use dmesg command to view them. OOM killer writes to this log every time it kills a process. Unfortunately, these log has limited storage space, and old entries are deleted, so only recent OOM kills will be visible there.

[MGlolenstine]

If a process is terminated by kernel OOM killer, its exit code is set to 137 (128 + SIGKILL), not 255. However, it is also possible that 255 is not an exit code of your application, but the exit code of e.g. the container entrypoint.

That is something I didn't know, thanks :)

AFAIK, the most robust way to check if your app was OOM killed is to check the kernel logs. You can use dmesg command to view them. OOM killer writes to this log every time it kills a process. Unfortunately, these log has limited storage space, and old entries are deleted, so only recent OOM kills will be visible there.

Would that work with docker containers though?

[MikailBag]

Would that work with docker containers though?

It should. Docker containers do not involve virtualization, so they use the host kernel. And any OOM kills (including those inside containers) will be reflected in host kernel logs.

[MGlolenstine]

I see, I'll make sure to test it out tomorrow and check Kernel logs as well!
I'll be back with more information when I get it :)