[Question] [HTTP Server] 🚀🚀 Handling multiple requests at the same time (IDFGH-9204)

[chipweinberger]

HTTP Server

Is there a way to handle multiple requests at the same time for the http server?

is it even possible?

In ESP-IDF today, only a single request handler can run at a time. And even with a separate thread, httpd_req_t is freed when the synchronous handler completes so prolonged communication is impossible.

How reasonable is it for me to hack ESP-IDF to support this?

Previous discussion: https://esp32.com/viewtopic.php?t=28000

[wuyuanyi135]

esp-idf/examples/protocols/http_server/advanced_tests/main/tests.c

Line 273 in 49551cc

httpd_socket_send(hd, fd, buf, strlen(buf), 0);

[chipweinberger]

Ive seen that example before but I don't understand how it would work in a real situation.

httpd_queue_work executes code on the same thread as the HTTP Server, and the example just returns a string synchronously. What is the reason for doing it this way?

Perhaps the example could show how to execute a long 30+ second calculation asynchronously on a separate thread and then return it on the http thread, without blocking other request handlers.

Or more challenging, how to download a large file over HTTP for a few minutes while still executing other requests.

[boarchuz]

+1 for this feature request

[wuyuanyi135]

You can save those file descriptor and handle to a structure and pass it to another thread for later use. In this case you cannot use those req-like functions. You have to assemble the http packet yourself. I can show you some c++ code to wrap around these functions when I got home

[wuyuanyi135]

#define HTTP_UTIL_CHECK(x)                                                                              \
  do {                                                                                                  \
    int ret = x;                                                                                        \
    if (ret == HTTPD_SOCK_ERR_INVALID || ret == HTTPD_SOCK_ERR_TIMEOUT || ret == HTTPD_SOCK_ERR_FAIL) { \
      return ret;                                                                                       \
    }                                                                                                   \
  } while (0)

int httpd_socket_send_common_header(httpd_handle_t hd, int fd, const std::string& content_type, size_t content_length, int status = 200) {
  char buf[80];
  size_t sz = snprintf(
      buf,
      sizeof(buf),
      "HTTP/1.1 %d OK\r\n"
      "Content-Type: %s\r\n"
      "Content-Length: %d\r\n",
      status,
      content_type.c_str(),
      content_length);
  return httpd_socket_send(hd, fd, buf, sz, 0);
}

int httpd_socket_header_finishes(httpd_handle_t hd, int fd) {
  const char* sep = "\r\n";
  return httpd_socket_send(hd, fd, sep, strlen(sep), 0);
}

int httpd_send_json(httpd_handle_t hd, int fd, const nlohmann::json& json, int status = 200) {
  auto str = json.dump();
  HTTP_UTIL_CHECK(httpd_socket_send_common_header(hd, fd, HTTPD_TYPE_JSON, str.size(), status));
  HTTP_UTIL_CHECK(httpd_socket_header_finishes(hd, fd));
  HTTP_UTIL_CHECK(httpd_socket_send(hd, fd, str.c_str(), str.size(), 0));
  return 0;
}

void httpd_send_json(httpd_req_t* req, const nlohmann::json& json, const char* status = HTTPD_200) {
  auto str = json.dump();

  CHK_E(httpd_resp_set_type(req, HTTPD_TYPE_JSON), "failed to set content type");
  CHK_E(httpd_resp_set_status(req, status), "failed to set status");
  CHK_E(httpd_resp_sendstr(req, str.c_str()), "failed to send json");
}

These are a part of my C++ http utility library. It depends on nlohmann::json for json encoding. Those CHK_E are macros for error checking. See how httpd_send_json works to assemble an asynchronous response.

If you are using C++, you can delegate the response to a thread like this (note must capture by value)

auto hd = req->handle;
auto fd = httpd_req_to_sockfd(req);
std::thread th([hd, fd]{
     json = ....
    httpd_send_json(hd, fd, json);
});

If you are using C, you can follow how the example works.

As far as how this works, IIRC, the http socket was kept open if no httpd_resp_* was used to send response. So if you do not use httpd_resp_* in handler callback and extract the fd and handle you can use them later in another thread.

I haven't test if the socket should be closed explicitly after sending all response. But from packet capture it looks this code works fine.

[chipweinberger]

@wuyuanyi135

Wow, thanks for the very impressive reply!!! Very helpful.

Must have taken some time to figure this out! Very clever!

Possible Improvements: Given that the fd still works in your approach, it might be easy to hack ESP-IDF into not freeing the httpd_req_t when the handler completes. Perhaps in our request handler we could call a new httpd_req_claim_ownership(httpd_req_t* req) function that makes us responsible for freeing the request. This would allow us to continue to use httpd_resp_xxx() calls in a new thread.

[wuyuanyi135]

@chipweinberger You are welcome!

One catch is that httpd_queue_work seems that it should do the work for the asynchronous response but actually it works fine with or without it. I raised a question in the post but no response received. I hope the developer could clarify the use case of httpd_queue_work.

For your proposed improvement, I do agree that there is no reason that the asynchronous response should use a different set of API. There should be some APIs to restore the httpd_req_t object or prolong its lifetime. Let's see what espressif's opinion on this.

[chipweinberger]

I'm begining to understand the http server code:

Where the req is fully deleted (not normally called - only called on error):
esp_err_t httpd_req_delete(struct httpd_data *hd)

Where the req is reset after a req handler completes, in order to be used again (this is the important one!):
static void httpd_req_cleanup(httpd_req_t *r)

Where the req handler is invoked by the server:
esp_err_t httpd_uri(struct httpd_data *hd)

The main task loop of the http server (just repeatedly calls httpd_server(hd))
static void httpd_thread(void *arg)

Where we create a new socket:
static esp_err_t httpd_accept_conn(struct httpd_data *hd, int listen_fd)

Where the socket is closed:
void httpd_sess_delete(struct httpd_data *hd, struct sock_db *session)

[chipweinberger]

A roadbump, it loops like the http server is designed around a single req object existing at a time. They are not actively freed & created.

This is the corresponding internal data for each httpd_handle_t:

struct httpd_data {
    ...
    struct httpd_req hd_req;                /*!< The current HTTPD request */
   ....
}

[chipweinberger]

Okay, after looking at more code. A httpd_req_t just holds basic info about that request. i.e the uri, the fd, the http req/resp headers.

It is my current understanding that we could just memcpy the httpd_req_t req and the req->aux object to somewhere we control, and then continue to use the request object as normal!

Note1: As you've found, each httpd_req_t gets assigned a new socket / fd, so it is safe to continue to use it's socket. The http server does not use HTTP 1.1 Pipelining (https://en.wikipedia.org/wiki/HTTP_pipelining) and all browsers have deprecated that feture.

Note2: HTTP 1.1 Persisent Connections (https://en.wikipedia.org/wiki/HTTP_persistent_connection) does allow multiple http requests per socket, but you are not allowed to send another request until you get a response back first, so your proposed method of sending on the fd manually should be fine.

lru_purge_enable: That said, if we are using lru_purge_enable we still need to be concerned about preventing the HTTP Server from closing our socket from underneath us! I have not found a good way to do prevent this, yet.

static esp_err_t httpd_accept_conn(struct httpd_data *hd, int listen_fd)
{
    /* If no space is available for new session, close the least recently used one */
    if (hd->config.lru_purge_enable == true) {
        if (!httpd_is_sess_available(hd)) {
            return httpd_sess_close_lru(hd);
 ...
}

[chipweinberger]

To alleviate questions about httpd_queue_work(), it is used to execute work on the httpd thread (in the httpd_server function).

static esp_err_t httpd_server(struct httpd_data *hd)
{
        ....
        // queued work will be processed here!
        httpd_process_ctrl_msg(hd);
        ....
}

I'm not sure exactly why you need to do this. But It seems like a simple way to get "thread safety" for doing various http related things. For example, you could do work on a separate (higher priority?) thread, and then come back to the httpd thread to finish sending a reply and be as "thread safe" as possible. Again, it seems not needed today -- httpd_req_t does not seem to care what thread you use with it as long as the request handler has not yet returned -- but probably that wont always be true forever. That's my understanding.

It does not allow simultaneous requests in and of itself, as we already know.

[boarchuz]

I also want to reiterate from the linked forum thread that the request body will be purged upon return from the handler. So while the above may work for requests with no/small body, a firmware upload, for example, where there is a lot of data to receive and a lot of processing time required to do so (ie. writing to flash) requires blocking the httpd task at least until the entire request is received.

[chipweinberger]

Open question: How does the http server handle closing of sockets? Will multiple async requests properly close sockets? Edit: yes, i think so.

I don't see any handling of Connection: close in the http server code (only http client).

And I've yet to find the place where the http server closes existing sockets.

My current understanding: the HTTP server will always keep the socket open forever until the client closes their tcp connection, in which case LWIP will handle the close for us. But I imagine the majority of sockets sockets are closed during LRU purge.

Edit: In http 1.1, browsers close sockets very rarely. Browsers typically keep sockets open even when the browser window is closed. This means the most common way for a socket to close will be the LRU purge on the server side.

I'd like to see Espressif be much more aggressive in server socket closing. A ESP32 server should close sockets regularly in order to keep a few free resources, i.e. after idle timeouts, or after hearing Connection: close from the client, etc. Currently we rely on lru_purge_enable, but that is a non-graceful way of doing socket shutdown. We should be more proactive, so that 0ms SO_LINGER is rarely needed.

Anyway, long story short, 4 conclusions:

1. your method for doing async requests looks okay to me
2. doing a memcpy of the req like I suggested before should work fine, so we can use the httpd_resp_xx() functions later.
3. there is no need for us to close the fd ourself
4. if the socket gets LRU purged, our async resp will fail. Ideally we should prevent this.

[wuyuanyi135]

As what I heard the esp-idf http server was a bit primitive. The mongoose http server was said to be more full-fledged. Mongoose server might not be cheap for commerial choice though. If working on a hobbist project, maybe check mongoose server and see if they provide the necessary features?

[chipweinberger]

I've opened a PR to add lru purge support. I also added example code for doing async requests.

#10602

[chipweinberger]

This feature has been merged.