Client connection time when a large number of clients connect.

[markregel2]

Hi @jchristn,

I'm using this as a server to support a communication platform for a system of IoT devices. For testing I wrote an app to create virtual clients, this allows me to have any number of clients connect to the server. The use case is any client can communicate with one or more (or all) of the currently connected clients. This is working well. My question is about the time it takes clients to connect. The virtual clients all attempt to connect at the same time (basically in a For Loop in the code). With a small number of clients (10 - 20), they all connect almost instantly. With 100 clients, the first few connect instantly, then things really slow down. It is taking one to two minutes for all 100 to connect. Do you have an idea why it is bogging down? I have another old native Win32 server (using TCPListner) running synchronously in a thread. It also slows down somewhat but it only takes a few seconds for 100 connections. Could it be related to the asynchronous methods used in this server, or perhaps that its built on .NET? Its not a deal breaker, but I would like to understand why it is so much slower accepting incoming connections.

Thanks and regards.

[jchristn]

Hi @markregel2 thanks for your note. In general, any server side interface, that is TCP-based is going to have a bit of headache with connection set up, teardown, and management. This one is no different. I assume it has to do with how the thread pool is managed, but it could also be because it's just using the underlying OS TCP mechanisms.

you can try changing thread, pool sizes and such to mitigate problems with the underlying reallocations that have to happen in the operating system for situations where you were operating with a large number of tasks. But I've never really had luck mitigating this exact issue. It's present in all of my libraries.

[markregel2]

Hi @jchristn

The largest workload my server will encounter aside from connection setup is sending live audio to all connected clients. Our audio is low bandwidth (8000 Bps, or 80,000 bps). Audio is received from one client and forwarded to all other connected clients. I'm performing the sending to each client inside the DataReceived event of the incoming audio client. The audio is received/sent in 500 byte frames at the rate of 62.5ms, to give the needed 8000 Bps rate. I wrote a test app to simulate any number of client connections. It all works well with approx 200 clients, but with more than 200 the server starts to bog down and can't deliver the packets fast enough. My server and clients are all connected directly to a local GBit switch, so network bandwidth should not be a constraint at these speeds. Do you have any ideas on how to improve data throughput? The time to decode received frames is negligible, the time is being consumed when sending the frame to all the connected clients, which occurs in a loop. We have 62.5ms to send each frame to all connected clients, any longer the system will get behind. So far I have tried setting UseAsyncDataReceivedEvents to true and false. I have also tired sending using _simpleTcpServer.Send and _simpleTcpServer.AsyncSend. These changes don't make a notcieable difference in performance. Do you think something like using old fashioned synchronous socket methods and threads might be faster? Do you have an idea on how much bandwidth the server should be able to send to n number of clients?

Thanks! Mark

[jchristn]

Hi @markregel2 thanks for your note. Generally with smaller data sizes maximum throughput is lower than with larger data sizes (more overhead per payload byte with smaller data sizes). The best advice I could give, really, would be to try adjusting thread pools. I haven't had a need to try and scale beyond dozens of connections, however, I could see how the allocated thread pool would be a bottleneck.

[jchristn]

You may also consider moving "down the stack": https://stackoverflow.com/questions/52176692/high-performance-tcp-socket-programming-in-net-c-sharp - notice the max throughput reached on SocketAsyncEventArgs/ReceiveAsync/SendAsync