kqueue: wrong API usage

[kristoff-it]

To test the issue:

Create a xev UDP socket bound to a local address (eg 0.0.0.0:1992) and use it to send two or more packets at a time (by calling udp.send sequentially twice). This will cause the event loop to drop some writes. You can test this more easily if you send to two different addresses in a row: only one of the two will receive a packet.

See this repro: https://github.com/kristoff-it/xev-kqueue-repro

The problem is based on the fact that in kqueue there can be only one write readiness event listener per file descriptor. From the kqueue manpage:

     EV_ADD         Adds the event to the kqueue.  Re-adding an existing event
                    will modify the parameters of the original event, and not
                    result in a duplicate entry.  Adding an event automati-
                    cally enables it, unless overridden by the EV_DISABLE
                    flag.

Looking at how xev is architected, it seems to suggest that this constraint was not taken into account when implementing that part of the library.

Unfortunately, given this limitation, AFAIK the only reasonable solution is to make xev.UDP pinned, make it hold an intrusive linked list of events, and then pass a pointer to the pinned struct to kqueue. Whenever you receive a notification from kqueue, you look into the linked list of events and process them until you receive EAGAIN.

Note that this problem is not unique to UDP (although to be fair you wouldn't easily experience this problem with TCP since parallel writes would be wrong in almost all cases).