Session expiration not threadsafe

[urso]

Session expiration is not threadsafe.

Documentation for time.AfterFunc says:

waits for the duration to elapse and then calls f in its own goroutine. It returns a Timer that can be used to cancel the call using its Stop method.

The expire functions simply drops the transaction from internal map. But maps are not thread safe. With golang 1.5 release on august 19th with GOMAXPROCS=(number of logical CPUs) by default tables might get corrupted by chance.

Some possible solutions:

1. use mutex
2. 'send' expired transaction to main loop for deletion.
3. create our own table of constant size (delete just overwrites pointer then - atomic op?)

[tsg]

I think an effect of this was just reported as #204.

[urso]

yes, traces of goroutine 10 and 1227 fully match the problem. This ticket is just more general, as similar code is different places.

[tsg]

I'm for a getting a fix quicker by putting a mutex around those maps (so solution 1). I'm thinking to make a "map + mutex" type in libbeat.common and provide some utilities so that it's easy to support our use cases. If you agree, I can start working on that.

[urso]

Agreed. Let's identify and fix our multithreading issues first.

[rshriram]

May I suggest to split the map per core to avoid the scalability bottleneck? Because, if one uses AF_PACKET in FANOUT_HASH mode, packets from a flow are delivered to the same core. So, session expiration could be sent out as transaction expired events to the main loop per core.

However, looking at the current code, there is no support for using AF_PACKET in fanout mode (which provides a lot more scalability than the stated 200K packets per second). So some machinery would have to be added to have multiple TPacket sockets and distributing them one per core. Gopacket has AF_PACKET fanout support builtin.

[urso]

Yes, fanout is not supported yet.

For all the different protocols we support the transaction expiration pattern using timeouts is very common. We might want to add some special special expiration (timer) support that helps contributors with these kinds of issues (like multiplexing timers on same goroutine processing packets).

When we add fanout I'm all in for reducing shared state aggressively.