Remove exit variable from all stages

[garious]

To gracefully shutdown a thread, just drop all its input channels. There's typically only one.

Getting rid of that variable would make stage tests more succinct with no loss of functionality.

[jackson-sandland]

@garious can you provide an example? I'd like to do this.

[garious]

Check out the tests in record_stage.rs.

[garious]

@jackson-sandland, I didn't get much further than you. :) . #252

[jackson-sandland]

@garious curious how you approached this. Taking a look :)
Thanks for the examples. This makes sense.

[bgorlick]

re: @garious comments

A stage listens to some receiver channel for data, processes it, and sends the processed data to its sender channel.

[garious]

Bumping this back a release. We can discuss this more after this month's release. @sakridge would prefer we not do it, and @aeyakovenko doesn't like how it would introduce an inconsistency between stages that listen on channels and stages that listen on UDP sockets.

[garious]

My opinion here is that a stage should shut itself down if any of its channels are dropped. This would allow either an upstream or a downstream stage to signal the stage to shut down. UDP sockets can be accompanied by an exit variable to provide that functionality.

The value in getting rid of the exit variables is for a more predictable pipeline shutdown. Closing the first stage's input channel would signal all stages to shut down after the last input had been processed by all stages. Using an exit variables means that the TPU is left in an inconsistent state where, for example, the banking stage state may reflect transactions that were not broadcast by the write stage. In production, that shouldn't matter, because the process is about to exit, but for unit tests, I'd expect it to be an annoyance.

[aeyakovenko]

I think it attaches a behavior to a channel lifetime that is hard to debug.

[garious]

Expanding on my previous comments, instead of an exit variable, I think stages containing UdpSockets as their only inputs should be constructed with a Receiver<()> as well, and should shutdown if its corresponding Sender is dropped.

[aeyakovenko]

For threads blocking on sockets, service will hold its own exit variable, and set it when you call join on the service?

If you do that, then you can't pull the threads out and aggregate them since the service join needs to be called to set exit. And aggregating services themselves would require you to hold vectors of heterogeneous types (a vector of Service traits)

[garious]

Can you reword please?

[garious]

Per #557, down to 122 instances of the word exit in the codebase.