Myhoard allows too many streams

[aiven-amartin]

What happened?

The server can accept requests for many backup streams. We have a few servers with 5+ open streams. It appears that https://github.com/aiven/myhoard/blob/f13a8750465a349bff6b68c63bf1a2533e04c7d1/myhoard/controller.py#L1778 might not be quite correct behaviour, as it appears to update streams, but not persist those updates back to controller state, then ask the controller to reload its state.

What did you expect to happen?

In https://github.com/aiven/myhoard/blob/f13a8750465a349bff6b68c63bf1a2533e04c7d1/myhoard/controller.py#L796, we prevent more than two streams from being open, so, that should not be the case. Rather than keep the extra streams that exist around, they should be discarded.

What else do we need to know?

This problem is exacerbated by the lack of an api to delete backups. Because these streams queue up, you can end up hitting your maximum backup count quite easily, which causes even more streams to queue up as they wait for room, snowballing the problem.