Refactoring: event stream based agent history

[enyst]

Memory component: short overview

This PR works with the short term history. It proposes an implementation of the event-stream based architecture for agent history, which:

• uses directly the event stream for history
  • it doesn't save its own copies of events (in state.history)
  • event stream is the source of truth
• filters and otherwise processes the events, for insertion in the prompt
• the stream is not pairs of (action, observation), and neither is history
  • when the agent returns an action, the runtime/backend returns an observation that has the field cause set to the action id
• a compatibility method with (action, observation) pairs is used for evals output.

Pre-requisite for #2021

[xingyaoww]

This is amazing efforts! Really hoping we can merge it soon - recently we did some big refractor on the eval inference side that might break this.. Lmk if there's anything i can help to fix the merge conflicts!

[enyst]

@xingyaoww Totally agree, I think it will be better to get this one in, sooner rather than later, because we need this architecture to continue building on.

Merge conflicts fixed, and it's ready for review.

[tobitege]

Great job!
Got no blockers, but a couple of nits. 😬

[tobitege]

LGTM! 🍰

[neubig]

I checked this briefly, but from my check it looks good!