AugMemo DeLorean Inconsistency

[kedz]

Rasa Open Source version

2.8.7

Rasa SDK version

No response

Rasa X version

No response

Python version

3.8

What operating system are you using?

OSX

What happened?

A customer has a bot working as expected with Rasa 2.3.3 and Rasa 2.6.1. For those versions their stories follow the AugmentedMemoizationPolicy. While upgrading to Rasa 2.8.7 they noticed inconsistency within the behavior of the AugmentedMemoizationPolicy where for the same stories they were using previously the AugMemo policy was now losing track of the conversation and deferring to the TEDPolicy to make a prediction, which they observed was largely a low confidence prediction.

Potential Source of the Problem

A potential explanation for this is because of the change introduced in Rasa 2.8 where the DeLorean phase of AugMemo began to trim the tracker history according to the max_history prior to constructing the prediction states. Previously the entire tracker history was being used until this commit applying the tracker trimming.

When the AugmentedMemoizationPolicy is trained all the states contain a component detailing the last user intent, like this: {user: {"intent": "x"}, in addition to the various other action and slot events.

For the customer's case their story has a bunch of sequential actions occurring between user utterances. Once the number of sequential actions exceeds max_history when the DeLorean trims the tracker the last user utterance is lost. So, when the DeLorean constructs the prediction states from the trimmed tracker the last user intent is not included in those states. As a result AugMemo will not find a match because the states constructed during training and at prediction time do not contain the same information.

Reproducing the Error

The problem can be reproduced with the repo linked in this slack discussion, which contains a similar bot to what the customer is using. You can train a model and follow this conversation:

User: Hi
Bot: What can I help you with?
User: /technical_support

The bot will process actions up to action_set_customer_service_path where the bot will no longer be able to follow the AugmentedMemoizationPolicy to make predictions. Instead TED makes a low confidence prediction. The expectation is the bot would follow the checkpoint to predict action_check_customer_service_path. Reviewing the construction of the mcfly_tracker after action_set_customer_service_path the trimmed tracker no longer contains the last user utterance because there are now 6 actions occurring after the utterance and the utterance gets trimmed.

Command / Request

No response

Relevant log output

No response

[kedz]

@samsucik @JEM-Mosig reviewers