mapping policy predicts action listen while running, but not in testing

[erohmensing]

Rasa version:

1.10.x

Issue:

If I have a mapping in my domain that is not in my stories, it is correctly predicted by the mapping policy, and the action listen is predicted by the mapping policy, as expected, while talking to the bot in rasa shell:

The predicted intent 'bot_challenge' is mapped to  action 'utter_iamabot' in the domain.
2020-10-05 12:12:03 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_2_MappingPolicy
2020-10-05 12:12:03 DEBUG    rasa.core.processor  - Predicted next action 'utter_iamabot' with confidence 1.00.
2020-10-05 12:12:03 DEBUG    rasa.core.processor  - Action 'utter_iamabot' ended with events '[BotUttered('I am a bot, powered by Rasa.', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {}, 1601892723.318448)]'.
2020-10-05 12:12:03 DEBUG    rasa.core.policies.memoization  - Current tracker state [{'intent_mood_unhappy': 1.0, 'prev_action_listen': 1.0}, {'intent_mood_unhappy': 1.0, 'prev_utter_cheer_up': 1.0}, {'intent_mood_unhappy': 1.0, 'prev_utter_did_that_help': 1.0}, {'intent_bot_challenge': 1.0, 'prev_action_listen': 1.0}, {'intent_bot_challenge': 1.0, 'prev_utter_iamabot': 1.0}]
2020-10-05 12:12:03 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2020-10-05 12:12:03 DEBUG    rasa.core.policies.mapping_policy  - The mapped action, 'utter_iamabot', for this intent, 'bot_challenge', was executed last so MappingPolicy is returning to action_listen.
2020-10-05 12:12:03 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_2_MappingPolicy
2020-10-05 12:12:03 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.

However, if I have the same training data (the one-turn interaction isn't included in the training data), and a test in which this one-turn interaction occurs, the utterance is correctly predicted by the mapping policy, but the action_listen is not:

2020-10-05 12:09:33 DEBUG    rasa.core.policies.mapping_policy  - The predicted intent 'bot_challenge' is mapped to  action 'utter_iamabot' in the domain.
2020-10-05 12:09:33 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_2_MappingPolicy
2020-10-05 12:09:33 DEBUG    rasa.core.processor  - Predicted next action 'utter_iamabot' with confidence 1.00.
2020-10-05 12:09:33 DEBUG    rasa.core.policies.memoization  - Current tracker state [{'intent_mood_unhappy': 1.0, 'prev_action_listen': 1.0}, {'intent_mood_unhappy': 1.0, 'prev_utter_cheer_up': 1.0}, {'intent_mood_unhappy': 1.0, 'prev_utter_did_that_help': 1.0}, {'intent_bot_challenge': 1.0, 'prev_action_listen': 1.0}, {'intent_bot_challenge': 1.0, 'prev_utter_iamabot': 1.0}]
2020-10-05 12:09:33 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2020-10-05 12:09:33 DEBUG    rasa.core.policies.mapping_policy  - The mapped action, 'utter_iamabot', for the intent, 'bot_challenge', was executed last, but it was predicted by another policy, 'None', so MappingPolicy is not predicting any action.
2020-10-05 12:09:33 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_TEDPolicy
2020-10-05 12:09:33 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 0.87.

In this case TED still predicted the right action, but if it predicts lower than my core fallback threshold, this would cause my test to fail although it works as expected with the same input data while actually talking to the bot.

[erohmensing]

closed in #6907