action_unlikely_intent prediction hides TED errors in failed_test_stories.yml #9057
Labels
area:rasa-oss 🎡
Anything related to the open source Rasa framework
feature:ml/intent-ted
type:bug 🐛
Inconsistencies or issues which will cause an issue or problem for users or implementors.
Rasa version: 2.7.1
Python version: 3.7.10
Operating system (windows, osx, ...): osx
Issue:
When running
rasa test
with a policy ensemble that includes UnexpecTEDIntent policy, if the ensemble fails after anaction_unlikely_intent
is predicted by UnexpecTEDIntentPolicy, the action will be recorded byrasa test
as being correct in the metrics and will not show up infailed_test_stories.yml
.Let’s take an example test story:
When a partial part of this test story:
is passed to the ensemble it can result in an
action_unlikely_intent
being triggeredSince, the last action triggered by ensemble was
action_unlikely_intent
, the ensemble should be queried again to see if other policies can actually predictutter_greet
. So, the following story:should be fed again to the ensemble. Now there are two cases possible:
Case 1:
utter_greet
gets predicted. So, the story looks like this -This story should go in
stories_with_warnings.yml
, as the expected action was correctly predicted afteraction_unlikely_intent
.Case 2:
utter_greet
does not get predicted but some other action gets predicted. So, the story looks like this -This story should end up in
failed_test_stories.yml
becausesome_other_action
is predicted instead ofutter_greet
Right now, case (2) is not being checked and can be reproduced with this example ….
Run
rasa test
on the TED only ensemblerasa test core -s test-bug/ -m models/ted --out results/ted
and you should see the following results:
which are correct.
Run the ensemble with UnexpecTEDIntentPolicy
rasa test core -s test-bug/ -m models/ited --out results/ited
and you get:
Looking in the failed test stories you can see TED's error when it's run in isolation
results/ted/failed_test_stories.yml
:However, the corresponding file for the combined TED+UnexpecTEDIntentPolicy (
results/ited/failed_test_stories.yml
) is empty while the action_unlikely_intent shows up inresults/ited/stories_with_warnings.yml
.WARNING: This does not change the case where
action_unlikely_intent
was actually expected to be triggered at a conversation turn but was not predicted by the ensemble. Such cases should still go tofailed_test_stories.yml
.The text was updated successfully, but these errors were encountered: