The result of markers evaluations doesn't seem to detect/include markers in all cases

[alwx]

Rasa Open Source version

3.0rc1

Rasa SDK version

No response

Rasa X version

No response

Python version

3.8

What operating system are you using?

OSX

What happened?

I was testing that a marker defined for each condition (action_executed, intent_detected, slot_set) is correctly extracted and accidentally found out that the result doesn't seem to contain all the markers that actually should be extracted.

I used the default bot (the one created with rasa init), and here is my markers.yml:

marker_mood:
 or:
   - intent_detected: mood_unhappy
   - intent_detected: mood_great

I had several conversations but the extracted_markers.csv contains only two markers:

sender_id,session_idx,marker_name,event_id,num_preceding_user_turns
345edc3430004f33ace142d33e327d43,0,marker_mood,7,1
5139e06d573a46f9825f19ec163b4960,0,marker_mood,7,1

The thing is my tracker store actually contains more than just two mood_unhappy or mood_great intents. It might also be that my understanding is wrong, but shouldn't all of them be extracted?
You can find my bot that also includes a database (simple_tracker_store.db) with all conversations in attachements here

Command / Request

No response

Relevant log output

No response

[aeshky]

It looks the there are three trackers in the tracker store.
These two are being read and processed correctly:

• 5139e06d573a46f9825f19ec163b4960
• 345edc3430004f33ace142d33e327d43

But this one should have ~60 events but only the first two are loaded.

• be03647713174631b14d1a8befd236f9

Will find out why.

[aeshky]

The tracker with the ID=be03647713174631b14d1a8befd236f9 contains three sessions and it looks like only the last session is loaded which only contains session started and action listen. The first two sessions in the tracker are not loaded, and therefore the events that should be marked aren't being marked.

This tracker also contains the event rewind. I'll look into whether it's the rewind that's causing problems or the multiple sessions.

[aeshky]

I removed the rewind events and this didn't make a difference. After debugging with different trackers, it looks like the problem is due to having multiple sessions in the same tracker.

Stepping through the code, I can see that the function _split_sessions in the classMarkers already receives a truncated list of events, so that's not where the problem is. MarkerTrackerLoader is invoking existing rasa code, which only seems to return the final session in the tracker. I wonder if this is a "feature" rather than a bug, because when I use rasa interactive and start a new session halfway through a dialogue, I can see that the story generated in stories.yml only contains the last session. So maybe the existing loader that we invoke is intended to be used differently. I'm continuing to look into this.

One thing I've come to realise while debugging this issue is that we should have kept the timestamp in the marker's output. It looks pretty unique to me, and would have been super helpful to identify which events were present and marked and which ones were dropped when the tracker wasn't fully loaded. An index that we generate after reading the tracker can't serve this function because we are already missing events.

[aeshky]

Ok I think I fixed it. Will submit a PR. 👍

[ka-bu]

It looks pretty unique to me, and would have been super helpful to identify which events were present and marked and which ones were dropped when the tracker wasn't fully loaded. An index that we generate after reading the tracker can't serve this function because we are already missing events.

The timestamps do not help much to identify missing events either: To know that something is missing we'd need to know which timestamps to expect. This might be true if we're adding a conversation ourselves and know that we talked to the bot at 3pm but there is no event logged for 3pm - but otherwise it gives us as much information as the indices and besides we'd also see that the words that we said in that conversation at 3pm are also not present in any of the logged messages.
Moreover, there is no analytics of any kind happening with timestamps so it seems wasteful to save that for every relevant event - and if we want to retrieve the full event that was marked / inspect the surrounding events in any other way, the indices are an straightforward way to get there (as opposed to searching for the list and comparing timestamps).

[aeshky]

I found it helpful because I was using a database viewer and can see the timestamps of events. I expected 4 events to be marked, but only got 2 markers in the output, and couldn't figure out from the indices which ones were missing. However, I could see this info in the debugger. But the question is whether customers would find it useful too or whether it is wasteful. At the moment you have to count the events in the database viewer to figure out where a marker is placed which iI found less conveniet than searching for a timestamp 😄