Tracker featurization during batch generation

[twerkmeister]

Description of Problem: Currently every tracker is featurized before the training and kept around in memory. This takes a lot of time, but also a lot of memory, keeping all the featurized trackers around at the same time. This issue is particularly big for large e2e datasets such as multiwoz.

Overview of the Solution: The trackers can be featurized during the batch generation. This leads to much lower memory consumption. However, featurization from the ground up is slower than the model training loop. This is why this issue first needs the lookup table issue to be resolved to be viable.

There is a prototypical implementation of this feature in the combined e2e fixes branch where the trackers are encoded inside the data loader class.

Open Issues

• Entity recognition does not work yet in the prototypical implementation

Blockers #9020 is needed for this proposal to be computationally viable

Definition of Done:

• Harmonize current prototypical implementation with the new architecture prototype
• Make code quality production ready
• Add memory / runtime test to confirm working
• training with entities works
• Feature mentioned in the changelog

[sync-by-unito]

➤ Maxime Verger commented:

💡 Heads up! We're moving issues to Jira: https://rasa-open-source.atlassian.net/browse/OSS.

From now on, this Jira board is the place where you can browse (without an account) and create issues (you'll need a free Jira account for that). This GitHub issue has already been migrated to Jira and will be closed on January 9th, 2023. Do not forget to subscribe to the corresponding Jira issue!

➡️ More information in the forum: https://forum.rasa.com/t/migration-of-rasa-oss-issues-to-jira/56569.