Investigate outlier in GPU memory usage with 2.6 (train): Hermit + cvf_diet_responset2t

[koernerfelicia]

Followup to #9433

As seen [here| https://docs.google.com/spreadsheets/d/14rIvQLxEdp1PJDqhM65n0__THKhvS5f4XPLoWz8SWSs/edit#gid=1230100797], there is a huge increase to GPU memory usage for train on Hermit with cvf_diet_responset2t. This is an outlier, and should be investigated.

Definition of Done (not necessarily in this order)

• confirm this is reproducible
• examine logs
• re-run test
• plot nvidia-smi logs to narrow down the increase
• (if relevant) document steps used as an extension of [this| https://github.com/Add scripts for 2.5 vs 2.3 runtime and memory usage experiments to research-experiments repo #9526] for future reference

Depending on what we learn from investigation create the appropriate issue:

• issue to address the problem OR issue to investigate further OR issue on TF repository

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sync-by-unito]

➤ Maxime Verger commented:

💡 Heads up! We're moving issues to Jira: https://rasa-open-source.atlassian.net/browse/OSS.

From now on, this Jira board is the place where you can browse (without an account) and create issues (you'll need a free Jira account for that). This GitHub issue has already been migrated to Jira and will be closed on January 9th, 2023. Do not forget to subscribe to the corresponding Jira issue!

➡️ More information in the forum: https://forum.rasa.com/t/migration-of-rasa-oss-issues-to-jira/56569.