Integrate Graph Architecture with rest of codebase #9803

wochinge · 2021-10-06T13:35:19Z

Proposed changes:

fix Implement Recipe to convert current config to GraphSchema #9277
fix Integrate rasa train and rasa run with new model architecture #9604
integrate the Graph Architecture with the rest of the codebase

Open Issues

Status (please check what you already did):

added some tests for the functionality
updated the documentation
updated the changelog (please check changelog for instructions)
reformat files using black (please check Readme for instructions)

…ipe' into 3.0-architecture-revamp/9604/train-and-run

…chitecture-revamp/9604/train

wochinge · 2021-10-14T06:24:42Z

Some model regression tests were failing over night due to poetry issues. As we have also seen no changes in previous runs and the entity testing works now, I just give it one final run on the Sara dataset before merging @joejuzl @ka-bu

github-actions · 2021-10-14T08:49:55Z

Commit: acd4601, The full report is available as an artifact.

Dataset: Sara, Dataset repository branch: architecture-revamp/modified-spacy-configs, commit: a88501268464439d1d058a8b77f16a1c752230bd

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `4m21s`, train: `7m3s`, total: `11m23s`	0.7179 (0.00)	0.7949 (0.00)	0.7898 (0.00)
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `4m46s`, train: `5m29s`, total: `10m14s`	0.7136 (0.00)	0.7925 (0.00)	0.7767 (0.00)
`Sparse + BERT + DIET(bow) + ResponseSelector(bow)` test: `4m33s`, train: `10m15s`, total: `14m47s`	0.6953 (0.00)	0.7949 (0.00)	0.8140 (0.00)
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `4m58s`, train: `7m49s`, total: `12m46s`	0.7020 (0.00)	0.7918 (0.00)	0.7922 (0.00)
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `1m38s`, train: `6m44s`, total: `8m21s`	0.6654 (0.00)	0.7949 (0.00)	0.7814 (0.00)
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `2m3s`, train: `5m26s`, total: `7m29s`	0.6822 (0.00)	0.7812 (0.00)	0.7851 (0.00)

Dialog Policy Configuration	Action Level Micro Avg. F1	Conversation Level Accuracy	Run Time Train	Run Time Test
`Rules`	0.1266 (0.00)	0.0000 (0.00)	`2m22s`	`1m16s`
`Rules + AugMemo`	0.9149 (-0.01)	0.6301 (-0.03)	`2m24s`	`1m30s`
`Rules + AugMemo + TED`	0.9728 (-0.00)	0.7603 (0.00)	`12m17s`	`2m58s`
`Rules + Memo`	0.3860 (0.00)	0.1438 (0.00)	`2m23s`	`1m22s`
`Rules + Memo + TED`	0.9575 (0.00)	0.6678 (-0.00)	`12m14s`	`2m50s`
`Rules + TED`	0.9566 (0.00)	0.6541 (0.00)	`12m16s`	`2m48s`

github-actions · 2021-10-14T10:14:13Z

Commit: df44e56, The full report is available as an artifact.

Dataset: Carbon Bot, Dataset repository branch: architecture-revamp/modified-spacy-configs, commit: a88501268464439d1d058a8b77f16a1c752230bd

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m31s`, train: `5m23s`, total: `6m54s`	0.7942 (0.00)	0.4235 (0.00)	0.5382 (0.00)
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m53s`, train: `4m47s`, total: `6m40s`	0.8078 (0.00)	0.7787 (0.00)	0.5430 (0.00)
`Sparse + BERT + DIET(bow) + ResponseSelector(bow)` test: `1m39s`, train: `5m23s`, total: `7m2s`	0.7922 (0.00)	0.4235 (0.00)	0.5497 (0.00)
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m1s`, train: `5m35s`, total: `7m36s`	0.7806 (0.00)	0.7880 (0.00)	0.5629 (0.00)
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `46s`, train: `3m2s`, total: `3m47s`	0.7456 (0.00)	0.4235 (0.00)	0.4901 (0.00)
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `1m9s`, train: `4m26s`, total: `5m34s`	0.7398 (0.00)	0.7022 (0.00)	0.5166 (0.00)

koernerfelicia · 2021-10-14T11:18:19Z

Run when compared to # 134 (last fully successful model reg. test) https://github.com/RasaHQ/rasa/actions/runs/1111032086

Dataset: Carbon Bot, Dataset repository branch: architecture-revamp/modified-spacy-configs, commit: a88501268464439d1d058a8b77f16a1c752230bd

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m28s`, train: `4m12s`, total: `5m40s`	0.7942 (0.00)	0.4235 (-0.33)	0.5382 (0.00)
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m41s`, train: `4m19s`, total: `5m59s`	0.8078 (0.01)	0.7787 (0.00)	0.5430 (0.00)
`Sparse + BERT + DIET(bow) + ResponseSelector(bow)` test: `1m31s`, train: `4m54s`, total: `6m24s`	0.7689 (-0.01)	0.4235 (-0.33)	0.5497 (-0.01)
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m48s`, train: `5m8s`, total: `6m55s`	0.7806 (0.00)	0.7880 (-0.01)	0.5497 (-0.03)
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `44s`, train: `2m50s`, total: `3m34s`	0.7398 (-0.00)	0.4235 (-0.33)	0.5249 (-0.01)
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `1m3s`, train: `4m9s`, total: `5m11s`	0.7398 (0.00)	0.7022 (0.02)	0.5497 (0.01)

Dataset: Hermit, Dataset repository branch: architecture-revamp/modified-spacy-configs, commit: a88501268464439d1d058a8b77f16a1c752230bd

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `2m30s`, train: `21m21s`, total: `23m51s`	0.8978 (0.00)	0.7504 (0.00)	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m46s`, train: `15m1s`, total: `17m47s`	0.8913 (-0.00)	0.8049 (-0.00)	`no data`
`Sparse + BERT + DIET(bow) + ResponseSelector(bow)` test: `2m40s`, train: `26m49s`, total: `29m28s`	0.8699 (0.00)	0.7504 (0.00)	`no data`
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m56s`, train: `19m0s`, total: `21m56s`	0.8727 (0.00)	0.7994 (0.00)	`no data`
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `1m6s`, train: `21m14s`, total: `22m19s`	0.8336 (0.00)	0.7504 (0.00)	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `1m21s`, train: `14m38s`, total: `15m58s`	0.8355 (0.00)	0.7561 (0.00)	`no data`

Dataset: Private 1, Dataset repository branch: architecture-revamp/modified-spacy-configs, commit: a88501268464439d1d058a8b77f16a1c752230bd

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m49s`, train: `4m1s`, total: `5m49s`	0.9096 (0.00)	0.9612 (0.00)	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m10s`, train: `3m46s`, total: `5m56s`	0.9148 (0.00)	0.9717 (0.00)	`no data`
`Spacy + DIET(bow) + ResponseSelector(bow)` test: `34s`, train: `3m2s`, total: `3m35s`	0.8420 (0.00)	0.9574 (0.00)	`no data`
`Spacy + DIET(seq) + ResponseSelector(t2t)` test: `53s`, train: `3m25s`, total: `4m17s`	0.8534 (-0.00)	0.9444 (0.00)	`no data`
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `29s`, train: `3m37s`, total: `4m6s`	0.8929 (-0.01)	0.9612 (0.00)	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `48s`, train: `3m32s`, total: `4m20s`	0.9085 (0.01)	0.9735 (0.00)	`no data`
`Sparse + Spacy + DIET(bow) + ResponseSelector(bow)` test: `38s`, train: `4m34s`, total: `5m11s`	0.8960 (0.01)	0.9574 (0.00)	`no data`
`Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)` test: `57s`, train: `4m21s`, total: `5m18s`	0.8929 (-0.00)	0.9700 (0.00)	`no data`

Dataset: Private 2, Dataset repository branch: architecture-revamp/modified-spacy-configs, commit: a88501268464439d1d058a8b77f16a1c752230bd

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m50s`, train: `11m17s`, total: `13m7s`	0.8745 (0.00)	`no data`	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m5s`, train: `6m18s`, total: `8m23s`	0.8830 (0.00)	`no data`	`no data`
`Spacy + DIET(bow) + ResponseSelector(bow)` test: `38s`, train: `5m59s`, total: `6m36s`	0.7253 (0.00)	`no data`	`no data`
`Spacy + DIET(seq) + ResponseSelector(t2t)` test: `47s`, train: `6m0s`, total: `6m47s`	0.7822 (0.00)	`no data`	`no data`
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `37s`, train: `5m41s`, total: `6m18s`	0.8498 (-0.01)	`no data`	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `44s`, train: `5m40s`, total: `6m24s`	0.8530 (0.00)	`no data`	`no data`
`Sparse + Spacy + DIET(bow) + ResponseSelector(bow)` test: `48s`, train: `8m35s`, total: `9m23s`	0.8594 (0.00)	`no data`	`no data`
`Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)` test: `52s`, train: `7m16s`, total: `8m8s`	0.8509 (0.00)	`no data`	`no data`

Dataset: Private 3, Dataset repository branch: architecture-revamp/modified-spacy-configs, commit: a88501268464439d1d058a8b77f16a1c752230bd

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `59s`, train: `1m10s`, total: `2m8s`	0.9177 (0.00)	`no data`	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m4s`, train: `52s`, total: `1m55s`	0.8436 (0.00)	`no data`	`no data`
`Spacy + DIET(bow) + ResponseSelector(bow)` test: `36s`, train: `54s`, total: `1m30s`	0.6132 (0.00)	`no data`	`no data`
`Spacy + DIET(seq) + ResponseSelector(t2t)` test: `42s`, train: `44s`, total: `1m25s`	0.6255 (0.00)	`no data`	`no data`
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `33s`, train: `1m6s`, total: `1m38s`	0.8683 (0.00)	`no data`	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `37s`, train: `49s`, total: `1m25s`	0.8642 (0.00)	`no data`	`no data`
`Sparse + Spacy + DIET(bow) + ResponseSelector(bow)` test: `37s`, train: `1m18s`, total: `1m55s`	0.8436 (-0.00)	`no data`	`no data`
`Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)` test: `43s`, train: `58s`, total: `1m40s`	0.8601 (0.00)	`no data`	`no data`

Dataset: Sara, Dataset repository branch: architecture-revamp/modified-spacy-configs, commit: a88501268464439d1d058a8b77f16a1c752230bd

Dialog Policy Configuration	Action Level Micro Avg. F1	Conversation Level Accuracy	Run Time Train	Run Time Test
`Rules`	0.1266 (0.00)	0.0000 (0.00)	`2m34s`	`1m24s`
`Rules + AugMemo`	0.9149 (-0.01)	0.6301 (-0.03)	`2m30s`	`1m33s`
`Rules + AugMemo + TED`	0.9727 (-0.00)	0.7534 (0.00)	`13m8s`	`3m6s`
`Rules + Memo`	0.3860 (0.00)	0.1438 (0.00)	`2m34s`	`1m27s`
`Rules + Memo + TED`	0.9555 (-0.00)	0.6644 (0.00)	`13m2s`	`3m5s`

koernerfelicia · 2021-10-14T11:28:41Z

Run when compared to TF 2.6 (note 11% drop for Rules+Memo for Sara is not explained yet, ignore this)

Dataset: Carbon Bot, Dataset repository branch: architecture-revamp/modified-spacy-configs, commit: a88501268464439d1d058a8b77f16a1c752230bd

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m28s`, train: `4m12s`, total: `5m40s`	0.7942 (0.00)	0.4235 (-0.33)	0.5382 (0.00)
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m41s`, train: `4m19s`, total: `5m59s`	0.8078 (0.01)	0.7787 (0.00)	0.5430 (0.01)
`Sparse + BERT + DIET(bow) + ResponseSelector(bow)` test: `1m31s`, train: `4m54s`, total: `6m24s`	0.7689 (-0.03)	0.4235 (-0.33)	0.5497 (-0.01)
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m48s`, train: `5m8s`, total: `6m55s`	0.7806 (0.00)	0.7880 (-0.01)	0.5497 (-0.03)
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `44s`, train: `2m50s`, total: `3m34s`	0.7398 (-0.01)	0.4235 (-0.33)	0.5249 (-0.01)
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `1m3s`, train: `4m9s`, total: `5m11s`	0.7398 (0.00)	0.7022 (0.02)	0.5497 (0.01)

Dataset: Hermit, Dataset repository branch: architecture-revamp/modified-spacy-configs, commit: a88501268464439d1d058a8b77f16a1c752230bd

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `2m30s`, train: `21m21s`, total: `23m51s`	0.8978 (-0.00)	0.7504 (0.00)	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m46s`, train: `15m1s`, total: `17m47s`	0.8913 (0.00)	0.8049 (0.00)	`no data`
`Sparse + BERT + DIET(bow) + ResponseSelector(bow)` test: `2m40s`, train: `26m49s`, total: `29m28s`	0.8699 (-0.00)	0.7504 (0.00)	`no data`
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m56s`, train: `19m0s`, total: `21m56s`	0.8727 (0.00)	0.7994 (0.00)	`no data`
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `1m6s`, train: `21m14s`, total: `22m19s`	0.8336 (0.00)	0.7504 (0.00)	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `1m21s`, train: `14m38s`, total: `15m58s`	0.8355 (0.00)	0.7561 (0.00)	`no data`

Dataset: Private 1, Dataset repository branch: architecture-revamp/modified-spacy-configs, commit: a88501268464439d1d058a8b77f16a1c752230bd

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m49s`, train: `4m1s`, total: `5m49s`	0.9096 (0.00)	0.9612 (0.00)	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m10s`, train: `3m46s`, total: `5m56s`	0.9148 (0.00)	0.9717 (0.00)	`no data`
`Spacy + DIET(bow) + ResponseSelector(bow)` test: `34s`, train: `3m2s`, total: `3m35s`	0.8420 (0.00)	0.9574 (0.00)	`no data`
`Spacy + DIET(seq) + ResponseSelector(t2t)` test: `53s`, train: `3m25s`, total: `4m17s`	0.8534 (-0.00)	0.9444 (0.00)	`no data`
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `29s`, train: `3m37s`, total: `4m6s`	0.8929 (0.00)	0.9612 (0.00)	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `48s`, train: `3m32s`, total: `4m20s`	0.9085 (0.01)	0.9735 (0.00)	`no data`
`Sparse + Spacy + DIET(bow) + ResponseSelector(bow)` test: `38s`, train: `4m34s`, total: `5m11s`	0.8960 (0.00)	0.9574 (0.00)	`no data`
`Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)` test: `57s`, train: `4m21s`, total: `5m18s`	0.8929 (-0.01)	0.9700 (0.00)	`no data`

Dataset: Private 2, Dataset repository branch: architecture-revamp/modified-spacy-configs, commit: a88501268464439d1d058a8b77f16a1c752230bd

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m50s`, train: `11m17s`, total: `13m7s`	0.8745 (0.00)	`no data`	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m5s`, train: `6m18s`, total: `8m23s`	0.8830 (0.00)	`no data`	`no data`
`Spacy + DIET(bow) + ResponseSelector(bow)` test: `38s`, train: `5m59s`, total: `6m36s`	0.7253 (-0.00)	`no data`	`no data`
`Spacy + DIET(seq) + ResponseSelector(t2t)` test: `47s`, train: `6m0s`, total: `6m47s`	0.7822 (0.00)	`no data`	`no data`
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `37s`, train: `5m41s`, total: `6m18s`	0.8498 (0.01)	`no data`	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `44s`, train: `5m40s`, total: `6m24s`	0.8530 (-0.00)	`no data`	`no data`
`Sparse + Spacy + DIET(bow) + ResponseSelector(bow)` test: `48s`, train: `8m35s`, total: `9m23s`	0.8594 (-0.00)	`no data`	`no data`
`Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)` test: `52s`, train: `7m16s`, total: `8m8s`	0.8509 (0.00)	`no data`	`no data`

Dataset: Private 3, Dataset repository branch: architecture-revamp/modified-spacy-configs, commit: a88501268464439d1d058a8b77f16a1c752230bd

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `59s`, train: `1m10s`, total: `2m8s`	0.9177 (0.00)	`no data`	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m4s`, train: `52s`, total: `1m55s`	0.8436 (0.00)	`no data`	`no data`
`Spacy + DIET(bow) + ResponseSelector(bow)` test: `36s`, train: `54s`, total: `1m30s`	0.6132 (-0.00)	`no data`	`no data`
`Spacy + DIET(seq) + ResponseSelector(t2t)` test: `42s`, train: `44s`, total: `1m25s`	0.6255 (0.00)	`no data`	`no data`
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `33s`, train: `1m6s`, total: `1m38s`	0.8683 (0.00)	`no data`	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `37s`, train: `49s`, total: `1m25s`	0.8642 (0.00)	`no data`	`no data`
`Sparse + Spacy + DIET(bow) + ResponseSelector(bow)` test: `37s`, train: `1m18s`, total: `1m55s`	0.8436 (-0.00)	`no data`	`no data`
`Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)` test: `43s`, train: `58s`, total: `1m40s`	0.8601 (0.00)	`no data`	`no data`

Dataset: Sara, Dataset repository branch: architecture-revamp/modified-spacy-configs, commit: a88501268464439d1d058a8b77f16a1c752230bd

Dialog Policy Configuration	Action Level Micro Avg. F1	Conversation Level Accuracy	Run Time Train	Run Time Test
`Rules`	0.1266 (0.00)	0.0000 (0.00)	`2m34s`	`1m24s`
`Rules + AugMemo`	0.9149 (0.00)	0.6301 (0.00)	`2m30s`	`1m33s`
`Rules + AugMemo + TED`	0.9727 (-0.00)	0.7534 (-0.00)	`13m8s`	`3m6s`
`Rules + Memo`	0.3860 (-0.11)	0.1438 (-0.03)	`2m34s`	`1m27s`
`Rules + Memo + TED`	0.9555 (-0.00)	0.6644 (-0.00)	`13m2s`	`3m5s`

github-actions · 2021-10-14T16:33:54Z

Hey @wochinge! 👋 To run model regression tests, comment with the /modeltest command and a configuration.

Tips 💡: The model regression test will be run on push events. You can re-run the tests by re-add status:model-regression-tests label or use a Re-run jobs button in Github Actions workflow.

Tips 💡: Every time when you want to change a configuration you should edit the comment with the previous configuration.

You can copy this in your comment and customize:

/modeltest

```yml
##########
## Available datasets
##########
# - "Carbon Bot" (NLU)
# - "Hermit" (NLU)
# - "Private 1" (NLU)
# - "Private 2" (NLU)
# - "Private 3" (NLU)
# - "Sara" (NLU, Core)
# - "financial-demo" (NLU, Core)
# - "helpdesk-assistant" (NLU, Core)
# - "insurance-demo" (NLU, Core)
# - "retail-demo" (NLU, Core)

##########
## Available NLU configurations
##########
# - "BERT + DIET(bow) + ResponseSelector(bow)"
# - "BERT + DIET(seq) + ResponseSelector(t2t)"
# - "Spacy + DIET(bow) + ResponseSelector(bow)"
# - "Spacy + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + BERT + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + BERT + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + Spacy + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)"

##########
## Available Core configurations
##########
# - "Rules"
# - "Rules + AugMemo"
# - "Rules + AugMemo + TED"
# - "Rules + Memo"
# - "Rules + Memo + TED"
# - "Rules + TED"

## Example configuration
#################### syntax #################
## include:
##   - dataset: ["<dataset_name>"]
##     config: ["<configuration_name>"]
#
## Example:
## include:
##  - dataset: ["Carbon Bot"]
##    config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]
#
## Shortcut:
## You can use the "all" shortcut to include all available configurations or datasets
#
## Example: Use the "Sparse + EmbeddingIntent + ResponseSelector(bow)" configuration
## for all available datasets
## include:
##  - dataset: ["all"]
##    config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]
#
## Example: Use all available configurations for the "Carbon Bot" and "Sara" datasets
## and for the "Hermit" dataset use the "Sparse + DIET + ResponseSelector(T2T)" and
## "BERT + DIET + ResponseSelector(T2T)" configurations:
## include:
##  - dataset: ["Carbon Bot", "Sara"]
##    config: ["all"]
##  - dataset: ["Hermit"]
##    config: ["Sparse + DIET(seq) + ResponseSelector(t2t)", "BERT + DIET(seq) + ResponseSelector(t2t)"]
#
## Example: Define a branch name to check-out for a dataset repository. Default branch is 'main'
## dataset_branch: "test-branch"
## include:
##  - dataset: ["Carbon Bot", "Sara"]
##    config: ["all"]
##
## Shortcuts:
## You can use the "all" shortcut to include all available configurations or datasets.
## You can use the "all-nlu" shortcut to include all available NLU configurations or datasets.
## You can use the "all-core" shortcut to include all available core configurations or datasets.

include:
 - dataset: ["Carbon Bot"]
   config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]

```

github-actions · 2021-10-14T16:33:58Z

/modeltest

dataset_branch: "architecture-revamp/modified-spacy-configs"
include:
 - dataset: ["Carbon Bot"]
   config:
   - "BERT + DIET(bow) + ResponseSelector(bow)"
   - "BERT + DIET(seq) + ResponseSelector(t2t)"
   - "Spacy + DIET(bow) + ResponseSelector(bow)"
   - "Spacy + DIET(seq) + ResponseSelector(t2t)"
   - "Sparse + BERT + DIET(bow) + ResponseSelector(bow)"
   - "Sparse + BERT + DIET(seq) + ResponseSelector(t2t)"
   - "Sparse + DIET(bow) + ResponseSelector(bow)"
   - "Sparse + DIET(seq) + ResponseSelector(t2t)"
   - "Sparse + Spacy + DIET(bow) + ResponseSelector(bow)"
   - "Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)"

github-actions · 2021-10-14T16:34:00Z

The model regression tests have started. It might take a while, please be patient.
As soon as results are ready you'll see a new comment with the results.

Used configuration can be found in the comment.

github-actions · 2021-10-14T17:19:41Z

Commit: 58cc760, The full report is available as an artifact.

Dataset: Carbon Bot, Dataset repository branch: architecture-revamp/modified-spacy-configs, commit: a88501268464439d1d058a8b77f16a1c752230bd

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m23s`, train: `4m12s`, total: `5m34s`	0.7942 (0.00)	0.7529 (0.00)	0.5382 (0.00)
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m42s`, train: `4m24s`, total: `6m5s`	0.8078 (0.00)	0.7787 (0.00)	0.5430 (0.00)
`Sparse + BERT + DIET(bow) + ResponseSelector(bow)` test: `1m31s`, train: `4m55s`, total: `6m25s`	0.7883 (0.00)	0.7529 (0.00)	0.5695 (0.00)
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m48s`, train: `5m7s`, total: `6m55s`	0.7806 (0.00)	0.7880 (0.00)	0.5695 (0.00)
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `42s`, train: `2m47s`, total: `3m29s`	0.7612 (0.00)	0.7529 (0.00)	0.5316 (0.00)
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `1m0s`, train: `4m0s`, total: `5m0s`	0.7398 (0.00)	0.7022 (0.00)	0.5099 (0.00)

wochinge added 30 commits September 16, 2021 13:11

implement recipes

71d23b8

rasa train

2e1893b

predict

fa2ea3e

fix recipe tests

a896608

implement train

a01e1bc

fix wrong imports

1e74df4

fix CLI tests for training

3dfc539

reduce code quality warnings

43639b7

fixes for model training

06f4a79

fixes for ted policy training

f385544

fix model train fixtures

7ed206a

fix test_model_training tests

56dec3f

Merge branch 'main' into 3.0-architecture-revamp/9277/recipe

741dd0e

fix e2e fingerprint tests

ec3d721

improve global cache directory fixture

ac47eda

register AugmentedMemoizationPolicy

b29801b

fix some broken tests

72c950e

delete old fingerprinting / finetuning detection code

521ce31

integrate FinetuningValidator

9682e31

Merge remote-tracking branch 'origin/3.0-architecture-revamp/9277/rec…

83f913d

…ipe' into 3.0-architecture-revamp/9604/train-and-run

Merge branch '3.0-architecture-revamp/9604/train-and-run' into 3.0-ar…

492c191

…chitecture-revamp/9604/train

implement finetuning

20751f6

fix some tests in test_train

d13b153

fix more finetuning tests

c7603e3

remove log statements

b45a520

more test fixes

acd4f1f

more finetuning fixes

5851b65

tests for response fingerprinting

6ced0fd

polish

7c13ff7

reset changes to agent / processor / run / loader / server

b867526

Merge branch 'main' into 3.0-architecture-revamp/9277/recipe

e9634b9

wochinge added runner:gpu status:model-regression-tests labels Oct 14, 2021

github-actions bot removed status:model-regression-tests runner:gpu labels Oct 14, 2021

wochinge added 2 commits October 14, 2021 10:59

remove unused imports

7b4522d

Merge branch 'main' into 3.0-architecture-revamp/9277/recipe

dd88a2c

wochinge added runner:gpu status:model-regression-tests and removed status:model-regression-tests labels Oct 14, 2021

github-actions bot removed status:model-regression-tests runner:gpu labels Oct 14, 2021

fix CRFEntityExtractor persistence / loading

bfe6575

wochinge added runner:gpu status:model-regression-tests labels Oct 14, 2021

add unit test to avoid regression

fd521e4

github-actions bot removed status:model-regression-tests runner:gpu labels Oct 14, 2021

wochinge merged commit e2d1e9b into main Oct 14, 2021

wochinge deleted the 3.0-architecture-revamp/9277/recipe branch October 14, 2021 19:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate Graph Architecture with rest of codebase #9803

Integrate Graph Architecture with rest of codebase #9803

wochinge commented Oct 6, 2021 •

edited

Loading

wochinge commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

koernerfelicia commented Oct 14, 2021

koernerfelicia commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

Integrate Graph Architecture with rest of codebase #9803

Integrate Graph Architecture with rest of codebase #9803

Conversation

wochinge commented Oct 6, 2021 • edited Loading

wochinge commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

koernerfelicia commented Oct 14, 2021

koernerfelicia commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

wochinge commented Oct 6, 2021 •

edited

Loading