speed up yaml parsing

[wochinge]

Problem
When parsing yaml files we currently look for environment variables to interpolate in every single line. While this is a useful feature for configuration files (endpoints.yml, config.yml, credentials.yml) and e.g. also used for Rasa X, this makes the training data reading very slow as stated here.

Note that removing this would be a breaking change as some users might rely on it. We need some sort of smooth transition for it.

Note that I've not investigated how much this line also contributes to the slowness

Ideas for solutions:

• disable environment variable interpolation for stories / nlu files by default
• add a switch to disable it

Definition of Done

• Environment variable interpolation no longer is a blocking factor when reading big training data files
• investigate if this line is still needed / what it's impact is on training data loading time
• document usage / migration path for environment variable interpolation

[alwx]

@wochinge I have a much simpler idea — what if we just check if the file contains ${...} before parsing it and use a faster parser (= the one that's without environment variable interpolation) if there is nothing? In this case, the migration process for end users won't be needed at all.

[wochinge]

That's also an option! Scanning a big file with open(...) as file should be quick, no?

[alwx]

yes! and we need to do this anyway — the YAML needs to be parsed no matter if it contains env variables or not