Rasa 1.10.14 Memory Leakage

[OmarFarag95]

Rasa version: 1.10.14

Rasa SDK version (if used & relevant):

Rasa X version (if used & relevant):

Python version: 3.6.9

Operating system (windows, osx, ...): linux

Issue:

• I was training CRF model on about 120K sentences for entity recognition on Rasa 1.3.5 and things were going just fine. However, upon upgrading to Rasa 1.10.14; I am always getting a memory leakage (whereasa RAM tends to increase significantly until the process is killed).

• I am using Google Colab with available RAM of 25GB

• One important note, is that training DIET classifier on the same dataset size and Rasa 1.10.14 works just fine. so the issue is within the CRF itself.

Error (including full traceback):

Command or request that led to error:

rasa train nlu

Content of configuration file (config.yml) (if relevant):

  - name: "SpacyNLP"
  - name: "SpacyTokenizer"
  - name: "SpacyFeaturizer"
  - name: "CRFEntityExtractor"
    max_iterations: 100
    features: [
    ["pos", "pos2"],
    [
      "bias",
      "prefix5",
      "prefix2",
      "suffix5",
      "suffix3",
      "suffix2",
      "pos",
      "pos2",
      "digit",
    ],
    ["pos", "pos2"],
    ]
    L1_c: 0.01,
    L2_c: 0.05

Content of domain file (domain.yml) (if relevant):

Definition of Done

• assert that problem still persists in last main state (consider using Vincent's spacy-3.0 branch)
• find what's causing the memory leak and propose next steps

[sara-tagger]

Thanks for the issue, @m-vdb will get back to you about it soon!

You may find help in the docs and the forum, too 🤗

[m-vdb]

👋🏻 hey @OmarFarag95 thanks a lot for your bug report. Would it be a possibility for you to upgrade to:

• either 1.10.20, which is the latest micro we currently have on the 1.10.x branch
• or even better 2.2.4, which is backwards-compatible with 1.x
  ?

That would help us diagnose the issue and make it easier to come up with a fix (especially if you upgrade to 2.2.4). LMK if it's not on the table 😄

[OmarFarag95]

Hi @m-vdb , Thank you for your reply.

I have tried to upgrade to 2.2.4. However, the bug is still persisting.

[m-vdb]

OK, (sadly) good to know. I'll circle back with the team and find out the best way forward. Sorry about the inconvenience 😓

[OmarFarag95]

Never mind!

I will be looking forward to your response 😸

[twerkmeister]

I just snooped around in the code for the crf entity extractor and had a quick look at the differences between v 1.3.5 and v 1.10.14.

Although it is not a change, in terms of memory generally a thing that caught my eye is all_possible_transitions=True in the crf function call. The doc for this feature reads:

    all_possible_transitions : bool, optional (default=False)
        Specify whether CRFsuite generates transition features that
        do not even occur in the training data (i.e., negative transition
        features). When True, CRFsuite generates transition features that
        associate all of possible label pairs. Suppose that the number
        of labels in the training data is L, this function will
        generate (L * L) transition features.
        This function is disabled by default.

One possible idea is, that something minor changed in sentence_to_features vs the new crf_tokens_to_features that added a few more features, but given this quadratic scale up and the size of the dataset in question, it could have an outsized effect. Just one hypothesis, could be wrong!

[twerkmeister]

@OmarFarag95 would it be possible for you to provide a few more stats on the dataset you are using for context? I know you are using a sizable dataset of 120k sentences. How about (roughly)

• the number of entity types?
• and number of examples per entity type ?

[twerkmeister]

Also, I am assuming that the system already crashes during the first iteration, is that right @OmarFarag95 ?

[OmarFarag95]

@twerkmeister Thanks for your response.

• The number of distinct entity types is 8
• The number of examples per entity is nearly equal = 120k/8 = 15k
• Yes, the training fails few seconds after starting the first iteration

[twerkmeister]

Hey @OmarFarag95, much appreciated! This will make tracking down the problem easier 👍