Upgrade to TF 2.6

[dakshvar22]

TF 2.4 is out and hence we should update the dependency inside Rasa OS to use it.
Introduces some breaking changes as mentioned in the changelog

Update: we are now targeting 2.6 see here

Note that there's a draft PR here that already contains some of the necessary changes. Known remaining tasks in order to actually update code for TF 2.5 are (this list contains only the known remaining tasks, there may be others hence it is not an exhaustive of all remaining tasks):

• Run model regression tests on all configs and datasets to see if training times / accuracy are the same
• see TODOs marked with 2.5 (ex: here)
• Investigate runtime and memory changes due to TF 2.5 #9433
• Investigate increased test runtime with TF 2.5 #9512

Temporarily de-prioritised followups:

• Investigate runtime and memory changes due to TF 2.6 on core #9514
• Investigate increase to CPU memory usage due to 2.6 for runs with cvf_diet_responset2t #9515
• Investigate outlier in GPU memory usage with 2.6 (train): Hermit + cvf_diet_responset2t #9516
• Investigate failed memory leak tests with upgrade to TF 2.6 #9764
• Investigate increased runtime of LMFeaturizer.parse with TF 2.6 #9666
• Investigate and address OOM due to TF 2.6 for CI model regression tests with Hermit and BERT + DIET(seq) + ResponseSelector(t2t) or Sparse + BERT + DIET(seq) + ResponseSelector(t2t) #9798
• Investigate and address OOM for Windows + test-nlu-predictors on CI with TF 2.6 #9734

Definition of done
No memory tests failing
No increased timeouts for tests
Model regression tests passing with approx. same time and accuracy

• Followup issue created: determine w DevRel how to communicate hits to community https://github.com/RasaHQ/research/issues/220
• Followup issue created: determine w CSE how to communicate hits to customers https://github.com/RasaHQ/research/issues/219
• Followup issue created: update CUDA drivers on GPU runners to be compatible with TF 2.5 : Update CUDA/cuDNN libraries on GPU runners #9611
• Model regression tests on GPU have approx. same accuracy
• Lockfile reflects correct TF dependency
• Create a PR for the bumped up dependency
• Create followup issue to investigate and address memory problems on Windows CI tests, so that this fix can be reverted Investigate and address OOM for Windows + test-nlu-predictors on CI with TF 2.6 #9734
• Create followup issue to investigate and address memory problems for Hermit CI model regression tests so that they can be run on schedule again
  Investigate and address OOM due to TF 2.6 for CI model regression tests with Hermit and BERT + DIET(seq) + ResponseSelector(t2t) or Sparse + BERT + DIET(seq) + ResponseSelector(t2t) #9798

[eikooc]

Rasa does not work on the new M1 chip before upgrading this dependency. Just thought I would leave that as a note here.

[Ghostvv]

blocked until a new version is released with this fix: https://github.com/tensorflow/tensorflow/pull/45534/files#diff-39eadfc771b57ef11bff67b85cc38a47b9c13ec203b145e3feaf556a43bdccb1R4762 - no, it doesn't seem to be the problem

[Ghostvv]

pushed the necessary code updates to tf-2.4 branch

[Ghostvv]

blocked by: tensorflow/tensorflow#46511

[koernerfelicia]

Once this is done should check this: #7762

[Ghostvv]

most probably we have to wait till 2.5

[koernerfelicia]

Related: #7793

[luzhongqiu]

same issue, wait for tf above 2.4 version, thanks

[rmiaouh]

Hi guys, same issue, we will need tf 2.4 version, ty for all.

[koernerfelicia]

Once this is done we should revisit this and see if it has been fixed:

#8004

[Ghostvv]

we cannot upgrade to 2.4, we need to wait till 2.5

[koernerfelicia]

Yeah, sorry I was being unclear, at this point when I say "done" I mean whatever tensorflow update we eventually manage

[marjoripomarole]

2.5 is now out!

https://developer.apple.com/metal/tensorflow-plugin/

https://github.com/tensorflow/tensorflow/releases/tag/v2.5.0

Seems like this can move forward?

[tmbo]

@TyDunn is this one still blocked? given that we'll need to buy m1s as well, would be great to sort this out 😅

[koernerfelicia]

@tmbo I'm trying to verify this atm, will update when I know more!

[marjoripomarole]

I am available to help migrate this over and test. Let me know if there is any ongoing work and how I can help speed it up. I have a M1 on hand and the desire to get rasa working asap on it :)

[koernerfelicia]

@mpomarole thank you for the kind offer! We'll let you know if we need any support :)

[dakshvar22]

@samsucik Do we have a small tensorflow code snippet with a very basic model and dummy data that could demonstrate this increase in training time? I think it would be worthwile to ask the tensorflow folks if we are doing something wrong or the increase is indeed expected.

Also, is the increase in time observable on both CPU and GPU?

[samsucik]

@dakshvar22 I'm coming back to this after almost a week and my previous comments basically summarise where I had left off. What you're describing is the state I'd like to reach asap 😉

Also, is the increase in time observable on both CPU and GPU?

Yes. I saw the regression tests taking 3x ~5x longer (on GPU runners) and I saw the same/similar thing with e2ebot trained locally on a CPU.

[samsucik]

Runtimes & failing model regression tests

To complete my previous update on model regression tests, I've re-run the tests on the Hermit dataset and tests involving the transformer version of DIET (i.e. DIET(seq) all failed again, the same way as before. It looks as if the failed runs were killed because they were taking too long -- they got killed after ~2 hours, whereas they normally take only around 15 minutes (see the most recent scheduled run). This needs further investigation -- it might be due to something hanging up, though I wouldn't necessarily link it to the hanging up we observed previously.

All in all, with TF 2.5, our regression test runs take ~5x longer and sometimes even get killed in the process (TF 2.5 run vs scheduled run).

[samsucik]

Overall status update & handover notes

My work is in the tf-2.5 branch and this associated PR, the old tf-2.4 branch and the associated PR are now behind and abandoned -- though the regression tests were run on the old PR (the new one didn't exist yet).

Having observed the regression tests taking so long, I profiled the training of our small e2ebot and took it from there (since this training was also taking much longer with TF 2.5). Using snakeviz, I noticed that a number of ops were taking particularly longer, especially IfGrad and WhileGrad, and the various TF CRF methods such as crf_log_norm or crf_log_likelihood. However, this later turned to be slightly misleading because the high runtime isn't really due to something like IfGrad, but potentially to some of its sub-calls. Still, seeing the CRF methods taking so long, I returned to the distilled example of our CRF layer used here, adapted it and used it for further investigation. Indeed, I saw the CRF methods still taking longer with TF 2.5, but I found it hard to actually pinpoint the precise issue (whether by searching through others' complaints on the internet, or by digging into the code, or by using the profiling tools). While I managed to make TF Profiler work with both the e2ebot-training example and with the CRF one, I'm quite new to the tool and felt like I wasn't getting too much value out of it in terms of pinpointing the precise TF ops. Importantly, in the Profiler stats, I didn't see the same ops topping the runtime list as I saw when analysing the general profiling results...

All of my code used for comparing TF 2.3 vs 2.5 runs of various Python examples is in this project dir. The README should help you with everything you need.

[samsucik]

Btw as for the "investigate failed memory leak tests.. these are flakey" task in the issue description, I didn't get to look into it. I did not do memory profiling, only runtime profiling.

Regarding the failing windows tests (e.g. here), in some cases these are seemingly due to memory errors, but sometimes all that we observe is that a worker crashes, so it's not all so clear. Additionally, there are some Ubuntu tests failing too (though I think that's a bit flakey and may not be related to TF 2.5). In any case, I think that memory profiling would help a lot here.

[koernerfelicia]

Related issue: #9129. This upgrade blocks our ability to fix since gelu was moved to tensorflow core in 2.4 and up

[koernerfelicia]

Dug a little deeper into the model regression test outputs. Looks like this also affects DIET without entity recognition. See here

[koernerfelicia]

I'm not sure what we can generalise about the effect on CPU performance. After yesterday's results re: DIET without entity recognition I ran some CPU regression tests to see whether this is true also on CPU. See here. Will start the full suite of CPU tests to dig into this further.

[ancalita]

On removing @training.enable_multi_worker (this TODO item)

@samsucik Sentry has flagged this error recently caused by this decorator - digging a bit through the forum, it seems to affect users who are upgrading to TF versions higher than the one currently specified in rasa poetry.lock.
Are you able to look into this as part of this issue or should I create a new issue for this decorator specifically and place it in the Research inbox?

[koernerfelicia]

@ancalita (Sam is currently out and I'm working on the issue with Daksh). We'll remove it as part of the upgrade, though this might be a while (about to write an update as my next comment). I think we can restrict users to 2.3.3 and wait until the upgrade, unless this error is urgent, in which case I think we can remove it, and possibly create an issue for someone to look into how to enable distributed training. Latter is out of scope for this project, and may be better handled by Engine.

[koernerfelicia]

Issue affecting our custom CRF layer has been communicated to tensorflow here. As far as we know there is no workaround. Waiting to hear otherwise or progress on the issue.

[ancalita]

@koernerfelicia the error is not urgent, it can wait until the upgrade.

[koernerfelicia]

investigate crashed workers on Windows 3.6 and 3.7 during test_e2e_with_entity_evaluation (test-other-unit-tests)

Crashed tests on Windows seem to be due to OOM (see here for Slack discussion, and here for possible solutions to the general issue of OOM on CI).

[koernerfelicia]

Tests for which we had to increase timeouts or memory threshold. Once TF addresses the issue above, we should aim to bring these timeouts back down.

test_train_persist_load_with_different_settings_non_windows
test_train_persist_load_with_different_settings
test_train_persist_load_with_only_entity_recognition
test_train_persist_load_with_composite_entities
test_lm_featurizer_shape_values_train
test_lm_featurizer_number_of_sub_tokens_process
TestNLULeakManyEpochs

[koernerfelicia]

See here for more workarounds: https://www.notion.so/rasa/tf-2-6-workarounds-timeouts-memory-f3706b6322214574b0e97b5689903524

[m-vdb]

Closing as this has been done as part of #9649