Merge 2.8.x into main

[rasabot]

🛺 This PR should be merged automatically once it has been approved. If it doesn't happen:

• Handle merge conflicts
• Fix build errors

💡 It has been opened automatically after changes were merged in a feature branch.

[wochinge]

💯 Thank you for fixing the logs! 🎉

[ka-bu]

@koernerfelicia @wochinge I'm repeatedly either seeing a windows access violation error or an OOM tf error it seems on e.g. "other" for windows 3.7 or windows 3.8 -- Any idea? Did we observe this before?
There are two warnings that are just filtered out at the moment. I was assuming that those warnings appeared in 2.8.x as well - and as far as I understood the tf.gather warning should be negligible and the np.array issue didn't seem to make a difference as things where still dtype=obj (i.e. no unexpected conversions whatever happening). But if those issues are new .... 🤔

[koernerfelicia]

Did we observe this before?

Sadly, yes, we observed this with test-nlu-predictors and windows after the TF 2.6 upgrade. The temporary fix was to run those in a single thread (as opposed to two). See here. IMO if we were okay with that temporary change, we could also do the same here, and then make an issue to follow up and revert the changes. I don't love that, but also don't have any great ideas to suggest otherwise. I know we've made some small wins to the testing suite memory consumption in the past, and Enable is planning on exploring other options (see here)

I think that the windows runners are a bit of a canary (imagery from Matt 🐤 ), in that they are the first to go OOM when we're close to the limit.

I was assuming that those warnings appeared in 2.8.x as well

I'm pretty sure you're right about that, though I'd have to check to be 100% sure.

[koernerfelicia]

Congratulations, @ka-bu! What a tough one 😬 If you haven't already, can you either make an issue or extend the one I linked to remind us to revert the monothreaded testing (like this one)

[ka-bu]

Congratulations, @ka-bu! What a tough one 😬 If you haven't already, can you either make an issue or extend the one I linked to remind us to revert the monothreaded testing (like this one)

Thanks so much for reminding me! :) I'll extend you issue