Investigate higher number of sentry bugs

[joejuzl]

We are hitting the limit of 50k errors/month.
We should see why this number is higher.
Is it because of:

• More users?
• User errors that should be ignored?
• More bugs?
  Should we upgrade to the next sentry plan?

Definition of done:

• Find the cause(es) for more bugs
• Determine if we should upgrade to next sentry plan
• Open followup issues if necessary

[m-vdb]

Received this Sentry report in my inbox on Monday:

[wochinge]

Investigation Results

• We had 280 000 error events in the last 30 days
• Tom already bumped the event limit to 500 000. This costs 77 $ per month more than 50k
• There has been no increase in users
• There has been an increase in daily (resolved + unresolved) events on April 29. We're on a high level ever since. We went from ~500 events to ~1500 per day.
  • The issue causing most events is already handled (~120k)
  • A few users are causing most of the bugs
    • Unfortunately there is no client side throttling . We can change the sampling rate but I'd rather pay more than lose visibility

• By ignoring OSErrors and fixing Pika errors we already made huge step in the right direction

Next steps

• Prioritize and tackle unresolved issue which cause a lot of events
  • 18 000: Telegram: https://sentry.io/organizations/rasahq/issues/2228387098/?end=2021-04-29T23%3A59%3A59&project=2801673&query=is%3Aunresolved&sort=freq&start=2021-04-29T00%3A00%3A00&utc=true (~ 700 per day)
  • 24 000, redis: https://sentry.io/organizations/rasahq/issues/2394425410/?project=2801673&query=&sort=freq&statsPeriod=30d
  • 12 0000, tracker-store:https://sentry.io/organizations/rasahq/issues/2260221914/?project=2801673&query=&sort=freq&statsPeriod=30d

Not suggested next steps

• Introduce a separate alert for issues with more than 5000 events per week. I wouldn't do this as these are mostly caused by very few users. Issues which impact more users are covered by existing rules.
• I suggest setting the quota back to 200 000. If we discard the ValueError and the OSErrors this accounts for roughly 162 000 errors which leaves us with ~120 000 errors. 200 000 errors should hence still give us plenty of buffer. Reason: see Tom's comment
• doesn't work as we don't have a business plan:
  • Ignored and discard the ValueError for environment variables which caused ~120k events. This means that this issue will no longer count towards our quota (see 4.1 here). This operation is revertible in case we think we made the wrong decision at same point.
  • Do the same for the OSErrors

[tmbo]

Happy to keep the limit at 500k, the cost difference is marginal to 200k

[m-vdb]

the ValueError seems a bit tricky: are we completely sure it's because of a usage error? I remember that there was a recent issue with how Rasa X discovered yaml files, and it tried to read the .github/ folder of a git repository (in this case the culprit was Rasa X).

[wochinge]

the ValueError seems a bit tricky: are we completely sure it's because of a usage error?

I think it might be possible that Rasa Open Source causes this error if users put completely unrelated files in the data directory which are very similar to Rasa training data files and contain env variables placeholder. I'd still classify that as a user error though. We've also changed to a RasaException (#8382) anyway. So discarding this issue in sentry would affect only what old Rasa Open Source versions (pre this fix) send anyway.

[wochinge]

I can't discard the ValueError 🤦🏻 It's a feature that requires a business plan. Let's keep it for now. Users will eventually move to newer releases which don't send a sentry event for this