fix: Increase process recurrence timeout

[Dschoordsch]

Fixes #9664

We know this might be long running, so adding a separate long running timeout. We're not reusing the adhoc timeout as this is excessive and might obfuscate issues in the query.

Testing scenarios

• add debug output to

  parabol/packages/server/utils/PubSubPromise.ts

  Line 57 in 4f49f95

  const timeout = isAdHoc ? ADHOC_TIMEOUT : longRunning ? LONG_TIMEOUT : STANDARD_TIMEOUT

• add CHRONOS_PROCESS_RECURRENCE='* * * * *' to your .env
• run yarn build && yarn start
• check the output

Final checklist

• I checked the code review guidelines
• I have added Metrics Representative as reviewer(s) if my PR invovles metrics/data/analytics related changes
• I have performed a self-review of my code, the same way I'd do it for any other team member
• I have tested all cases I listed in the testing scenarios and I haven't found any issues or regressions
• Whenever I took a non-obvious choice I added a comment explaining why I did it this way
• I added the label Skip Maintainer Review Indicating the PR only requires reviewer review and can be merged right after it's approved if the PR introduces only minor changes, does not contain any architectural changes or does not introduce any new patterns and I think one review is sufficient'
• PR title is human readable and could be used in changelog

[rafaelromcar-parabol]

I'm testing in an on-demand-env and I'm still seeing some error 🤔

• Namespace and branch names are od-chronos-recurrence.
• Chronos is running and has the following in the logs

Trace: SEND TO SENTRY TIMEOUT {"query":"\n          mutation ProcessRecurrence {\n          "} undefined                                                                                                           
    at sendToSentry (/home/node/parabol/dist/chronos.js:17376:11)                                                                                                                                                  
    at publishWebhookGQL (/home/node/parabol/dist/chronos.js:17347:28)

Docker image is built using our Github Action.

[Dschoordsch]

Can I somehow see the datadog logs of that instance?

[rafaelromcar-parabol]

We do not have Datadog logs on that environment to save costs. You can see the logs using k9s! You have already do that in the past I think!

[Dschoordsch]

That doesn't show me the graphql traces sadly.

[rafaelromcar-parabol]

You could also use Google Console logging here for example

[Dschoordsch]

It seems the job runs every second, that's a bit much. I suggest setting it to run every minute at most and then we can check again.

[rafaelromcar-parabol]

Do the changes yourself on the on-demand-env branch, apply using the pipeline and test!

[Dschoordsch]

Seems to work just fine with the lower frequency.