feat(logs): timeout old operations

[bodinsamuel]

Describe your changes

Fixes NAN-1036

Since we have only 15days retention and syncs are unlimited it's mostly for cosmetic and some smaller better time scoped operations (action, oauth, webhook).

• Add expiration date to all operations
• Cron to set timeout to all expired operations

How to test?

• Trigger an oauth but don't complete the process and wait 5-10minutes
  or
• Add an operation, change the expiresAt manually, wait for the cron

(The query in curl)

#!/usr/bin/env bash

curl -X POST "http://localhost:55132/20240528_messages.2024-05-30/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
            "bool": {
                "must": [{ "range": { "expiresAt": { "lt": "now" } } }],
                "must_not": { "exists": { "field": "parentId" } },
                "should": [{ "term": { "state": "waiting" } }, { "term": { "state": "running" } }]
            }
        }
}
'

[linear]

NAN-103 [Talkbase] Fix list records error

[TBonnin]

why 7 days by default. Can any operation last longer than 24hours?

[bodinsamuel]

technically yes I believe, syncs can run as long as they need if the heartbeat works no?

[khaliqgant]

syncs can run as long as they need if the heartbeat works no?

A sync can't run longer than 24 hours https://github.com/NangoHQ/nango/blob/master/packages/jobs/lib/workflows.ts#L6

[TBonnin]

1 day for an action that's pretty generous

[bodinsamuel]

it's just for the heartbeat, unfortunately I have no way to have something more precise. But if everything works well everything should be handled by temporal/v2 :D

[khaliqgant]

Would be nice to abstract out these times to one place in case we need to adjust

[bodinsamuel]

yeah good point, actually I wasn't aware of the constant you shared on previous message so maybe I can expose them globally somehow

[linear]

NAN-1036 Timeout old running operations

[khaliqgant]

Feel like this expiresAt should be a constant somewhere since it is repeated in so many places

new Date(Date.now() + 300 * 1000).toISOString()

[khaliqgant]

Abstraction request

[bodinsamuel]

Abstracted the values, I had to expose them through utils let me know.
(could also be helpful for you thomas)

[bodinsamuel]

Note I didn't removed them from jobs/workflows.ts because temporal can't compile utils 🤦🏻

[TBonnin]

I think it is fine. I just have to remember to remove it when getting rid of temporal

[bodinsamuel]

Okay this time should be fine, I could not replace webhooks timing because I can't import logs in shared

[khaliqgant]

Thanks