-
Notifications
You must be signed in to change notification settings - Fork 427
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(logs): timeout old operations #2220
Conversation
packages/logs/lib/models/helpers.ts
Outdated
startedAt: data.startedAt || null, | ||
endedAt: data.endedAt || null | ||
endedAt: data.endedAt || null, | ||
expiresAt: data.operation ? data.expiresAt || new Date(now.getTime() + 7 * 86400 * 1000).toISOString() : null |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why 7 days by default. Can any operation last longer than 24hours?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
technically yes I believe, syncs can run as long as they need if the heartbeat works no?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
syncs can run as long as they need if the heartbeat works no?
A sync can't run longer than 24 hours https://github.com/NangoHQ/nango/blob/master/packages/jobs/lib/workflows.ts#L6
operation: { type: 'action' }, | ||
message: 'Start action', | ||
expiresAt: new Date(Date.now() + 86400 * 1000).toISOString() | ||
}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1 day for an action that's pretty generous
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's just for the heartbeat, unfortunately I have no way to have something more precise. But if everything works well everything should be handled by temporal/v2 :D
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be nice to abstract out these times to one place in case we need to adjust
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah good point, actually I wasn't aware of the constant you shared on previous message so maybe I can expose them globally somehow
id: String(activityLogId), | ||
operation: { type: 'auth', action: 'create_connection' }, | ||
message: 'Authorization OAuth', | ||
expiresAt: new Date(Date.now() + 300 * 1000).toISOString() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Feel like this expiresAt should be a constant somewhere since it is repeated in so many places
new Date(Date.now() + 300 * 1000).toISOString()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Abstraction request
Abstracted the values, I had to expose them through utils let me know. |
|
||
export const WEBHOOK_TIMEOUT = '15m'; | ||
export const WEBHOOK_MAX_ATTEMPTS = 3; | ||
export const MAX_WEBHOOK_DURATION = ms(WEBHOOK_TIMEOUT); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note I didn't removed them from jobs/workflows.ts because temporal can't compile utils 🤦🏻
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it is fine. I just have to remember to remove it when getting rid of temporal
Okay this time should be fine, I could not replace webhooks timing because I can't import logs in shared |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks
Describe your changes
Fixes NAN-1036
Since we have only 15days retention and syncs are unlimited it's mostly for cosmetic and some smaller better time scoped operations (action, oauth, webhook).
timeout
to all expired operationsHow to test?
or
(The query in curl)