fix(core): Filter out certain executions from crash recovery #9904
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
n8n-assistant
bot
added
core
ivov
added
the
release/backport
ivov
changed the title
ivov
commented
Jul 2, 2024
There was a problem hiding this comment.
Copied from existing ones. At some point we need to simplify all this.
ivov
commented
Jul 2, 2024
There was a problem hiding this comment.
Only change in this file is adding a dependency.
…-executions-from-crash-recovery
netroy
approved these changes
Jul 2, 2024
Comment on lines
+207
to
+211
| if (config.getEnv('executions.mode') === 'regular') { | ||
| // removal of active executions will no longer release capacity back, | ||
| // so that throttled executions cannot resume during shutdown | ||
| this.concurrencyControl.disable(); | ||
| } |
1 flaky test on run #5743 ↗︎
Details:
|
|||||||||||||||||||||
| Test | Artifacts | |
|---|---|---|
| NDV > should not retrieve remote options when required params throw errors |
Screenshots
Video
|
|
Review all test suite changes for PR #9904 ↗︎
|
✅ All Cypress E2E specs passed |
cstuncsik
pushed a commit
that referenced
this pull request
Jul 3, 2024
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <[email protected]>
Merged
|
Got released with |
Merged
adrian-martinez-onestic
pushed a commit
to onesdata/n8n-fork
that referenced
this pull request
Jul 8, 2024
…9904) Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes an unwanted interaction between concurrency control and crash recovery, by logging new events (execution throttled, execution started during bootup) and filtering out those executions from the log writer's check.
Context
During startup we start running any executions that had been enqueued at the time of shutdown, running them up to the concurrency limit and enqueuing the excess. This does not block, so that concurrency control can let through or throttle these executions as usual.
During startup we also have the execution recovery service, which receives event logs about unfinished executions and uses them to amend data in executions truncated by an instance crash. It is the log writer that collects these event logs about unfinished executions, by checking for events written to
n8nEventLog-{n}.log.Hence during startup we move executions to
runningstatus up to the limit and keep the excess withnew(i.e. enqueued) status, but because all these executions have not finished yet, the log writer reports them to the recovery service, which marks them as crashed.Testing
Currently the log writer is not easily testable. To test manually:
neworrunningexecutions. SetN8N_CONCURRENCY_PRODUCTION_LIMIT=2and start the instance.runningexecutions and a handful ofnewexecutions to accumulate.successand a handful ofnewexecutions. Start the instance again.newexecutions are moved torunningup to the limit, without being marked ascrashed.Related Linear tickets, Github issues, and Community forum posts
https://n8nio.slack.com/archives/C069HS026UF/p1719846780406959?thread_ts=1719825397.534179&cid=C069HS026UF
Review / Merge checklist
release/backport(if the PR is an urgent fix that needs to be backported)