Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(core): Stop binary data restoration from preventing execution from finishing #8082

Merged
merged 5 commits into from
Dec 21, 2023

Conversation

krynble
Copy link
Contributor

@krynble krynble commented Dec 19, 2023

In the case of a filesystem failure to rename the binary files as part of the execution's cleanup process, the execution would fail to be saved and would never finish. This catch prevents it.

Summary

Whenever an execution is wrapping u to save the data, if it uses binary data n8n will try to find possibly misallocated files and place them in the right folder. If this process fails, the execution fails to finish.

Given the execution has already finished at this point, and we cannot handle the binary data errors more gracefully, all we can do at this point is log the message as it's a filesystem issue. The rest of the execution saving process should remain as normal.

Related tickets and issues

https://linear.app/n8n/issue/HELP-430

Review / Merge checklist

  • PR title and summary are descriptive. Remember, the title automatically goes into the changelog. Use (no-changelog) otherwise. (conventions)
  • Docs updated or follow-up ticket created.
  • Tests included.

    A bug is not considered fixed, unless a test is added to prevent it from happening again.
    A feature is not complete without tests.

@n8n-assistant n8n-assistant bot added core Enhancement outside /nodes-base and /editor-ui n8n team Authored by the n8n team labels Dec 19, 2023
try {
await restoreBinaryDataId(fullRunData, this.executionId, this.mode);
} catch (e) {
logger.debug('Failed to restore binary data ID', {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to handle only the specific error here instead of everything.

Also, since this isn't really something that is expected, wouldn't we want to log this either as an error or warning?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, Tomi - changed the level to errror

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed - looks good? Would you mind approving?

ivov added a commit that referenced this pull request Dec 19, 2023
@krynble krynble changed the title fix: Prevent binary data restoration from preventing execution from finishing fix: Stop binary data restoration from preventing execution from finishing Dec 19, 2023
Comment on lines 416 to 419
logger.error('Failed to restore binary data ID', {
executionId: this.executionId,
mode: this.mode,
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are still ignoring the thrown error. Could we at least include it in the logging so we know what the underlying reason for the failure was?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ivov added your suggestions and I rebased my branch to contain it

const promises = Object.keys(runData).map(async (nodeName) => {
const binaryDataId = runData[nodeName]?.[0]?.data?.main?.[0]?.[0]?.binary?.data?.id;
const promises = Object.keys(runData).map(async (nodeName) => {
const binaryDataId = runData[nodeName]?.[0]?.data?.main?.[0]?.[0]?.binary?.data?.id;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make more sense to have the try catch inside the map function, as the tasks are independent of each others and any one or multiple of them could in theory throw. Otherwise if one of them throws the function will return even if the other promises are still going. Might as well extract it to its own function while at it.

Other option would be to use Promise.allSettled and check the result afterwards and log possible errors

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH @tomi I did implement the change but given we have to then update the binary data id after the renaming process happens, it turned out so ugly by having to use then as well as allSettled in a .forEach pushing things to a promise array that I was weighing the pros and cons.

If we fail to rename some files even though this is independent, the impact is that you might end up with some files not being deleted when the execution is removed.

This is not great, but this also only happens if there's something else went wrong with the execution, where the binary file was not written to disk in the first place, so this is an edge case and the impact is minimal, so I went for code legibility.

Copy link
Contributor

@tomi tomi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Contributor

⚠️ Some Cypress E2E specs are failing, please fix them before merging

Copy link

cypress bot commented Dec 21, 2023

1 flaky test on run #3438 ↗︎

0 306 5 0 Flakiness 1

Details:

🌳 🖥️ browsers:node18.12.0-chrome107 🤖 krynble 🗃️ e2e/*
Project: n8n Commit: ddc2051c6c
Status: Passed Duration: 46:45 💡
Started: Dec 21, 2023 8:51 AM Ended: Dec 21, 2023 9:38 AM
Flakiness  cypress/e2e/27-two-factor-authentication.cy.ts • 1 flaky test

View Output Video

Test Artifacts
Two-factor authentication > Should be able to login with MFA token Screenshots Video

Review all test suite changes for PR #8082 ↗︎

Copy link
Contributor

✅ All Cypress E2E specs passed

@krynble krynble merged commit 5ffff1b into master Dec 21, 2023
19 checks passed
@krynble krynble deleted the help-430 branch December 21, 2023 09:40
@github-actions github-actions bot mentioned this pull request Dec 21, 2023
@netroy netroy changed the title fix: Stop binary data restoration from preventing execution from finishing fix(core): Stop binary data restoration from preventing execution from finishing Dec 21, 2023
ivov added a commit that referenced this pull request Dec 21, 2023
#
[1.22.0](https://github.com/n8n-io/n8n/compare/[email protected]@1.22.0)
(2023-12-21)


### Bug Fixes

* **core:** Close db connection gracefully when exiting
([#8045](#8045))
([e69707e](e69707e))
* **core:** Consider timeout in shutdown an error
([#8050](#8050))
([4cae976](4cae976))
* **core:** Do not display error when stopping jobless execution in
queue mode ([#8007](#8007))
([8e6b951](8e6b951))
* **core:** Fix shutdown if terminating before hooks are initialized
([#8047](#8047))
([6ae2f5e](6ae2f5e))
* **core:** Handle multiple termination signals correctly
([#8046](#8046))
([67bd8ad](67bd8ad))
* **core:** Initialize queue once in queue mode
([#8025](#8025))
([53c0b49](53c0b49))
* **core:** Prevent axios from force setting a form-urlencoded
content-type ([#8117](#8117))
([bba9576](bba9576))
* **core:** Remove circular references before serializing executions in
public API ([#8043](#8043))
([989888d](989888d))
* **core:** Restore workflow ID during execution creation
([#8031](#8031))
([c5e6ba8](c5e6ba8))
* **core:** Use relative imports for dynamic imports in
SecurityAuditService ([#8086](#8086))
([785bf99](785bf99))
* **core:** Stop binary data restoration from preventing execution from
finishing ([#8082](#8082))
([5ffff1b](5ffff1b))
* **editor:** Add back credential `use` permission
([#8023](#8023))
([329e5bf](329e5bf))
* **editor:** Cleanup Executions page component
([#8053](#8053))
([2689c37](2689c37))
* **editor:** Disable auto scroll and list size check when clicking on
executions ([#7983](#7983))
([fcb8b91](fcb8b91))
* **editor:** Ensure execution data overrides pinned data when copying
in executions view ([#8009](#8009))
([1d1cb0d](1d1cb0d))
* **editor:** Fix copy/paste issue when switch node is in workflow
([#8103](#8103))
([4b86926](4b86926))
* **editor:** Make keyboard shortcuts more strict; don't accept extra
Ctrl/Alt/Shift keys ([#8024](#8024))
([8df49e1](8df49e1))
* **editor:** Show credential share info only to appropriate users
([#8020](#8020))
([b29b4d4](b29b4d4))
* **editor:** Turn off executions list auto-refresh after leaving the
page ([#8005](#8005))
([e3c363d](e3c363d))
* **editor:** Update image sizes in template description not to be full
width always ([#8037](#8037))
([63a6e7e](63a6e7e))
* **ActiveCampaign Node:** Fix pagination issue when loading tags
([#8017](#8017))
([1943857](1943857))
* **HTTP Request Node:** Do not create circular references in HTTP
request node output ([#8030](#8030))
([5b7ea16](5b7ea16))
* Upgrade axios to address CVE-2023-45857
([#7713](#7713))
([64eb9bb](64eb9bb))


### Features

* Add option to `returnIntermediateSteps` for AI agents
([#8113](#8113))
([7806a65](7806a65))
* **core:** Add config option to prefer GET request over LIST when using
Hashicorp Vault ([#8049](#8049))
([439a22d](439a22d))
* **core:** Add N8N_GRACEFUL_SHUTDOWN_TIMEOUT env var
([#8068](#8068))
([614f488](614f488))
* **editor:** Add lead enrichment suggestions to workflow list
([#8042](#8042))
([36a923c](36a923c))
* **editor:** Finalize workers view
([#8052](#8052))
([edfa784](edfa784))
* **editor:** Gracefully ignore invalid payloads in postMessage handler
([#8096](#8096))
([9d22c7a](9d22c7a))
* **editor:** Upgrade frontend tooling to address a few vulnerabilities
([#8100](#8100))
([19b7f1f](19b7f1f))
* **Filter Node:** Overhaul UI by adding the new filter component
([#8016](#8016))
([3d53052](3d53052))
* **Respond to Webhook Node:** Overhaul with improvements like returning
all items ([#8093](#8093))
([32d397e](32d397e))


### Performance Improvements

* **editor:** Improve canvas rendering performance
([#8022](#8022))
([b780436](b780436))

Co-authored-by: ivov <[email protected]>
@janober
Copy link
Member

janober commented Dec 21, 2023

Got released with [email protected]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Enhancement outside /nodes-base and /editor-ui n8n team Authored by the n8n team Released
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants