Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(core): Stop binary data restoration from preventing execution from finishing #8082

Merged
merged 5 commits into from
Dec 21, 2023
Merged

fix(core): Stop binary data restoration from preventing execution from finishing #8082

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5e632ca
fix: Prevent binary data restoration from preventing execution from f…
krynble Dec 19, 2023
a7898e6
refactor: Change logging level to error
krynble Dec 19, 2023
944f6c6
refactor(core): Suggestions for #8082
ivov Dec 19, 2023
66b8ef2
Update packages/cli/src/executionLifecycleHooks/restoreBinaryDataId.ts
krynble Dec 20, 2023
ddc2051
fix: Lint issue
krynble Dec 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 1 addition & 8 deletions packages/cli/src/WorkflowExecuteAdditionalData.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -410,14 +410,7 @@ function hookFunctionsSave(parentProcessMode?: string): IWorkflowExecuteHooks {
workflowId: this.workflowData.id,
});

try {
await restoreBinaryDataId(fullRunData, this.executionId, this.mode);
} catch (e) {
logger.error('Failed to restore binary data ID', {
executionId: this.executionId,
mode: this.mode,
});
}
await restoreBinaryDataId(fullRunData, this.executionId, this.mode);

const isManualMode = [this.mode, parentProcessMode].includes('manual');

Expand Down
41 changes: 27 additions & 14 deletions packages/cli/src/executionLifecycleHooks/restoreBinaryDataId.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { BinaryDataService } from 'n8n-core';
import type { IRun, WorkflowExecuteMode } from 'n8n-workflow';
import type { BinaryData } from 'n8n-core';
import config from '@/config';
import { Logger } from '@/Logger';

/**
* Whenever the execution ID is not available to the binary data service at the
Expand Down Expand Up @@ -32,28 +33,40 @@ export async function restoreBinaryDataId(
return;
}

const { runData } = run.data.resultData;
try {
const { runData } = run.data.resultData;

const promises = Object.keys(runData).map(async (nodeName) => {
const binaryDataId = runData[nodeName]?.[0]?.data?.main?.[0]?.[0]?.binary?.data?.id;
const promises = Object.keys(runData).map(async (nodeName) => {
const binaryDataId = runData[nodeName]?.[0]?.data?.main?.[0]?.[0]?.binary?.data?.id;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make more sense to have the try catch inside the map function, as the tasks are independent of each others and any one or multiple of them could in theory throw. Otherwise if one of them throws the function will return even if the other promises are still going. Might as well extract it to its own function while at it.

Other option would be to use Promise.allSettled and check the result afterwards and log possible errors

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH @tomi I did implement the change but given we have to then update the binary data id after the renaming process happens, it turned out so ugly by having to use then as well as allSettled in a .forEach pushing things to a promise array that I was weighing the pros and cons.

If we fail to rename some files even though this is independent, the impact is that you might end up with some files not being deleted when the execution is removed.

This is not great, but this also only happens if there's something else went wrong with the execution, where the binary file was not written to disk in the first place, so this is an edge case and the impact is minimal, so I went for code legibility.


if (!binaryDataId) return;
if (!binaryDataId) return;

const [mode, fileId] = binaryDataId.split(':') as [BinaryData.StoredMode, string];
const [mode, fileId] = binaryDataId.split(':') as [BinaryData.StoredMode, string];

const isMissingExecutionId = fileId.includes('/temp/');
const isMissingExecutionId = fileId.includes('/temp/');

if (!isMissingExecutionId) return;
if (!isMissingExecutionId) return;

const correctFileId = fileId.replace('temp', executionId);
const correctFileId = fileId.replace('temp', executionId);

await Container.get(BinaryDataService).rename(fileId, correctFileId);
await Container.get(BinaryDataService).rename(fileId, correctFileId);

const correctBinaryDataId = `${mode}:${correctFileId}`;
const correctBinaryDataId = `${mode}:${correctFileId}`;

// @ts-expect-error Validated at the top
run.data.resultData.runData[nodeName][0].data.main[0][0].binary.data.id = correctBinaryDataId;
});
// @ts-expect-error Validated at the top
run.data.resultData.runData[nodeName][0].data.main[0][0].binary.data.id = correctBinaryDataId;
});

await Promise.all(promises);
await Promise.all(promises);
} catch (e) {
const error = e instanceof Error ? e : new Error(`${e}`);
const logger = Container.get(Logger);

if (error.message.includes('ENOENT')) {
logger.warn('Failed to restore binary data ID - No such file or dir', { executionId, error });
return;
}

logger.warn('Failed to restore binary data ID - Unknown error', { executionId, error });
krynble marked this conversation as resolved.
Show resolved Hide resolved
}
}
22 changes: 22 additions & 0 deletions packages/cli/test/unit/execution-lifecyle/restoreBinaryDataId.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,28 @@ for (const mode of ['filesystem-v2', 's3'] as const) {

expect(binaryDataService.rename).not.toHaveBeenCalled();
});

it('should ignore error thrown on renaming', async () => {
const workflowId = '6HYhhKmJch2cYxGj';
const executionId = 'temp';
const binaryDataFileUuid = 'a5c3f1ed-9d59-4155-bc68-9a370b3c51f6';

const incorrectFileId = `workflows/${workflowId}/executions/temp/binary_data/${binaryDataFileUuid}`;

const run = toIRun({
binary: {
data: { id: `s3:${incorrectFileId}` },
},
});

binaryDataService.rename.mockRejectedValueOnce(new Error('ENOENT'));

const promise = restoreBinaryDataId(run, executionId, 'webhook');

await expect(promise).resolves.not.toThrow();

expect(binaryDataService.rename).toHaveBeenCalled();
});
});
}

Expand Down
Loading