fix(core): Improve the performance of last 2 sqlite migrations #6522

netroy · 2023-06-22T16:11:46Z

No description provided.

github-actions · 2023-06-22T16:13:31Z

Great PR! Please pay attention to the following items before merging:

Files matching packages/**:

If fixing bug, added test to cover scenario.
If addressing forum or Github issue, added link to description.

Files matching packages/**/*.ts:

Added unit tests to cover new or updated functionality.

Files matching packages/cli/src/databases/migrations/**:

Requested review from at least two engineers on migration.
Avoided irreversible data migrations.
Avoided deleting or updating data keys.
Wrote 'down' migration if possible.

Make sure to check off this list before asking for review.

codecov · 2023-06-22T16:29:19Z

Codecov Report

Patch coverage: 85.71% and project coverage change: -0.07 ⚠️

Comparison is base (76aca62) 28.91% compared to head (50c729a) 28.85%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6522      +/-   ##
==========================================
- Coverage   28.91%   28.85%   -0.07%     
==========================================
  Files        3070     3070              
  Lines      188256   188270      +14     
  Branches    20877    20879       +2     
==========================================
- Hits        54440    54322     -118     
- Misses     132931   133062     +131     
- Partials      885      886       +1

Impacted Files	Coverage Δ
...ckages/cli/src/databases/utils/migrationHelpers.ts	`65.21% <85.71%> (+3.67%)`	⬆️

... and 7 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

packages/cli/src/databases/utils/migrationHelpers.ts

…he migration starts

…eMigration

krynble

LGTM, just left a comment whether we'd like to remove the console.time. I find ithelpful to stay, so up to you.

krynble · 2023-07-05T11:28:14Z

packages/cli/src/databases/migrations/sqlite/1690000000002-MigrateIntegerKeysToString.ts

+			return;
+		}
+
+		console.time('pruningData');


We might want to remove the console.time for production deployment

we can do that after the release. I would like to be able to have a bit more visibility for the patch releases.

cypress · 2023-07-05T11:39:45Z

Passing run #1382 ↗︎

0	228	0	0	0

Details:

🌳 PruneAndVacuumBeforeMigration 🖥️ browsers:node18.12.0-chrome107 🤖 netroy 🗃...
Project: n8n	Commit: `50c729abad`
Status: Passed	Duration: 07:39 💡
Started: Jul 5, 2023 11:32 AM	Ended: Jul 5, 2023 11:39 AM

_{This comment has been generated by cypress-bot as a result of this
project's GitHub integration settings.}

github-actions · 2023-07-05T11:39:46Z

✅ All Cypress E2E specs passed

janober · 2023-07-05T13:30:04Z

Got released with [email protected]

* master: (50 commits) fix(Google Drive Node): Fix regex in file RLC (#6607) feat(HTML Node): 'Convert to table operation (#6540) ci: e2e tests don't need to enable saving manual executions 🚀 Release 1.0.1 (#6606) fix(core): Make node execution order configurable, and backward-compatible (#6507) 🚀 Release 0.236.0 (#6602) fix: Fix migrations for postgres and mysql (no-changelog) (#6600) fix(core): Improve the performance of last 2 sqlite migrations (#6522) refactor: Switch cloud plan retrieval endpoint (no-changelog) (#6529) fix(Salesforce Node): Fix typo for adding a contact to a campaign (#6598) ci: Hide v1 banner in e2e tests to fix 2-credentials.cy.ts (no-changelog) fix(editor): Restore expression completions (#6566) refactor: Remove basic auth env vars from docker compose stacks (#6570) fix(core): Fix migrations for MySQL/MariaDB (#6591) fix(editor): Show retry information in execution list only when it exists (#6587) fix(Strapi Node): Fix issue with pagination (#4991) fix(core): Route `/rest/workflows/new` correctly (no-changelog) (#6572) fix(XML Node): Fix issue with not returning valid data (#6565) fix(core): Fix credentials test (#6569) fix(core): Ensure valid `logger` is passed to every migration (no-changelog) (#6563) ... # Conflicts: # packages/design-system/src/components/N8nCallout/Callout.vue # packages/editor-ui/src/App.vue # packages/editor-ui/src/plugins/i18n/locales/en.json # packages/editor-ui/src/stores/ui.store.ts

* ADO-814-trial-banner: (58 commits) ⚡ Updating permanent dismiss logic to work with all banners fix(editor): Make Source control branch select required (#6619) ⚡ Minor refactoring of banner logic refactor(core): Load `cookieParser` middleware only once (no-changelog) (#6614) fix(editor): Prevent keyboard shortcuts to edit workflows in readonly mode (#6613) fix(editor): Show appropriate empty workflow list content when instance environment is readonly (#6610) ✨ Implemented permanent/temporary dismiss logic for v1 flag ci: Fix linting issues (no-changelog) fix(Google Drive Node): Fix regex in file RLC (#6607) feat(HTML Node): 'Convert to table operation (#6540) ci: e2e tests don't need to enable saving manual executions 🚀 Release 1.0.1 (#6606) fix(core): Make node execution order configurable, and backward-compatible (#6507) 🚀 Release 0.236.0 (#6602) fix: Fix migrations for postgres and mysql (no-changelog) (#6600) fix(core): Improve the performance of last 2 sqlite migrations (#6522) refactor: Switch cloud plan retrieval endpoint (no-changelog) (#6529) fix(Salesforce Node): Fix typo for adding a contact to a campaign (#6598) ci: Hide v1 banner in e2e tests to fix 2-credentials.cy.ts (no-changelog) fix(editor): Restore expression completions (#6566) ...

netroy assigned krynble Jun 22, 2023

netroy force-pushed the PruneAndVacuumBeforeMigration branch from 0db3383 to 9eb58ba Compare June 22, 2023 16:13

n8n-assistant bot added core Enhancement outside /nodes-base and /editor-ui n8n team Authored by the n8n team labels Jun 22, 2023

krynble force-pushed the PruneAndVacuumBeforeMigration branch from dfee533 to c88cd09 Compare June 26, 2023 13:40

netroy commented Jun 26, 2023

View reviewed changes

packages/cli/src/databases/utils/migrationHelpers.ts Outdated Show resolved Hide resolved

netroy commented Jun 26, 2023

View reviewed changes

packages/cli/src/databases/utils/migrationHelpers.ts Outdated Show resolved Hide resolved

netroy force-pushed the PruneAndVacuumBeforeMigration branch 3 times, most recently from fd3ff4a to e0f678b Compare July 4, 2023 14:48

netroy and others added 12 commits July 4, 2023 17:47

update typeorm version and the patch

34c08cf

let migrations define if we should run pruning and vacuuming before t…

6ddd067

…he migration starts

wip: Add pruning and vacuuming to migration

92a8cf8

feat: Implement vacuum and pruning

76ae088

feat: Add timer to pruning and vacuuming

2346c7e

fix: Temporarily add pruning to second migration

eec358e

add an additional console.time around all migrations

f8ec267

also log when the DB file size is not large enough

fbcb784

prune once is enough

6de258b

skip vacuuming

6e724b1

don't use a timer to determine if migrations finished running

35d2bef

remove pnpm patches. they aren't available in releases

915cfb2

netroy force-pushed the PruneAndVacuumBeforeMigration branch from e0f678b to 915cfb2 Compare July 4, 2023 15:49

netroy added 3 commits July 4, 2023 17:52

downgrade typeorm

ad3350f

remove pruneBeforeRunning

f05ee8e

use batching when copying data in sqlite migrations

5696d76

netroy changed the title ~~feat(core): Add support for pruning and vacuuming before individual migrations~~ feat(core): Remove typeorm patches, and improve the performance of sqlite migrations Jul 4, 2023

netroy marked this pull request as ready for review July 4, 2023 16:52

netroy requested a review from a team as a code owner July 4, 2023 16:52

netroy mentioned this pull request Jul 4, 2023

fix(core): Remove typeorm patches, but still enforce transactions on every migration #6594

Merged

This comment was marked as resolved.

Sign in to view

netroy marked this pull request as draft July 4, 2023 20:25

Merge remote-tracking branch 'origin/master' into PruneAndVacuumBefor…

f80bf18

…eMigration

netroy changed the title ~~feat(core): Remove typeorm patches, and improve the performance of sqlite migrations~~ feat(core): Improve the performance of sqlite migrations Jul 5, 2023

netroy marked this pull request as ready for review July 5, 2023 11:00

netroy requested a review from krynble July 5, 2023 11:01

netroy added 2 commits July 5, 2023 13:14

Merge remote-tracking branch 'origin/master' into PruneAndVacuumBefor…

2a623c1

…eMigration

remove unnecessary console.time

50c729a

krynble approved these changes Jul 5, 2023

View reviewed changes

netroy changed the title ~~feat(core): Improve the performance of sqlite migrations~~ fix(core): Improve the performance of last 2 sqlite migrations Jul 5, 2023

netroy merged commit 31cba87 into master Jul 5, 2023

netroy deleted the PruneAndVacuumBeforeMigration branch July 5, 2023 11:46

netroy added a commit that referenced this pull request Jul 5, 2023

fix(core): Improve the performance of last 2 sqlite migrations (#6522)

d2b8207

This was referenced Jul 5, 2023

🚀 Release 0.234.1 #6599

Merged

🚀 Release 0.235.0 #6601

Closed

🚀 Release 0.236.0 #6602

Merged

janober added the Released label Jul 5, 2023

CajuCLC mentioned this pull request Jul 7, 2023

Release/0.234.1 CajuCLC/n8n-puppeteer-docker#6

Merged

netroy added a commit that referenced this pull request Jul 12, 2023

fix(core): Improve the performance of last 2 sqlite migrations (#6522)

718a89d

github-actions bot mentioned this pull request Jul 12, 2023

🚀 Release 1.0.2 #6651

Merged

github-actions bot mentioned this pull request Jul 26, 2023

🚀 Release 1.1.0 #6746

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(core): Improve the performance of last 2 sqlite migrations #6522

fix(core): Improve the performance of last 2 sqlite migrations #6522

netroy commented Jun 22, 2023

github-actions bot commented Jun 22, 2023

codecov bot commented Jun 22, 2023 •

edited

Loading

This comment was marked as resolved.

krynble left a comment

krynble Jul 5, 2023

netroy Jul 5, 2023

cypress bot commented Jul 5, 2023

github-actions bot commented Jul 5, 2023

janober commented Jul 5, 2023

fix(core): Improve the performance of last 2 sqlite migrations #6522

fix(core): Improve the performance of last 2 sqlite migrations #6522

Conversation

netroy commented Jun 22, 2023

github-actions bot commented Jun 22, 2023

codecov bot commented Jun 22, 2023 • edited Loading

Codecov Report

This comment was marked as resolved.

krynble left a comment

Choose a reason for hiding this comment

krynble Jul 5, 2023

Choose a reason for hiding this comment

netroy Jul 5, 2023

Choose a reason for hiding this comment

cypress bot commented Jul 5, 2023

Passing run #1382 ↗︎

github-actions bot commented Jul 5, 2023

janober commented Jul 5, 2023

codecov bot commented Jun 22, 2023 •

edited

Loading