feat(core): Lazy-load nodes and credentials to reduce baseline memory usage

[netroy]

over 50% drop in baseline memory usage, by loading nodes and credentials as they are needed, instead of loading them all at startup.

[linear]

N8N-4532

[ivov]

1. This breaks the dev flow for node builders in the team and in the community, because watch transpiles changes to dist but does not reflect them in the root dir cache. We have to keep the cache fresh or bypass it for watch.
2. In /dist/known/credentials.json the mapping from credential type name to className and sourcePath looks inferrable. Do we need these two details to load a credential type? In /dist/known/nodes.json, the mapping from node type name to className looks inferrable but sourcePath becomes unpredictable because of grouping dirs like Aws, which btw I am not sure we benefit much from.

[ivov]

We are copying icons from /nodes-base/src to /nodes-base/dist with Gulp during build, and here we are copying those icons from /nodes-base/dist to the root dir cache. Can we copy to both destinations in the same step, either with Gulp or otherwise?

[netroy]

the first part happens during the build process for nodes-base, and the second one happens when loading nodes-base and community nodes that might also have icons.
We need the first step in the build, because we don't publish the icons in the src folder, and we need these icons in the published package.

[netroy]

1. This breaks the dev flow for node builders in the team and in the community, because watch transpiles changes to dist but does not reflect them in the root dir cache. We have to keep the cache fresh or bypass it for watch.

Done. for now, we'll be using tsc-watch to regenerate the types after tsc finishes successfully. This however does not update the already-loaded nodes in a running server, and nor does it restart the server. But, that's the same behavior on master as well.
I'd like to make the type-generation script incremental, and would also like to automatically unload nodes/credential classes from a running server, so we don't actually need to restart the server. But, that's another big change, and I'd rather not do that in this PR.
We don't need to worry about refreshing the known files for now either as we still need to add files names explicitly in the n8n block in the package.json. I plan to tackle that as well soon.

• In /dist/known/credentials.json the mapping from credential type name to className and sourcePath looks inferrable. Do we need these two details to load a credential type? In /dist/known/nodes.json, the mapping from node type name to className looks inferrable but sourcePath becomes unpredictable because of grouping dirs like Aws, which btw I am not sure we benefit much from.

In master we are inferring className from the fileName. I changed this to avoid additional string manipulation and garbage generation at startup. But, I'm not sure if the impact of that is significant. If you prefer, I can remove className from the generated files, and infer it before setting it in the objects stores in this.known in /home/aditya/Work/n8n/n8n/packages/cli/src/LoadNodesAndCredentials.ts. WDYT?

[janober]

Got released with [email protected]