Frequent License Checks Causing Service Disruption in n8n Multi-Main Mode

[dominik-mukrecki]

Bug Description

n8n encounters a service disruption due to frequent license renewal requests leading to a "Too many requests" error. This problem is exacerbated in environments utilizing the multi-main instance feature, potentially due to increased check frequency in such configurations. Although the license is valid, the aggressive checking disrupts service operation, demanding an alternative strategy for license verification that does not impair production environments.

https://github.com/8gears/n8n-helm-chart - deployed using this chart with some custom tweaks.

To Reproduce

1. Operate n8n in an environment with multi-main instance mode enabled.
2. Observe the frequency and handling of license checks during the startup and operation.
3. Experience service disruption when the system encounters a "Too many requests" error during license verification.

Expected behavior

The license verification process should be robust enough to handle environments with multiple instances without exceeding API limits, ensuring continuous service operation. The system should implement a throttling mechanism or cache license status to reduce the frequency of checks against the license server.

Operating System

v1.27.11 +k3s1 (Rancher Kubernetes distro)

n8n Version

n8nio/n8n:1.38.0

Node.js Version

default for given image

Database

PostgreSQL

Execution mode

main (default)

[Joffcom]

Hey @dominikmukrecki,

How many main instances are you running?

[dominik-mukrecki]

Three pods, all operational right now.

Also, check logs:

User
13:20:55.412Z | warn     | [license] license renewal failed: Too many requests. Slow down! "{ file: 'LicenseManager.js', function: 'log' }"
2024-04-18T13:20:55.418773134Z 2024-04-18T13:20:55.418Z | debug    | Skipping license activation "{ file: 'BaseCommand.js', function: 'initLicense' }"
2024-04-18T13:20:55.423972440Z Error: Your license does not allow for feat:multipleMainInstances. To enable feat:multipleMainInstances, please upgrade to a license that supports this feature.
2024-04-18T13:20:55.424018157Z     at Start.initOrchestration (/usr/local/lib/node_modules/n8n/dist/commands/start.js:155:19)
2024-04-18T13:20:55.424024230Z     at Start.init (/usr/local/lib/node_modules/n8n/dist/commands/start.js:136:20)
2024-04-18T13:20:55.424030447Z     at processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-04-18T13:20:55.424036325Z     at Start._run (/usr/local/lib/node_modules/n8n/node_modules/@oclif/core/lib/command.js:303:13)
2024-04-18T13:20:55.424041696Z     at Config.runCommand (/usr/local/lib/node_modules/n8n/node_modules/@oclif/core/lib/config/config.js:417:25)
2024-04-18T13:20:55.424046918Z     at run (/usr/local/lib/node_modules/n8n/node_modules/@oclif/core/lib/main.js:85:16)
2024-04-18T13:20:55.424052338Z     at /usr/local/lib/node_modules/n8n/bin/n8n:49:2
2024-04-18T13:20:55.425755902Z 2024-04-18T13:20:55.425Z | error    | Error: Exiting due to an error. "{ file: 'LoggerProxy.js', function: 'exports.error' }"
2024-04-18T13:20:55.426419621Z 2024-04-18T13:20:55.426Z | error    | FeatureNotLicensedError: Your license does not allow for feat:multipleMainInstances. To enable feat:multipleMainInstances, please upgrade to a license that supports this feature. "{ file: 'LoggerProxy.js', function: 'exports.error' }"

[dominik-mukrecki]

@Joffcom My idea is - rapid license check during booting all pods at once throws too many requests error.

Proposal - provide some sort of "soft check" enterprise license, set timeout (Portainer for example gives 10 days to upgrade license after exceeding node count).

I also wrote email to my license provider directly.

[dominik-mukrecki]

After few restarts everything works, check this screenshot from Rancher please

[Joffcom]

Hey @dominikmukrecki,

Looking at that you are running 3 main instances is that correct? When you say you have emailed your license provider is that someone internally here at n8n?

We should be checking the license on startup but I am not sure how often it is actually checked I would have assumed only the once then whenever it is needed so maybe we need to relax the firewall infront of the license check to allow for the connections.

[dominik-mukrecki]

Looking at that you are running 3 main instances - exactly When you say you have emailed your license provider is that someone internally here at n8n? - directly from your HQ, by mail. Also, using n8n without persistent storage could result in such behaviour? I will be testing it with RWX volume, but this solution is complicated in setup and maintenance and against HA idea.

[Joffcom]

Hey @dominikmukrecki,

Who did you email here so I can check what they are doing already as there may be no need for me to look into this if someone else is already doing it and we can close this issue if it has been directly reported.

[Joffcom]

Hey @dominikmukrecki,

I have managed to catch up with JC and he has taken the issue to the team to have this looked into. As you are in direct contact about this I am going to mark this as closed for now.

[ivov]

@dominikmukrecki Thank you for your feedback, tracking internally: https://linear.app/n8n/issue/PAY-1517

[janober]

Fix got released with [email protected]