Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add support for large files with declarative nodes #6461

Merged
merged 2 commits into from
Jun 19, 2023

Conversation

Joffcom
Copy link
Member

@Joffcom Joffcom commented Jun 16, 2023

Github issue / Community forum post (link here to close automatically): https://community.n8n.io/t/maxbodylenght-limit-with-google-cloud-storage-node/27216

Declarative nodes don't appear to use proxyRequestToAxios() which was updated in 0.179.0 to allow files over 10mb to be uploaded. This PR adds the same option for the newer node style, Tested with a 32mb PDF and a 210mb zip to Google Cloud Storage.

To test...

export N8N_PAYLOAD_SIZE_MAX=400 

Test Workflow

{
  "meta": {
    "instanceId": "8c8c5237b8e37b006a7adce87f4369350c58e41f3ca9de16196d3197f69eabcd"
  },
  "nodes": [
    {
      "parameters": {},
      "id": "60f89e61-93e2-42ed-8ad8-405d4572e62b",
      "name": "When clicking \"Execute Workflow\"",
      "type": "n8n-nodes-base.manualTrigger",
      "typeVersion": 1,
      "position": [
        540,
        420
      ]
    },
    {
      "parameters": {
        "resource": "object",
        "operation": "create",
        "bucketName": "n8n-test-bucket",
        "objectName": "=Test Object {{ $execution.id }}",
        "createData": {},
        "createQuery": {},
        "encryptionHeaders": {}
      },
      "id": "2a4906af-69da-4e3d-af93-32dfc0e0b59f",
      "name": "Google Cloud Storage",
      "type": "n8n-nodes-base.googleCloudStorage",
      "typeVersion": 1,
      "position": [
        1040,
        420
      ],
      "credentials": {
        "googleCloudStorageOAuth2Api": {
          "id": "23",
          "name": "Google Cloud Storage account"
        }
      }
    },
    {
      "parameters": {
        "url": "https://cdn1.papercut.com/web/products/ng-mf/manuals/mf/22.x/pcmf-manual-22.0.0.pdf",
        "options": {}
      },
      "id": "3e6bafbc-23f2-426d-9732-533b1e789d13",
      "name": "HTTP Request",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.1,
      "position": [
        760,
        280
      ]
    },
    {
      "parameters": {
        "content": "30mb PDF\n",
        "height": 233,
        "width": 218
      },
      "id": "3126fd76-8ecb-4557-8fc7-089b92e16467",
      "name": "Sticky Note",
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        702,
        226
      ]
    },
    {
      "parameters": {
        "url": "https://downloads.n8n.io/file/n8n-downloads/n8n-mac.zip",
        "options": {}
      },
      "id": "31bb1ff7-9b48-44b7-9098-dd1ad36f81cf",
      "name": "HTTP Request1",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.1,
      "position": [
        760,
        560
      ]
    },
    {
      "parameters": {
        "content": "210mb zip\n",
        "height": 242
      },
      "id": "92d82298-fa7b-4f06-85be-13cb204894aa",
      "name": "Sticky Note1",
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        700,
        500
      ]
    }
  ],
  "connections": {
    "When clicking \"Execute Workflow\"": {
      "main": [
        [
          {
            "node": "HTTP Request",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "HTTP Request": {
      "main": [
        [
          {
            "node": "Google Cloud Storage",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}

@github-actions
Copy link
Contributor

Great PR! Please pay attention to the following items before merging:

Files matching packages/**:

  • If fixing bug, added test to cover scenario.
  • If addressing forum or Github issue, added link to description.

Files matching packages/**/*.ts:

  • Added unit tests to cover new or updated functionality.

Make sure to check off this list before asking for review.

@codecov
Copy link

codecov bot commented Jun 16, 2023

Codecov Report

Patch coverage has no change and project coverage change: -0.08 ⚠️

Comparison is base (16f707d) 28.44% compared to head (61ffafc) 28.37%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6461      +/-   ##
==========================================
- Coverage   28.44%   28.37%   -0.08%     
==========================================
  Files        2991     2991              
  Lines      185943   185943              
  Branches    20508    20506       -2     
==========================================
- Hits        52889    52758     -131     
- Misses     132260   132391     +131     
  Partials      794      794              
Impacted Files Coverage Δ
packages/core/src/NodeExecuteFunctions.ts 14.65% <ø> (ø)

... and 4 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@n8n-assistant n8n-assistant bot added core Enhancement outside /nodes-base and /editor-ui n8n team Authored by the n8n team labels Jun 16, 2023
@cypress
Copy link

cypress bot commented Jun 16, 2023

1 flaky tests on run #1228 ↗︎

0 227 0 0 Flakiness 1

Details:

🌳 support-large-files-with-declarative 🖥️ browsers:node18.12.0-chrome107 🤖 Jo...
Project: n8n Commit: 61ffafc83e
Status: Passed Duration: 07:30 💡
Started: Jun 16, 2023 3:23 PM Ended: Jun 16, 2023 3:30 PM
Flakiness  cypress/e2e/16-webhook-node.cy.ts • 1 flaky test

View Output Video

Test Artifacts
Webhook Trigger node > should listen for a GET request Output Screenshots Video

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings.

@github-actions
Copy link
Contributor

✅ All Cypress E2E specs passed

@Joffcom Joffcom merged commit e0f109f into master Jun 19, 2023
@Joffcom Joffcom deleted the support-large-files-with-declarative branch June 19, 2023 07:41
MiloradFilipovic added a commit that referenced this pull request Jun 20, 2023
* master: (34 commits)
  feat(editor): Replace root events with event bus events (no-changelog) (#6454)
  feat(DebugHelper Node): Fix and include in main app (#6406)
  feat(Webhook Node): Stream binary response in `lastNode.firstEntryBinary` mode (#6463)
  fix(editor): Update git repo URL validation to prevent using https protocol (#6475)
  fix(editor): Remove tooltip about SMTP being required to invite user (no-changelog) (#6474)
  feat: Add support for large files with declarative nodes (#6461)
  fix(core): Fix the url sent in the password-reset emails (#6466)
  fix(HTML Node): Prevent XSS in execution-data preview (#6432)
  fix(Snowflake Node): Upgrade snowflake-sdk to address CVE-2023-34232 (no-changelog) (#6458)
  refactor(core): Replace lodash's "soft-deprecated" individual packages with `lodash` to resolve CVE (no-changelog) (#6450)
  fix(editor): Remove `$if`, `$min` and `$max` from code node autocomplete (#6460)
  fix(editor): Fix DNV header disappearing when scrolling the code editor content (#6459)
  feat: Remove vue-fragment (no-changelog) (#6456)
  ci: Prevent e2e failure on `commented` type review (no-changelog) (#6452)
  fix(LinkedIn Node): Remove unsupported description from image posts (#6446)
  fix(Split In Batches Node): Add "done" context to allow simple reset (#6437)
  feat(Gmail Node): Add reply to email (#6453)
  fix: Remove Vue.component usage and refactor plugins into Vue Plugins (no-changelog) (#6445)
  fix(editor): Show confirm on pull only when http response status is 409 (#6451)
  fix(editor): Update data pinning tooltip to match current behaviour (#6436)
  ...

# Conflicts:
#	packages/nodes-base/nodes/CompareDatasets/GenericFunctions.ts
#	packages/nodes-base/nodes/Merge/v2/GenericFunctions.ts
@janober
Copy link
Member

janober commented Jun 22, 2023

Got released with [email protected]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Enhancement outside /nodes-base and /editor-ui n8n team Authored by the n8n team Released
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants