Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(core): Implement wrapping of regular nodes as AI Tools #10641

Merged
merged 18 commits into from
Sep 4, 2024
Merged

feat(core): Implement wrapping of regular nodes as AI Tools #10641

merged 18 commits into from
Sep 4, 2024

Conversation

OlegIvaniv
Copy link
Contributor

@OlegIvaniv OlegIvaniv commented Sep 3, 2024 •
edited
Loading

Summary

This PR introduces the ability to wrap regular n8n nodes as AI Tools for use with AI agents. Key changes include:

  1. Added usableAsTool property to node definitions to indicate eligibility for tool wrapping.

  2. Implemented createNodeAsTool function:

    • Analyzes node parameters to identify placeholders for dynamic inputs.
    • Generates a Zod schema based on these placeholders for input validation.
    • Creates a DynamicStructuredTool that wraps the node's functionality.
    • Handles nested parameters and maintains the original node's structure.
    • Provides a proxy for getNodeParameter to integrate LLM-provided values.

  3. Modified NodeExecuteFunctions:

    • Enhanced getInputConnectionData to handle tool nodes.
    • If a node doesn't have a supplyData method but is marked as usable as a tool, it creates a wrapper function that calls getNodeAsTool.
    • This allows tool nodes to be executed within the existing n8n framework.

  4. Updated NodeHelpers with convertNodeToAiTool function to modify node descriptions for tool use, including adding a custom description field for LLMs.

Right now, usableAsTool is not set for any of the nodes, we'll enable this in the next phase once we also have placeholder parameters implemented so that LLM could actually pass data to these tools.

Example with usableAsTool: true and HackerNews node:

CleanShot.2024-09-03.at.09.46.53.mp4

Related Linear tickets, Github issues, and Community forum posts

Review / Merge checklist

  • PR title and summary are descriptive. (conventions)
  • Docs updated or follow-up ticket created.
  • Tests included.
  • PR Labeled with release/backport (if the PR is an urgent fix that needs to be backported)

OlegIvaniv and others added 17 commits July 29, 2024 14:20
@OlegIvaniv
WIP: Connect execute nodes as tools
092724b
@jeanpaul
WIP: Ensure we pass context properly to converted tools
4f033f0 
Before this change, any tools connected to the agent wouldn't run with
updated parameters. This was because the context would've been cached by
the closure in the newly added `supplyData` function. Now, we derive the
context and parameters from the passed context.
@jeanpaul
WIP: Use a wrapper instead of converting on the fly
b34912b
@jeanpaul
Allow versioned nodes to be passed to NodeToTool
cd059c1
@jeanpaul
Fix typing in toolWrapper
6934d82
@jeanpaul
Add tool description to the tool properties when wrapping
a5fc12c
@jeanpaul
Enable wrapping tools from the description
5982587
@jeanpaul
Remove wrappedTools.node.js from the langchain nodes package list
c7b825c
@OlegIvaniv
WIP: Allow to use some nodes as tools
c3e1290
@OlegIvaniv
Add comments and fix parameters caching
01e2436
@OlegIvaniv
Fix test
eb0d73a
@OlegIvaniv
Merge branch 'master' into ai-284-implement-node-as-tool-wrapper
c57e91c
@OlegIvaniv
Remove renamed fiel
185124e
@OlegIvaniv
Fix merging issues
7a8f7e3
@OlegIvaniv
Merge branch 'master' into ai-284-implement-node-as-tool-wrapper
710b46b
@OlegIvaniv
Do not set usableAsTool for HackerNews and Airtable, cleanup console log
896cbee
@OlegIvaniv
Remove console log
56389b6
@n8n-assistant n8n-assistant bot added core Enhancement outside /nodes-base and /editor-ui n8n team Authored by the n8n team ui Enhancement in /editor-ui or /design-system labels Sep 3, 2024
burivuhster
burivuhster requested changes Sep 3, 2024
View reviewed changes
Copy link
Contributor

@burivuhster burivuhster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing work! 💪

Not sure it is relevant (feel free to discard), but lodash have some utility methods for working with object properties by path (e.g. get, set, toPath, property). Although lodash path format is slightly different for array elements, maybe it is possible to replace some custom code with methods from lodash.

I left some minor comments and questions.

@OlegIvaniv OlegIvaniv force-pushed the ai-284-implement-node-as-tool-wrapper branch from 134f288 to 576d601 Compare September 3, 2024 14:23
OlegIvaniv
OlegIvaniv commented Sep 3, 2024
View reviewed changes
Copy link
Contributor Author

@OlegIvaniv OlegIvaniv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@burivuhster Thank you for the review! Could you have one more look, please?

packages/core/test/CreateNodeAsTool.test.ts Show resolved Hide resolved
packages/workflow/src/NodeHelpers.ts Show resolved Hide resolved
packages/workflow/src/NodeHelpers.ts Show resolved Hide resolved
packages/cli/src/node-types.ts Outdated Show resolved Hide resolved
packages/core/src/CreateNodeAsTool.ts Outdated Show resolved Hide resolved
burivuhster
burivuhster approved these changes Sep 4, 2024
View reviewed changes
Copy link
Contributor

@burivuhster burivuhster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing my comments!
LGTM 🚀

@cypress Cypress
Copy link

cypress bot commented Sep 4, 2024

n8n    Run #6726

Run Properties:  status check passed Passed #6726  •  git commit 576d6015e7: 🌳 🖥️ browsers:node18.12.0-chrome107 🤖 OlegIvaniv 🗃️ e2e/*
Project n8n
Branch Review ai-284-implement-node-as-tool-wrapper
Run status status check passed Passed #6726
Run duration 04m 47s
Commit git commit 576d6015e7: 🌳 🖥️ browsers:node18.12.0-chrome107 🤖 OlegIvaniv 🗃️ e2e/*
Committer Oleg Ivaniv
View all properties for this run ↗︎
Test results
Tests that failed  Failures 0
Tests that were flaky  Flaky 0
Tests that did not run due to a developer annotating a test with .skip  Pending 0
Tests that did not run due to a failure in a mocha hook  Skipped 0
Tests that passed  Passing 422
View all changes introduced in this branch ↗︎

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 4, 2024

✅ All Cypress E2E specs passed

@OlegIvaniv OlegIvaniv merged commit da44fe4 into master Sep 4, 2024
51 of 64 checks passed
@OlegIvaniv OlegIvaniv deleted the ai-284-implement-node-as-tool-wrapper branch September 4, 2024 10:06
MiloradFilipovic added a commit that referenced this pull request Sep 5, 2024
@MiloradFilipovic
Merge branch 'master' into ADO-2545-ai-assistant-support-chat
f6d99fb 
* master:
  refactor(RabbitMQ Trigger Node): Improve type-safety, add tests, and fix issues with manual triggers (#10663)
  feat(editor): Add support for nodes with multiple main inputs in new canvas (no-changelog) (#10659)
  fix(editor): Set minimum zoom to 0 to allow fitting very large workflows in new canvas (no-changelog) (#10666)
  feat(editor): Change selection to be default canvas behaviour (no-changelog) (#10668)
  feat: More hints to nodes  (#10565)
  fix(editor): Fix opening executions tab from a new, unsaved workflow (#10652)
  fix(AI Agent Node): Fix tools agent when using memory and Anthropic models (#10513)
  feat(editor): Make highlighted data pane floating (#10638)
  fix(editor): Fix workflow loading after switching to executions view in new canvas (no-changelog) (#10655)
  refactor(benchmark): Separate cloud env provisioning from running benchmarks (#10657)
  feat(core): Implement wrapping of regular nodes as AI Tools (#10641)
  refactor(editor): Remove Trial logic in personalization modal and port to script setup (#10649)
  fix(core): Declutter webhook insertion errors (#10650)
  feat: Reintroduce collaboration feature (#10602)
  feat(benchmark): Add scenario for expressions with Set node (#10647)
  feat(benchmark): Add benchmark scenario for binary files (#10648)
  build: Add `reset` script (#10627)
  feat(editor): Overhaul node insert position computation in new canvas (no-changelog) (#10637)
@github-actions github-actions bot mentioned this pull request Sep 5, 2024
Merged
@janober
Copy link
Member

janober commented Sep 5, 2024

Got released with [email protected]

@ShireenMissi ShireenMissi mentioned this pull request Sep 5, 2024
Merged
4 tasks
riascho pushed a commit that referenced this pull request Sep 23, 2024
@OlegIvaniv @jeanpaul @riascho
feat(core): Implement wrapping of regular nodes as AI Tools (#10641)
de24fe7 
Co-authored-by: JP van Oosten <[email protected]>
netroy
netroy reviewed Oct 15, 2024
View reviewed changes
packages/workflow/src/NodeHelpers.ts
* as an AI Agent Tool.
* Returns the modified item (not copied)
*/
export function convertNodeToAiTool<
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why did we decide to put this here? can we instead move this to core?
We have way much code in the workflow package. So unless some code actually needs to be shared between backend, frontend, and nodes, we should avoid adding it to this package 🙏🏽

@netroy netroy mentioned this pull request Oct 15, 2024
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@netroy netroy netroy left review comments

@burivuhster burivuhster burivuhster approved these changes

Assignees
No one assigned
Labels
core Enhancement outside /nodes-base and /editor-ui n8n team Authored by the n8n team Released ui Enhancement in /editor-ui or /design-system
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@OlegIvaniv @janober @netroy @burivuhster @jeanpaul