feat: HTTP request tool

[michael-radency]

Summary

This tool enables the model to make HTTP requests with various parameters, send in path, body, headers or query . It also allows the model to utilize credentials from other nodes for authentication purposes. Optimize Response option allows some optimization to reduce context send to model.

Related tickets and issues

https://linear.app/n8n/issue/AI-162/tool-to-visit-a-website

[netroy]

We could use @mozilla/readability and jsdom here to cleanly extract the content that's likely relevant to an end-user.

Something like this perhaps:

import { JSDOM } from 'jsdom'
import { Readability } from '@mozilla/readability'

const dom = await JSDOM.fromURL(url)
const article = new Readability(dom.window.document, {
    keepClasses: true,
}).parse()

and then use article.content.

we could also consider using turndown to convert the html into markdown, which LLM tend to handle better than html IMO.

import Turndown from 'turndown'
const markdown = turndown.turndown(article.content)

[michael-radency]

@netroy
what would be advantages over html-to-text + Cheerio? since we already using such setup for Html node

[netroy]

cheerio is great to either use css selectors to extract text, but leaves the burden of determining the semantics in the markup to the end user.

First download an article via curl https://www.bbc.com/news/articles/cldd6x6gglxo > news.html.

Then try this with cheerio:

const fs = require('fs');
const cheerio = require('cheerio');
const html = fs.readFileSync('news.html', 'utf8');
const $ = cheerio.load(html);
console.log($('body').text());

I got this

With just readability:

(async () => {
	const { JSDOM } = require('jsdom');
	const { Readability } = require('@mozilla/readability');
	const Turndown = require('turndown');

	const dom = await JSDOM.fromFile('news.html');
	const article = new Readability(dom.window.document, {
		keepClasses: true,
	}).parse();
	console.log(article.textContent);
})();

I got this

With readability + turndown:

(async () => {
	const { JSDOM } = require('jsdom');
	const { Readability } = require('@mozilla/readability');
	const Turndown = require('turndown');

	const dom = await JSDOM.fromFile('news.html');
	const article = new Readability(dom.window.document, {
		keepClasses: true,
	}).parse();
	const turndown = new Turndown({
		headingStyle: 'atx',
		hr: '---',
		bulletListMarker: '-',
		codeBlockStyle: 'fenced',
	});
	const markdown = turndown.turndown(article.content);
	console.log(markdown);
})();

I got this

Perhaps we should add a "Extract as Markdown" option in the node to determine if we want to use markup to reduce semantic noise in the extracted text?

[OlegIvaniv]

+1 for using @mozilla/readability

[OlegIvaniv]

+1 for using @mozilla/readability

[OlegIvaniv]

Suggested change

	if (!Object.keys(options.headers as IDataObject).length) {
	delete options.headers;
	}
	if (!Object.keys(options.qs as IDataObject).length) {
	delete options.qs;
	}
	if (!Object.keys(options.body as IDataObject).length) {
	delete options.body;
	}
	if (options) {
	options.url = encodeURI(options.url);
	if (options.headers && !Object.keys(options.headers).length) {
	delete options.headers;
	}
	if (options.qs && !Object.keys(options.qs).length) {
	delete options.qs;
	}
	if (options.body && !Object.keys(options.body).length) {
	delete options.body;
	}
	}

[OlegIvaniv]

We might also want to add a .dark icon to improve contrast

[cypress]

1 flaky test on run #5573 ↗︎

0

395

0

0

1

Details:

🌳 🖥️ browsers:node18.12.0-chrome107 🤖 michael-radency 🗃️ e2e/*
Project: n8n	Commit: 853da9022c
Status: Passed	Duration: 04:29 💡
Started: Jun 19, 2024 7:47 AM	Ended: Jun 19, 2024 7:51 AM

  e2e/5-ndv.cy.ts • 1 flaky test

View Output Video

Test		Artifacts
NDV > should not retrieve remote options when required params throw errors		Screenshots Video

Review all test suite changes for PR #9228 ↗︎

[github-actions]

✅ All Cypress E2E specs passed

[netroy]

This PR also updated packages/nodes-base/nodes/Splunk/splunk.svg. Is that intentional?

[janober]

Got released with [email protected]

[janober]

Got released with [email protected]