Q: How to import the bindings in Node.js?

[Mickael-van-der-Beek]

Hi @indutny,

It used to be possible to import the old HTTP parser by loading the bindings in Node.js.
I looked into llhttp and the parser generator project but couldn't find an immediate way to do this.
Would you have an example snippet that would let you run the HTTP parser manually in Node.js?
Something similar to: https://stackoverflow.com/a/30180574

Thank you a lot in advance!

[indutny]

Hello!

That stackoverflow page is referring to http-parser-js which is a JavaScript port of http-parser. I believe it is still possible to use this module, because it doesn't use any C/C++ bindings at all.

It is not clear what you'd like to do, however. Do you want to run old HTTP parser or new HTTP parser? Do you want to run it manually (i.e. outside of http.createServer()/http.request)?

[Mickael-van-der-Beek]

@indutny Thanks for the quick answer!

You're right, I linked to the wrong snippet, I meant to link to the native bindings: https://stackoverflow.com/a/42003256

Yes, the idea would be to run the new HTTP parser that you wrote, llhttp, but manually in Node.js in a similar way than in the stackoverflow answer.
I'd like to pipe a Node.js stream into the parser that would then pipe the response headers and body back into a stream.
I know how to write the stream interfaces so that's ok, I'm just looking for a way to run the llhttp part.

I know that this is pretty much what the HTTP module does, but I'd like to be able to recreate something similar on my own.

[indutny]

The process.binding(...) should work without any changes on your end since llhttp and http-parser are interoperable (at least when it comes to their JS APIs).

[Mickael-van-der-Beek]

Hmm, then I'm encountering a bug I believe in [email protected] and [email protected] on OSX, for this short test case (which would work on the old parser):

cconst HTTPParser = process.binding('http_parser').HTTPParser;

const httpParser = new HTTPParser(HTTPParser.REQUEST);

httpParser[HTTPParser.kOnHeadersComplete] = () => console.log('headers=', arguments);

httpParser.execute(Buffer.from(
  'GET /test HTTP/1.1\r\n' +
  'Host: www.example.com\r\n' +
  '\r\n'
));

I'm getting the following error:

node(55269,0x10df1a5c0) malloc: can't allocate region
*** mach_vm_map(size=9007066110754816) failed (error code=3)
node(55269,0x10df1a5c0) malloc: *** set a breakpoint in malloc_error_break to debug
libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
fish: 'node http-parser-test.js' terminated by signal SIGABRT (Abort)

[Mickael-van-der-Beek]

Wow, it's actually even weirder than I thought.
If I run the program multiple times, it will sometimes fail with the error I posted above:

node(55465,0x1097d65c0) malloc: can't allocate region
*** mach_vm_map(size=27105212221095936) failed (error code=3)
node(55465,0x1097d65c0) malloc: *** set a breakpoint in malloc_error_break to debug
libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
Abort trap: 6

sometimes with only the abort trap error:

Abort trap: 6

and then a few times:

Segmentation fault: 11

Even on the latest version, v13.7.0.

[indutny]

It is likely that there is some mandatory callback missing. Let me see.

As for your case, wouldn't it be easier (and less dependent on the internals) to just create a server and do server.emit('connection', yourInputStream) and then send request events to the output stream?

[indutny]

Ah, you haven't initialized the parser:

  parser.initialize(
    HTTPParser.REQUEST,
    new HTTPServerAsyncResource('HTTPINCOMINGMESSAGE', socket)
  );

As I said before, it is not easy to use this outside of Node.js core...

[Mickael-van-der-Beek]

Thanks for the follow-up!
Even with the correct initialisation parameters (which I mocked), I seem to keep getting the same errors described above.
The updated snippet is now:

const { Socket } = require('net');
const HTTPParser = process.binding('http_parser').HTTPParser;

class HTTPServerAsyncResource {
  constructor(type, socket) {
    this.type = type;
    this.socket = socket;
  }
}

const httpParser = new HTTPParser(
  HTTPParser.REQUEST,
  new HTTPServerAsyncResource('HTTPINCOMINGMESSAGE', new Socket())
);

httpParser[HTTPParser.kOnHeadersComplete] = () => console.log('headers=', arguments);

httpParser.execute(Buffer.from(
  'GET /test HTTP/1.1\r\n' +
  'Host: www.example.com\r\n' +
  '\r\n'
));

I understand that it's tricky but I would still like to try to call the parser manually because it was possible in the past and it let's you try out a lot of things on the parser directly.

[indutny]

Could you try calling .initialize() on the parser?

[Mickael-van-der-Beek]

@indutny It seems to work now. Thank you very much for the support and sorry for asking silly questions. :)

Have a nice evening.

[indutny]

Np at all. Glad it worked! You have a good evening as well!

[leppert]

As for your case, wouldn't it be easier (and less dependent on the internals) to just create a server and do server.emit('connection', yourInputStream) and then send request events to the output stream?

For anyone else curious about a minimal way to utilize these parsers, as mentioned above, here's a code snippet.

import { createServer, request } from 'http'
import { PassThrough } from 'node:stream'

const parseRequest = (data) => new Promise(resolve =>
  createServer()
    .on('request', resolve)
    .on('connection', stream => stream.write(data))
    .emit('connection', new PassThrough())
)
const parseResponse = (data) => new Promise(resolve =>
  request({ createConnection: () => new PassThrough() })
    .on('socket', stream => stream.write(data))
    .on('response', resolve)
)

const parsedRequest = await parseRequest(Buffer.from('GET / HTTP/1.1\r\nHost: example.com\r\n\r\n'))
const parsedResponse = await parseResponse(Buffer.from('HTTP/1.1 200 OK\r\n\r\n'))

For other ways of accessing these parsers, see here: gr3p1p3/transparent-proxy#30

[kettanaito]

For those wondering, getting HTTPParser from the bindings has been deprecated. If you absolutely must, import it from the node:_http_common module instead:

import { HTTPParser } from 'node:_http_common'

[rickmed]

is import { HTTPParser } from 'node:_http_common' a public api? if not, can it be?

[ShogunPanda]

It's not a public API and there is no need for it to be.
llhttp can be used in JS by using its WASM version, just like undici does.
See: https://github.com/nodejs/undici/blob/219da8b7b3fea7e38a7644b8bc35fe6fec97d66e/lib/dispatcher/client-h1.js#L139