parsing json & the Stream interface

[dominictarr]

hi!

I'm looking for a practical streaming json parser.

basically, what I think would be incredibly useful would be a parser that you could pipe into from a raw stream:

   //(load all docs from local couchdb)

   request('http://localhost:5984/tests/_all_docs)
   .pipe(new StreamingJsonParser())
   .pipe(anotherStream)

  //(note, in 0.5.x pipe returns the dest pipe, so it is chainable)

now, i'd expect StreamingJsonParser to take a raw stream, and emit objects.

I think for this to actually be useful, the root of the json stream should be on an array,
then, the 'data' events are the members of the array.

emitting the members of the first array the parser sees would work for the cases that I have examined so far
(github, twitter, rackspace, and couchdb)

unfortunately, couchdb views do not actually have an array at the root, but instead it's like this:

{total_rows: 1000, rows: [
...
]}

which is why I am advocating emitting a stream of the members of the bottom most array.

what do you think?

[dominictarr]

also, there is another potential streaming parser here: https://github.com/Floby/node-json-streams

and i've opened a similar issue there

Floby/node-json-streams#4

[creationix]

This is something you can easily implement using this library. Using the event hook you can do any x-path style query and emit your own custom events. I don't want to limit the usefulness of this library or bloat it.

[dominictarr]

ah, yes.

just reading the code, after learning from floby's project.

the onValue function is called when the object is finished parsing, and then I can look at the stack to see where it is in the object.

thanks!