What is your biggest pain point w/ Node?

[mikeal]

What is the biggest pain point right now using Node?

This doesn't have to be specifically related to Node core.

Please refrain from suggesting features yet.

Please include a little bit about your background and what you're doing with Node.

[trevnorris]

There's too much surface area to the API that's tightly coupled with the internals. By writing a lower level binding (what @domenic has dubbed "libuv.js") that much of the current API can be implemented on top of, Node can allow developers to implement their own interfaces.

There's sentiment from one group that Node should have full support for Promises. While at the same time another group wants generator syntax support (e.g. var f = yield fs.stat(...)). It seems these parties miss the fact Node can't simultaneously support all syntaxes for all upcoming features.

That's why I believe core code needs to minimize the API and allow these parties to implement the interface of choice. These are easily enough installed via a module, and it reduces the surface area core maintainers have to maintain. This will lead to more secure and stable code from core, while also allowing interface implementors to 1) iterate faster than core releases occur; 2) bypass the current overhead of needing to implement any other interface on top of the current EventEmitter/Streams model.

[trevnorris]

Heh, seems I didn't make it to point #3 in the OP.

[jonathanong]

btw generator support === promises. if they ask for something different, they don't know what they're talking about.

i'd be pretty interested in libuv.js.

my biggest issue is right now is that it seems as though the node lead only spends a single day out of the month to work on node.js _

[bodokaiser]

I think the biggest problem with node is that it lost a lot dynamic. Just look at the release cycle: We are more or less waiting (or using) 0.11 for a year now and for me it does not look as 0.12 will be released 2014. I understand that the core team wants to ship a product enterprise ready by freezing apis and rejecting any major changes but in the long term a lot of people will loose interest as PRs get rejected and bad api decisions maintained.

Regarding libuv.js I see potential to decouple dependency from the nodes api decision but am not sure if the community is able to support various architectures in terms of documentation and reliability.

[feross]

I agree with @jonathanong and @bodokaiser - node 0.12 is way past overdue. The goal of shipping an "enterprise ready" product is laudable, but having 0.12 be in a perpetual "right about to be released" state actually hurts credibility way more than releasing 0.11 right now as 0.12 would.

[juliangruber]

I'm listing my 2/3 points without thinking about whether they make sense to be fixed in node:

• not having several call stacks / threads: you can't just let a block of execution crash, knowing safely that you don't need to crash the process. like: each http request is completely independent and a crash like an uncaught "error" event somewhere wouldn't affect the other requests
• versioning of node core modules, i want readable-stream style versioning for every core module. that would also help with release cycles
• too long release cycles, when I need change X i need to use unstable node

[Qard]

I'd like to be able to use es6 tech with core.

Streams could be simpler:

var upcaseStream = passthroughStream(function* (read, write) {
  var bytes = 1024
  var chunk
  while ((chunk = yield read(bytes))) {
    yield write(chunk.toString().toUpperCase())
  }
})

process.stdin.pipe(upcaseStream).pipe(process.stdout)

Servers could be simpler:

// textual interface (get foo, set foo bar)
tcp.createServer(function* (read, write) {
  var auth = yield read()
  if ( ! (yield db.auth(auth))) {
    return
  }

  var message
  while ((message = yield read())) {
    var match = message.match(/^(get|set) (\S*)( (\S*))?$/)
    var command = match[1]
    var key = match[2]
    var val = match[4]

    if (match) {
      yield write(yield db[command](key, val))
    }
  }
})

At the same time, I agree with @trevnorris that this sort of concern of promises vs generators should probably not be directly in core. However, having some of these features enabled by default in V8 would promote wider acceptance of their use--I think co/koa would be a lot more popular if you didn't need to use a special flag on an unstable build.

I really like the libuv.js idea. As someone whose job is instrumenting node, having deeper access into the guts of it would be really valuable. I'd really like to see visibility into uv_check_t, uv_prepare_t and uv_idle_t at JS level.

Having something like Zones in core would be great. Perhaps with libuv.js we could store file descriptor references directly on JS objects so cleaning up file descriptors of things in a zone could be as simple as socket.fd.destroy().

I'd also love to see core facilities designed in a way that they are more pluggable. For example, with browserify you can use transforms to make require('./path/to/template.jade') return a compiled jade template function. Also, monkey-patching require() behaviour is currently rather difficult to do while retaining correct path resolution--it'd be good if there was a way around that.

My biggest gripe, however, and the reasoning for my strong desire for generator-based streams is the fragility of event emitters propagating error events. Having to special case error events is kind of obnoxious. With generators, you can simply let the user try/catch it, or leave it to throw in normal JS fashion.

Cleanup also kind of sucks. If you forget to end a stream, bad stuff can happen. With the generator example above, you can simply assume that reaching the end of the generator function means the stream is done, since there is no longer code to operate on it.

[rvagg]

@Qard your streams suggestion is absolutely something that should be experimented with in user-land in order to prove its utility. Major departures like this should never again be dumped straight into core without having a long burn-in time in user-land where they can be embraced or rejected by a wider group.

Easy to put together a wrapper, then promote it and see if you can get any traction. Personally I'd like to see core streams stripped back even further to something like what http client is now such that its a logical choice to add an abstraction over the top of it to get real work done. Then we avoid lock-in, complexity in core, and allow the ecosystem to come up with the best patterns rather than having it forced on them.

[Qard]

@rvagg Yep, I totally agree that is a userland thing. I feel like the need for a harmony flag and unstable build currently is holding people back a bit from experimenting with it though. I'm more just advocating for at least a stable release at some point with the harmony flags, though I'd prefer to see some features on by default.

As @mikeal put it, I'm not suggesting we start implementing my pet feature ideas. I'm just sharing my views. I do think the whole error event thing still kind of sucks though. Generators may be a possible solution, but, as you say, it's yet to be seen as the solution.

In line with what you are saying, and the libuv.js discussion, stripping core down even further would be really cool. I think sometimes core does too much. Having lower-level control of things would be great.

[rvagg]

Agreed on the special-case 'error' event, I just don't see a good solution to that in Promises or Generators yet, but I'll try and reserve too much judgement. In short, Promises treats errors like a second-class citizen which is exactly the 'error' event problem so it buys you nothing, Generators takes us back to codebases littered with try/catch which I certainly don't want to see become an idiomatic Node style. But, I look forward to being convinced of a great new pattern that someone clever comes up with to solve all our woes.

[Qard]

I certainly prefer manual try/catch to promises just eating errors. It is a bit verbose though. A rescue expression, like ruby, would be really nice.

[calvinmetcalf]

@Qard https://github.com/calvinmetcalf/es6-stream

[Qard]

Nice. I actually have an implementation of my own half done. I'll probably finish it on the long bus ride I'll be on tonight. Mine includes readable and writeable streams too. :)

[calvinmetcalf]

Figured you were writing one but couldn't help myself.

I mean writable is easy you just do thunkify with the native .write method, readable could be trickeir, but a thunk version of noms (which is pretty much how I wish streams worked) might be a starting point.

edit: spell my name right

[calvinmetcalf]

on second though I know exactly what you mean by read and write streams because I wrote them to make that library work and can just separate them out

[Fishrock123]

I think most of my pain points actually come from npm.
Additionally, if we could get node-forward to eventually release a npm-fixed 0.8, that would be great.

Aside from that, faster releases. I know way too many people running 0.11 in production for generator stuff haha.

[trevnorris]

@Qard Node won't be enabling any harmony flags by default. We leave it up to V8 to tell us when those features are "ready". Though, if it's any consolation, we'll probably upgrade to v3.29 before v0.12 is released. This removes generators from behind the harmony flag.

As for your streams implementation and comments about the event emitter, I agree that the current implementation is not as it should be. Though no one can agree on the Right Way. Hence the need for a lower level API to allow module authors to implement their own interfaces.

Whether devs like it or not the basic callback style is the fastest way for JS and C++ to interact, and even using that we have too much overhead.

[joeybaker]

My biggest pain point is debugging.

1. Errors suck. They're a mixture of unclear and infrequently useful in an async world (because the stack traces get broken). Domains weren't the answer. I've heard a rumor that async stack traces are coming in 0.12 but… I don't think I've see it in 0.11 yet?
2. "dev tools" are lacking. Things like node-inspector are close, but I want call stack flame graphs and memory profiling. I don't want to learn dtrace if I don't have to.

IMHO

About me

I've been working with JS professionally for 5+ years now and node for about 2.5. I like callbacks, streams, tape tests, and hiking. Oh, and beer. 🍺

[Qard]

@trevornorris Tracking latest V8 versions is all I hope for in terms of es6
features. Node has lagged behind quite a bit lately.

I agree that the generator implementation may not be the right way to do
streams. It has some obvious advantages, but may also have subtle
disadvantages. JIT optimization performance is as-yet unknown.

A lower-level interface for streams would be great. I'd like to see the
current evented design abstracting something deeper and more manual.

[rvagg]

This discussion on streams is exactly the kind of discussion that I'd like to see moved out of core and into a more focused group around readable-stream so we don't have API design being driven by core implementation concerns but the other way around.

[trevnorris]

@joeybaker

I've heard a rumor that async stack traces are coming in 0.12

Sort of. That was part of the async listener patch. Though the user-facing API has been scrapped for further iteration. There is still a "less public" API user can tap into, and we hope the will so we can get feedback on the best approach.

I want call stack flame graphs and memory profiling

There's definitely more we can do, but asking for a flame graph via a function call is a bit of a stretch. :)

@Qard

Node has lagged behind [v8] quite a bit lately.

Main reason for the lag of V8 updates is because in v3.26 V8 removed the internal debugger. Which meant we needed to implement our own. Took a while but @indutny stepped up and got it working.

JIT optimization performance is as-yet unknown [for generators].

It's an unknown because generators can't be optimized. :-P Like, straight up are never sent to the optimizing compiler. Maybe one day they will be, but as of right now there is no question of whether they are slower.

@rvagg

This discussion on streams is exactly the kind of discussion that I'd like to see moved out of core

I agree, hence why I want to create a non-streams, callbacks only, based interface for the lower-level API that everyone can use to implement their own streams interface.

[bodokaiser]

+1 for callbacks in core
+1 for streams out of core - most implementations are anyway out of stack

Am 14.10.2014 um 01:46 schrieb Trevor Norris [email protected]:

@joeybaker

I've heard a rumor that async stack traces are coming in 0.12

Sort of. That was part of the async listener patch. Though the user-facing API has been scrapped for further iteration. There is still a "less public" API user can tap into, and we hope the will so we can get feedback on the best approach.

I want call stack flame graphs and memory profiling

There's definitely more we can do, but asking for a flame graph via a function call is a bit of a stretch. :)

@Qard

Node has lagged behind [v8] quite a bit lately.

Main reason for the lag of V8 updates is because in v3.26 V8 removed the internal debugger. Which meant we needed to implement our own. Took a while but @indutny stepped up and got it working.

JIT optimization performance is as-yet unknown [for generators].

It's an unknown because generators can't be optimized. :-P Like, straight up are never sent to the optimizing compiler. Maybe one day they will be, but as of right now there is no question of whether they are slower.

@rvagg

This discussion on streams is exactly the kind of discussion that I'd like to see moved out of core

I agree, hence why I want to create a non-streams, callbacks only, based interface for the lower-level API that everyone can use to implement their own streams interface.

—
Reply to this email directly or view it on GitHub.

[Qard]

Yep. I was aware the debugger was gone.

Good to know about the generator performance. I haven't taken the time yet
to dig into V8 internals to see how the new stuff really works.

[Fishrock123]

It's an unknown because generators can't be optimized. :-P

I still wonder if regenerator doesn't actually make faster code.

[joeybaker]

@trevnorris Yea, I don't expect a flame graph to be a function call away. I do think it should be easier to get a CPU profile out of V8 from node though :)

I'll go back to my main point though: ease-of-use is one the key factors in picking a platform. I'd define that as:

1. easy to maintain:
   • backward-compatible core releases: we'll get there.
   • actionable errors: core errors are pretty opaque
   • stability: spin up a node process. It doesn't die easily.
2. ramping up new hires
   • easy to grok new code: JS has no standard format, has many ways to do the same thing, and has no common framework like ruby. There are pros/cons to each of these, but it does make new code harder to quickly understand.
   • easy to understand the language: JS async is still a mindbender for beginners which we could make better, but JS is pretty good, and that's a JS problem, not a node problem. Also: not erlang.
   • good docs: they're somewhere between barely-sufficient to a mess
3. open source support (combined with the quality of that code)
   • have something like npm: we have npm
   • packages are generally of high quality: thank you unix philosophy
   • good community: e.g. this repo
   • community has agreed upon standards for common problems: code style, async handling, error propagation.
4. debugability: writing new stuff
   • great debugging tools: unless you're a dtrace wizard (and even then), node-inspector is about as good as it gets; but it's slow, and not very full-featured.
   • it's easy to get down to the metal if need-be: I love me the high-order languages, but sometimes I know better than v8 when to run GC; and other such problems.
   • stack traces are useful: async stack traces please.

Node-core may not be responsible for all of these. But it should probably, at the very least, have an opinion on all of them. IMHO

[ronkorving]

My biggest non-code related pain point is how Node 0.12 release was supposedly "imminent" for the past year or what, and here we are. I'm quite frustrated by the lack of communication from the project lead to the community. In fact, it feels like the community is pretty much being ignored right now, including those who contribute the most. Now I know I'm preaching to the choir here so I'll stop ranting and move on to what you're really asking.

My biggest pain point is asynchronous error handling. It's a real pain, and generators give a way to solve that. I would really like to see those in. I don't see a point in libuv.js as a standalone product, but who knows, if that does the trick. I'm not against asynchronous syntax in cases like EventEmitter on a server.

I'll go against the flow here a bit and give my 2 cents on Streams. I would like them to stay in core. It's so fundamental to thousands of modules, that taking it out and letting it "evolve" in different directions is asking for trouble. I think the fragmentation and incompatibilities could destroy what is currently a very successful ecosystem (npm). I'm already frustrated when I see coffeescript modules (which I refuse in my codebase) that are otherwise really useful. My frustration will reach new heights when I see a "tar" module that is only compatible with "substack-streams", but not "ogden-streams". For that I will need to download "ogden-tar". Please don't let that happen.

About me:

I'm VPoE at a company in Tokyo called Wizcorp, and we've been Node.js users since the days of 0.4, and have been using it on (game) projects with literally millions of users. We embrace open source, have open sourced a lot of work and contributed to a lot of others.

[joeybaker]

@ronkorving You convinced on streams. 👍 for them staying in core. Once you got into it, I realized: yes the streams implementation has been changed a few times, and yes it was tested in userland first, but they're such a core concept, that there really oughta be a "blessed" implementation.

[Qard]

I don't think they should be outright removed from core. Just that the
existing API should be an abstraction of something lower-level that we can
bypass, if we so choose.

Frankly, streams are super janky, but the current implementation is central
to many, many modules. Removing the current API from core would break all
of those, which would be unacceptable.

[bnoordhuis]

@mythagon Apropos V8's memory limit, doesn't --max_old_space_size=4096 work for you?

[mythagoras]

@bnoordhuis nope, it didn't work for me, and from what I read it doesn't really work, max memory remains ~1.5-2GB, it's seems to be an eternal oath from v8's part and don't expect them to change it.

[bnoordhuis]

@mythagon Is that with joyent/node v0.10? I know some of our (i.e. StrongLoop's) customers are successfully running with 4 or 8 GB heaps. It's possible that only works well with newer V8 versions than what is available in v0.10.

[mythagoras]

@bnoordhuis yeap, I'm currently on an old joyent/node v0.10, getting ready to migrate to iojs. I was waiting on nvm support and will now do some tests specifically on this.

[mkliu]

Not sure if it's already tracked, but this is THE biggest problem I have
nodejs/node-v0.x-archive#6960

[unbornchikken]

Gyp.

• If I want to integrate a complex library to node as a native addon, first I have to somehow hack it to build right with gyp on every platform (*.h.in files anyone?)
• It's Python 2 dependency is a shame
• It still lacks of essential build features, you have to write custom Python 2 scripts to achieve anything complex.
• It still not uses Ninja (node-gyp)

That's why I created CMake.js module. I think it's time to give a place to other build systems than gyp in node/io.js ecosystem.

Does somebody support my idea or is it just stupid, gyp should be enough for anyone?

[tychotatitscheff]

I would say that debug (trace and debug tools are way to low level).
They are hard to read in order to identify the source of the bug.

[mgol]

My biggest pain point recently is that Node is... moving too fast. Yes, the Joyent times were too slow but current situation looks like that when you use any compiled dependencies like node-sass (and a lot of people do):

1. New stable V8 comes out.
2. New io.js/Node.js version comes out (~2 weeks later)
3. Waiting for a compatible version of compiled deps (sometimes 3-4 weeks later).
4. Some time for internal testing & updating the used io.js version to the new one.
5. New stable V8 comes out (shortly after point 4).

The effect is that for 80% of the time I'm on an unsupported io.js/Node.js version as I wait for package updates. This is worse than in Joyent times - back then the V8 used didn't have security support from the V8 team but Joyent was patching things. Now theoretically it should be better as io.js is on a supported V8 version but so what if I can update to it only slightly before its EOL?

The effect is that using the latest io.js/Node.js is fine for task automation but on any public server it's a security risk and only an LTS version would work. And I assume that's not an intention of the Node team.

[bnoordhuis]

the V8 used didn't have security support from the V8 team but Joyent was patching things

That is a misconception. There are glaring bugs in the V8 that ships with v0.10 and v0.12, some possibly exploitable, that are almost certainly never going to get fixed.

The few fixes that have landed have mostly been contributed by outside people. I think me and @indutny are the top contributors in that respect. Joyent itself does hardly anything with V8.

only an LTS version would work

LTS releases are in the works: https://github.com/nodejs/LTS - participation very welcome.

[mgol]

@bnoordhuis Thanks for the clarification.

I know about the LTS plan, it's good! However, it would be cool if every new Node major release didn't require a significant refactoring of native modules as then non-LTS versions having upstream security support are unreachable for most people using packages like node-sass.

Maybe a good idea would be to have security support not only for the latest Node & LTS releases but also for one previous Node version? Native modules are usually updated for compatibility with Node n before Node n+1 comes out so this would solve the security issue of always trying to use the latest available Node when using native modules.

[bnoordhuis]

have security support not only for the latest Node & LTS releases but also for one previous Node version?

If I understand you right, that's the case: the support lifecycles of current and current-1 overlap.

[mgol]

If I understand you right, that's the case: the support lifecycles of current and current-1 overlap.

That solves my problem, thanks! :) Is it written up somewhere?

[bnoordhuis]

The table at the bottom of the README lists the support schedules. You can see how the release branches overlap by a year.

(Minor aside: v0.12's LTS EOL is going to be moved three months to the fore; that was only decided yesterday and the README hasn't been updated yet to reflect that. EDIT: nodejs/Release#35)

[mgol]

The table at the bottom of the README lists the support schedules.

That lists LTS releases. What I mean that when Node 6.0.0 is released, Node.js 4.x.y will be a still supported (security-wise) LTS version as long as 0.10 & 0.12. However, many people won't be able to upgrade to Node 6.0.0 because native modules will not have been updated yet. Therefore, they'll need to stay on Node 5 for some time and switch to Node 6 not long before Node 7 is released. Therefore, it'd be good that not only LTS releases & the latest stable one receive security fixes but also the previous stable version, i.e. here both Node 6 & 5 lines should receive security support.

Otherwise relying on stable non-LTS node for production would mean you cannot use native modules or you'll get stuck on an unsupported Node version.

[mikeal]

However, it would be cool if every new Node major release didn't require a significant refactoring of native modules as then non-LTS versions having upstream security support are unreachable for most people using packages like node-sass

There are two parts to our answer for this.

The first is that we actually have pulled back the number of major releases each year (which would require, at the least, a dependency version bump of nan) to two. Our hope is that 6 months is enough time for the ecosystem to "catch up." Also, only half of those releases are targets for LTS so that means that most enterprises will only be updating once a year and we'll be doing a bigger push to get the ecosystem to update around those releases.

The second part of this answer is simple: we need to move certain sections of the community and ecosystem off of native modules entirely. You bring up a good example, node-sass, which has pure JS alternatives that are just as good and don't suffer the problems native modules do. The frontend tools community should be able to move off of native modules entirely. For obvious reasons the IoT community and the level database community will have a much harder time and we're unlikely to see this happen for quite a while, but frontend compile tools shouldn't need native deps and we're working on incentives to move this community away from native modules.

[mgol]

Our hope is that 6 months is enough time for the ecosystem to "catch up."

6 months should be enough so the problem doesn't affect LTS releases too much, only latest stable Node ones.

You bring up a good example, node-sass, which has pure JS alternatives that are just as good and don't suffer the problems native modules do.

I assume you mean Less or Stylus. I've seen more people moving from Less to Sass than otherwise (due to Sass's expressiveness) and with Sass you can choose between a slow Ruby implementation and a fast C++ one. I don't imagine people moving off Sass in a predictably near future.

But yes, if the front-end community stopped needing native modules, the problem would be resolved as well. But if that's not currently possible, supporting the previous-to-latest-stable Node version security-wise would also solve the problem for most people.

we're working on incentives to move this community away from native modules.

Do you have anything publicly available to read about those incentives?

[domenic]

The third part of the answer is that V8 will likely not undergo as much churn in the future as it has in the recent past. We had some bad timing, with io.js 1.0 being released right before V8 4.3/4.4's Maybepocalypse and SetIndexedData change. The two big jumps in quick succession, from years-out-of-date V8 with Node to V8 4.2 in io.js 1.0, and then from io.js 2.0 to io.js 3.0, have given a false impression that V8 will undergo drastic changes every release. Such changes may still happen, but it shouldn't be very often. See the header at https://bit.ly/v8-api-changes to get a better sense of what changes are coming; they are in general less disruptive.

[mgol]

@domenic That's very good to hear.

[mikeal]

I assume you mean Less or Stylus. I've seen more people moving from Less to Sass than otherwise (due to Sass's expressiveness) and with Sass you can choose between a slow Ruby implementation and a fast C++ one. I don't imagine people moving off Sass in a predictably near future.

I didn't mean alternatives to Sass I meant pure JavaScript implementations like https://github.com/medialize/sass.js rather than the native one. I've been told there are others.

[mikeal]

Do you have anything publicly available to read about those incentives?

So, the plan is for master to be representative of the next major release of node.js (not the current) and for this to keep up with v8's 6 week cycle and produce regular "alpha" releases. The plan is to promote this alpha channel aggressively so that we get a broad array of testing. Being that new JS features will land here first, and the frontend community tends to lead the charge for new JS features, we should be able to get a lot of that community on to this channel even though there will be regular native breaks. Those native breaks will nature incentivize the community to get off of native modules.

The basic value proposition is: if you want new JS features on the regular you can't depend on native modules.

[creationix]

I wonder if efforts could be focused on some generic ffi module that stayed compatible between node releases? Then "native" addons could be built using this ffi interface in pure JS.

That said, as you know, I'm all for re-implementing popular tools to pure JS.

[mikeal]

@creationix nodejs/node#1865 :)

[trevnorris]

@bnoordhuis Seems that diagram hasn't been updated to follow the current release process.

/cc @jasnell here: https://github.com/nodejs/LTS/blob/master/README.md

[jasnell]

Yes, that page needs to be updated. I'll be working on that today or tomorrow

[Zayelion]

My latest pain point with node has been trying to npm install sqlite3 on a windows box with iojs2+. I gave up. @mikeal and those before me pointed out native dependencies are painful. There needs to be like a movement or something and gyp changed into a JS script. Its using python WHY?

[StoneCypher]

@Zayelion - because you haven't submitted a patch yet.

[Trott]

Closing all issues in this archived repository.