Expand working group and repo to all diagnostics.

Add additional top level to repo with subfolders for diagnostic domains. Add initial README.md for each domain. Add some initial docs and samples for domains beyond tracing. Move and minor cleanup of AsyncWrap docs and background info on TRACE_EVENT.
nodejs · Jun 9, 2016 · ab45c0e · ab45c0e
1 parent d3a41e9
commit ab45c0e
Show file tree

Hide file tree

Showing 14 changed files with 176 additions and 72 deletions.
diff --git a/README.md b/README.md
@@ -1,9 +1,24 @@
-# tracing-wg
+# Diagnostics Working Group  
+The goal of this WG is to ensure Node provides a set of comprehensive, documented, extensible diagnostic protocols, formats, and APIs to enable tool vendors to provide reliable diagnostic tools for Node.
 
-Tracing Working Group
+Work is divided into several domains:
+- [Tracing](./tracing)
+- [Profiling](./profiling)
+- [Heap and Memory Analysis](./heap-memory)
+- [Step Debugging](./debugging)
 
-# Members
+Background, reference documentation, samples, and discussion for each domain is contained within its folder.
 
+Work needed includes:  
+- Collect, understand, and document existing diagnostic capabilities and entry-points throughout Node, V8, and other components.
+- Collect and document projects and products providing diagnostics for Node with brief description of their technical architecture and sponsoring organizations.
+- Identify opportunities and gaps, then propose and implement solutions.
+
+### Logistics
+- Monthly Meetings
+- Biannual F2F
+
+### Members  
 * @bnoordhuis 
 * @sam-github 
 * @othiym23 
@@ -28,3 +43,5 @@ Tracing Working Group
 * @matthewloring
 * @pmuellr
 * @jeffolfert
+* @joshgav
+
diff --git a/debugging/README.md b/debugging/README.md
@@ -0,0 +1,29 @@
+## Debugging API
+- [Chrome Debugging Protocol](https://chromedevtools.github.io/debugger-protocol-viewer/)
+- [V8 Debugging Protocol](https://github.com/v8/v8/wiki/Debugging-Protocol)
+
+Node currently relies on V8's internal Debug API and associated commands and events. The Debug API is published at [v8/include/v8-debug.h](https://github.com/v8/v8/blob/master/include/v8-debug.h), and the message protocol is documented in prose in [the V8 wiki](https://github.com/v8/v8/wiki/Debugging-Protocol) and in code in [v8/src/debug/debug.js#L2341-2372](https://github.com/v8/v8/blob/master/src/debug/debug.js#L2341-L2372). 
+
+Node also provides an agent [node/src/debug-agent.h](https://github.com/blob/master/src/debug-agent.h) which relays requests, responses, and events through a TCP socket, and a command-line debugger [node/lib/_debugger.js](https://github.com/blob/master/lib/_debugger.js).
+
+The Chrome/V8 team has deprecated the internal V8 API and command set and proposed replacing it with a subset of the Chrome Debugging Protocol (CrDP), see https://github.com/nodejs/node/issues/2546. CrDP is documented at [https://chromedevtools.github.io/debugger-protocol-viewer/]() and the backing GitHub repo.
+
+The [cyrus-and/chrome-remote-interface](https://github.com/cyrus-and/chrome-remote-interface) library provides a JS proxy for CrDP.
+
+## Step Debugging Tools  
+Name | Sponsor
+-----|--------
+[node-inspector][] | StrongLoop
+[JetBrains WebStorm][] | JetBrains
+[Visual Studio Code][] | Microsoft
+[Node CLI Debugger][] | Node Foundation
+[Chrome DevTools][] | Google
+[Theseus][] | Adobe Research
+
+[node-inspector]: https://github.com/node-inspector/node-inspector 
+[JetBrains WebStorm]: https://www.jetbrains.com/help/webstorm/2016.1/running-and-debugging-node-js.html
+[Visual Studio Code]: https://github.com/Microsoft/vscode
+[Node CLI Debugger]: https://nodejs.org/api/debugger.html
+[Chrome DevTools]: https://github.com/ChromeDevTools/devtools-frontend
+[Theseus]: https://github.com/adobe-research/theseus
+
diff --git a/docs/README.md b/docs/README.md
diff --git a/docs/perf-events/README.md b/docs/perf-events/README.md
diff --git a/docs/platform-tracing/README.md b/docs/platform-tracing/README.md
diff --git a/heap-memory/README.md b/heap-memory/README.md
@@ -0,0 +1,10 @@
+## Heap and Memory Analysis  
+The [nodejs/post-mortem](https://github.com/nodejs/post-mortem) WG is responsible for this domain.
+
+### Heap Analysis  
+- [mdb_v8](https://github.com/joyent/mdb_v8)
+- [node-heapdump](https://github.com/bnoordhuis/node-heapdump)
+- [llnode](https://github.com/indutny/llnode)
+- [IBM IDDE](https://www.ibm.com/developerworks/java/jdk/tools/idde/)
+- [Snapshot Utils](https://github.com/bmeck/snapshot-utils)
+
diff --git a/profiling/README.md b/profiling/README.md
@@ -0,0 +1,26 @@
+## Profiling for Node  
+- CPU Profiling
+  - V8 CPU Profiler: [v8/include/v8-profiler.h](https://github.com/v8/blob/master/include/v8-profiler.h)
+    - Accessible globally through `--prof` runtime flag, basic processing and display through `--prof-processor` runtime flag
+  - Intel vTune profiler. Accessible through `--enable-vtune-profiling` compile-time flag.
+
+- Heap Profiling
+  - V8 Heap Profiler: [v8/include/v8-profiler.h](https://github.com/v8/blob/master/include/v8-profiler.h)
+    - Heap memory usage stats are available through built-in 'v8' module.
+
+Name | Sponsor
+-----|--------
+[v8-profiler][] | StrongLoop
+[Chrome DevTools][] | Google
+
+[v8-profiler]: https://github.com/node-inspector/v8-profiler
+[Chrome DevTools]: https://github.com/ChromeDevTools/devtools-frontend
+
+### Stack Analysis  
+- [0x](https://github.com/davidmarkclements/0x)
+- [StackVis](https://github.com/joyent/node-stackvis)
+
+## Docs  
+[guides/simple-profiling](https://nodejs.org/en/docs/guides/simple-profiling/)
+[paulirish/automated-chrome-profiling](https://github.com/paulirish/automated-chrome-profiling)
+
diff --git a/docs/AsyncWrap/README.md → tracing/AsyncWrap/README.md b/docs/AsyncWrap/README.md → tracing/AsyncWrap/README.md
diff --git a/docs/AsyncWrap/example-trace/run.js → tracing/AsyncWrap/example-trace/run.js b/docs/AsyncWrap/example-trace/run.js → tracing/AsyncWrap/example-trace/run.js
diff --git a/docs/AsyncWrap/example-trace/trace.js → tracing/AsyncWrap/example-trace/trace.js b/docs/AsyncWrap/example-trace/trace.js → tracing/AsyncWrap/example-trace/trace.js
diff --git a/tracing/README.md b/tracing/README.md
@@ -0,0 +1,7 @@
+## Tracing  
+- [AsyncWrap](./AsyncWrap/README.md) - Most native objects are represented indirectly by the [AsyncWrap class](https://github.com/nodejs/node/blob/master/src/async-wrap.h), so hooks have been added to that class to trace lifecycle (init/destroy) and callback activity (pre/post) related to these objects.
+
+- [OS Tracing](./os-tracing/README.md) - LTTng (Linux), SystemTap (Linux), DTrace (OSX), ETW (Windows)
+
+- [VM Tracing](./vm-tracing/README.md) - Tracing native to JS VMs, such as V8's TRACE_EVENT.
+
diff --git a/tracing/os-tracing/README.md b/tracing/os-tracing/README.md
@@ -0,0 +1,26 @@
+## OS Tracing  
+
+* [DTrace](http://dtrace.org/)  
+  Available on IllumOS, Solaris, Mac
+
+  - The [DTrace blog](http://dtrace.org/blogs/blog/category/node-js/) has some articles on using DTrace with Node.js
+
+* [LTTng](http://lttng.org/)  
+  Linux only.  
+
+  - [Tracing Node on Linux with LTTNG](http://nearform.github.io/tracing-node-lttng-nodejsdublin/)
+  - [thekemkid/magic-tracing](https://github.com/thekemkid/magic-tracing) is an example of a Node.js application which implements LTTng tracepoints. A demo is available [here](https://asciicinema.org/a/16785).
+
+* [ETW](https://msdn.microsoft.com/en-us/library/windows/desktop/aa363668.aspx)  
+  Windows only.  
+
+  - Some information was provided by Bill Ticehurst (@billti) in [issue #10](https://github.com/nodejs/tracing-wg/issues/10#issuecomment-137145822) about how he integrated ETW into Node.js for Windows
+
+* [Perf](https://perf.wiki.kernel.org/)  
+  Linux only.  
+
+  - Can be enabled with `--perf_basic_prof` v8 run-time flag.
+
+* [SystemTap](https://sourceware.org/systemtap/)  
+  Linux only.
+
diff --git a/docs/v8-tracing/README.md → tracing/vm-tracing/README.md b/docs/v8-tracing/README.md → tracing/vm-tracing/README.md
@@ -1,13 +1,24 @@
-Node.js tracing - native v8 tracing
-================================================================================
+## VM Native Tracing  
+- See [v8-tracing.md](./v8-tracing.md) for technical details and crude implementation.
 
-Google is restructuring the tracing in V8 to accommodate restructuring of
-tracing in Chrome.
+- All trace events are streamed through the same native component, which can in turn publish as appropriate to its subscribers.
+    - Ideally should be possible to subscribe and receive events in JavaScript too.
+- Requires implementation in Node.js, could learn from Chromium's implementation.
+- For maximum utility, we should add trace events to Node's operations, especially async ones.
+- Possible to handle other trace sources, such as OS (LTTng, ETW) alongside V8 trace source, since they all are event streams.
 
-----
+## Resources  
+- [google/trace-viewer](https://github.com/google/trace-viewer) - `chrome://tracing` UI.
+- [chromium/src/base/trace_event/](https://code.google.com/p/chromium/codesearch#chromium/src/base/trace_event/)
+- [Chromium Speed-Infra](https://www.chromium.org/developers/speed-infra)
+- [Chromium Telemetry](https://catapult.gsrc.io/telemetry)
+- [Trace Platform Explainer](https://docs.google.com/a/chromium.org/document/d/1l0N1B4L4D94andL1BY39Rs_yXU8ktJBrKbt9EvOb-Oc/edit)
+- [Implementations for V8, D8, Chromium](https://docs.google.com/a/chromium.org/document/d/1_4LAnInOB8tM_DLjptWiszRwa4qwiSsDzMkO4tU-Qes/edit#)
+- [Trace Event Format](https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit)
+
+### Background  
 
-Some older-ish comments from @natduca in
-[nodejs/node issue #671](https://github.com/nodejs/node/issues/671#issuecomment-73191538):
+@natduca in [nodejs/node#671_comment](https://github.com/nodejs/node/issues/671#issuecomment-73191538):
 
 I thought I'd provide some notes on the direction we're taking in chrome and indirectly v8 around tracing & stack sampling. Since we both intersect at v8 and do run into some of the same "guhh i need a tracing api but its gotta be low overhead," I hope there's some useful context!
 
@@ -37,8 +48,7 @@ Anyway, thats a lotta words. Hope it makes sense! This was all designed with cli
 
 ----
 
-@ofrobots added some additional detail in
-[issue #21](https://github.com/nodejs/tracing-wg/issues/21#issuecomment-143891637):
+@ofrobots in [nodejs/tracing-wg#21_comment](https://github.com/nodejs/tracing-wg/issues/21#issuecomment-143891637):
 
 Here are the main components, as per my understanding:
 
@@ -54,6 +64,3 @@ Basically this trace-event buffer becomes the 'single pipe' through all trace-ev
 
 Once the above linked CL lands, traces being gathered in V8 will show up there. Node.js can start putting its own traces into the same pipe. I would imagine we would also need an API in Node.js to be exposed to JavaScript to allow JS code to start sending the trace events.
 
-----
-
-If you have more info to provide, please send us a pull request!
diff --git a/tracing/vm-tracing/v8-tracing.md b/tracing/vm-tracing/v8-tracing.md
@@ -0,0 +1,39 @@
+## V8/Chromium Tracing  
+Generic macro definitions intended to be shared by many projects are in `trace_event_common.h`.  
+Source of truth for `trace_event_common.h` is in `chromium:src/base/trace_event/trace_event_common.h`.  
+This file is in Chromium but not in V8. @ofrobots copied it into `nodejs:deps/v8`. Perhaps it should be moved to `nodejs:deps/v8/include` or `nodejs:deps/v8/src/trace_event`.  
+
+Implementation-specific ("INTERNAL_*") macro definitions are in `v8:src/trace_event.h` and `chromium:src/base/trace_event.h`.  
+Chromium: `trace_event.h` macros utilize `v8::base::TraceLog::AddTraceEvent` to write traces.  
+V8: `trace_event.h` macros utilize `v8::Platform::AddTraceEvent` to write traces.  
+Node uses `DefaultPlatform` from `v8:src/libplatform/default-platform.h` which currently simply returns 0.  
+A simplistic implementation which logs all trace events to stdout illustrates the basics:
+
+```
+// from v8:src/libplatform/default-platform.cc
+// add `#include <iostream>` to includes
+
+uint64_t DefaultPlatform::AddTraceEvent(
+    char phase, const uint8_t* category_enabled_flag, const char* name,
+    /* const char* scope, */ uint64_t id, uint64_t bind_id, int num_args,
+    const char** arg_names, const uint8_t* arg_types,
+    const uint64_t* arg_values, unsigned int flags) {
+
+  time_t timestamp = std::time(nullptr);
+
+  std::cout << "phase: " << phase << std::endl;
+  std::cout << "category_enabled_flag: " << category_enabled_flag << std::endl;
+  std::cout << "name: " << name << std::endl;
+  std::cout << "id: " << id << std::endl;
+  std::cout << "epoch_timestamp: " << timestamp << std::endl;
+  std::cout << "human_timestamp: " << std::asctime(std::localtime(&timestamp)) << std::endl;
+
+  return 0;
+}
+
+const uint8_t* DefaultPlatform::GetCategoryGroupEnabled(const char* name) {
+  static uint8_t yes = CategoryGroupEnabledFlags::kEnabledForRecording_CategoryGroupEnabledFlags | 
+    CategoryGroupEnabledFlags::kEnabledForEventCallback_CategoryGroupEnabledFlags;
+  return &yes;
+}
+```