node-heapdump captures <1% of the heap

[kevinburke]

Hi, I'm running a Node process in a Docker container; it's the only process running in the container.

I observe via docker stats that the container is using 1.77GiB of memory. The memory usage is pretty constantly this high; it's not growing.

CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O               BLOCK I/O           PIDS
9f53cc02a2e5        21.03%              1.774 GiB / 3.275 GiB   54.15%              1.846 GB / 1.865 GB   1.211 GB / 0 B      10

I set it up to take heap snapshots by requiring heapdump when the process starts and then sending a USR2 signal to the process (via docker kill --signal=USR2). I haven't configured the writeSnapshot function to do anything - I just load the heapdump tool by requiring it in the main javascript file that runs.

When I do this, for some reason two heap snapshot files get generated:

-rw-r--r--    1 root     root       10.7M Jan 26 21:08 heapdump-170516916.755571.heapsnapshot
-rw-r--r--    1 root     root       10.7M Jan 26 21:08 heapdump-170517368.954191.heapsnapshot

They seem to have close to identical contents.

When I open up the heap snapshot in Google Chrome, it reports 18.3MB of memory usage. Which is less than 1% of the 1.7GiB usage reported by "docker stats."

I'm running Node 6.9.2. The application pulls data from NSQ, transforms it, and makes HTTP requests to two or three domains in total.

• Is it possible for node-heapdump to miss that much of the heap, or is there some other problem with the reported data?
• Any idea why it is leaving two files? (It's possible I am calling require('heapdump') twice in my code base).
• Is it possible for a partial snapshot to have been written? E.g. if the heap snapshot failed halfway through, would I see a percentage of the heap snapshot, or would the entire operation fail?

[bnoordhuis]

Is it possible for node-heapdump to miss that much of the heap, or is there some other problem with the reported data?

node-heapdump can only report about objects from the JS heap. Memory allocations from native modules (C++ add-ons) won't normally be visible in heapdumps, not unless the add-on makes them visible; most don't.

It could also be memory fragmentation, see nodejs/node#8871 (comment) for an example; that won't be visible in a heapdump either.

Any idea why it is leaving two files? (It's possible I am calling require('heapdump') twice in my code base).

No idea, sorry.

Is it possible for a partial snapshot to have been written? E.g. if the heap snapshot failed halfway through, would I see a percentage of the heap snapshot, or would the entire operation fail?

Partial snapshots can happen but it's something you can check. A heapdump is just a big JSON object. Inspect in a text editor and see if it's a complete object.

I think Chrome accepts partial snapshots but I'm not 100% sure about that.

[kevinburke]

It turns out this was related to transparent huge pages! So yes the memory usage reported by node-heapdump was correct.

It might be good to put something in the README explaining what could be the cause if the snapshot doesn't account for everything.