@node-rs/crc32: rss memory leak

[twoeths]

The below code snippet can reproduce the memory leak issue

const nodeRs = require("@node-rs/crc32");
const crc32c = nodeRs.crc32c;

const bytes = Buffer.alloc(1000);


async function memTest() {
  const count = 1_000_000_000;
  for (let i = 0; i < count; i++) {
    crc32c(bytes);
    if (i % 100_000 === 0) {
      await new Promise((resolve) => setTimeout(resolve, 100));
      const {heapTotal, rss} = process.memoryUsage();
      console.log(
        "Memory usage",
        Math.floor((i * 100) / count) + "%",
        "heapTotal",
        toMem(heapTotal),
        "rss",
        toMem(rss)
      );
    }
  }
}

function toMem(n) {
  const bytes = Math.abs(n);
  const sign = n > 0 ? "+" : "-";
  if (bytes < 1e6) return sign + Math.floor(bytes / 10) / 100 + " KB";

  if (bytes < 1e9) return sign + Math.floor(bytes / 1e4) / 100 + " MB";

  return sign + Math.floor(bytes / 1e7) / 100 + " GB";
}

console.log("Start node-rs crc32c mem test");

memTest().then(() => console.log("Done"));

Heap memory is kept stable but rss keeps increasing, some sample outputs:

Memory usage 1% heapTotal +7.63 MB rss +2.27 GB
Memory usage 1% heapTotal +7.63 MB rss +2.25 GB
Memory usage 1% heapTotal +7.63 MB rss +2.23 GB

This happens even with the latest version (1.5.1)

Found it from ChainSafe/node-snappy-stream#16

[Brooooooklyn]

@tuyennhv these are two separate issues, memory leak could be resolved by upgrade to latest NAPI-RS: napi-rs/napi-rs#1331, I can confirm it was fixed in local.

The second is globally declared Buffer would cause memory leak too in your code snippets, it's the expected Node.js behavior: nodejs/node#39915

the memory goes to normal if I move the const bytes = Buffer.alloc(1000) into the loop:

- const bytes = Buffer.alloc(1000);


async function memTest() {
  const count = 1_000_000_000;
  for (let i = 0; i < count; i++) {
+   const bytes = Buffer.alloc(1000);
    crc32c(bytes);
    if (i % 100_000 === 0) {
      await new Promise((resolve) => setTimeout(resolve, 100));
      const {heapTotal, rss} = process.memoryUsage();
      console.log(
        "Memory usage",
        Math.floor((i * 100) / count) + "%",
        "heapTotal",
        toMem(heapTotal),
        "rss",
        toMem(rss)
      );
    }
  }
}

[twoeths]

@Brooooooklyn thanks for your quick response and investigation, looking forward to seeing your new release!

[BlackGlory]

I've noticed memory leaks on @node-rs/xxhash too, when will a new version be released?

[Brooooooklyn]

Publishing

[Brooooooklyn]

Published

[twoeths]

thanks @Brooooooklyn 🙏