Decoding of serialized object fails when there are about 70,000 keys or more

[rotemdan]

System info

• msgpackr version: 1.11.0
• node version: v22.9.0
• OS: Windows 11 x64

Reduced test case

import { encode, decode } from 'msgpackr'

export function test() {
	let obj: any = {}

	for (let i = 0; i < 70000; i++) {
		obj['x' + i] = true
	}

	const encoded = encode(obj)

	const decoded = decode(encoded)

	console.log(decoded)
}

test()

decode(encoded) fails with error:

Error: Data read, but end of buffer not reached 
{"x0":true,"x1":true,"x2":true,"x3":true,"x4":true,"x5":true,"x6":true,"x7":true,"x8":true,"x9":true

If reducing loop to 60,000 iterations (object keys), there is no failure on decoding, meaning the issue is likely related to the size of the object.

[kriszyp]

As stated in the docs: https://github.com/kriszyp/msgpackr?tab=readme-ov-file#options, if you want more than 64K of properties, you need to use the variableMapSize option.

[rotemdan]

Thanks. I assumed that the default options work with any input. I was testing several libraries so I assumed this one behaves the same.

Other libraries I tried, like msgpack-lite, don't require special options for particular inputs. I don't think having more than 64,000 keys is that unusual (especially for the kind of data that is serialized to disk).

The error message looks more like a failure then a settings issue. Maybe the error message can suggest to set the option instead?

[kriszyp]

Good suggestion, thank you.