identify/manage XS size limits

[warner]

What is the Problem Being Solved?

The XS engine was originally built for embedded systems (think thermostats and toasters), so it's optimized for pretty small programs. As a result, it occasionally has size limits on surprising things, or surprisingly small size limits on normal things. We need to catalog all those limits and make sure we're comfortable with them.

Some limits will be in the "you should never need more than 640K" category: normal programs/contracts won't hit the limit, so we 1: document it, 2: provide a clear error if it gets exceeded, and 3: move on with our lives. The error will probably be vat termination, but we should provide a way to discover what limit got exceeded.

Other limits will be in the "only limited by available memory" category, which means our metering code ought to catch any programs that attempt excessive use.

We know that there's a surprisingly small size limit on the table that contains the interned strings used for property names (probably 64K, to fit in an unsigned short. This will lead to an engine failure for programs that use objects as maps and assign a large number of distinct property names. The Moddable folks tell us it'll be easy to make this be 32 bits instead of 16, which would move it into the "you shouldn't need more" category.

Most of the other table sizes are already using 32 bit indices.

This ticket is for us to think about other limits that we might need to remain aware of, and to think about how excession errors are reported.

cc @dckc

[warner]

also cc @michaelfig

[dckc]

The XS machine creation parameters are currently hard-coded.

• some, such as initialHeapCount, are not limits; the heap goes up by incrementalHeapCount on demand
• parserBufferSize is a limit and we have definitely run into cases where we had to increase it
  • I don't recall for certain what happens when it's not big enough... maybe SyntaxError due to truncation?
• keyCount is the max number of property names, I think (though today's discussion makes me wonder. Am I confused, @michaelfig ?) I don't recall whether we ran into this limit in practice.

Should parserBufferSize and keyCount be cli args and xsnap() creation options? @kriskowal I think we considered that but left it alone because YAGNI.

	xsCreation _creation = {
		16 * 1024 * 1024,	/* initialChunkSize */
		16 * 1024 * 1024,	/* incrementalChunkSize */
		1 * 1024 * 1024,	/* initialHeapCount */
		1 * 1024 * 1024,	/* incrementalHeapCount */
		4096,				/* stackCount */
		32000,				/* keyCount */
		1993,				/* nameModulo */
		127,				/* symbolModulo */
		8192 * 1024,		/* parserBufferSize */
		1993,				/* parserTableModulo */
	};

agoric-sdk/packages/xsnap/src/xsnap.c

Lines 207 to 218 in b9c1a69

	xsCreation _creation = {
	16 * 1024 * 1024, /* initialChunkSize */
	16 * 1024 * 1024, /* incrementalChunkSize */
	1 * 1024 * 1024, /* initialHeapCount */
	1 * 1024 * 1024, /* incrementalHeapCount */
	4096, /* stackCount */
	32000, /* keyCount */
	1993, /* nameModulo */
	127, /* symbolModulo */
	8192 * 1024, /* parserBufferSize */
	1993, /* parserTableModulo */
	};

[kriskowal]

Ideally, xsnap would OOM and die if we tried to pass a string too large, rather than having a hard coded limit. Practically, I think we should just make it as big as necessary for the biggest program we’ve ever needed, and bump it higher until it’s so high it becomes a problem for cases that don’t need it to be high. Then, we might need a CLI flag.

I don’t recall how xsnap died if the source was too large, but I remember it segfaults sometimes.

[dckc]

from moddable Fri, 19 Mar 2021 13:06:35 -0700:

The maximum block size is a positive 32-bit signed integer
max = 0x7FFFFFFF
Subtract the size of the block structure to get the maximum chunk size
max -= sizeof(txBlock) // 16 in 64-bit xst
XS rounds up the chunk size to a multiple of the incremental chunk size
max -= max % (16 * 1024 * 1024) // in xst
Subtract the size of the chunk structure to get the maximum data size
max -= sizeof(txChunk) // 16 in 64-bit xst
XS rounds up the data size to a multiple of 32-bit
max -= max % 4

For xst the maximum data size is 2_130_706_416. For typed arrays, the maximum length depends of course on the size of the items. For dense arrays, the maximum length depends on the size of a slot (32 in 64-bit xst). Here are the maximum lengths in 64-bit xst:

Uint8Array        2_130_706_416
Uint16Array       1_065_353_208
Uint32Array         532_676_604
BigUint64Array      266_338_302
Array                66_584_575