Improve memory usage for large files

[alexdima]

e.g.

• open the file and wait for ~ 30s

---

1. file A

Download Loan Data for 2016 Q3 from https://www.lendingclub.com/info/download-data.action

	empty file	73 MB 99127 lines	actual usage	overhead	overhead per line
OSX RESIDENT	190 MB	353 MB	163 MB	90 MB	952.0 B
OSX Heap Snapshot	50 MB	138 MB	88 MB	15 MB	158.6 B

---

2. file B

Download checker.ts from https://github.com/Microsoft/TypeScript/blob/296660a2a077c32a2ed41cb762ef530031e56417/src/compiler/checker.ts

Generate a 128x times checker:

var fs = require('fs');
var str = fs.readFileSync('checker.ts').toString();
for (var i = 0; i < 7; i++) {
    str = str + '\n' + str;
}
fs.writeFileSync('checker-out.ts', str);

	empty file	177 MB 3171200 lines	actual usage	overhead	overhead per line
OSX RESIDENT	190 MB	1474 MB	1284 MB	1107 MB	366.0 B
OSX Heap Snapshot	50 MB	716 MB	666 MB	489 MB	161.7 B

---

• OSX shallow size taken by a ModelLine (via Heap Snapshot) - 72 bytes

[alexdima]

Removing ModelLine._lineNumber

• OSX shallow size taken by a ModelLine (via Heap Snapshot) - 64 bytes

1. file A

	empty file	73 MB 99127 lines	actual usage	overhead	overhead per line
OSX RESIDENT	190 MB	348 MB	158 MB	85 MB	899.1 B
OSX Heap Snapshot	50 MB	136 MB	86 MB	13 MB	137.51 B

2. file B

	empty file	177 MB 3171200 lines	actual usage	overhead	overhead per line
OSX RESIDENT	190 MB	1194 MB	1004 MB	827 MB	273.45 B
OSX Heap Snapshot	50 MB	692 MB	642 MB	465 MB	153.75 B

[alexdima]

Using MinimalModelLine

• OSX shallow size taken by a MinimalModelLine (via Heap Snapshot) - 40 bytes

1. file A

	empty file	73 MB 99127 lines	actual usage	overhead	overhead per line
OSX RESIDENT	190 MB	347 MB	157 MB	84 MB	888.6 B
OSX Heap Snapshot	50 MB	133 MB	83 MB	10 MB	105.8 B

2. file B

	empty file	177 MB 3171200 lines	actual usage	overhead	overhead per line
OSX RESIDENT	190 MB	1078 MB	888 MB	711 MB	235.1 B
OSX Heap Snapshot	50 MB	577 MB	527 MB	350 MB	115.7 B

[alexdima]

Using MinimalModelLine with lazy markers

• OSX shallow size taken by a MinimalModelLine (via Heap Snapshot) - 40 bytes or 32 bytes

1. file A

	empty file	73 MB 99127 lines	actual usage	overhead	overhead per line
OSX RESIDENT	190 MB	348 MB	158 MB	85 MB	899.1 B
OSX Heap Snapshot	50 MB	132 MB	82 MB	9 MB	95.2 B

2. file B

	empty file	177 MB 3171200 lines	actual usage	overhead	overhead per line
OSX RESIDENT	190 MB	991 MB	801 MB	624 MB	206.3 B
OSX Heap Snapshot	50 MB	553 MB	503 MB	326 MB	107.8 B

[alexdima]

Fixing the array leak of #30189

• OSX shallow size taken by a MinimalModelLine (via Heap Snapshot) - 40 bytes or 32 bytes

1. file A

	empty file	73 MB 99127 lines	actual usage	overhead	overhead per line
OSX RESIDENT	190 MB	344 MB	154 MB	81 MB	856.8 B
OSX Heap Snapshot	50 MB	132 MB	82 MB	9 MB	95.2 B

2. file B

	empty file	177 MB 3171200 lines	actual usage	overhead	overhead per line
OSX RESIDENT	190 MB	971 MB	781 MB	604 MB	199.71 B
OSX Heap Snapshot	50 MB	519 MB	469 MB	292 MB	96.55 B

[alexdima]

Using IdentityLinesCollection

• OSX shallow size taken by a MinimalModelLine (via Heap Snapshot) - 40 bytes or 32 bytes

1. file A

	empty file	73 MB 99127 lines	actual usage	overhead	overhead per line
OSX RESIDENT	190 MB	344 MB	154 MB	81 MB	856.8 B
OSX Heap Snapshot	50 MB	129 MB	79 MB	6 MB	63.5 B

2. file B

	empty file	177 MB 3171200 lines	actual usage	overhead	overhead per line
OSX RESIDENT	190 MB	901 MB	711 MB	534 MB	176.5 B
OSX Heap Snapshot	50 MB	461 MB	411 MB	234 MB	77.4 B

[alexdima]

Using a hack to avoid (sliced string)

• OSX shallow size taken by a MinimalModelLine (via Heap Snapshot) - 40 bytes or 32 bytes

1. file A

	empty file	73 MB 99127 lines	actual usage	overhead	overhead per line
OSX RESIDENT	190 MB	341 MB	151 MB	78 MB	825.1 B
OSX Heap Snapshot	50 MB	127 MB	77 MB	4 MB	42.31 B

2. file B

	empty file	177 MB 3171200 lines	actual usage	overhead	overhead per line
OSX RESIDENT	190 MB	928 MB	738 MB	561 MB	185.5 B
OSX Heap Snapshot	50 MB	425 MB	375 MB	198 MB	65.5 B

[alexdima]

Conclusions

The resident memory usage is ... random, so looking at the Heap Snapshot (which is something that makes sense):

Edit: turns out the resident memory usage measurements were all skewed by the web worker receiving these large files. The web worker no longer receives such large files.

model memory = c0 + file size + (x + 32) * lineCount , where 32 comes from our allocating a MinimalModelLine per line and x depends on the line length (i.e. the overhead of a string object around a character array). x is smaller for long lines, and larger for short lines; it is around 12 - 36 bytes.

	file A	gain	file B	gain
initial	88 MB	-	666 MB	-
_lineNumber	86 MB	2.27%	642 MB	3.60%
MinimalModelLine	83 MB	3.49%	527 MB	17.91%
lazy markers	82 MB	1.2%	503 MB	4.55%
leak #30189	82 MB	0%	469 MB	6.76%
IdentityLinesCollection	79 MB	3.66%	411 MB	12.37%
(sliced string) hack	77 MB	2.53%	375 MB	8.76%

Totals - file A - 73 MB - 99,127 lines

OSX resident memory usage for the file: 163 MB -> 87 MB : 46.6% less
Heap total memory usage for the file: 88 MB -> 77 MB : 12.5% less
Heap overhead over file size: 15 MB -> 4 MB : 73.3% less

Totals - file B - 177 MB - 3,171,200 lines

OSX resident memory usage for the file: 1284 MB -> 340 MB : 73.5% less
Heap total memory usage for the file: 666 MB -> 375 MB : 43.7% less
Heap overhead over file size: 489 MB -> 198 MB : 59.5% less

---

Final test on win 32 bits

File	Private Bytes	Working Set	Heap (via Timeline)
empty.txt	107 MB	147 MB	33 MB
file A - 73 MB - 99,127 lines	165 MB	219 MB	105 MB
file B - 177 MB - 3,171,200 lines	361 MB	415 MB	336 MB
file 2*B - 351 MB - 6,342,399 lines	627 MB	679 MB	561 MB
file 4*B - 700 MB - 12MM lines	failed	failed	failed

Final test on win 64 bits

File	Private Bytes	Working Set	Heap (via Timeline)
empty.txt	128 MB	201 MB	56.5 MB
file A - 73 MB - 99,127 lines	225 MB	268 MB	125 MB
file 2*A - 139 MB - 198,253 lines	266 MB	287 MB	201 MB
file 4*A - 278 MB - 396,505 lines	418 MB	435 MB	352 MB
file 8*A - 557 MB - 793,009 lines	737 MB	749 MB	658 MB
file 16*A - 1,115 MB - 1,586,017 lines	1,365 MB	1,367 MB	1,264 MB
file 32*A - 2,230 MB - x lines	failed	failed	failed
file B - 177 MB - 3,171,200 lines	612 MB	688 MB	414 MB
file 2*B - 351 MB - 6,342,399 lines	856 MB	919 MB	776 MB
file 4*B - 700 MB - 12MM lines	failed	failed	failed

---

It's hard to say what our limits should be based on the above results, since failure is not correlated with file size, nor with line count, but a function of the two. I suggest:

• we refuse to open files above 300 MB on win 32 bit
• we open any file size everywhere else

[Chillee]

This seems like a pretty big improvement! I'm surprised to not see it on the iteration plan.

[egamma]

@Chillee it is on the July iteration plan: #30209