Load contents of source files only when needed

[matthijskooijman]

When compiling a big sketch, I noticed that arduino-builder took up a lot of RAM (over 1G, sometimes it even went over 1.7G and crashed with an out of memory error).

Investigating, it turns out that this is because arduino-builder and arduino-cli load all source file contents into memory when loading the sketch. In practice, only the .ino file contents are actually used, so the rest is just dead weight. For sketches that have a lot of extra baggage (we have libraries, a core and some host-system tools in the sketch directory), this can take up a lot of unneeded space.

This PR modifies sketch.Item to remove the Source string, which is then no longer filled at sketch load time. Instead, the GetSourceStr() method loads the contents on demand (which is really only when merging the sketch files).

This PR moves around some error handling (since I/O operations have moved), but I've managed to split most of the trivial code movements into separate commits.

To reproduce this problem, you can simply create an empty sketch and add some baggage (https://github.com/vancegroup/arduino-boost adds some 300MB of source files and is good for testing):

matthijs@grubby:~$ mkdir sketch
matthijs@grubby:~$ cd sketch/
matthijs@grubby:~/sketch$ echo "void setup() { } void loop() { }" > sketch.ino
matthijs@grubby:~/sketch$ git clone https://github.com/vancegroup/arduino-boost.git                                                                           
[ ... snip output ... ]
matthijs@grubby:~/sketch$ /usr/bin/time -f '%MkB' arduino-cli compile -b arduino:avr:uno .                                                                    
Sketch uses 444 bytes (1%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.
379120kB

This uses time to show that the maximum resident memory size is nearly 400MB.

With the patch applied, this is reduced to 70MB:

matthijs@grubby:~/sketch$ /usr/bin/time -f '%MkB' arduino-cli compile -b arduino:avr:uno .                                                                    
Sketch uses 444 bytes (1%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.                                           
71176kB

This still seems rather much. When running arduino-builder on the same sketch, the memory usage drops from 375MB to 37MB, so it seems that arduino-builder is a bit more efficient even (and on our much bigger production sketch things improved from 1.1G to just 27M, which is unexpectedly even less). But I guess this is something for another time.

[rsora]

Thanks @matthijskooijman !
This is, with no doubts, high quality contributing 😄

[matthijskooijman]

Thanks for merging and the compliment :-)