Investigate whether we can postpone all raster interaction to threedi-tables

[caspervdw]

We currently read the complete raster in memory here in threedigrid-builder. This effectively limits the size of the model you can make to about 50000x50000 pixels (50x50 km with a 1m resolution DEM). Also it is the main performance bottleneck in threedigrid-builder.

Can we make the grid based only on a model area (a multipolygon)? What info are we missing if we don't have the DEM? I think it is only these:

1. We can't say what cells and crosssections are active
2. We can't align the cells precisely on the raster

Presumeably, the cells inside the model area are are mostly active. There may be holes in the DEM, but I think in practice >90% of the area will be active. Does it matter, performance wise, if we have a bunch of inactive nodes and lines in the calculation core?

The precise cell alignment is not so important. A DEM is always an arbitrary discretization of the real measurements (point cloud, mostly) which also have an uncertainty. The exact location of a pixel has no real meaning. So whether your cell is exactly on a pixel or not, it does not matter. Interpolating on an already interpolated dataset does loose some accuracy, but my feeling is that this is negligible if there are >10 pixels in your cell. For the crosssections this is a bit harder, but again, the DEM disretization is arbitrary to start with, so picking "1 pixel on one side and 1 on the other" is as sensible as "1 meter on one side and 1 on the other".

I already discussed this with @martijn-siemerink , I think it is good to follow up.

[jpprins1]

In any case I think it's good to set a hard limit for the maximum amount of memory usage allowed during grid building, I would propose 5 GB.

In the current situation this means a limit of 5 GB = 2.684.354.560 pixels, if DEM is square around (51810, 51810).

Note: because this limit is not in place, laptops can easily crash when generation grid's in the MI.