feat(python): Set smaller defaults on row group and file size

[wjones127]

Description

Arrow defaults max_rows_per_group to 1024 * 1024, but in many instances this will lead to files only having one row group, limiting parallelism in future reads. So I set the max to 128 * 1024, close to what DuckDB defaults to (100,000 I think).

I also set a 10 million row cap on file size, so that we also split across files. This is helpful for other readers like Spark, which IIRC use files not row groups as their level of parallelism.

Related Issue(s)

• closes delta write should generate a sensible max_rows_per_group defaults #816

Documentation

Mytherin/duckdb@3b8ad03#diff-a95d5e017c81184e18f0f04c5df3b72061fd80555d581a4ce163af5deca3dac0R394

[roeap]

Eventually we should probably try and consider the targetFileSize setting on the delta table if it exists, but I think we need to go through the settings handling once more before doing this.