[C++][Parquet] Write parquet page index

[wgtmac]

Describe the enhancement requested

Parquet C++ reader has supported reading page index from file. Now it is time to implement the write logic.

Component(s)

C++, Parquet

[XinyuZeng]

Just curious, would page index optimization be added to the Arrow interface in the long term after the low level reader/writer are finished? I'd expect that also requires change the I/O unit from row group to page.

[mapleFU]

Just curious, would page index optimization be added to the Arrow interface in the long term after the low level reader/writer are finished? I'd expect that also requires change the I/O unit from row group to page.

Sounds ok, but seems it requires high performance IO-merging and requires some benchmarks/testing

[XinyuZeng]

Just curious, would page index optimization be added to the Arrow interface in the long term after the low level reader/writer are finished? I'd expect that also requires change the I/O unit from row group to page.

Sounds ok, but seems it requires high performance IO-merging and requires some benchmarks/testing

There is already IO coalesce, but its range unit is ColumnChunk .

arrow/cpp/src/arrow/io/caching.cc

Line 175 in 39bad54

ranges = internal::CoalesceReadRanges(std::move(ranges), options.hole_size_limit,

Perhaps it is not necessary to breakdown the IO to page, since Parquet-format states ColumnChunk is the IO unit.

[mapleFU]

Perhaps it is not necessary to breakdown the IO to page, since Parquet-format states ColumnChunk is the IO unit.

Yes, although the standard says so, but we can use it. Currently parquet-cpp implemention use both Page-IO and Chunk-IO:

• arrow can use ReadRangeCache to serving chunk-level io, and, however, I don't think it provides an good performance. And currently, it will read whole buffer, and caching them in a ::arrow::io::BufferReader
• If no cache is used, an ArrowInputStream would be created directly on input, and PageReader will try do create read buffer page-by-page

Besides, I think currently the implemention of ReadRangeCache is naive. I'm not sure it will works well.

[wgtmac]

Page index is a pretty new concept in the history of parquet specs. In the old days when page index is unavailable, the I/O unit is always column chunk because we can only drop row groups based on the column statistics. Once page index is supported, finer-grained I/O unit is a must if we want to leverage page index to efficiently skip specific pages. @XinyuZeng