User Experience Punchlist

[danking]

In Progress:

Short term:

• Teach PyArray __getitem__ which just delegates to scalar_at.
• Teach PyVortex to use parallelism during decompression.
• Make reading a Vortex file into an Arrow Array as fast as Parquet (at least partly needs to address read/write disagreement on chunk size, see below we side stepped this by removing buffer sizes when implementing filter pushdown).
• Make reading a Vortex file with Polars as fast as Parquet. Reading a Vortex array in Polars is slower than Parquet #1071
• read should have high throughput on files written by write (currently, write does not enforce chunking whereas read does which can degrade throughput on arrays for which slice is not free again, sidestepped by removing buffer sizing from filter pushdown).
• (docs) Write a comparison section which describes similarities and differences from other file formats.
• Expose Vortex compute methods in Python API by way of new classes (e.g. Array.as_struct() and StructArray which permits column selection).
• Teach Vortex IsNull & IsNotNull and plumb substrait into them.
• Do not expose modules with confusing names such as vortex.encoding.
• Expose more functions on scalar values such as __eq__, array indexing, or getting a memoryview.
• Teach PyVortex (really: Layout readers and writers) to read/write non-struct arrays.
• Teach RecordBatchReader to read from multiple files.
• Teach Polars to write Vortex files.
• Teach DuckDB to write Vortex files.
• Support multiple files and/or directories in the Vortex Dataset API
• Consider using the Rufo theme for the docs

Long term:

• For Torch, expose a method to read from a Vortex file directly into a mutable NumPy array. Torch does not support immutable NumPy arrays.
• Reduce Vortex array metadata size. This primarily benefits very small datasets (e.g. PBI AirlineSentiment).
• Implement a RecordBatchReader for Vortex arrays and Vortex files.
• Implement a Pandas ExtensionArray which permits compute on the compressed array representations (thus avoiding the cost of decompression).
• Integrate with DuckDB.
• (docs) Finish specification and move into docs.

Complete:

• Pushdown row filters Filter pushdown over layouts #1124 .
• (docs) Write a Vortex quick start which demonstrates read, write, compress, and integration with pandas, polars, arrow, and duckdb. feat: expanded Python docs + Rust docs #1137
• (docs) Move current documentation into an API Reference section. feat: expanded Python docs + Rust docs #1137
• (docs) Write (or gather from Notion & Docs) an enduser-friendly explanation of the Vortex file format. feat: expanded Python docs + Rust docs #1137
• write should either warn or compress by default so that naive users understand why write(vortex.array(...), ...) gives no benefit. feat: expanded Python docs + Rust docs #1137
• Teach PyVortex to read from Object Storage. feat: teach PyVortex to read from object storage (attempt two) #1151
• Teach PyArray scalar_at (requested by Rob) feat: teach PyArray scalar_at #1095.
• Disable or fix RoaringIntArray and RoaringBoolArray to be no-copy. Vortex should support zero-copy roaring int and roaring bool array construction #1075, fix: disable roaring compressors #1076
• Remove Vortex "cannot find compressor" log statements when calling compress. feat: vortex.dataset.Dataset: deep integration with Polars & DuckDB #1089
• Implement the Arrow Dataset API feat: vortex.dataset.Dataset: deep integration with Polars & DuckDB #1089