Parquet Writer: Make column descriptor public on the writer #3002
Conversation
parquet/src/file/writer.rs
Outdated
| @@ -473,9 +479,14 @@ impl<'a> SerializedColumnWriter<'a> { | |||
| /// optional callback to be invoked on [`Self::close`] | |||
| pub fn new( | |||
| inner: ColumnWriter<'a>, | |||
| descriptor: ColumnDescPtr, | |||
parquet/src/file/writer.rs
Outdated
| @@ -488,6 +499,11 @@ impl<'a> SerializedColumnWriter<'a> { | |||
| get_typed_column_writer_mut(&mut self.inner) | |||
| } | |||
|
|
|||
| /// Returns a clone to a [`ColumnDescPtr`] | |||
| pub fn descriptor(&self) -> ColumnDescPtr { | |||
There was a problem hiding this comment.
Returning &ColumnDescPtr would be more idiomatic so the caller can clone only if they need to
|
@tustvold Maybe it's possible to do what I'm trying to do using a different API, but currently, what I'm working on is a way for me to aggregate events that comes in from a different thread, and package data into a parquet file using the SerializedColumnWriter. What this means, is that I have an in-memory cache that I'm holding, and then constructing the parquet file (row group) when it reaches a certain size. Since the ColumnDescriptor is not really available, as far as I can tell, I have to basically create an index myself to figure out which column I am referring to when I call If I had the possibility to gather some kind of information from the columnWriter, it would help understand the context I'm in and avoid some bookkeeping on my side. Does that make sense? I apologize for the breaking change, I guess I misinterpreted what "User-facing changes" meant. Edit: It's possible that adding a getter to the |
Makes sense 👍
Sounds good to me |
This is so that it's possible to gather information from the column we're about to write to. That information was already present in the column but not publicly available. The getter returns a reference of `ColumnDescPtr`.
|
Benchmark runs are scheduled for baseline = 01ea8c7 and contender = 8ca4e65. 8ca4e65 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
There's no issue currently, because I just had this issue this morning and decided to make a PR for it. I can open up an issue if it helps.
Rationale for this change
This is so that it's possible to gather information from the column we're about to write to. That information was already present in the column but buried inside the internal of the GenericColumnWriter.
The getter returns a reference of
ColumnDescPtr.What changes are included in this PR?
The PR is missing tests, I wasn't sure if the test would add any value/safety. I'm more than happy to provide some tests if you think it's important.
Are there any user-facing changes?
None, I believe.