feat: support reading and writingStringView and BinaryView in parquet (part 2)

[ariesdevil]

Which issue does this PR close?

Closes #5530

Rationale for this change

Use view_buffer instead of offset_buffer for view type parquet reader

What changes are included in this PR?

Use view_buffer instead of offset_buffer for view type parquet reader

Are there any user-facing changes?

Yes

[ariesdevil]

The parquet build for wasm32 check always failed, but it works on my machine...

[ariesdevil]

Hi @alamb @tustvold , this PR is not really ready for review, however, I may need some advice, and then I'll continue.

[alamb]

Thanks @ariesdevil -- I think @tustvold is away for a few days. I will try and give this PR a look over the next day or two. Very exciting!

cc @XiangpengHao

[alamb]

Thank you @ariesdevil -- I think this is looking close to something we can merge 🙏

To merge this, I think we need:

1. Benchmarks for reading/writing StringView / BinaryViews
2. Cover a few more cases for round tripping

I think there is quite a bit more performance to be had as well by avoiding string copies. This PR will copy the data I think twice (once out of parquet buffer and once out of the offset buffer).

It would be nice to avoid at least one of those copies prior to merge, but I also think once we have the test coverage it would be ok to merge this PR and then do additional optimizations as follow on PRs

We can add benchmarks here:

• reading

  arrow-rs/parquet/benches/arrow_reader.rs

  Lines 867 to 868 in 7e134f4

  	// string benchmarks
  	//==============================

• writing:

  arrow-rs/parquet/benches/arrow_writer.rs

  Lines 382 to 392 in 7e134f4

  	let batch = create_string_bench_batch(4096, 0.25, 0.75).unwrap();
  	group.throughput(Throughput::Bytes(
  	batch
  	.columns()
  	.iter()
  	.map(|f| f.get_array_memory_size() as u64)
  	.sum(),
  	));
  	group.bench_function("4096 values string", |b| {
  	b.iter(|| write_batch(&batch).unwrap())
  	});

End to end round trip tests:

[alamb]

This is really nice step forward -- thank you @ariesdevil

[alamb]

Thank you @ariesdevil -- I think this PR looks like a really nice incremental step.

As I think we have identified, it can be significantly improved performance wise but having the basic tests and benchmarks in place we are in a good space to optimize it.

I left some other smaller style / documentation comments that might be nice to cleanup, but I don't think they are strictly required

Let's wait another day for @tustvold to see if he wants to review (i think he is still not back yet) but otherwise we'll merge this PR and add additional features as a follow on PR

[alamb]

I plan to give this another review today

[alamb]

The integration test https://github.com/apache/arrow-rs/actions/runs/8553472637/job/23436722309?pr=5557 seems to have failed for reasons unrelated to this PR. I have restarted it

################# FAILURES #################
FAILED TEST: union Rust producing,  Rust consuming
<class 'RuntimeError'>: Command failed: /build/rust/debug/flight-test-integration-client --host localhost --port=34799 --path /tmp/arrow-integration-92iwbbih/generated_union.json
With output:
--------------
Error: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })))

[alamb]

I am sorry @ariesdevil -- I need more time and a fresh set of eyes to review the new implementation in this PR. I will find time over the next few days.

[ariesdevil]

Hi @alamb ,the new implementation has indeed changed a lot. It should have been implemented in another PR, but the original performance is really poor.

Thanks for your patience and have a nice weekend.

[alamb]

• Validate utf8 data (I left comments). I think we need a test for that too -- I'll make a PR to make that easier to add.

I wrote new tests in #5639

[alamb]

I am on vacation this week but I hope to find some time later today or tomorrow to give this another look. Thanks @ariesdevil

[ariesdevil]

No need to rush, enjoy your vacation, just so I can have time to rethink how to continue optimization 😃

[alamb]

I see some new updates -- I plan to look at this PR later this week (probably won't be until Wednesday)

[XiangpengHao]

Hi @ariesdevil, thanks for the great work! I'm looking at this PR these days and trying to push it forward. However, I do feel the code change is too large for a single PR that it is quite difficult to understand every detail of the implementation.

I suggest we break the PR into even smaller pieces and incrementally add the features/optimizations so it poses less mental burdens on the reviewers, what do you think?

I'll try to consolidate a few low-hanging fruit changes and try to get them merged first; let me know your thoughts!

[ariesdevil]

Hi @XiangpengHao, thanks for your reply. I don't think this PR is very large, the previous reviewers are already very familiar with this PR, this PR has been submitted for almost three months, and the reviewer has had enough time to get familiar with it.

[alamb]

Hi @XiangpengHao, thanks for your reply. I don't think this PR is very large, the previous reviewers are already very familiar with this PR, this PR has been submitted for almost three months, and the reviewer has had enough time to get familiar with it.

Indeed -- I am sorry this PR is stalled @ariesdevil -- it has been very hard for me to find enough contiguous time to get familar enough with this PR and really understand it (and ensure the test coverage is adequate, etc).

@ariesdevil do you think we should attempt to merge this entire PR at once? I do think it would help speed up review if we could break it into some smaller PRs

[ariesdevil]

Hi @alamb, I understand how busy you must be maintaining multiple projects and writing an important paper simultaneously.

As we now have try_append_view implemented, we no need ViewBuffer any more, I'll rebase this PR after #5877 merged. After that, this PR should be smaller.

[alamb]

Hi @alamb, I understand how busy you must be maintaining multiple projects and writing an important paper simultaneously.

Thank you for understanding -- I feel bad whenever PRs get stuck.

As we now have try_append_view implemented, we no need ViewBuffer any more, I'll rebase this PR after #5877 merged. After that, this PR should be smaller.

That is awesome. THank you. I will make every effort to help get this over the line and now that @XiangpengHao is helping I am optimistic we can get it in shortly

[alamb]

#5877 is merged now!

[ariesdevil]

closed by #5877