Arbitrary size concat elements utf8

[ismailmaj]

Which issue does this PR close?

Closes #1748.

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Should be merged after #1780.

[codecov-commenter]

Codecov Report

Merging #1787 (759d4b5) into master (9e8c1bf) will increase coverage by 0.00%.
The diff coverage is 98.11%.

@@           Coverage Diff           @@
##           master    #1787   +/-   ##
=======================================
  Coverage   83.47%   83.48%           
=======================================
  Files         221      221           
  Lines       57042    57094   +52     
=======================================
+ Hits        47616    47665   +49     
- Misses       9426     9429    +3     

Impacted Files	Coverage Δ
arrow/src/compute/kernels/concat_elements.rs	99.17% <98.11%> (-0.83%)	⬇️
arrow/src/datatypes/datatype.rs	65.42% <0.00%> (-0.38%)	⬇️
parquet_derive/src/parquet_field.rs	65.75% <0.00%> (-0.23%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9e8c1bf...759d4b5. Read the comment docs.

[tustvold]

I would be very interested to see some benchmarks of this, this has introduced a number of additional memory allocations and a significantly more complicated loop body, it may not matter, but it would be good to verify either way...

[ismailmaj]

Thank you for the review @tustvold!
I'll work on a benchmark, I'm wondering if we could use smallvec to avoid heap allocating with only a few arrays as input.

[alamb]

FYI I think @tustvold is out this week

Another way to move this PR forward might be to make this a new kernel (rather than modifying the existing concat_elements_utf8), perhaps something like concat_n_elements_utf8 or concat_elements_utf8_many?

That way we aren't worried about a potential regression in performance and people can optimize the new kernel in the future if that is desired

[HaoYang670]

I'll work on a benchmark.

@Ismail-Maj This could be done as a follow-up PR.

[tustvold]

Sorry for the delay, thank you 👍