Document arrow-rs architecture and structure

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The underlying implementation of arrow arrays has changed significantly due to the work in #3880

There are now several important classes such as ScalarBuffer, NullBuffer, Buffer PrimitiveArray, etc that underlying the arrays in addition to the "classic" ArrayData. In addition after #3879 is complete I believe these types will be more publicly exposed through the various Arrow APIs.

Describe the solution you'd like
I would like documentation / diagrams / something that briefly explains the key structures ad how they are related to each other.

Perhaps we can take inspiration (or copy/modify) the wonderful guide that @jorgecarleitao wrote for arrow2: https://jorgecarleitao.github.io/arrow2/main/guide/

Describe alternatives you've considered

Additional context
See details on #4061 (comment)

[alamb]

cc @tustvold

[xxchan]

It would be helpful to have such a doc 👀. Specifically I feel a little confused about "buffer", "data" (is it still useful after recent refactoring?) and "array", and not sure what to use. arrow-2's description about "low-level"/"high-level" API is easier to understantd 😄 .

[tustvold]

https://docs.rs/arrow/latest/arrow/#columnar-format and by extension the linked https://docs.rs/arrow-array/40.0.0/arrow_array/index.html I believe is such a doc, but please let me know if anything isn't clear or could do with additional clarification

[xxchan]

Thanks. The docs are indeed very helpful! However, my user journey is like:

Wow, there are so many sub-crates! And ... what on earth do they mean? (So I didn't browse the introductory content below at the beginning..)

Well, let me go to arrow-buffer / arrow-data and check it. Oops, there aren't many useful infomation either.. What is "Buffer abstraction"?

So it seems good doc exists, but are not accessible enough.

Ways to improve I can come up with now:

• Better "introduction" section in crate-level docs.
  • root arrow crate can group sub-crates (e.g., low-level/high-level/compute kernal, like the arrow2's doc) and add more description, instead of simply enumerating them.
  • sub-crate may need some more explanations (at least mention that refer to some struct's doc for more details)
• It seems high-level API (Array) explains low-level API (Buffer), but not the reverse. Maybe a backlink can be added to explain the relationship.

[tustvold]

Thank you that is very useful feedback, I'll try to get something to address this up in the coming days 👍