[c++] Arrow utils with current-domain option

[johnkerl]

Issue and/or context: As tracked on issue #2407 / [sc-51048].
The intended Python and R API changes are all agreed on and finalized as described in #2407.

Changes:

• create_arrow_schema and create_index_column_info take flags for status quo (use_current_domain=false) as well as new-shape (use_current_domain=true -- [python/r/c++] Revisit shape for component arrays #2407) . These flags will allow for a reasoned, well-tested transition to new-shape functionality at the R and Python levels, implemented on upcoming PRs.
• For knowledge management, I'll introduce some comment blocks about how these two methods work. They're an ingenious technique for sharing schema information in a language-neutral way between R/C++ and Python/C++.
• Unit-test support on this PR is mainly confined to demonstrating that no backward compatibility is broken: the Python and R APIs continue to work as they have before.
• Unit-test cases at the libtiledbsoma level will validate correctness of the libtiledbsoma-to-core contracts, thereby increasing confidence of the correctness of the libtiledbsoma-level implementation.
• Further unit-test expansion on subsequent PRs will introduce the resize logic to libtiledbsoma, and unit-test its correctness. However, this PR is already more than big enough as-is.

Notes for Reviewer:

As of 2024-08-18, this is stacked on top of #2910. #2910 should be approved and merged first, then, this should be rebased on the new main. (What should not happen is for this PR to be merged into #2910.)

As of 2024-08-18 this incorporates #2915 without being stacked on top of it. Here, too, #2915 should be approved and merged first, then, this should be rebased on the new main.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.02%. Comparing base (08193f2) to head (3517a1e).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2911      +/-   ##
==========================================
+ Coverage   89.87%   90.02%   +0.15%     
==========================================
  Files          38       38              
  Lines        3999     3999              
==========================================
+ Hits         3594     3600       +6     
+ Misses        405      399       -6     

Flag	Coverage Δ
python	90.02% <ø> (+0.15%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
python_api	90.02% <ø> (+0.15%)	⬆️
libtiledbsoma	∅ <ø> (∅)

[eddelbuettel]

A more general comment orthogonal to this PR: we really should think about making more use of nanoarrow (as I have) to remove most if not all of this arrow hand-holding boilerplate code (ie the bits that have to set each element of the struct explicitly).

[johnkerl]

Thanks @eddelbuettel . That's a good idea for another PR. At the moment I want to separate refactors / dependency changes from the careful sequencing of steps necesary for confident rollout of the current-domain-backed new-shape feature. Introducing nanoarrow here would be too many variables: a dependent-package change, along with the current-domain support. As follow-on -- separately -- yes, I am behind this. I'll file an issue to track this great idea.

[eddelbuettel]

I know where you are coming from here, and I slept over it one night, but I am not sure I am fully on-board.

We have the factory methods. We have the SOMA definitions. It is clear what a well-formed SOMA object is.

But I still cannot shake the feeling that this seems "wrong" in the sense of working, testing, testable, useful, ... general code being removed.

I'll sleep over it another night. Not objecting to the change, but scratching my chin. 🤔

[johnkerl]

The removed code is dead ... unit tests would be throwing if it were not. :)

[eddelbuettel]

That was not really my point / I understand that. But such a converter is also useful outside of SOMA, the test is fine, and I have tested and developed too with non-soma dataframes. That can be useful.

Again, not vetoing anything here, just saying that I simultaneously understand why you propose what you and that I likely would not do it. No more, no less, and surely no veto.

(And as an aside, "we can remove because doing so does not break tests" is not really a viable rule because we all know that coverage metrics are never perfect and there may well be some code that potentially calls this (this is open source after all). Anyway all good by me either way ...)