Migrate TileDB-Py to scikit-build-core and don't check for deprecations

[jdblischak]

Fixes #747

The nightly builds are no longer failing due to the use of deprecated APIs. However, now it is failing to build dev TileDB-Py from source (#747 (comment)).

I updated the nightly workflow to build TileDB-Py with scikit-build-core and also to set TILEDB_REMOVE_DEPRECATIONS=OFF. However, despite this being essentially identical to my setup in my centralized nightlies, TileDB-Py can't find libtiledb at runtime (at least that is my assumption for the failed import test). The only difference is that my centralized nightlies perform an editable install.

@dudoslav do you have any insight in how to fix this runtime error? I set both TILEDB_PATH and LD_LIBRARY_PATH. And the build finds libtiledb and sets the RPATH

  -- Found external TileDB core library
  -- Using libtiledb path: /home/runner/work/TileDB-VCF/TileDB-VCF/install//lib/cmake/TileDB
  -- Setting RPATH for targets "main" and "libtiledb" to /home/runner/work/TileDB-VCF/TileDB-VCF/install/lib
  -- Setting RPATH for target "cc" to /home/runner/work/TileDB-VCF/TileDB-VCF/install/lib

Here is an example failed build from my fork.

[dudoslav]

@jdblischak I believe this is due to running this command:

+ python -c 'import tiledb; print('\''successful import'\'')'

in a folder that contains folder tiledb. So either move cd .. or delete that folder. This happened even before scikit-build-core rework, I just don't understand why it is an issue now.

[jdblischak]

I believe this is due to running this command + python -c 'import tiledb; print('\''successful import'\'')' in a folder that contains folder tiledb.

@dudoslav you are absolutely right. I keep stepping on this same rake

xref: TileDB-Inc/TileDB-Py#1988 (comment), jdblischak/centralized-tiledb-nightlies@81849b9

[jdblischak]

Changing the directory fixed the import error. Now I am only getting an error for macos-12-libtiledb-release-2.24

TileDB-VCF/ci/nightly/build-tiledbvcf-py.sh: line 26: 78997 Abort trap: 6           pytest
tests/test_tiledbvcf.py ........s............s....
Error: Process completed with exit code 134.

I'm not sure what is causing this. It's strange it only happened against tiledb 2.24 and not nightly tiledb. The last time I saw Abort trap was in a feedstock build with VCF 0.27.1 and TileDB 2.18 TileDB-Inc/tiledb-vcf-feedstock#108

[jdblischak]

Both the passing and failed builds installed pyarrow-17.0.0-cp310-cp310-macosx_10_15_x86_64.whl. And looking at all the installed packages, the only difference is that the failing build re-installed packaging-24.1 (in both builds it is installed in an earlier step)

# passed
Successfully installed MarkupSafe-2.1.5 click-8.1.7 cloudpickle-3.0.0 dask-2024.8.0 distributed-2024.8.0 exceptiongroup-1.2.2 fsspec-2024.6.1 importlib-metadata-8.2.0 iniconfig-2.0.0 jinja2-3.1.4 locket-1.0.0 msgpack-1.0.8 numpy-1.26.4 pandas-2.2.2 partd-1.4.2 pluggy-1.5.0 psutil-6.0.0 pyarrow-17.0.0 pyarrow-hotfix-0.6 pytest-8.3.2 python-dateutil-2.9.0.post0 pytz-2024.1 pyyaml-6.0.2 six-1.16.0 sortedcontainers-2.4.0 tblib-3.0.0 tiledbvcf-0.34.2.dev3+gf64ff03b tomli-2.0.1 toolz-0.12.1 tornado-6.4.1 tzdata-2024.1 urllib3-2.2.2 zict-3.0.0 zipp-3.19.2

# failed
Successfully installed MarkupSafe-2.1.5 click-8.1.7 cloudpickle-3.0.0 dask-2024.8.0 distributed-2024.8.0 exceptiongroup-1.2.2 fsspec-2024.6.1 importlib-metadata-8.2.0 iniconfig-2.0.0 jinja2-3.1.4 locket-1.0.0 msgpack-1.0.8 numpy-1.26.4 packaging-24.1 pandas-2.2.2 partd-1.4.2 pluggy-1.5.0 psutil-6.0.0 pyarrow-17.0.0 pyarrow-hotfix-0.6 pytest-8.3.2 python-dateutil-2.9.0.post0 pytz-2024.1 pyyaml-6.0.2 six-1.16.0 sortedcontainers-2.4.0 tblib-3.0.0 tiledbvcf-0.34.2.dev3+gf64ff03b tomli-2.0.1 toolz-0.12.1 tornado-6.4.1 tzdata-2024.1 urllib3-2.2.2 zict-3.0.0 zipp-3.19.2

[gspowley]

Thanks @jdblischak, the macos failure is something we're debugging and the reason for TileDB-Inc/tiledb-vcf-feedstock#134 (which we closed). It is interesting that the error did not happen when building with the nightly tiledb, because we have seen that error with the nightly tiledb before.

For the VCF nightly, can we pin pyarrow (<13) until the macos issue is resolved?

[jdblischak]

Ready for review. I confirmed in another branch that pinning pyarrow==12 fixes the issue (jdblischak@be78767, build). Note that previously the nightly build was pinning pyarrow==11 in ci/nightly/requirements.txt, which explains why this error reappeared in this PR.