Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python] Coalesce wheels-build and PyPI publish #700

Closed
johnkerl opened this issue Jan 11, 2023 · 2 comments
Closed

[python] Coalesce wheels-build and PyPI publish #700

johnkerl opened this issue Jan 11, 2023 · 2 comments

Comments

@johnkerl
Copy link
Member

johnkerl commented Jan 11, 2023

This needs to be resolved before we publish a 0.5.0a4.

Currently this is split into two files, one of them stubbed:

  • wheels:
    # Note: tries all supported Python versions as specified in apis/python/setup.py
    name: Build wheels on ${{ matrix.os }}
    needs: sdist
    runs-on: ${{ matrix.os }}
    strategy:
    matrix:
    include:
    - os: ubuntu-20.04
    cibw_build: '*-manylinux_x86_64'
    platform: manylinux2014
    - os: macos-11
    cibw_build: '*-macosx_x86_64'
    platform: macosx
    steps:
    - name: Download sdist artifact
    uses: actions/download-artifact@v3
    with:
    name: sdist
    - name: rename sdist
    run: cp tiledbsoma-*.tar.gz tiledbsoma.tar.gz && ls -lh
    - name: Build wheels
    uses: pypa/[email protected]
    with:
    package-dir: tiledbsoma.tar.gz
    env:
    CIBW_BUILD: ${{ matrix.cibw_build }}
    CIBW_BUILD_VERBOSITY: 1
    CIBW_BEFORE_BUILD: bash -x -c 'rm -rf tiledbsoma*/dist_links/dist/lib'
    # ^ Delete lib folder that apis/python/setup.py:find_or_build() looks for in deciding to
    # run CMake build or not. Otherwise it'll keep reusing the library file built in the
    # first iteration of cibuildwheel's outer loop, resulting in wheels with the library
    # built for the wrong python version.
    - name: Upload wheels-${{ matrix.platform }}
    uses: actions/upload-artifact@v3
    with:
    name: wheels-${{ matrix.platform }}
    path: ./wheelhouse/*.whl
  • - name: Build wheel distribution
    run: python -m pip -v wheel --no-deps --wheel-dir=dist-wheel apis/python
    env:
    CC: ${{ matrix.cc }}
    CXX: ${{ matrix.cxx }}
    - name: Publish package to TestPyPI
    if: matrix.os == 'ubuntu-22.04' && github.event_name == 'release'
    uses: pypa/gh-action-pypi-publish@master
    continue-on-error: true
    with:
    repository_url: https://test.pypi.org/legacy/
    user: __token__
    password: ${{ secrets.TEST_PYPI_TOKEN }}
    packages_dir: dist-wheel
    verbose: true
    # Until general release of the main branch of tiledbsoma we'll follow the convention of tagging releases
    # 0.5.0a1, 0.5.1a1, etc -- always with the "a1" suffix -- that way PyPI will automagically make these
    # "prereleases". Then:
    # pip install tiledbsoma -> 0.1.12 (or whatever the main-old branch is at)
    # pip install --pre tiledbsoma -> 0.5.0a1 (or whatever the main branch is at)
    # When we're ready for prime time:
    # if: matrix.os == 'ubuntu-22.04' && github.event_name == 'release' && !github.event.release.prerelease
    # # For now:
    # - name: Publish package to PyPI
    # if: matrix.os == 'ubuntu-22.04' && github.event_name == 'release' && (contains(${{github.event.release.tag_name}}, "a") || contains(${github.event.release.tag_name}}, "b"))
    # uses: pypa/gh-action-pypi-publish@master
    # with:
    # user: __token__
    # password: ${{ secrets.PYPI_TOKEN }}
    # packages_dir: dist-wheel
    # verbose: true
    # Notes:
    # The above is fine for releases; notes here are for manual ops.
    # https://towardsdatascience.com/create-your-own-python-package-and-publish-it-into-pypi-9306a29bc116
    # TL;DR:
    # o cd apis/python
    # o python setup.py sdist
    # o python -m twine upload --repository-url https://test.pypi.org/legacy/ dist/*
    # - Be sure to use username __token__ and password being the token you set up at test.pypi.org
    # o pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple tiledbsoma

This needs to be done in one place, and done completely.

See also
https://github.com/single-cell-data/TileDB-SOMA/wiki#branches-and-releases

Update 2023-01-12: see also
https://github.com/single-cell-data/TileDB-SOMA/wiki/PyPI-packaging-WIP
for important information about the migration path

Update 2023-01-13: see also
#701 (comment)
for a great sketch of remaining work to be done

@johnkerl
Copy link
Member Author

cc @mlin

@johnkerl
Copy link
Member Author

#704 takes care of the dedupe per se.

However there's more work here to be done than I realized. Issue #706 has been opened to track that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant