Merge branch 'master' into ref/pybedtools-pysam

.github/workflows/tests.yml

@@ -0,0 +1,52 @@
+name: Tests
+
+on: [pull_request, workflow_call]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  tests:
+    name: Test with py${{ matrix.python-version }}
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.8", "3.9", "3.10", "3.11"]
+
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up mamba
+        uses: conda-incubator/setup-miniconda@v2
+        with:
+          activate-environment: test
+          auto-activate-base: false
+          python-version: ${{ matrix.python-version }}
+          miniforge-version: latest
+          miniforge-variant: Mambaforge
+          use-mamba: true
+          mamba-version: "*"
+      - name: Install test dependencies
+        shell: bash -el {0}
+        run: |
+          mamba install -c conda-forge -c bioconda bcftools pip pytest pytest-cov
+      - name: Upgrade pip and install our package
+        shell: bash -el {0}
+        run: |
+          python -m pip install --upgrade pip setuptools wheel setuptools-scm
+          pip install -e .
+      - name: Test with pytest
+        shell: bash -el {0}
+        run: |
+          python -m pytest --cov=. --cov-report term-missing --cov-fail-under 89 --doctest-modules --junitxml=junit/test-results-${{ matrix.python-version }}.xml
+      - name: Test command line
+        shell: bash -el {0}
+        run: |
+          ./test/cmdline_tests.sh
+      - name: Upload pytest test results
+        uses: actions/upload-artifact@v3
+        with:
+          name: pytest-results-${{ matrix.python-version }}
+          path: junit/test-results-${{ matrix.python-version }}.xml
+        # Use always() to always run this step to publish test results when there are test failures
+        if: ${{ always() }}

.gitignore

@@ -36,6 +36,7 @@ c57*
 build/
 dist/
 env.yaml
+trtools/version.py
 
 # Files generated by examples
 NA12878_eh_reader.vcf.gz

.readthedocs.yml

@@ -5,9 +5,15 @@
 # Required
 version: 2
 
+build:
+  os: "ubuntu-22.04"
+  tools:
+    python: "mambaforge-22.9"
+
 # Build documentation in the docs/ directory with Sphinx
 sphinx:
   configuration: doc/conf.py
+  fail_on_warning: true
 
 # Optionally build your docs in additional formats such as PDF
 formats:

.readthedocs_conda_env.yml

@@ -2,7 +2,7 @@ name: trtools_3.6
 channels:
   - conda-forge
   - bioconda
-  - defaults
+  - nodefaults
 dependencies:
   - python=3.6
   - sphinx=3.0.4
@@ -16,4 +16,5 @@ dependencies:
   - scikit-learn
   - sphinx-autodoc-typehints
   - cyvcf2
-
+  - sphinx_rtd_theme
+  - statsmodels

Dockerfile

@@ -41,7 +41,7 @@ RUN ldconfig
 WORKDIR ..
 
 # Download and install TRTools
-RUN git clone https://github.com/gymreklab/TRTools
+RUN git clone https://github.com/gymrek-lab/TRTools
 WORKDIR TRTools
 RUN python3 setup.py install
 WORKDIR ..

PUBLISHING.rst

@@ -22,7 +22,7 @@ New Dependencies
 ----------------
 If you've added dependencies to trtools or its tests, those dependencies should be listed in
 
-  * setup.py
+  * pyproject.toml
   * the .readthedocs_conda_env.yml file in the root of the repository that's used for building
     TRTool's Read The Docs webpage.
   * the appropriate section of the bioconda recipe (see below)
@@ -31,12 +31,12 @@ If you've added dependencies to trtools or its tests, those dependencies should
 Publishing Steps
 ----------------
 
-Once changes have been made to develop that are ready to be published, first choose the new version number. Then set up the environment you're going to publish TRTools from:
+Once changes have been made to develop that are ready to be published, first choose the new version number according to `semantic versioning <https://semver.org>`_. Then set up the environment you're going to publish TRTools from:
 
 #. Create a clean environment.
 #. Install setuptools with version >= 40.8.0
-#. Additionally, install ``pytest``, ``wheel`` and ``twine``
-#. Clone the `trtools repo <https://github.com/gymreklab/TRTools>`_
+#. Additionally, install ``pytest``, ``wheel``, ``build``, and ``twine``
+#. Clone the `trtools repo <https://github.com/gymrek-lab/TRTools>`_
 #. Check out the develop branch
 #. Run :code:`pip install --upgrade pip && pip install -e .`
 
@@ -45,26 +45,16 @@ Then go through the steps of merging the changes into the master branch:
 #. Run :code:`pytest` and make sure all the tests pass. Then run :code:`./test/cmdline_tests.sh` and make sure those tests pass.
 #. Change the 'Unreleased Changes' section of :code:`RELEASE_NOTES.rst` to the new version number.
 #. Check if any changes have been made that have not yet been documented in the release notes. If so, document them.
-#. Update the version number in setup.py
-#. Run ``python setup.py sdist bdist_wheel`` (this ensures that trtools/version.py contains the updated version number)
-#. Commit the changes to setup.py and trtools/version.py and push them.
 #. Submit a pull request from develop into master on the github webiste.
-#. If the code review and travis checks pass, merge the pull request.
+#. If the code review checks pass, merge the pull request.
 #. Tag the merge commit with the package version in vX.Y.Z format. (For more details on tagging, see `below`)
 
 Then go through the steps of publishing the changed code to PyPI:
 
-1. :code:`cd` into the root of your clone of the trtools repo, checkout master and pull the latest change.
+1. :code:`cd` into the root of your clone of the trtools repo, checkout master and pull the latest change. Note that the most recent commit *must* be tagged.
 2. Run :code:`rm -rf build dist *.egg-info` to make sure all previous build artifacts are removed
-3. Run :code:`python setup.py sdist bdist_wheel` to build the package.
-
- This will create the warning::
-
-   UserWarning: Unknown distribution option: 'license_file'  warnings.warn(msg)
-
- You can ignore this warning: the 'license_file' option is necessary for creating the build artifacts
-
-4. Run :code:`twine upload dist/*` to upload the build to PyPI
+3. Run :code:`python -m build` to build the package with the version number you just tagged. (Note: you might need to install ``build`` first.)
+5. Run :code:`twine upload dist/*` to upload the distribution to PyPI
 
 Lastly, the change needs to be published to bioconda.
 

README.rst

@@ -2,12 +2,12 @@
 .. a location that the doc/index.rst uses for including this file
 .. before_header
 
-.. image:: https://travis-ci.org/gymreklab/TRTools.svg?branch=master
-    :target: https://travis-ci.org/gymreklab/TRTools
+.. image:: https://github.com/gymrek-lab/trtools/workflows/Tests/badge.svg
+    :target: https://github.com/gymrek-lab/trtools/workflows/Tests/badge.svg
 
 
-.. image:: https://codecov.io/gh/gymreklab/TRTools/branch/master/graph/badge.svg
-  :target: https://codecov.io/gh/gymreklab/TRTools
+.. image:: https://codecov.io/gh/gymrek-lab/TRTools/branch/master/graph/badge.svg
+  :target: https://codecov.io/gh/gymrek-lab/TRTools
 
 
 .. a location that the doc/index.rst uses for including this file
@@ -33,13 +33,13 @@ With conda
 
 ::
 
-        conda install -c bioconda trtools
+        conda install -c conda-forge -c bioconda trtools
 
 Optionally install :code:`bcftools` which is used to prepare input files for TRTools by running:
 
 ::
 
-        conda install -c bioconda bcftools
+        conda install -c conda-forge -c bioconda bcftools
 
 Note: Bioconda only supports python versions 3.6-3.8 currently,
 so that is all TRTools supports in conda.
@@ -64,10 +64,10 @@ Note: TRTools installation may fail for pip version 10.0.1, hence the need to up
 From source
 ^^^^^^^^^^^
 
-To install from source (only recommended for development) download the TRTools repository from `github <https://github.com/gymreklab/TRTools/>`_,
+To install from source (only recommended for development) download the TRTools repository from `github <https://github.com/gymrek-lab/TRTools/>`_,
 checkout the branch you're interested in, and run the following command from the base directory of the repo. e.g.::
 
-        git clone https://github.com/gymreklab/TRTools
+        git clone https://github.com/gymrek-lab/TRTools
         cd TRTools/
         pip install --upgrade pip
         pip install -e .
@@ -127,15 +127,15 @@ Development Notes
 
 Contact Us
 ----------
-Please submit an issue on the `trtools github <https://github.com/gymreklab/TRTools>`_
+Please submit an issue on the `trtools github <https://github.com/gymrek-lab/TRTools>`_
 
 .. _Contributing:
 
 Contributing
 ------------
 We appreciate contributions to TRTools. If you would like to contribute a fix or new feature, follow these guidelines:
 
-1. Consider `discussing <https://github.com/gymreklab/TRTools/issues>`_ your solution with us first so we can provide help or feedback if necessary.
+1. Consider `discussing <https://github.com/gymrek-lab/TRTools/issues>`_ your solution with us first so we can provide help or feedback if necessary.
 #. Install TRTools from source `as above <From source_>`_.
 #. Additionally, install :code:`pytest`, `pytest-cov <https://anaconda.org/conda-forge/pytest-cov>`_ and :code:`sphinx>=3` in your environment.
 #. Fork the TRTools repository.

RELEASE_NOTES.rst

@@ -79,7 +79,7 @@ Misc:
 
 Bug fixes:
 
-* https://github.com/gymreklab/TRTools/issues/146 fixed record positions being compared twice
+* https://github.com/gymrek-lab/TRTools/issues/146 fixed record positions being compared twice
 * CompareSTR: Decision on which records are comparable is now based on data from harmonized TRRecords,
   and not from the records directly from VCF readers. Thanks to this, HipSTR records which have different starting positions,
   but position of their repeat is at the same position are compared correctly (harmonization step removes this difference).
@@ -92,7 +92,7 @@ Bug fixes:
 
 Bug fixes:
 
-* https://github.com/gymreklab/TRTools/issues/143 Fix HipstrMinSuppReads filter when
+* https://github.com/gymrek-lab/TRTools/issues/143 Fix HipstrMinSuppReads filter when
   there are called samples but none have ALLREADS
 
 4.0.0

doc/VIGNETTE-AFREQ.rst

@@ -3,7 +3,7 @@ Plotting allele length distributions by population group
 
 Tools used: mergeSTR, statSTR
 
-This vignette shows how to use :code:`mergeSTR` to merge two VCF files and :code:`statSTR` to plot allele frequencies across different sample groups for an example TR locus. It uses the example VCF files :code:`ceu_ex.vcf.gz` and :code:`yri_ex.vcf.gz` available at https://github.com/gymreklab/TRTools/tree/master/example-files. These VCFs were generated by GangSTR on samples sequenced by the 1000 Genomes Project. They have already been sorted and indexed.
+This vignette shows how to use :code:`mergeSTR` to merge two VCF files and :code:`statSTR` to plot allele frequencies across different sample groups for an example TR locus. It uses the example VCF files :code:`ceu_ex.vcf.gz` and :code:`yri_ex.vcf.gz` available at https://github.com/gymrek-lab/TRTools/tree/master/example-files. These VCFs were generated by GangSTR on samples sequenced by the 1000 Genomes Project. They have already been sorted and indexed.
 
 After downloading the VCF files, we can use :code:`mergeSTR` to merge them into a single VCF::
 

doc/VIGNETTE-COMPARE-CALLERS.rst

@@ -3,7 +3,7 @@ Comparing TR calls across different genotypers
 
 Tools used: mergeSTR, compareSTR
 
-This vignette shows how to use :code:`mergeSTR` to merge VCFs from multiple samples into a single VCF, and :code:`compareSTR` to compare VCF files generated by different genotypers (HipSTR and ExpansionHunter) using the same set of reference TRs. In this example, we use VCF files available at https://github.com/gymreklab/TRTools/tree/master/example-files:
+This vignette shows how to use :code:`mergeSTR` to merge VCFs from multiple samples into a single VCF, and :code:`compareSTR` to compare VCF files generated by different genotypers (HipSTR and ExpansionHunter) using the same set of reference TRs. In this example, we use VCF files available at https://github.com/gymrek-lab/TRTools/tree/master/example-files:
 
 * :code:`NA12878_chr21_eh.sorted.vcf.gz`, :code:`NA12891_chr21_eh.sorted.vcf.gz`, and :code:`NA12892_chr21_eh.sorted.vcf.gz` generated using ExpansionHunter on three separate samples
 * :code:`trio_chr21_hipstr.sorted.vcf.gz` generated using HipSTR run jointly on all three samples.

doc/VIGNETTE-COMPARE.rst

@@ -3,7 +3,7 @@ Comparing TR calls across different parameter sets
 
 Tools used: compareSTR
 
-This vignette shows how to use :code:`compareSTR` to compare two VCF files generated using the same set of reference TRs. In this example, we use VCF files :code:`c57_ex1.vcf.gz` and :code:`c57_ex2.vcf.gz` available at https://github.com/gymreklab/TRTools/tree/master/example-files. These VCF files were generated by GangSTR on a mouse dataset using two different sets of stutter parameters.
+This vignette shows how to use :code:`compareSTR` to compare two VCF files generated using the same set of reference TRs. In this example, we use VCF files :code:`c57_ex1.vcf.gz` and :code:`c57_ex2.vcf.gz` available at https://github.com/gymrek-lab/TRTools/tree/master/example-files. These VCF files were generated by GangSTR on a mouse dataset using two different sets of stutter parameters.
 
 To run :code:`compareSTR`::
 

doc/VIGNETTE-FILTER-QC.rst

@@ -3,7 +3,7 @@ Filtering and QC of VCFs
 
 Tools used: dumpSTR, qcSTR
 
-This vignette shows how to use :code:`dumpSTR` to filter a VCF and :code:`qcSTR` to visualize some basic QC metrics. For this example, we use the file :code:`trio_chr21_popstr.sorted.vcf.gz` available at https://github.com/gymreklab/TRTools/tree/master/example-files. This file was generated on samples NA12878, NA12891, and NA12892 using popSTR.
+This vignette shows how to use :code:`dumpSTR` to filter a VCF and :code:`qcSTR` to visualize some basic QC metrics. For this example, we use the file :code:`trio_chr21_popstr.sorted.vcf.gz` available at https://github.com/gymrek-lab/TRTools/tree/master/example-files. This file was generated on samples NA12878, NA12891, and NA12892 using popSTR.
 
 First, let's perform some filtering on the VCF::
 

doc/VIGNETTE-STATSTR.rst

@@ -3,7 +3,7 @@ Computing per-locus TR statistics
 
 Tools used: mergeSTR, statSTR
 
-This vignette shows how to use :code:`mergeSTR` to merge two VCF files and :code:`statSTR` to compute statistics across different sample groups for an example TR locus. It uses the example VCF files :code:`ceu_ex.vcf.gz` and :code:`yri_ex.vcf.gz` available at https://github.com/gymreklab/TRTools/tree/master/example-files. These VCFs were generated by GangSTR on samples sequenced by the 1000 Genomes Project. They have already been sorted and indexed.
+This vignette shows how to use :code:`mergeSTR` to merge two VCF files and :code:`statSTR` to compute statistics across different sample groups for an example TR locus. It uses the example VCF files :code:`ceu_ex.vcf.gz` and :code:`yri_ex.vcf.gz` available at https://github.com/gymrek-lab/TRTools/tree/master/example-files. These VCFs were generated by GangSTR on samples sequenced by the 1000 Genomes Project. They have already been sorted and indexed.
 
 After downloading the VCF files, we can use :code:`mergeSTR` to merge them into a single VCF::
 

doc/conf.py

@@ -13,18 +13,13 @@
 import os
 import sys
 sys.path.insert(0, os.path.abspath('..'))
-import trtools.version
 
 # -- Project information -----------------------------------------------------
 
 project = 'TRTools'
 copyright = '2020, Gymreklab'
 author = 'Gymreklab'
 
-# The full version, including alpha/beta/rc tags
-version = trtools.version.version
-release = trtools.version.version
-
 master_doc = 'index'
 
 # -- General configuration ---------------------------------------------------
@@ -73,9 +68,6 @@
         'searchbox.html'
     ]
 }
-html_theme_options = {
-    'fixed_sidebar' : True
-}
 
 
 # Add any paths that contain custom static files (such as style sheets) here,