Skip to content

Commit

Permalink
many enhancements to the contributing guide, some cleanups for the Ma…
Browse files Browse the repository at this point in the history
…kefile

including a new target: `make cleanup`

Co-authored-by: Michael R. Crusoe <[email protected]>
  • Loading branch information
jjkoehorst and mr-c authored Aug 17, 2022
1 parent ceb38d9 commit f3a35f3
Show file tree
Hide file tree
Showing 2 changed files with 78 additions and 39 deletions.
78 changes: 57 additions & 21 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,68 @@
Style guide:
- PEP-8
- Python 3.7+ compatible code
- PEP-8 (as implemented by the `black` code formatting tool)
- Python 3.6+ compatible code
- PEP-484 type hints
- Vertically align the type hints in function definitions

The development is done using ``git``, we encourage you to get familiar with it.
The development is done using `git`, we encourage you to get familiar with it.
Here's a rough guide (improvements are welcome!)
To get the code and start working on the changes you can start a console and:
- Clone the cwltool: ``git clone https://github.com/common-workflow-language/cwltool.git``
- Switch to cwltool directory: ``cd cwltool``
- Clone the cwltool: `git clone https://github.com/common-workflow-language/cwltool.git`
- Switch to cwltool directory: `cd cwltool`

In order to contribute to the development of ``cwltool``, the source code needs to pass the test before your changes are accepted.
There are a couple ways to test the code with your changes: let `tox` manage installation and test running in virtual environments, or do it manually (preferably in a virtual environment):
- Install ``tox`` preferably using the OS' package manager, otherwise it can be installed with ``pip install --user -U tox``
In order to contribute to the development of `cwltool`, the source code needs to
pass the test before your changes are accepted. There are a couple ways to test
the code with your changes: let `tox` manage installation and test running in
virtual environments, or do it manually (preferably in a virtual environment):
- Install `tox` preferably using the OS' package manager, otherwise it can be
installed with `pip install --user -U tox`
- Make your changes to the code and add tests for new cool things you're adding!
- Run the tests with the command ``tox``, it's recommended to use some parameters as tox will try to run all the checks in all available python interpreters.
- The important tests to run are ``unit tests`` and ``type tests``.
To run these two in Python 3.7, we can tell tox to run only those tests by running: ``tox -e py37-unit,py37-mypy2,py37-mypy3``.
- Run ``tox -l`` to see all available tests and runtimes
- Run the tests with the command `tox`, it's recommended to use some parameters
as tox will try to run all the checks in all available python interpreters.
- The important tests to run are "unit tests" and "type tests".
To run these two in Python 3.7 (for example), we can tell tox to run only those
tests by running: `tox -e py37-mypy`. (the `mypy` tox target also runs the
unit tests; to just run the type checker use `make mypy` as shown below)
- Run `tox -l` to see all available tests and runtimes

For the more traditional workflow:
- Create a virtual environment: ``python3 -m venv cwltool``
- To begin using the virtual environment, it needs to be activated: ``source bin/activate``
- To check if you have the virtual environment set up: ``which python`` and it should point to python executable in your virtualenv
- Install cwltool: ``pip install -e .``
- Check the version which might be different from the version installed in general on any system: ``cwltool --version``
- Create a virtual environment: `python3 -m venv cwltool`
(if you have multiple version of Python installed, then you can replace `python3`
in this command with the version you want to use, `python3.10` for example)
- To begin using the virtual environment, it needs to be activated: `source venv/bin/activate`
- To check if you have the virtual environment set up: `which python`
and it should point to python executable in your virtualenv
- Install cwltool in development mode, along with development tools:
`make install-dep dev`
- Check the `cwltool` version which might be different from the version installed in
general on your system: `cwltool --version`
- Make your changes to the code and add tests for new cool things you're adding!
- Run the unit-tests to see : ``python setup.py test``
- After you're done working on ``cwltool``, you can deactivate the virtual environment: ``deactivate``
- Run the unit-tests to see : `make test`
- After you're done working on `cwltool`, you can deactivate the virtual
environment: `deactivate`

When tests are passing, you can simply commit and create a PR on ``cwltool`` repo
Before you commit your code (or at least before you push to GitHub) it is
recommended to run `make cleanup`
(which is a shortcut for `make sort_imports format flake8 diff_pydocstyle_report`)
to fix common issues and point out any remaining code formatting issues
that cannot be fixed automatically.

When tests are passing, you can create git commits in a new branch and push to
GitHub to make a pull request on `cwltool` repo

Useful `make` commands:
The `make` commands below can help you to install the dependencies, format and test your code

- `make help` to show the list of commands that can be used with the make function
- `make install-dep` will install the dependencies needed by `cwltool`
- `make format` will clean up your code according to the accepted python standard
- `make test` will perform the tests of `cwltool` (can take a while to run all tests)
- `make sort_imports` will clean up and sort your import statements
- `make remove_unused_imports` will remove any unneeded `import` statements
- `make flake8` will catch format issues that `make format` cannot fix automatically
- `make diff_pydocstyle_report` check Python docstring style for changed files only
- `make diff-cover` to run the tests and point out which lines of code that have been
changed as compared to the `main` branch are missing test coverage. This is also calculated
automatically after all the automatic tests run when you open a Pull Request on GitHub;
running it locally can help you confirm that any changes or additions
to the tests are sufficient before pushing to GitHub.
- `make cleanup` is a shortcut for `make sort_imports format flake8 diff_pydocstyle_report`
39 changes: 21 additions & 18 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -36,14 +36,17 @@ VERSION=3.1.$(shell TZ=UTC git log --first-parent --max-count=1 \
mkfile_dir := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
UNAME_S=$(shell uname -s)

## all : default task
## all : default task (install cwltool in dev mode)
all: dev

## help : print this help message and exit
## help : print this help message and exit
help: Makefile
@sed -n 's/^##//p' $<

## install-dep : install most of the development dependencies via pip
## cleanup : shortcut for "make sort_imports format flake8 diff_pydocstyle_report"
cleanup: sort_imports format flake8 diff_pydocstyle_report

## install-dep : install most of the development dependencies via pip
install-dep: install-dependencies

install-dependencies: FORCE
Expand All @@ -53,20 +56,20 @@ install-dependencies: FORCE
install-doc-dep:
pip install -r docs/requirements.txt

## install-deb-dep: install most of the dev dependencies via apt-get
## install-deb-dep : install many of the dev dependencies via apt-get
install-deb-dep:
sudo apt-get install $(DEBDEVPKGS)

## install : install the ${MODULE} module and schema-salad-tool
## install : install the cwltool module with the "deps" dependency (galaxy-tool-util)
install: FORCE
pip install .[deps]

## dev : install the ${MODULE} module in dev mode
## dev : install the cwltool module in dev mode
dev: install-dep
pip install -e .[deps]


## dist : create a module package for distribution
## dist : create a module package for distribution
dist: dist/${MODULE}-$(VERSION).tar.gz

check-python3:
Expand All @@ -76,11 +79,11 @@ check-python3:
dist/${MODULE}-$(VERSION).tar.gz: check-python3 $(SOURCES)
python setup.py sdist bdist_wheel

## docs : make the docs
## docs : make the docs
docs: FORCE
cd docs && $(MAKE) html

## clean : clean up all temporary / machine-generated files
## clean : clean up all temporary / machine-generated files
clean: check-python3 FORCE
rm -f ${MODULE}/*.pyc tests/*.pyc *.so ${MODULE}/*.so
rm -Rf ${MODULE}/__pycache__/
Expand All @@ -89,37 +92,37 @@ clean: check-python3 FORCE
rm -f diff-cover.html

# Linting and code style related targets
## sorting imports using isort: https://github.com/timothycrosley/isort
## sort_import : sorting imports using isort: https://github.com/timothycrosley/isort
sort_imports: $(PYSOURCES) mypy-stubs
isort $^

remove_unused_imports: $(PYSOURCES)
autoflake --in-place --remove-all-unused-imports $^

pep257: pydocstyle
## pydocstyle : check Python docstring style
## pydocstyle : check Python docstring style
pydocstyle: $(PYSOURCES)
pydocstyle --add-ignore=D100,D101,D102,D103 $^ || true

pydocstyle_report.txt: $(PYSOURCES)
pydocstyle setup.py $^ > $@ 2>&1 || true

## diff_pydocstyle_report : check Python docstring style for changed files only
## diff_pydocstyle_report : check Python docstring style for changed files only
diff_pydocstyle_report: pydocstyle_report.txt
diff-quality --compare-branch=main --violations=pydocstyle --fail-under=100 $^

## codespell : check for common mispellings
## codespell : check for common mispellings
codespell:
codespell -w $(shell git ls-files | grep -v cwltool/schemas | grep -v cwltool/jshint/ | grep -v mypy-stubs)

## format : check/fix all code indentation and formatting (runs black)
## format : check/fix all code indentation and formatting (runs black)
format:
black --exclude cwltool/schemas setup.py cwltool.py cwltool tests mypy-stubs

format-check:
black --diff --check --exclude cwltool/schemas setup.py cwltool.py cwltool tests mypy-stubs

## pylint : run static code analysis on Python code
## pylint : run static code analysis on Python code
pylint: $(PYSOURCES)
pylint --msg-template="{path}:{line}: [{msg_id}({symbol}), {obj}] {msg}" \
$^ -j0|| true
Expand Down Expand Up @@ -154,18 +157,18 @@ diff-cover: coverage.xml
diff-cover.html: coverage.xml
diff-cover --compare-branch=main $^ --html-report $@

## test : run the ${MODULE} test suite
## test : run the ${MODULE} test suite
test: check-python3 $(PYSOURCES)
python -m pytest -rs ${PYTEST_EXTRA}

## testcov : run the ${MODULE} test suite and collect coverage
## testcov : run the ${MODULE} test suite and collect coverage
testcov: check-python3 $(PYSOURCES)
python -m pytest -rs --cov --cov-config=.coveragerc --cov-report= ${PYTEST_EXTRA}

sloccount.sc: $(PYSOURCES) Makefile
sloccount --duplicates --wide --details $^ > $@

## sloccount : count lines of code
## sloccount : count lines of code
sloccount: $(PYSOURCES) Makefile
sloccount $^

Expand Down

0 comments on commit f3a35f3

Please sign in to comment.