Research feasibility of 100% test coverage

[simonw]

Inspired by https://twitter.com/mikeal/status/1271473021593636866

Almost every library I’ve written in the last 2 years has had 100% coverage and that’s probably not going to change in the future. It’s not that hard to start at 100% and hold onto it and the workflow it enables is so much nicer.

[simonw]

Starlette achieves this. https://github.com/encode/starlette

[simonw]

Using https://pypi.org/project/pytest-cov/ and running pytest --cov=datasette:

---------- coverage: platform darwin, python 3.7.7-final-0 -----------
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
datasette/__init__.py                    3      0   100%
datasette/__main__.py                    3      3     0%
datasette/_version.py                  277    152    45%
datasette/actor_auth_cookie.py          19      3    84%
datasette/app.py                       499     27    95%
datasette/cli.py                       157     45    71%
datasette/database.py                  233     17    93%
datasette/default_permissions.py        39      0   100%
datasette/facets.py                    209     24    89%
datasette/filters.py                   122      7    94%
datasette/hookspecs.py                  19      0   100%
datasette/inspect.py                    37     23    38%
datasette/plugins.py                    34      6    82%
datasette/publish/__init__.py            0      0   100%
datasette/publish/cloudrun.py           55      2    96%
datasette/publish/common.py             19      1    95%
datasette/publish/heroku.py             95     13    86%
datasette/renderer.py                   63      4    94%
datasette/sql_functions.py               4      0   100%
datasette/tracer.py                     85     16    81%
datasette/utils/__init__.py            503     31    94%
datasette/utils/asgi.py                253     25    90%
datasette/utils/shutil_backport.py      44     40     9%
datasette/version.py                     4      0   100%
datasette/views/__init__.py              0      0   100%
datasette/views/base.py                288     19    93%
datasette/views/database.py            120      2    98%
datasette/views/index.py                57      2    96%
datasette/views/special.py              72     16    78%
datasette/views/table.py               418     18    96%
--------------------------------------------------------
TOTAL                                 3731    496    87%

[simonw]

Added a .coveragerc file:

[run]
omit = datasette/_version.py, datasette/utils/shutil_backport.py

And ran again: pytest --cov=datasette --cov-config=.coveragerc

Name                               Stmts   Miss  Cover
------------------------------------------------------
datasette/__init__.py                  3      0   100%
datasette/__main__.py                  3      3     0%
datasette/actor_auth_cookie.py        19      3    84%
datasette/app.py                     499     27    95%
datasette/cli.py                     157     45    71%
datasette/database.py                233     17    93%
datasette/default_permissions.py      39      0   100%
datasette/facets.py                  209     24    89%
datasette/filters.py                 122      7    94%
datasette/hookspecs.py                19      0   100%
datasette/inspect.py                  37     23    38%
datasette/plugins.py                  34      6    82%
datasette/publish/__init__.py          0      0   100%
datasette/publish/cloudrun.py         55      2    96%
datasette/publish/common.py           19      1    95%
datasette/publish/heroku.py           95     13    86%
datasette/renderer.py                 63      4    94%
datasette/sql_functions.py             4      0   100%
datasette/tracer.py                   85     16    81%
datasette/utils/__init__.py          503     31    94%
datasette/utils/asgi.py              253     25    90%
datasette/version.py                   4      0   100%
datasette/views/__init__.py            0      0   100%
datasette/views/base.py              288     19    93%
datasette/views/database.py          120      2    98%
datasette/views/index.py              57      2    96%
datasette/views/special.py            72     16    78%
datasette/views/table.py             418     18    96%
------------------------------------------------------
TOTAL                               3410    304    91%

[simonw]

Added --cov-report html and got this report: https://static.simonwillison.net/static/2020/htmlcov-issue-841/index.html

[simonw]

From digging through that report it looks like the majority stuff that isn't fully covered is corner-cases... which are the kind of things I really do want the tests to catch.

I'm not entirely ready to commit to 100%, but I'm going to start digging through and seeing how close I can get. If I can get to 98% (I'm on 91% already) I may as well push all the way to 100.

[simonw]

This one looks easy enough to fix:

[simonw]

Looking at options for publishing coverage reports:

• https://github.com/codecov/codecov-action
• https://github.com/coveralls-clients/coveralls-python

I'm going to try https://codecov.io/

[simonw]

I ran export CODECOV_TOKEN="f7935cad...", then ran this:

datasette $ bash <(curl -s https://codecov.io/bash)  
  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
                              Bash-20200602-f809a24


x> No CI provider detected.
    Testing inside Docker? http://docs.codecov.io/docs/testing-with-docker
    Testing with Tox? https://docs.codecov.io/docs/python#section-testing-with-tox
    project root: .
--> token set from env
    Yaml not found, that's ok! Learn more at http://docs.codecov.io/docs/codecov-yaml
==> Running gcov in . (disable via -X gcov)
==> Searching for coverage reports in:
    + .
    -> Found 1 reports
==> Detecting git/mercurial file structure
==> Reading reports
    + ./coverage.xml bytes=139174
==> Appending adjustments
    https://docs.codecov.io/docs/fixing-reports
    -> No adjustments found
==> Gzipping contents
==> Uploading reports
    url: https://codecov.io
    query: branch=master&commit=0e49842e227a0f1f69d48108c87d17fe0379e548&build=&build_url=&name=&tag=&slug=simonw%2Fdatasette&service=&flags=&pr=&job=
    -> Pinging Codecov
https://codecov.io/upload/v4?package=bash-20200602-f809a24&token=secret&branch=master&commit=0e49842e227a0f1f69d48108c87d17fe0379e548&build=&build_url=&name=&tag=&slug=simonw%2Fdatasette&service=&flags=&pr=&job=
    -> Uploading
    -> View reports at https://codecov.io/github/simonw/datasette/commit/0e49842e227a0f1f69d48108c87d17fe0379e548

But https://codecov.io/github/simonw/datasette/commit/0e49842e227a0f1f69d48108c87d17fe0379e548 is a 404, so it doesn't seem to have worked?

UPDATE: It works now, took about 30 seconds before the report showed up at that URL.

[simonw]

Closing this because I've researched feasibility. I may start a milestone in the future to help me get to 100%.