WIP: ci: Travis: revisit/fix coverage reporting #4846

blueyed · 2019-02-27T11:20:26Z

This should bring coverage back that got missing with 9dcd6f2.

Continuation of #4839.

codecov · 2019-02-27T12:31:24Z

Codecov Report

Merging #4846 into master will increase coverage by 0.54%.
The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master    #4846      +/-   ##
==========================================
+ Coverage   94.28%   94.82%   +0.54%     
==========================================
  Files         113      113              
  Lines       25160    25161       +1     
  Branches     2497     2497              
==========================================
+ Hits        23723    23860     +137     
+ Misses       1100      971     -129     
+ Partials      337      330       -7

Impacted Files	Coverage Δ
src/_pytest/warnings.py	`87.65% <ø> (ø)`	⬆️
src/_pytest/compat.py	`96.98% <ø> (ø)`	⬆️
src/_pytest/unittest.py	`87.3% <ø> (ø)`	⬆️
src/_pytest/terminal.py	`91.74% <0%> (+2.64%)`	⬆️
testing/test_pdb.py	`95.84% <100%> (+47.69%)`	⬆️
testing/test_argcomplete.py	`20.28% <0%> (-47.83%)`	⬇️
src/_pytest/_argcomplete.py	`33.33% <0%> (-41.67%)`	⬇️
src/_pytest/pytester.py	`80.93% <0%> (-5.41%)`	⬇️
testing/deprecated_test.py	`96.77% <0%> (-3.23%)`	⬇️
testing/python/fixture.py	`98.5% <0%> (-0.58%)`	⬇️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df8869c...3d0ef12. Read the comment docs.

blueyed · 2019-02-27T12:53:20Z

After 1st commit:

increase coverage by 0.99%.

blueyed · 2019-02-27T13:27:41Z

Now without windows:

will increase coverage by 0.31%.

blueyed · 2019-02-27T14:36:35Z

Now:

will increase coverage by 0.46%.

Need to investigate/compare coverage again.

blueyed · 2019-02-27T16:56:34Z

btw: so apparently tests are failing with PYTEST_ADDOPTS=-vv on Travis.. :/
(https://travis-ci.org/pytest-dev/pytest/builds/499355099?utm_source=github_status&utm_medium=notification)

nicoddemus · 2019-02-27T17:13:38Z

btw: so apparently tests are failing with PYTEST_ADDOPTS=-vv on Travis.. :/

😞

blueyed · 2019-02-27T17:14:39Z

codecov/patch link still times out, with only 5 uploads now.. https://codecov.io/gh/pytest-dev/pytest/commit/11b531be0a38e3c9ce9e43e52fbc3c56fb72b6a5/build
So the base might also be relevant here.. :/

blueyed · 2019-02-27T17:16:00Z

@nicoddemus
But we can see that only ~5 build jobs are necessary to have ~current coverage.

blueyed · 2019-02-27T17:16:46Z

The diff status is pretty much expected, since I've added comments where coverage regressed due to #4839.

nicoddemus · 2019-02-27T17:18:56Z

codecov/patch link still times out, with only 5 uploads now.. codecov.io/gh/pytest-dev/pytest/commit/11b531be0a38e3c9ce9e43e52fbc3c56fb72b6a5/build
So the base might also be relevant here.. :/

Not sure how to proceed then. Can we reach out to them perhaps? Or even drop the current base and start fresh, if that's the issue?

blueyed · 2019-02-27T17:23:00Z

I'm slacking with them since September last year about this at least.. :/

I will compare this manually again, clean it up, and then we should merge it.

I will bring back the jobs as before, but only ~5 then with coverage reporting.

I also plan to e.g. replace py37 with py37-pexpect (to get one less job) etc.

nicoddemus · 2019-02-27T17:25:32Z

I'm slacking with them since September last year about this at least.. :/

Bummer. They have no idea why our coverage report view times out?

blueyed · 2019-02-27T17:27:48Z

Yeah, not really apparently.. the suggested it might be flags, or combinations in general.
I've also seen it with other projects though (awesome/awesomeWM), where using less flags appeared to help.. there are less jobs though.

blueyed · 2019-02-27T17:30:30Z

@nicoddemus
FWIW I could invite you on the slack conversation they've created with me, but it would not really help I guess.

nicoddemus · 2019-02-27T17:43:08Z

Yeah, I agree.

Feel free to merge this then when you clean it up and deem it good enough.

This should bring coverage back that got missing with 9dcd6f2. 1. add PYTEST_COVERAGE to py37-pexpect,… job For `_pytest.debugging`, `Testdir.spawn`, `_pytest.terminal`. 2. re-add py37-xdist macos job This covers e.g. `_disable_output_capturing_for_darwin` again. NOTES: I have compared the coverage manually on Codecov, e.g. for `_pytest.debugging`: Before: https://codecov.io/gh/pytest-dev/pytest/src/ee62674322206299e0caaa36c172d9bb63feeaaf/src/_pytest/debugging.py After: https://codecov.io/gh/pytest-dev/pytest/src/9dcd6f2a87650f31f5b3b754e38a1cf4d6a14fe4/src/_pytest/debugging.py

This reverts commit 9957183.

This reverts commit a850cd9.

blueyed · 2019-02-28T21:21:46Z

Nearly there - will hopefully finish it tomorrow or on the weekend then.

blueyed · 2019-03-01T12:30:30Z

With master being merged here this also now does not time out anymore on codecov.io.

nicoddemus · 2019-03-01T12:41:34Z

So in summary flags were to blame then?

blueyed · 2019-03-01T12:54:21Z

So in summary flags were to blame then?

Looks like it, yes.

nicoddemus · 2019-03-01T12:58:34Z

Good, I'm sure they would like to know that. 👍

blueyed · 2019-03-01T13:06:00Z

It is something they suspect already. But it is not the only factor apparently.

This should bring coverage back that got missing with 9dcd6f2. Continuation of pytest-dev#4839 / pytest-dev#4846.

blueyed · 2019-03-01T14:38:49Z

Continued / restarted in #4865.

This should bring coverage back that got missing with 9dcd6f2. Continuation of pytest-dev#4839 / pytest-dev#4846.

This brings coverage back that got missing with 9dcd6f2. Continuation of pytest-dev#4839 / pytest-dev#4846.

blueyed changed the title ~~ci: Travis: revisit/fix coverage reporting~~ WIP: ci: Travis: revisit/fix coverage reporting Feb 27, 2019

blueyed force-pushed the revisit-cov branch from 8db76ee to f18ff3d Compare February 27, 2019 15:40

blueyed mentioned this pull request Feb 27, 2019

ci: PYTEST_ADDOPTS=-vv #4851

Merged

nicoddemus approved these changes Feb 27, 2019

View reviewed changes

blueyed added 11 commits February 27, 2019 20:04

Azure: no coverage

45d6663

Azure: use xdist with py34,py35,py36

2cd9faa

TEMP: Travis: only coverage jobs

bdbe13f

TEMP: Azure: only single non-coverage job

6e07279

azure: py37-xdist with coverage

0af34ab

py27-xdist via Azure

17d2943

TODO/mark (un)covered code

b22de71

Azure: coverage with py35-xdist

e03b552

test_pdb_interaction_on_internal_error: pragma: no branch

4197252

Travis: py37-pexpect-xdist for coverage of flush

7359606

blueyed added 8 commits February 28, 2019 20:58

Azure: py36-xdist for _py36_windowsconsoleio_workaround

56358d9

Revert "ignore branch coverage with flush"

428e30b

This reverts commit 9957183.

TODO

75e88bc

Azure: py34-xdist

4361b3c

Revert "Travis: TOXENV=py27-lsof-nobyte"

0e4b355

This reverts commit a850cd9.

Azure: py27-nobyte

ef569a9

doc

f9f4614

Travis: remove py27-nobyte - covered by Azure

dd81bde

blueyed added 6 commits February 28, 2019 22:26

Azure: use existing py27-numpy/nobyte

67cc480

fixup! Azure: py34-xdist

699f3fc

Bring back non-coverage jobs

8867164

TEMP: only coverage jobs

13dde1a

TODO: check py27-trial

a4d82c1

Merge branch 'master' into revisit-cov

3d0ef12

blueyed added a commit to blueyed/pytest that referenced this pull request Mar 1, 2019

ci: revisit coverage reporting

24909d9

This should bring coverage back that got missing with 9dcd6f2. Continuation of pytest-dev#4839 / pytest-dev#4846.

blueyed mentioned this pull request Mar 1, 2019

ci: revisit coverage reporting #4865

Merged

blueyed closed this Mar 1, 2019

blueyed deleted the revisit-cov branch March 1, 2019 14:38

blueyed added a commit to blueyed/pytest that referenced this pull request Mar 1, 2019

ci: revisit coverage reporting

9766a9a

This should bring coverage back that got missing with 9dcd6f2. Continuation of pytest-dev#4839 / pytest-dev#4846.

blueyed added a commit to blueyed/pytest that referenced this pull request Mar 1, 2019

ci: revisit coverage reporting

cf933eb

This should bring coverage back that got missing with 9dcd6f2. Continuation of pytest-dev#4839 / pytest-dev#4846.

blueyed added a commit to blueyed/pytest that referenced this pull request Mar 4, 2019

ci: revisit coverage reporting

f94b7bb

This brings coverage back that got missing with 9dcd6f2. Continuation of pytest-dev#4839 / pytest-dev#4846.

blueyed added a commit to blueyed/pytest that referenced this pull request Mar 5, 2019

ci: revisit coverage reporting

66cc05a

This brings coverage back that got missing with 9dcd6f2. Continuation of pytest-dev#4839 / pytest-dev#4846.

blueyed added a commit to blueyed/pytest that referenced this pull request Mar 5, 2019

ci: revisit coverage reporting

8481e43

This brings coverage back that got missing with 9dcd6f2. Continuation of pytest-dev#4839 / pytest-dev#4846.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: ci: Travis: revisit/fix coverage reporting #4846

WIP: ci: Travis: revisit/fix coverage reporting #4846

blueyed commented Feb 27, 2019

codecov bot commented Feb 27, 2019 •

edited

Loading

blueyed commented Feb 27, 2019

blueyed commented Feb 27, 2019

blueyed commented Feb 27, 2019

blueyed commented Feb 27, 2019 •

edited

Loading

nicoddemus commented Feb 27, 2019

blueyed commented Feb 27, 2019

blueyed commented Feb 27, 2019

blueyed commented Feb 27, 2019

nicoddemus commented Feb 27, 2019

blueyed commented Feb 27, 2019

nicoddemus commented Feb 27, 2019

blueyed commented Feb 27, 2019

blueyed commented Feb 27, 2019

nicoddemus commented Feb 27, 2019

blueyed commented Feb 28, 2019

blueyed commented Mar 1, 2019

nicoddemus commented Mar 1, 2019

blueyed commented Mar 1, 2019

nicoddemus commented Mar 1, 2019

blueyed commented Mar 1, 2019

blueyed commented Mar 1, 2019

WIP: ci: Travis: revisit/fix coverage reporting #4846

WIP: ci: Travis: revisit/fix coverage reporting #4846

Conversation

blueyed commented Feb 27, 2019

codecov bot commented Feb 27, 2019 • edited Loading

Codecov Report

blueyed commented Feb 27, 2019

blueyed commented Feb 27, 2019

blueyed commented Feb 27, 2019

blueyed commented Feb 27, 2019 • edited Loading

nicoddemus commented Feb 27, 2019

blueyed commented Feb 27, 2019

blueyed commented Feb 27, 2019

blueyed commented Feb 27, 2019

nicoddemus commented Feb 27, 2019

blueyed commented Feb 27, 2019

nicoddemus commented Feb 27, 2019

blueyed commented Feb 27, 2019

blueyed commented Feb 27, 2019

nicoddemus commented Feb 27, 2019

blueyed commented Feb 28, 2019

blueyed commented Mar 1, 2019

nicoddemus commented Mar 1, 2019

blueyed commented Mar 1, 2019

nicoddemus commented Mar 1, 2019

blueyed commented Mar 1, 2019

blueyed commented Mar 1, 2019

codecov bot commented Feb 27, 2019 •

edited

Loading

blueyed commented Feb 27, 2019 •

edited

Loading