ENH: Test full output and coverage

[larsoner]

Over in Sphinx-gallery for about a year we've been testing full builds of sphinx using Pytest:

https://github.com/sphinx-gallery/sphinx-gallery/blob/master/sphinx_gallery/tests/test_full.py

It's convenient because:

1. You can add assertions about the HTML that you get in pytest form.
2. You can easily interactively test by doing make clean; make html from within the numpydoc/tests/tinybuild directory and looking at the rendered HTML.
3. You can easily add more test cases by adding new things to be documented in the nd_test_mod module in tinybuild.

I've shown how to use this framework in some basic cases here. I've also added some assertions that are bad (marked by XXX) that #221 should fix (@thequackdaddy after this PR is merged and you rebase, you can change a couple of ins below to not ins to assert that the *args, **kwargs situation has been fixed).

This PR also attempts to enable codecov.

[larsoner]

Okay all green, ready for review/merge from my end. @thequackdaddy perhaps you could look?

[larsoner]

Looks like we are already at 93.4% coverage, not bad!

[larsoner]

@jnothman if you can take a look, it would be good to get in so that #221 can proceed.

[thequackdaddy]

@larsoner Sorry for my non-response! Busy week.

I took a look and I think its a good start. I'll totally confess you are clearly more up-to-speed with some of this than I am. I wasn't thinking about adding in a mini-sphinx build to test against, so I think it looks good. That looks impressive and 93.4% coverage is excellent.

Obviously, the maintainers would be a better reviewer than me though.

[jnothman]

I generally think this is an awesome improvement to the testing. Though I don't really like how the test fixtures, i.e. the numpydoc, is far from the assertions about it, and that at the same time the purpose of each test case is very mixed... I.e. the tests aren't very legible, which is not improved by the use of string matching over html.

I don't have any constructive suggestions at this point, and would be happy to see this merged.

[jnothman]

No licensing requirements?

[jnothman]

It took me a while to work out what nd is. Maybe expand it?

[jnothman]

Suggested change

	"""Numpdoc test module.
	"""Numpydoc test module.

[jnothman]

Suggested change

	def test_function(sphinx_app):
	def test_my_function(sphinx_app):

Otherwise this sounds too generic

[jnothman]

Suggested change

	def test_class(sphinx_app):
	def test_MyClass(sphinx_app):

[larsoner]

Though I don't really like how the test fixtures, i.e. the numpydoc, is far from the assertions about it, and that at the same time the purpose of each test case is very mixed... I.e. the tests aren't very legible, which is not improved by the use of string matching over html.

Basically the idea is to test that the end output (HTML) is as it should be. I agree that this is not so great for inspecting where in numpydoc the failures arose, but it is good for ensuring that we don't introduce failures in the final output (what is correct). Ideally tests for new functionality are built at the fine-grained level as encapsulated as possible (like what we have at master) to help with the first case, and not just at the full-sphinx-build level. But sometimes this is difficult.

Another way to look at it is that at the end of the day we want numpydoc to give us correct HTML outputs, so it's reasonable to check that the HTML output looks correct.

[larsoner]

Pushed a commit with the suggested changes, plus a tiny bit more explanatory text comments for the HTML assertions

[larsoner]

Since comments were addressed, this does not change any functionality (just expands our testing capabilities a bit and enables codecov), and @jnothman was +1 for merge I'll go ahead and put this in. Hopefully it helps #221!

[jnothman]

Thanks!