TEST: Parallelize pytest

[effigies]

Was curious how much parallelizing could squeeze down pytest runtime, acknowledging that it's already not super long (~10 minutes out of a 40 minute build and test cycle).

This led down a doctest-shaped rabbit hole, because some workers were running the module-level doctests that changed directories and others were running the function-level doctests, and the context wasn't being shared so there were a couple stochastic crashes each build. So I figured out how to set a fixture that would chdir into the testing/data directory on doctests and only doctests. This seems cleaner than the old way of doing it anyway.

Changes proposed in this pull request

• Add pytest-xdist to testing dependencies
• Add -n auto to pytest config
• Add in_testing fixture to chdir for doctests
• Remove old header chdir boilerplate

[oesteban]

On my experience those two cores will perform as just one....

[effigies]

Well, looks like we're dropping from 6:30 to about 4:15, so about a 33% speedup, which isn't too bad for 2 cores.

[djarecka]

I think nipype.test() won't work.

also, one of the reason why I didn't remove imports etc. from files to auto-fixture was that i thought it will be harder for users to follow these examples. Not sure, how important it is..

[effigies]

I think nipype.test() won't work.

Yeah, I get one failure, but I'm not sure if that's related. I've never tried testing that way. I'll check on master.

also, one of the reason why I didn't remove imports etc. from files to auto-fixture was that i thought it will be harder for users to follow these examples. Not sure, how important it is..

Doesn't seem super useful to me, since those docstrings are never rendered in the published docs. But I'm okay being overruled here.

[satra]

the docstrings are meant to guide individuals - they are not meant to execute a real computation. so i'm +1 for fixture as long as it doesn't get in the way of the docstrings (which at least in this PR it doesn't look like).

[djarecka]

@effigies - i also don't usually test like that, but @satra always wanted to include this option, so users can easily test nipype. I don't remember right now (would have to read again and check my previous PR), but it is related how pytest searching for root directory etc. Never fully understood all details..

regarding docstrings, I just wanted explain my point, I thought that they should be design in a way that users can search the file, copy-paste and try to execute

[effigies]

Okay, the failure introduced by this in nipype.test() is in the following test:

nipype/nipype/pipeline/plugins/tests/test_callback.py

Lines 49 to 67 in 6ca791d

	def test_callback_exception(tmpdir):
	tmpdir.chdir()
	so = Status()
	wf = pe.Workflow(name='test', base_dir=tmpdir.strpath)
	f_node = pe.Node(
	niu.Function(function=bad_func, input_names=[], output_names=[]),
	name='f_node')
	wf.add_nodes([f_node])
	wf.config['execution'] = {'crashdump_dir': wf.base_dir}
	try:
	wf.run(plugin="Linear", plugin_args={'status_callback': so.callback})
	except:
	pass
	assert len(so.statuses) == 2
	for (n, s) in so.statuses:
	assert n.name == 'f_node'
	assert so.statuses[0][1] == 'start'
	assert so.statuses[1][1] == 'exception'

The failing line is:

nipype/nipype/pipeline/plugins/tests/test_callback.py

Line 63 in 6ca791d

assert len(so.statuses) == 2

Don't see what should be affecting it, immediately.

[djarecka]

@effigies sorry, I think this time it's not related to directory structure, you didn't change anything with it, but doesn't like the flag -n auto. do you need it?

[djarecka]

@satra ok

[djarecka]

(almost) all tests are passing on my OSX when I removed -n auto from pytest.ini

• almost because I've recreated the error in test_core.py mentioned in Matlab command cannot be found #2436

[effigies]

Oh, I see. We were overwriting some of what was in the pytest.ini. For consistency, I dropped most of that to run the same tests inside as outside, but added arguments for disabling both doctests and parallelism.

The 3.4 and 3.5 issue that's showing up on Travis is that dictionaries don't have guaranteed order pre-3.6. See pytest-dev/pytest#1075. Someone there suggests adding pytest-env, but I'd rather not add yet another dependency if I don't have to.

[effigies]

Went ahead and used pytest-env because I couldn't come up with anything better without wasting more effort than avoiding it was worth. This is ready for review.