Fix #762: PyNEST test result parsing confused by NEST output #788

gtrensch · 2017-07-19T06:53:39Z

This PR installs the nosetests capturestderr plugin and corrcets some of the nosetests log messages. For more details please see issue #762.

Silmathoron

The PR is straightforward enough, I just have a few questions, but I don't think there are any problems with the changes.

Silmathoron · 2017-07-26T01:04:00Z

.travis.yml

  - sudo apt-get install -y python-nose
+  - git clone https://github.com/sio2project/nose-capturestderr.git capturestderr


Given how small this plugin is, do you think it would be possible to include this in the nest repo directly after asking the guy about the license? It would avoid problems if the repo was moved or closed...

This would indeed be an option. There is also hope that the stderr-plugin will be part of nosetests itself. Unclear is also whether we stay with nosetests or move to another unit test framework.

@gtrensch This solution will only work on Travis and makes us dependent on yet another external repository. Users running the testsuite would need to install the plugin manually, complicating things. See also my comment on the issue.

@gtrensch : Have you tried to use the pypi package for nose-capturestderr 1.2
https://pypi.python.org/pypi/nose-capturestderr/1.2

Silmathoron · 2017-07-26T01:05:13Z

pynest/nest/tests/test_aeif_lsodar.py

-        '''
+    def testClosenessNestLSODAR(self):
+
+        # Compare models to the LSODAR implementation.


Why would you remove standard docstrings to replace them with comments? Does this have to do with the docstrings being printed somewhere during the test? I'm confused...

If present, nostests takes the first line of the docstring for reporting on the test cases. This led to something like this: The models should behave as iaf_cond_* if a == 0., b == 0. and ... ok
Most of the test cases make no use of a docstring. In that case the function name is reported. Hence I turned the docstring into line comments to make it clear and consistent.

@gtrensch Tests should be commented using docstrings and the first line should give a concise description of the test. We haven't been particularly systematic about this, but changing docstrings to comments, is in my opinion the wrong way to go. In this particular case, the text is also fine as a docstring.

@heplesser using the docstring here was one of the confusions. As stated above, nosetests takes the first line, and only the first line of the docstring, to report on the test. I agree, well written docstrings are good coding practice. Just a few test cases make use of it. I'm not sure if it should be of the scope of this PR to do this refactoring task here.

Silmathoron

Apart from the question regarding what we should do with capturestderr, I'm fine with this PR.
I think we should also discuss that in the context of #761 and check whether this is not already implemented in Nose2 (we are using pretty dated stuff, here)

heplesser · 2017-07-31T12:46:08Z

pynest/nest/tests/test_aeif_lsodar.py

-        '''
+    def testClosenessNestLSODAR(self):
+
+        # Compare models to the LSODAR implementation.


@gtrensch Tests should be commented using docstrings and the first line should give a concise description of the test. We haven't been particularly systematic about this, but changing docstrings to comments, is in my opinion the wrong way to go. In this particular case, the text is also fine as a docstring.

heplesser · 2017-07-31T12:47:53Z

pynest/nest/tests/test_aeif_lsodar.py

+    def testIafBehaviour(self):
+
+        # The models should behave as iaf_cond_* if a == 0., b == 0. and
+        # Delta_T == 0.


This should remain a docstring, but with a better text, e.g.

""" Ensure consistency with iaf_cond_* models. The models shall behave as iaf_cond_* if a == b == Delta_T == 0. """

@heplesser using the docstring here was one of the confusions. As stated above, nosetests takes the first line, and only the first line of the docstring, to report on the test. I agree, well written docstrings are good coding practice. Just a few test cases make use of it. I'm not sure if it should be of the scope of this PR to do this refactoring task here.

I think I misunderstood. So even if nosetest only outputs a single line that still breaks our output parsing? I had though/hoped that our parsing would work as long as nose did no insert any line breaks. We should discuss how to proceed in the next VC.

@heplesser I might be wrong but I think the problem had nothing to do with line breaks contained in the docstring. Nest's outpout to stdout/stderr just simply comes in between, the report on the unit test printed by nosetests and the "... ok" string at the end of single test run. The line breaks came from the Nest output.
If the first line of the docstring would contain a meaningful test description, things are perfect. We have > 500 nose-based unit tests (I believe) where most of it do not make use of a docstring. There is some effort adding docstrings to all tests.
.. but yes, let's discuss this in the VC.

heplesser · 2017-07-31T12:51:03Z

.travis.yml

  - sudo apt-get install -y python-nose
+  - git clone https://github.com/sio2project/nose-capturestderr.git capturestderr


@gtrensch This solution will only work on Travis and makes us dependent on yet another external repository. Users running the testsuite would need to install the plugin manually, complicating things. See also my comment on the issue.

gtrensch · 2017-07-31T13:58:02Z

@heplesser indeed I'm also not happy with this approach. On the other hand we do similar things in the PIP Cython and Music install sections. I see this as an indermediate solution to fix the bug. If we move to nose2 or perhaps pytest this would be gone.

lekshmideepu · 2017-08-03T11:42:03Z

.travis.yml

  - sudo apt-get install -y python-nose
+  - git clone https://github.com/sio2project/nose-capturestderr.git capturestderr


@gtrensch : Have you tried to use the pypi package for nose-capturestderr 1.2
https://pypi.python.org/pypi/nose-capturestderr/1.2

gtrensch · 2017-08-03T12:05:33Z

@lekshmideepu We can also download and install the tar file but isn't it the same source?

Silmathoron · 2017-08-03T12:58:03Z

It think it is not: using pip to access Pypi means accessing a "repository of software for the Python programming language", which means that the package is supposed to be a lot more stable in time, whatever happens to the GitHub repo.

gtrensch · 2017-08-03T14:40:31Z

@lekshmideepu @Silmathoron Thanks, I got the point. Installing from the Python package index means you can simply use "pip install". I didn't know this. This simplifies things a lot and as a nice side effect the plugin also appears in the list of the plugins installed.

lekshmideepu · 2017-08-04T12:39:27Z

@gtrensch : Yes, @Silmathoron is right. Pypi packages are supposed to be stable when compared to installing a package from a GitHub repo.
You could see here the following:
Author: The SIO2 Project Team
Home Page: http://github.com/sio2project/nose-capturestderr

lekshmideepu · 2017-08-04T12:47:17Z

@gtrensch : Plugin capturestderr was also shown in the earlier case when you were installing from the GitHub repo.

gtr added 2 commits July 18, 2017 17:26

Issue762: install the nosetests capturestderr plugin

365eeb8

Issue762: make neosetests output consistent across test cases

ad16987

terhorstd requested review from lekshmideepu and Silmathoron July 24, 2017 09:51

Silmathoron reviewed Jul 26, 2017

View reviewed changes

Silmathoron approved these changes Jul 26, 2017

View reviewed changes

Silmathoron mentioned this pull request Jul 26, 2017

Decide on one Python test approach for NEST #761

Closed

heplesser requested changes Jul 31, 2017

View reviewed changes

lekshmideepu suggested changes Aug 3, 2017

View reviewed changes

install the nose-acpturestderr plugin from the python package index

15aeb28

heplesser approved these changes Aug 7, 2017

View reviewed changes

heplesser added this to the NEST 2.12.1 milestone Aug 21, 2017

heplesser merged commit f2304c9 into nest:master Aug 21, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix #762: PyNEST test result parsing confused by NEST output #788

Fix #762: PyNEST test result parsing confused by NEST output #788

gtrensch commented Jul 19, 2017

Silmathoron left a comment

Silmathoron Jul 26, 2017

gtrensch Jul 26, 2017

heplesser Jul 31, 2017

lekshmideepu Aug 3, 2017

Silmathoron Jul 26, 2017

gtrensch Jul 26, 2017

heplesser Jul 31, 2017

gtrensch Jul 31, 2017

Silmathoron left a comment •

edited

Loading

heplesser Jul 31, 2017

heplesser Jul 31, 2017

gtrensch Jul 31, 2017

heplesser Jul 31, 2017

gtrensch Aug 3, 2017

heplesser Jul 31, 2017

gtrensch commented Jul 31, 2017

lekshmideepu Aug 3, 2017

gtrensch commented Aug 3, 2017

Silmathoron commented Aug 3, 2017 •

edited

Loading

gtrensch commented Aug 3, 2017

lekshmideepu commented Aug 4, 2017

lekshmideepu commented Aug 4, 2017

		- sudo apt-get install -y python-nose
		- git clone https://github.com/sio2project/nose-capturestderr.git capturestderr

Fix #762: PyNEST test result parsing confused by NEST output #788

Fix #762: PyNEST test result parsing confused by NEST output #788

Conversation

gtrensch commented Jul 19, 2017

Silmathoron left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Silmathoron left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gtrensch commented Jul 31, 2017

Choose a reason for hiding this comment

gtrensch commented Aug 3, 2017

Silmathoron commented Aug 3, 2017 • edited Loading

gtrensch commented Aug 3, 2017

lekshmideepu commented Aug 4, 2017

lekshmideepu commented Aug 4, 2017

Silmathoron left a comment •

edited

Loading

Silmathoron commented Aug 3, 2017 •

edited

Loading