Unit tests fail

[stuarteberg]

Apparently the unit tests in this repo used to pass, but now I'm seeing failures. Are we concerned about that?

Here's the test log I get. (The tests take a really long time to run, so I stopped them early, hence the ^C from when I hit Ctrl+C.)

ping: @akreshuk @FabianIsensee

+ nosetests .
..........F..FF^C
======================================================================
FAIL: test_initial_set (test_wrapper_selection.TestWrapperMethod)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/miniconda/conda-bld/work/tests/test_wrapper_selection.py", line 96, in test_initial_set
    53]))
AssertionError: Items in the first set but not the second:
35
36
37
38
45
51
20
54
23
26
Items in the second set but not the first:
33
43
53
-------------------- >> begin captured logging << --------------------
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 21])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 0.915250, overall best score is 0.749500
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 21])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 21, 27])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.001000, overall best score is 0.915250
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 21, 27])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 35, 21, 27])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.044750, overall best score is 1.001000
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 35, 21, 27])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 42, 35, 21, 27])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.066500, overall best score is 1.044750
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 42, 35, 21, 27])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 42, 18, 35, 21, 27])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.084250, overall best score is 1.066500
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 42, 18, 35, 21, 27])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 42, 45, 18, 35, 21, 27])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.118000, overall best score is 1.084250
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 21, 27, 35, 42, 45])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 21, 26, 27, 35, 42, 45])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.113750, overall best score is 1.118000
ilastik_feature_selection.wrapper_feature_selection: INFO: best set has not changed in 1 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 21, 27, 35, 42, 45])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 21, 26, 27, 35, 37, 42, 45])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.127500, overall best score is 1.118000
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 21, 26, 27, 35, 37, 42, 45])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 21, 26, 27, 35, 37, 42, 45, 51])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.123250, overall best score is 1.127500
ilastik_feature_selection.wrapper_feature_selection: INFO: best set has not changed in 1 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 21, 26, 27, 35, 37, 42, 45])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 21, 26, 27, 35, 37, 42, 45, 51, 54])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.135000, overall best score is 1.127500
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 21, 26, 27, 35, 37, 42, 45, 51, 54])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 21, 26, 27, 35, 36, 37, 42, 45, 51, 54])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.130750, overall best score is 1.135000
ilastik_feature_selection.wrapper_feature_selection: INFO: best set has not changed in 1 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 21, 26, 27, 35, 37, 42, 45, 51, 54])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 20, 21, 26, 27, 35, 36, 37, 42, 45, 51, 54])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.140500, overall best score is 1.135000
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 20, 21, 26, 27, 35, 36, 37, 42, 45, 51, 54])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 20, 21, 23, 26, 27, 35, 36, 37, 42, 45, 51, 54])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.134250, overall best score is 1.140500
ilastik_feature_selection.wrapper_feature_selection: INFO: best set has not changed in 1 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 20, 21, 26, 27, 35, 36, 37, 42, 45, 51, 54])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 20, 21, 23, 26, 27, 35, 36, 37, 38, 42, 45, 51, 54])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.128000, overall best score is 1.140500
ilastik_feature_selection.wrapper_feature_selection: INFO: best set has not changed in 2 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 20, 21, 26, 27, 35, 36, 37, 42, 45, 51, 54])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 20, 21, 23, 26, 27, 30, 35, 36, 37, 38, 42, 45, 51, 54])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.143750, overall best score is 1.140500
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 20, 21, 23, 26, 27, 30, 35, 36, 37, 38, 42, 45, 51, 54])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 20, 21, 23, 26, 27, 30, 35, 36, 37, 38, 42, 45, 51, 54, 58])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.125500, overall best score is 1.143750
ilastik_feature_selection.wrapper_feature_selection: INFO: best set has not changed in 1 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 20, 21, 23, 26, 27, 30, 35, 36, 37, 38, 42, 45, 51, 54])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 20, 21, 23, 26, 27, 30, 31, 35, 36, 37, 38, 42, 45, 51, 54, 58])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.123250, overall best score is 1.143750
ilastik_feature_selection.wrapper_feature_selection: INFO: best set has not changed in 2 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 20, 21, 23, 26, 27, 30, 35, 36, 37, 38, 42, 45, 51, 54])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 20, 21, 23, 26, 27, 30, 31, 35, 36, 37, 38, 42, 44, 45, 51, 54, 58])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.127000, overall best score is 1.143750
ilastik_feature_selection.wrapper_feature_selection: INFO: best set has not changed in 3 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best feature set set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 20, 21, 23, 26, 27, 30, 35, 36, 37, 38, 42, 45, 51, 54])
ilastik_feature_selection.wrapper_feature_selection: INFO: curr set is now: set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 18, 20, 21, 23, 26, 27, 30, 31, 35, 36, 37, 38, 39, 42, 44, 45, 51, 54, 58])
ilastik_feature_selection.wrapper_feature_selection: INFO: local best score is 1.130750, overall best score is 1.143750
ilastik_feature_selection.wrapper_feature_selection: INFO: best set has not changed in 4 iterations
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_wrapper_BFS_adv (test_wrapper_selection.TestWrapperMethod)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/miniconda/conda-bld/work/tests/test_wrapper_selection.py", line 33, in test_wrapper_BFS_adv
    self.assertEqual(a[1], 1.2064999999999999)
AssertionError: 1.2090000000000001 != 1.2065
-------------------- >> begin captured logging << --------------------
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([]) with score -inf
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([20])   score: 0.669750
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best node thanks to compound operators
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best node thanks to compound operators
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best node thanks to compound operators
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best node thanks to compound operators
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best node thanks to compound operators
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best node thanks to compound operators
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best node thanks to compound operators
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best node thanks to compound operators
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best node thanks to compound operators
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 42, 44, 46, 19, 20, 21, 54, 58, 29]) with score 1.147500
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([34, 42, 43, 44, 46, 19, 20, 21, 54, 58, 29])   score: 1.163250
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 42, 43, 44, 46, 19, 20, 21, 54, 58, 29]) with score 1.163250
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([34, 42, 43, 44, 46, 19, 20, 21, 54, 58, 27, 29])   score: 1.175000
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 42, 43, 44, 46, 19, 20, 21, 54, 58, 27, 29]) with score 1.175000
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([34, 42, 43, 44, 46, 20, 21, 54, 58, 27, 29])   score: 1.179250
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 42, 43, 44, 46, 20, 21, 54, 58, 27, 29]) with score 1.179250
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([34, 42, 43, 44, 46, 20, 21, 54, 58, 27, 29, 30])   score: 1.183000
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 42, 43, 44, 46, 20, 21, 54, 58, 27, 29, 30]) with score 1.183000
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([34, 42, 43, 44, 46, 20, 21, 54, 58, 27, 30])   score: 1.185250
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 42, 43, 44, 46, 20, 21, 54, 58, 27, 30]) with score 1.185250
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([34, 42, 43, 44, 46, 20, 21, 54, 58, 27, 26, 30])   score: 1.193000
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 42, 43, 44, 46, 20, 21, 54, 58, 27, 26, 30]) with score 1.193000
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([34, 38, 42, 43, 44, 46, 20, 21, 54, 58, 27, 26, 30])   score: 1.194750
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 38, 42, 43, 44, 46, 20, 21, 54, 58, 27, 26, 30]) with score 1.194750
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([34, 38, 42, 43, 44, 46, 20, 21, 54, 24, 58, 27, 26, 30])   score: 1.200500
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 38, 42, 43, 44, 46, 20, 21, 54, 24, 58, 27, 26, 30]) with score 1.200500
ilastik_feature_selection.wrapper_feature_selection: INFO: The feature set has not been updated in the last 1 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 38, 42, 43, 44, 46, 20, 21, 54, 24, 58, 27, 26, 30]) with score 1.200500
ilastik_feature_selection.wrapper_feature_selection: INFO: The feature set has not been updated in the last 2 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 38, 42, 43, 44, 46, 20, 21, 54, 24, 58, 27, 26, 30]) with score 1.200500
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([34, 42, 43, 44, 20, 21, 54, 62, 58, 27, 26, 30])   score: 1.209000
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 42, 43, 44, 20, 21, 54, 62, 58, 27, 26, 30]) with score 1.209000
ilastik_feature_selection.wrapper_feature_selection: INFO: The feature set has not been updated in the last 1 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 42, 43, 44, 20, 21, 54, 62, 58, 27, 26, 30]) with score 1.209000
ilastik_feature_selection.wrapper_feature_selection: INFO: The feature set has not been updated in the last 2 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 42, 43, 44, 20, 21, 54, 62, 58, 27, 26, 30]) with score 1.209000
ilastik_feature_selection.wrapper_feature_selection: INFO: The feature set has not been updated in the last 3 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 42, 43, 44, 20, 21, 54, 62, 58, 27, 26, 30]) with score 1.209000
ilastik_feature_selection.wrapper_feature_selection: INFO: The feature set has not been updated in the last 4 iterations
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_wrapper_BFS_no_adv (test_wrapper_selection.TestWrapperMethod)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/miniconda/conda-bld/work/tests/test_wrapper_selection.py", line 21, in test_wrapper_BFS_no_adv
    self.assertEqual(a[1], 1.1995)
AssertionError: 1.2075 != 1.1995
-------------------- >> begin captured logging << --------------------
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([]) with score -inf
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([20])   score: 0.669750
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([20]) with score 0.669750
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([20, 30])   score: 0.761500
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([20, 30]) with score 0.761500
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([43, 20, 30])   score: 0.881250
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([43, 20, 30]) with score 0.881250
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([34, 43, 20, 30])   score: 1.003000
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 43, 20, 30]) with score 1.003000
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([27, 34, 43, 20, 30])   score: 1.062750
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([27, 34, 43, 20, 30]) with score 1.062750
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([34, 36, 43, 20, 27, 30])   score: 1.110500
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 36, 43, 20, 27, 30]) with score 1.110500
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([34, 27, 36, 21, 20, 43, 30])   score: 1.128250
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 27, 36, 21, 20, 43, 30]) with score 1.128250
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([34, 43, 36, 21, 20, 27, 30, 5])    score: 1.164000
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 43, 36, 21, 20, 27, 30, 5]) with score 1.164000
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([34, 36, 5, 10, 43, 20, 21, 27, 30])    score: 1.193750
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 36, 5, 10, 43, 20, 21, 27, 30]) with score 1.193750
ilastik_feature_selection.wrapper_feature_selection: INFO: updated best feature set: set([34, 36, 5, 10, 43, 20, 21, 23, 27, 30])    score: 1.207500
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 36, 5, 10, 43, 20, 21, 23, 27, 30]) with score 1.207500
ilastik_feature_selection.wrapper_feature_selection: INFO: The feature set has not been updated in the last 1 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 36, 5, 10, 43, 20, 21, 23, 27, 30]) with score 1.207500
ilastik_feature_selection.wrapper_feature_selection: INFO: The feature set has not been updated in the last 2 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 36, 5, 10, 43, 20, 21, 23, 27, 30]) with score 1.207500
ilastik_feature_selection.wrapper_feature_selection: INFO: The feature set has not been updated in the last 3 iterations
ilastik_feature_selection.wrapper_feature_selection: INFO: current best set: set([34, 36, 5, 10, 43, 20, 21, 23, 27, 30]) with score 1.207500
ilastik_feature_selection.wrapper_feature_selection: INFO: The feature set has not been updated in the last 4 iterations
--------------------- >> end captured logging << ---------------------

Traceback (most recent call last):
  File "/miniconda/bin/conda-build", line 5, in <module>
----------------------------------------------------------------------
Ran 16 tests in 906.054s

FAILED (failures=3)

[FabianIsensee]

Hi,
these tests run feature selection and compare the result they obtain
with the feature set that I got when I created these tests quite a while
ago. Since the random forest is non-deterministic, feature sets may vary
between runs in real applications. I tried to prevent this intrinsic
variation by setting the random_state of the random forest to a fixed
value. However, if the implementation of the random forest changed in
the meantime (new sklearn version?) or something else is shady with the
rng then this may be the reason the tests fail.
What I can see from the log is that the actual functionality of
test_initial_set is working properly, i.e. the initially set features
(features 0-9) are still included in the final set.
Cheers,
Fabian

edit: removed cited text for readability

[FabianIsensee]

What I see from the logs of the other two tests (test_wrapper_BFS_adv
and test_wrapper_BFS_no_adv) is a simple assertion error: The score of
the computed feature set is not equal to the score I obtained way back.
I guess the test aborts before comparing the obtained feature sets with
the 'ground truth' which is why it does not tell us that the sets are
different. The reason for the failure on these tests is most certainly
the same one as before.

edit: removed cited text for readability

[stuarteberg]

OK, that makes sense. I upgraded to scikit-learn 0.17.1. Is there an automated way to generate the new "expected" results and update the test suite?

[FabianIsensee]

There is no automated way but you should be able to run the code from
the tests and get the result it expects (see
https://github.com/ilastik/ilastik-feature-selection/blob/master/tests/test_wrapper_selection.py).
Just open an IPython shell and run what you need. It should not be much code
Here is an example for test_wrapper_BFS_no_adv

/import sys//
//import utils//
//import numpy as np//
//import ilastik_feature_selection//
//import sklearn//
//import unittest//
//import os//
//import logging//
//
//rf = sklearn.ensemble.RandomForestClassifier(random_state = 14271, 
n_estimators = 10)//
//X = np.load(os.path.dirname(os.path.realpath(**file**)) 
+"/digits_data.npy")//
//Y = np.load(os.path.dirname(os.path.realpath(**file**)) 
+"/digits_target.npy")//
//
//eval_fct = 
ilastik_feature_selection.wrapper_feature_selection.EvaluationFunction(rf, 
complexity_penalty=0.4)//
//feat_selector = 
ilastik_feature_selection.wrapper_feature_selection.WrapperFeatureSelection(X, 
Y,  eval_fct.evaluate_feature_set_size_penalty, method="BFS")//
//a = feat_selector.run(do_advanced_search=False)/

(make sure that the path to the .npy arrays is correct. You can find the
files here:
https://github.com/ilastik/ilastik-feature-selection/tree/master/tests)

The test expects:

/self.assertEqual(a[1], 1.1995)//
//self.assertEqual(set(a[0]), set([10, 13, 20, 29, 35, 37, 42, 44, 51, 
53]))/

These are the lines that need to be updated.

Cheers,

Fabian

edit: removed cited text for readability, added code formatting

[k-dominik]

So, I'm really unhappy with these tests, they take ages. Also it seems that @FabianIsensee suggests to update the tests by regenerating the results with the test-code? How can we then, be ever sure, that nothing has significantly changed? Right now the differences are striking!