different behavior for pytest xfail in 2.9.2 version and 3.0.6

[SergeyPirogov]

I noticed that pytest.mark.xfail works differently on versions of pytest mentioned in subj.

Example: I have two tests

@pytest.mark.xfail(reason='ololo')
def test_can_xfail():
    assert True


@pytest.mark.xfail(reason='ololo')
def test_Y():
    assert False

When I run them with pytest==2.9.2 version first test fails and second is skipped.
But when I run same test with pytest==3.0.6 first test is passing and second is skipped.

[The-Compiler]

I see an XPASS and xfail as expected:

========================================== test session starts ==========================================
platform linux -- Python 3.6.0, pytest-2.9.2, py-1.4.32, pluggy-0.3.1 -- /home/florian/tmp/.venv/bin/python3
cachedir: .cache
rootdir: /home/florian/tmp, inifile: 
collected 2 items 

x.py::test_can_xfail XPASS
x.py::test_Y xfail

================================= 1 xfailed, 1 xpassed in 0.00 seconds ==================================

========================================== test session starts ==========================================
platform linux -- Python 3.6.0, pytest-3.0.6, py-1.4.32, pluggy-0.4.0 -- /home/florian/tmp/.venv/bin/python3
cachedir: .cache
rootdir: /home/florian/tmp, inifile: 
collected 2 items 

x.py::test_can_xfail XPASS
x.py::test_Y xfail

================================= 1 xfailed, 1 xpassed in 0.02 seconds ==================================

[SergeyPirogov]

Well maybe this can help:

def test_xpass(report_for):
        report = report_for("""
        import pytest
    
        @pytest.mark.xfail(reason='ololo')
        def test_Y():
            assert True
        """)
    
        assert_that(report, has_error(message='xpassed',
>                                     trace='ololo'))
E       AssertionError: 
E       Expected: an object with a property '{}test-cases' matching an object with a property 'test-case' matching (an object with a property 'attrib' matching a dictionary containing ['status': 'failed'] and an object with a property 'failure' matching (an object with a property 'message' matching 'xpassed' and an object with a property 'stack-trace' matching an object with str 'ololo'))
E            but: property '{}test-cases' property 'test-case' an object with a property 'attrib' matching a dictionary containing ['status': 'failed'] property 'attrib' was <{'start': '1486636993426', 'status': 'passed', 'stop': '1486636993428'}>

When I run the same test with pytest==2.9.2 assertion doesn't happen

[The-Compiler]

What's report_for? Is this related to JUnitXML? If so, that was a bugfix, see #1546.

[SergeyPirogov]

@The-Compiler yes, it relates to xml generation for allure reports. Is this fix already included in v 3.0.6?
Could you please suggest, what should I do to fix such behavior for this test?

[The-Compiler]

The fix is that xpass now shows up as passed instead of failed, because that's the case elsewhere in pytest too.

You can use xfail with strict=True, but what the right solution is really depends what the test is testing.

Either way, this doesn't look like a pytest issue, so I'm closing this.