Tests silently consume AssertionError in Python

[FArekkusu]

AssertionError doesn't show up in the output when it's raised inside a function decorated by test.describe, test.it, or inside any function called by those. It correctly stops the execution, emits a non-zero exit code, and results in a test failure, but there's no error message. The issue is present both in Python 3.6 and Python 3.8.

---

This works:

import codewars_test as test

assert 0

@test.describe("T")
def _():
    @test.it("t")
    def _():
        test.pass_()

---

While any of these doesn't:

import codewars_test as test

@test.describe("T")
def _():
    
    assert 0
    
    @test.it("t")
    def _():
        test.pass_()

import codewars_test as test

@test.it("t")
def _():
    assert 0
    test.pass_()

import codewars_test as test

# whether this is defined in the test file or imported from the solution doesn't matter
def f():
    assert 0

@test.it("t")
def _():
    f()
    test.pass_()

[kazk]

The first one works because it doesn't involve this test framework at all. This test framework wasn't made to work with AssertionError (we did try to retrofit it, but it's not fully functional).

https://github.com/codewars/python-test-framework/blob/v0.1.0/codewars_test/test_framework.py is the deployed version.

[namedots]

Looks to me like it's supposed to catch AssertionException, which is defined at the top of the module and raised in expect but is not caught anywhere. Presumably an auto-complete failure.

from https://docs.codewars.com/languages/python/authoring/#calling-assertions :

One somewhat distinctive feature of Python assertions is that by default, a failed assertion does not cause a test case to fail early. It can lead to unexpected crashes when an actual value had already been asserted to be invalid, but the execution of the current test case was not stopped and following assertions continue to refer to it. This behavior can be overridden by passing the allow_raise=True argument to the assertion functions that support it.

allow_raise does not actually have this behaviour, but it WOULD if AssertionError was changed to AssertionException here:

python-test-framework/codewars_test/test_framework.py

Lines 112 to 113 in 072ae38

	except AssertionError as e:
	display('FAILED', str(e))

should likely be:

except AssertionException:
    pass

and allow_raise should actually be named abort_on_failure or some such. Its current behaviour does not match its name at all, nor does it match the above docs:

python-test-framework/codewars_test/test_framework.py

Lines 23 to 24 in 072ae38

	if allow_raise:
	raise AssertException(message)

(if it says ALLOW raise, then it should not be raising itself, it should ... ALLOW someone else to do it, except, that's probably not the intention of the flag, rather that seems to be aborting early -- that's the only reason why expect would raise - to abort the function - it's "allow", but from the wrong perspective, it's allowing expect to raise, and it does so to abort, and then there needs to be a matching catch in it -- which is this except AssertionError typo)