Add support for soft assertions #19

baev · 2017-03-01T13:34:52Z

No description provided.

scripnichenko · 2018-12-17T13:22:44Z

Hi @baev. Any plans to get it implemented? 💡

baev · 2018-12-17T15:53:03Z

@scripnichenko we are not using soft assertions in our code and you are the first one who are asking for such issue during last few years. Seems like everyone uses AssertJ instead of TestNG assertions this days. But I'm understand that issue is important for people who support old projects written years ago and have no possibility for refactoring.

BTW I have no idea how to fix this, some research work is required. Probably the easiest way to fix the issue add an aspect. If you have any ideas feel free to share.

Little-Elephant · 2019-02-12T18:44:26Z

I'll be the second one.

Lyisko · 2019-06-27T09:15:42Z

I'll be the third one.

vlayon · 2020-02-25T07:50:56Z

Hello there.
I want to ask also for implementing SoftAsserts in Allure. Are you going to do it?

..because now the bahavior is : current step, which has failed soft assert in it, does not fail. The case fails when assertAll method is called.

bereg2k · 2020-11-03T14:46:32Z

Yep, that's an important feature.

Guter1988 · 2021-01-19T15:25:02Z

I use it too.
And due to this, add attachments when it failed to find a better solution.

sskorol · 2021-01-19T20:44:06Z

Can anyone explain the expected behavior and attach the minimal code to reproduce the issue? As I don't really understand what kind of SA support is required.

Guter1988 · 2021-01-21T12:49:01Z

I will try from my point of usage:
When in the an of API (for example), I'm checking the JSON response, I need to test all fields that exist as expected.
I can run soft assertion or run a new test to validate one filed.
Or in UI - when by click on an element will be an open window with 3 parameters that I need to validate, I will use SA.

I hope it helped :)

sskorol · 2021-01-21T15:40:29Z

@guterscooter I know how SA works. I wonder what do you expect to see in Allure while using SA?

Guter1988 · 2021-01-24T12:07:35Z

@sskorol To see all parts where it failed.
I see only the first one, while the next can failed will not be shown at Allure.

twixdouble · 2021-02-17T12:23:32Z

@sskorol mark each failed step (and each of its ancestors) with red color, add assertion message within failed step content, add stack-trace within assertion error content, add summary error-message as test result, e.g. 'N assertions failed: <error 1> (expandable with stack-trace), <error 2> (expandable) ...

lbastil · 2021-11-09T12:29:16Z

@sskorol: I think in #529 everything is explained quite well:

What is the current behavior?
current step, which has failed soft assert in it, does not fail. The case fails when assertAll method is called.

What is the expected behavior?
hope it show failed in each step and make the test failed

Example + Expectation:

On a very general level an example would be:

public class MyTestClass {

        private SoftAssert softAssert;
    
        @BeforeMethod(alwaysRun = true)
        public void initialize(Method method) {
    
            softAssert = new SoftAssert();
        }
    
        @Test
        public void myComplexTest_1() {
    
            reusableTestcasePart_1();
            reusableTestcasePart_2();
            softAssert.assertAll();
        }
    
        @Step("...")
        protected void reusableTestcasePart_1() {
            // do something directly ...
    
            // use reusable Steps:
            testStep_1();
            testStep_2();
        }
    
        @Step("...")
        protected void reusableTestcasePart_2() {
            // do something directly ...
    
            // use reusable Steps:
            testStep_3();
            testStep_4();
        }
    
        @Step("...")
        protected void testStep_1() {
    
            boolean stepResult = true;// in real case check some sophisticated ui part
            // soft assertion
            softAssert.assertTrue(stepResult);
        }
    
        @Step("...")
        protected void testStep_2() {
    
            boolean stepResult = false;// in real case check some sophisticated ui part
            // soft assertion
            softAssert.assertTrue(stepResult);
        }
    
        @Step("...")
        protected void testStep_3() {
    
            boolean stepResult = false;// in real case check some sophisticated ui part
            // soft assertion
            softAssert.assertTrue(stepResult);
        }
    
        @Step("...")
        protected void testStep_4() {
    
            boolean stepResult = true;// in real case check some sophisticated ui part
            // soft assertion
            softAssert.assertTrue(stepResult);
        }
    }

One would expect in this case the following report result:

(test method) myComplexTest_1 : FAILED
- (step) reusableTestcasePart_1 : FAILED
  - (step) testStep_1 : SUCCESS
  - (step) testStep_2 : FAILED
- (step) reusableTestcasePart_2 : FAILED
  - (step) testStep_3 : FAILED
  - (step) testStep_4 : SUCCESS

But it is currently:

(test method) myComplexTest_1 : FAILED
- (step) reusableTestcasePart_1 : SUCCESS
  - (step) testStep_1 : SUCCESS
  - (step) testStep_2 : SUCCESS
- (step) reusableTestcasePart_2 : SUCCESS
  - (step) testStep_3 : SUCCESS
  - (step) testStep_4 : SUCCESS

Which is misleading and hard to review for the test team which evaluates the allure report details

lbastil · 2021-11-09T12:33:24Z

@sskorol : if I can I would try to support the feature enhancement. Can you give me some advise about entry points where to start? I guess there is a kind of post-processing for the final test / step state (updating the TestResult information).

deisenberg-remedy · 2023-06-12T18:57:35Z

What is the status on implementing Soft Assertion support as described above?

Phlegethonyarre · 2023-07-21T06:45:47Z

Hi guys. A little workaround here I created long time ago, this could be helpful for your own ideas, probably with aspects :)

public class StepUtil {

	private final List<Throwable> troubles = new ArrayList<>();
	private final List<String> causes = new ArrayList<>();

	public void runStep(Object name, Runnable runnable) {
		final String uuid = UUID.randomUUID().toString();
		final List<String> localCauses = new ArrayList<>();
		final List<Throwable> localExceptions = new ArrayList<>();

		try {
			Allure.getLifecycle().startStep(uuid, new StepResult().setName(name.toString()));
			runnable.run();
		} catch (Throwable e) {
			localCauses.add(String.format("\r\nFailed step name: %s", name.toString()));
			localExceptions.add(e);
		}

		if (!localExceptions.isEmpty()) {
			Allure.getLifecycle().updateStep(uuid, (step) -> step.setStatus(Status.FAILED));
			troubles.addAll(localExceptions);
			causes.addAll(localCauses);
		} else {
			Allure.getLifecycle().updateStep(uuid, (step) -> step.setStatus(Status.PASSED));
		}

		Allure.getLifecycle().updateStep(step -> {
			List<StepResult> stepResults = step.getSteps();
			if (stepResults.stream().anyMatch(stepResult -> stepResult.getStatus().equals(Status.FAILED))) {
				step.setStatus(Status.FAILED);
			} else if (stepResults.stream().anyMatch(stepResult -> stepResult.getStatus().equals(Status.BROKEN))) {
				step.setStatus(Status.BROKEN);
			}
		});

		Allure.getLifecycle().stopStep(uuid);
	}

	public void checkTroubles() {
		if (!troubles.isEmpty()) {
			throw new AssertJMultipleFailuresError(causes.toString(), troubles);
		}
	}

Achitheus · 2023-11-10T19:08:17Z

Hi guys. A little workaround here I created long time ago, this could be helpful for your own ideas, probably with aspects :)

public class StepUtil {

	private final List<Throwable> troubles = new ArrayList<>();
	private final List<String> causes = new ArrayList<>();

	public void runStep(Object name, Runnable runnable) {
		final String uuid = UUID.randomUUID().toString();
		final List<String> localCauses = new ArrayList<>();
		final List<Throwable> localExceptions = new ArrayList<>();

		try {
			Allure.getLifecycle().startStep(uuid, new StepResult().setName(name.toString()));
			runnable.run();
		} catch (Throwable e) {
			localCauses.add(String.format("\r\nFailed step name: %s", name.toString()));
			localExceptions.add(e);
		}

		if (!localExceptions.isEmpty()) {
			Allure.getLifecycle().updateStep(uuid, (step) -> step.setStatus(Status.FAILED));
			troubles.addAll(localExceptions);
			causes.addAll(localCauses);
		} else {
			Allure.getLifecycle().updateStep(uuid, (step) -> step.setStatus(Status.PASSED));
		}

		Allure.getLifecycle().updateStep(step -> {
			List<StepResult> stepResults = step.getSteps();
			if (stepResults.stream().anyMatch(stepResult -> stepResult.getStatus().equals(Status.FAILED))) {
				step.setStatus(Status.FAILED);
			} else if (stepResults.stream().anyMatch(stepResult -> stepResult.getStatus().equals(Status.BROKEN))) {
				step.setStatus(Status.BROKEN);
			}
		});

		Allure.getLifecycle().stopStep(uuid);
	}

	public void checkTroubles() {
		if (!troubles.isEmpty()) {
			throw new AssertJMultipleFailuresError(causes.toString(), troubles);
		}
	}

Maybe I'm wrong, but it looks like the run() method never throws errors because it has a soft assertion inside?

Achitheus · 2023-11-10T19:17:56Z

I did this for my needs:

public class AllureSelenideCustom {
    public static void markOuterStepAsFailedAndStop() {
        AllureLifecycle lifecycle = getLifecycle();
        lifecycle.updateStep(step -> step.setStatus(Status.FAILED));
        lifecycle.stopStep();
    }

    public static void stepSoftAssert(final String passMessage, final String failMessage,
                                      final Consumer<String> consumer, final boolean markStepAsFailed) {
        Allure.step(markStepAsFailed ? failMessage : passMessage
                , () -> {
                    consumer.accept(failMessage);
                    if (markStepAsFailed) {
                        markOuterStepAsFailedAndStop();
                    }
                });
    }
}

The bad new is you should pass to this method boolean markStepAsFailed. It means you should additionally ("manually") check result before use it.
Its ok for me in cases when I looking for "bad items" and pass as argument something like badItem.exists()

baev mentioned this issue Mar 1, 2017

The problem is TestNG and finding steps error installed using soft asserts allure-framework/allure1#700

Closed

baev added this to the AJava Release milestone Mar 14, 2017

baev added type:new feature Change that add something new for end users work:backlog No work on this issue at the moment theme:testng TestNG related issue labels Oct 31, 2018

baev changed the title ~~[TestNG] Add support for soft asserions~~ Add support for soft assertions Nov 12, 2018

lbastil mentioned this issue Nov 5, 2021

is any plan to support soft assertion? allure-framework/allure-python#529

Open

2 tasks

baev mentioned this issue Nov 10, 2022

Parent step is marked as passed with failed child step using AssertionMode = SOFT + .includeSelenideSteps(true) allure-framework/allure2#1765

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for soft assertions #19

Add support for soft assertions #19

baev commented Mar 1, 2017

scripnichenko commented Dec 17, 2018

baev commented Dec 17, 2018

Little-Elephant commented Feb 12, 2019

Lyisko commented Jun 27, 2019

vlayon commented Feb 25, 2020

bereg2k commented Nov 3, 2020

Guter1988 commented Jan 19, 2021

sskorol commented Jan 19, 2021

Guter1988 commented Jan 21, 2021

sskorol commented Jan 21, 2021

Guter1988 commented Jan 24, 2021

twixdouble commented Feb 17, 2021

lbastil commented Nov 9, 2021

lbastil commented Nov 9, 2021 •

edited

Loading

deisenberg-remedy commented Jun 12, 2023

Phlegethonyarre commented Jul 21, 2023 •

edited

Loading

Achitheus commented Nov 10, 2023

Achitheus commented Nov 10, 2023 •

edited

Loading

Add support for soft assertions #19

Add support for soft assertions #19

Comments

baev commented Mar 1, 2017

scripnichenko commented Dec 17, 2018

baev commented Dec 17, 2018

Little-Elephant commented Feb 12, 2019

Lyisko commented Jun 27, 2019

vlayon commented Feb 25, 2020

bereg2k commented Nov 3, 2020

Guter1988 commented Jan 19, 2021

sskorol commented Jan 19, 2021

Guter1988 commented Jan 21, 2021

sskorol commented Jan 21, 2021

Guter1988 commented Jan 24, 2021

twixdouble commented Feb 17, 2021

lbastil commented Nov 9, 2021

lbastil commented Nov 9, 2021 • edited Loading

deisenberg-remedy commented Jun 12, 2023

Phlegethonyarre commented Jul 21, 2023 • edited Loading

Achitheus commented Nov 10, 2023

Achitheus commented Nov 10, 2023 • edited Loading

lbastil commented Nov 9, 2021 •

edited

Loading

Phlegethonyarre commented Jul 21, 2023 •

edited

Loading

Achitheus commented Nov 10, 2023 •

edited

Loading