AutomationIntegrationJsonTest unstable

[wborn]

This test failed in a Jenkins build: https://ci.openhab.org/job/openHAB-Core/1107/

TEST org.openhab.core.automation.integration.test.AutomationIntegrationJsonTest#assertThatARuleFromJsonFileIsExecutedCorrectly() <<< ERROR: 
Expected: is not null
     but: was null
java.lang.AssertionError: 
Expected: is not null
     but: was null
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
	at org.openhab.core.automation.integration.test.AutomationIntegrationJsonTest.lambda$17(AutomationIntegrationJsonTest.java:383)
	at org.openhab.core.test.java.JavaTest.waitForAssert(JavaTest.java:245)
	at org.openhab.core.test.java.JavaTest.waitForAssert(JavaTest.java:213)
	at org.openhab.core.test.java.JavaTest.waitForAssert(JavaTest.java:166)
	at org.openhab.core.automation.integration.test.AutomationIntegrationJsonTest.assertThatARuleFromJsonFileIsExecutedCorrectly(AutomationIntegrationJsonTest.java:382)

[wborn]

Another test in this class failed today, see: https://ci.openhab.org/job/openHAB-Core/1135

TEST org.openhab.core.automation.integration.test.AutomationIntegrationJsonTest#assertThatARuleFromJsonFileIsAddedAutomaticallyAndTheRuntimeRuleHasResolvedModuleReferences() <<< ERROR: 
Expected: is not null
     but: was null
java.lang.AssertionError: 
Expected: is not null
     but: was null
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
	at org.openhab.core.automation.integration.test.AutomationIntegrationJsonTest.lambda$15(AutomationIntegrationJsonTest.java:325)
	at org.openhab.core.test.java.JavaTest.waitForAssert(JavaTest.java:245)
	at org.openhab.core.test.java.JavaTest.waitForAssert(JavaTest.java:213)
	at org.openhab.core.test.java.JavaTest.waitForAssert(JavaTest.java:177)
	at org.openhab.core.automation.integration.test.AutomationIntegrationJsonTest.assertThatARuleFromJsonFileIsAddedAutomaticallyAndTheRuntimeRuleHasResolvedModuleReferences(AutomationIntegrationJsonTest.java:324)

[wborn]

The saga continues: https://ci.openhab.org/job/PR-openHAB-Core/5505/

TEST org.openhab.core.automation.integration.test.AutomationIntegrationJsonTest#assertThatARuleFromJsonFileIsExecutedCorrectly() <<< ERROR: 
Expected: is not null
     but: was null
java.lang.AssertionError: 
Expected: is not null
     but: was null
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
	at org.openhab.core.automation.integration.test.AutomationIntegrationJsonTest.lambda$17(AutomationIntegrationJsonTest.java:370)
	at org.openhab.core.test.java.JavaTest.waitForAssert(JavaTest.java:245)
	at org.openhab.core.test.java.JavaTest.waitForAssert(JavaTest.java:213)
	at org.openhab.core.test.java.JavaTest.waitForAssert(JavaTest.java:166)
	at org.openhab.core.automation.integration.test.AutomationIntegrationJsonTest.assertThatARuleFromJsonFileIsExecutedCorrectly(AutomationIntegrationJsonTest.java:369)

[wborn]

This one too: https://ci.openhab.org/job/PR-openHAB-Core/5484/

TEST org.openhab.core.automation.integration.test.AutomationIntegrationJsonTest#assertThatARuleFromJsonFileIsAddedAutomaticallyAndTheRuntimeRuleHasResolvedModuleReferences() <<< ERROR: 
Expected: is not null
     but: was null
java.lang.AssertionError: 
Expected: is not null
     but: was null
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
	at org.openhab.core.automation.integration.test.AutomationIntegrationJsonTest.lambda$15(AutomationIntegrationJsonTest.java:325)
	at org.openhab.core.test.java.JavaTest.waitForAssert(JavaTest.java:245)
	at org.openhab.core.test.java.JavaTest.waitForAssert(JavaTest.java:213)
	at org.openhab.core.test.java.JavaTest.waitForAssert(JavaTest.java:177)
	at org.openhab.core.automation.integration.test.AutomationIntegrationJsonTest.assertThatARuleFromJsonFileIsAddedAutomaticallyAndTheRuntimeRuleHasResolvedModuleReferences(AutomationIntegrationJsonTest.java:324)

[J-N-K]

I wonder why those tests now fail so often. I can't remember that something changed in the way events are handled in rules.

[J-N-K]

I had a look and I believe the issue is that we first do registerService(itemEventHandler); and then immediately eventPublisher.post(ItemEventFactory.createStateEvent("myMotionItem", OnOffType.ON));. I guess on slower systems the event is posted before the new event handler was registered with the event manager. In that case we miss the event and the test fails.

The problem is that it's difficult to say when the registration has finished, I tried to spy on the addEventSubscriber method in OSGiEventManager, but that doesn't work because it is in an internal package and can't be accessed. Moving the class to an exported package just for testing feels wrong.

[wborn]

I think it's back from vacation: https://ci.openhab.org/job/openHAB-Core/1253/

TEST org.openhab.core.automation.integration.test.AutomationIntegrationJsonTest#assertThatARuleFromJsonFileIsExecutedCorrectly() <<< ERROR: 
Expected: is not null
     but: was null
java.lang.AssertionError: 
Expected: is not null
     but: was null
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
	at org.openhab.core.automation.integration.test.AutomationIntegrationJsonTest.lambda$17(AutomationIntegrationJsonTest.java:383)
	at org.openhab.core.test.java.JavaTest.waitForAssert(JavaTest.java:245)
	at org.openhab.core.test.java.JavaTest.waitForAssert(JavaTest.java:213)
	at org.openhab.core.test.java.JavaTest.waitForAssert(JavaTest.java:166)
	at org.openhab.core.automation.integration.test.AutomationIntegrationJsonTest.assertThatARuleFromJsonFileIsExecutedCorrectly(AutomationIntegrationJsonTest.java:382)