fix flaky integration tests in xample-domain

[lisac]

What was the problem?

The continuous Integration (CI) tests for xample-domain are flaky: they appear to fail randomly for no apparent reason.

As reported in #2345, this error is consistent in the logs of those failed tests:

o.s.a.r.l.SimpleMessageListenerContainer : Failed to check/redeclare auto-delete queue(s). [...] Unexpected ERROR logs in svc-xample-j container.

Theory: 1) the domain-xample integration test code (java) occasionally fails to connect to the rabbitMQ service, even if the service is observed to be running*; 2) initial connection failures are not uncommon; and 3) that if given a modest number of retry attempts, the connection would succeed.

Associated tickets or Slack threads:

• Fix Flickering **xample-domain / integration-test #2345

How does this fix it?

1. Introduce a retry policy on rabbitMQ actions, limited to the scope of the integration test.
2. Refine error interpretation: rather than assess an integration test failure if the word "ERROR" is found in a log file, ignore cases of "ERROR" that are associated with AMQP's "SimpleMessageListenerContainer" - with the theory that initial connection failures are not uncommon.
3. To mitigate the previous item being too lax, assess an integration failure if the word "Test failures:" is found in the log file.. (test failures will in fact cause the the log file assessment to be skipped)

This PR has an additional change to the integration test, I think minor (removal of an action I thought redundant).

How to test this PR

• in particular, please assess whether the change to the error interpretation could be more refined
• Step 2¹

*the CI workflow verifies the rabbitMQ service is ready before moving on to executing the integration tests.

abd-vro/.github/workflows/xample-integration-test.yml

Lines 46 to 50 in e3ec82a

	- name: 'Check for RabbitMQ to be ready'
	uses: department-of-veterans-affairs/[email protected]
	with:
	url: 'http://localhost:15672/api/vhosts'
	method: 'GET'

Footnotes

1. To check if a PR will succeed in the SecRel workflow, test PRs in the SecRel pipeline. ↩

[lisac]

verified that the CI fails (failure was intentional, per 93cb964) - although wasn't able to get insight into the log file handling, as that step was skipped, due to the failure detected at the integration test step.

Will rely on code review for assessing whether the change to log file handling is acceptable.

[lisac]

This placement of setRetryTemplate() is not ideal, for a few reasons:

1. it should be executed just once, such as in a @BeforeAll function. However, the method signature for @BeforeAll is static, which is incompatible with the rabbitTemplate instance.
2. it should be configured in spring (or whatever it is that sets up rabbitTemplate)

... and I'm stumped on how to implement either of these. Very open to pointers. Thanks.

[lisac]

i removed this method as I thought it redundant to the purgeQueue() performed in setUp()

[msnwatson]

Great questions in this PR about implementation. I know the static method stuff I've bumped into a lot and don't really have an answer and unfortunately I'm not much of a spring expert either.

[lisac]

Proceeding with merging. The implementation could be better. On the other hand, it's scoped to a test class that wouldn't affect production; and it is effective with respect to reducing flakiness. Given my limited springboot comfort, I'm not confident in the value of spending more time on this.

Forgive me, whoever next handles this code.