[FEATURE] Tests should be able to detect if warnings/errors were logged

[peternied]

Is your feature request related to a problem?
There are many condictions within the security plugin that warn/error on bad configuration or help guide users away from deprecated features. There should be a way to confirm that the warnings/errors were actually fired for users/administrators to see.

What solution would you like?
We could add a mechanism to listen to the logger output and make it available for tests to assert on, this would involve catching this state from running nodes, aggregating it, and supplying for a test in progress. It would be complex to implement in a reliable fashion as it is a similiar scenario to TestAuditlogImpl

What alternatives have you considered?
OpenSearchTestCase has this functionality, and migrating to these base configurations is an option for some test configurations, see
OpenSearchTestCase.java#L506

[cwperks]

[Triage] @peternied Re-surfacing this issue. Are there any tests in particular you can think of that would benefit from this functionality? Can you provide more info to make this issue more actionable or close the issue?

[peternied]

@cwperks I don't think we need to make this issue more actionable. If we triaged it, we are saying 'we would accept a change like this into the codebase' - not a commitment to get this work done.