[Custom threshold] Improve the custom threshold rule request/response logging

[maryam-saeidi]

Related to #187271

Summary

As mentioned in this PR, when logging a JSON stringified string for debugging or tracing, we need to wrap it in a function. This PR does this for the custom threshold rule and removes extra request/response logging, relying on alerting framework logging for that purpose.

Before, we were able to have a similar log using plugins.observability and plugins.alerting.observability.rules.custom_threshold configs (as shown above), but now we only have plugins.alerting.observability.rules.custom_threshold which can be enabled by either of the following configs:

logging:
  loggers:
    - name: plugins.alerting
      level: debug
    - name: plugins.alerting.observability.rules.custom_threshold
      level: trace

Thanks @dgieselaar for bringing this to our attention!

How to test

• Enable the trace logging config as mentioned above
• Create a custom threshold rule and check the server logs; you should be able to see the logs for the request and response of the rule execution.

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[obltmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[maryam-saeidi]

Created a ticket for improving logging msearch requests and responses: #192445

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 8c510f3
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-192443-8c510f362be9

Failed CI Steps

• Jest Tests #3
• FTR Configs #88

Test Failures

• [job] [logs] Jest Tests #3 / AllCasesSelectorModal Closing modal when pressing the x icon
• [job] [logs] FTR Configs #88 / Rules Management - Prebuilt Rules - Large Prebuilt Rules Package @ess @serverless @skipInServerlessMKI install_large_prebuilt_rules_package should install a package containing 15000 prebuilt rules without crashing

Metrics [docs]

✅ unchanged

History

• 💔 Build #233128 failed 0c33ce5
• 💛 Build #232987 was flaky 77f822b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @maryam-saeidi

[ymao1]

response ops changes LGTM

[benakansara]

LGTM