Surface config parsing error under EA managed mode

[carsonip]

Motivation/summary

By wrapping reloader errors in a way that EA manager understands (i.e. in a MultiError of UnitError), newRunner initialization errors, including config parsing errors, will be surfaced and logged.

Checklist

- [ ] Update CHANGELOG.asciidoc I expect this to go into 8.17 first, then 8.16, but there may be uncertainty around this. We can add changelog before the release.

• Documentation has been updated

How to test these changes

Start apm-server in EA managed mode (e.g. on ESS), pass in an invalid integration policy, e.g. setting apm-server.rum.response_headers to a string instead of a list. Check that in fleet UI that EA input unit has failed. Check that an error is logged.

Related issues

Fixes #14560

[mergify]

This pull request does not have a backport label. Could you fix it @carsonip? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-7.17 is the label to automatically backport to the 7.17 branch.
• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• backport-8.x is the label to automatically backport to the 8.x branch.

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label.

[carsonip]

@mergify backport 8.17

[mergify]

backport 8.17

✅ Backports have been created

• #14807 Surface config parsing error under EA managed mode (backport #14574) has been created for branch 8.17

[rubvs]

Hi @carsonip, I only get the error message shown below when testing for these changes. The main concern I have is the "source": "elastic-agent" field. Should the source not be apm-server?

{
  "log.level": "error",
  "@timestamp": "2024-12-11T22:56:28.639Z",
  "log.origin": {
    "function": "github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents",
    "file.name": "coordinator/coordinator.go",
    "file.line": 663
  },
  "message": "Unit state changed apm-default-1e9d923a-9507-414e-bc8e-1737cc7ca6fc (CONFIGURING->FAILED): failed to load input config: Error processing configuration: required object, but found string in field apm-server.rum.response_headers",
  "log": {
    "source": "elastic-agent"
  },
  "component": {
    "id": "apm-default",
    "state": "HEALTHY"
  },
  "unit": {
    "id": "apm-default-1e9d923a-9507-414e-bc8e-1737cc7ca6fc",
    "type": "input",
    "state": "FAILED",
    "old_state": "CONFIGURING"
  },
  "ecs.version": "1.6.0"
}

Steps Used

• Create a deployment in cloud.elastic.co
• Add a new Policy via: Management -> Integrations -> Elastic APM
• Open a Linux VM in Multipass to create a new Elastic Agent.
• Within this VM Terminal 1:
  • curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.16.1-linux-arm64.tar.gz
  • tar xzvf elastic-agent-8.16.1-linux-arm64.tar.gz
  • cd elastic-agent-8.16.1-linux-arm64
  • sudo ./elastic-agent run
• VM Terminal 2:
  • Copy config from ESS and replace content in VM's elastic-agent.yml
  • Change the rum field to a string as suggested.
  • Observe the logs in sudo nvim data/elastic-agent-b6da7f/logs/elastic-agent-20241211-1.ndjson

Thanks @1pkg for the help on this

[carsonip]

elastic-agent-8.16.1-linux-arm64.tar.gz

8.16.1 is not a version that contains this fix, but I've confirmed with @rubvs offline that the apm-server binary is swapped to 8.17 for testing.

Should the source not be apm-server?

The log you see is expected. The source is EA because apm-server returns an error to EA (over grpc), and EA is the one responsible for reporting a failure. This log line confirms the bug fix is working as intended.