Failed to validate metadata at cloud.instance.name (AWS/EKS)

[dznicol]

Describe the bug

Installing the agent on a local machine, data is sent to APM server, but when deployed to AWS validation of metadata fails.

Steps to reproduce

1. Run locally with data is sent to APM server:

ELASTIC_APM_SERVER_URL=[redacted] ELASTIC_APM_VERIFY_SERVER_CERT=false ELASTIC_APM_SECRET_TOKEN=[redacted] foreman start

2. Running via Docker container in a Kubernetes Deployment POD in EKS with env vars below, all agent attempts to send data fail:

ELASTIC_APM_SECRET_TOKEN=[redacted]
ELASTIC_APM_VERIFY_SERVER_CERT=false
ELASTIC_APM_SERVICE_NAME=[redacted]
ELASTIC_APM_SERVER_URL=[redacted]

Expected behavior

In both cases above I should receive data into my APM server. However when running in AWS I see this in the Rails log:

E, [2020-11-06T15:18:41.290424 #1] ERROR -- : [ElasticAPM] APM Server responded with an error:
"{\"accepted\":0,\"errors\":[{\"message\":\"failed to validate metadata: error validating JSON: I[#] S[#] doesn't validate with \\\"metadata#\\\"\\n  I[#/cloud/instance/name] S[#/properties/cloud/properties/instance/properties/name/type] expected string, but got null\",\"document\":\"{\\\"metadata\\\":{\\\"service\\\":{\\\"name\\\":\\\"[redacted]\\\",\\\"environment\\\":\\\"production\\\",\\\"version\\\":null,\\\"agent\\\":{\\\"name\\\":\\\"ruby\\\",\\\"version\\\":\\\"3.11.0\\\"},\\\"framework\\\":{\\\"name\\\":\\\"Ruby on Rails\\\",\\\"version\\\":\\\"6.0.3.4\\\"},\\\"language\\\":{\\\"name\\\":\\\"ruby\\\",\\\"version\\\":\\\"2.6.5\\\"},\\\"runtime\\\":{\\\"name\\\":\\\"ruby\\\",\\\"version\\\":\\\"2.6.5\\\"}},\\\"process\\\":{\\\"pid\\\":1,\\\"title\\\":\\\"[redacted]\\\",\\\"argv\\\":[\\\"-b\\\",\\\"0.0.0.0\\\"]},\\\"system\\\":{\\\"hostname\\\":\\\"[redacted]\\\",\\\"architecture\\\":\\\"x86_64\\\",\\\"platform\\\":\\\"linux\\\",\\\"kubernetes\\\":{\\\"namespace\\\":null,\\\"node\\\":{\\\"name\\\":null},\\\"pod\\\":{\\\"name\\\":null,\\\"uid\\\":\\\"[redacted]\\\"}}},\\\"labels\\\":null,\\\"cloud\\\":{\\\"provider\\\":\\\"aws\\\",\\\"account\\\":{\\\"id\\\":\\\"[redacted]\\\",\\\"name\\\":null},\\\"availability_zone\\\":\\\"[redacted]\\\",\\\"instance\\\":{\\\"id\\\":\\\"i-[redacted]\\\",\\\"name\\\":null},\\\"machine\\\":{\\\"type\\\":\\\"t3.medium\\\"},\\\"project\\\":{\\\"id\\\":null,\\\"name\\\":null},\\\"region\\\":\\\"eu-west-2\\\"}}}\"}]}\n"

Environment

• OS: Mac (locally) Linux (AWS)
• Ruby version: 2.6.5
• Framework and version: Rails 6.0.1
• APM Server version: 7.9.2
• Agent version: 3.11.0

Additional Information

Investigating this I see that there is a package called filebeat, and cloud.instance.name is not metadata that is exported other than on Azure and Openstack Nova.

Is this perhaps a bug that the agent needs to cope with cloud.instance.name being null on AWS?

Or do I need to install filebeat package into my docker container that is running on our AWS kubernets pods? I am seeing some EC2 data though, such as region and availability_zone.

[estolfo]

Hi @dznicol Thanks for reporting this. It seems that you are experiencing a bug in the APM server that will be fixed in the next version (7.10.0). We released an agent version 3.11.1 to fix compatibility after another user also reported this (#884). Would you mind upgrading to the latest agent version and letting us know if the error goes away?

[estolfo]

Hi @dznicol Is there any chance you've tried agent version 3.11.1? We've released 3.12.0 since I wrote my comment so you could even use that agent version.

[estolfo]

I'm going to close this issue as APM server version 7.10.0 is released with a fix for this. Agent version >= 3.11.1 should fix this as well, even for server versions < 7.10.0