Fix metadata.service.* fields requirements in json schema

[simitt]

Motivation/summary

Validations on metadata.service.* fields were not run correctly. This PR changes the schema to properly run validations for v2 and rum-v3:

• metadata.service.name,metadata.service.agent.name: key required, value must be non-empty string ,
• metadata.service.agent.version: key must be sent, value must not be null
• metadata.service.runtime.name,metadata.service.runtime.version: keys must be sent, values must not be null if metadata.service.runtime is sent
• metadata.service.language.name: keys must be sent, values must not be null if metadata.service.language is sent

The PR also fixes an issue where metadata.service.cloud.* fields that are not required were not allowed to be sent as null values.

Checklist

- [ ] I have signed the Contributor License Agreement.

• I have updated CHANGELOG.asciidoc

I have considered changes for:
- [ ] documentation
- [ ] logging (add log lines, choose appropriate log selector, etc.)
- [ ] metrics and monitoring (create issue for Kibana team to add metrics to visualizations, e.g. Kibana#44001)

• automated tests (add tests for the code changes, all unit tests pass locally)
  - [ ] telemetry
  - [ ] Elasticsearch Service (https://cloud.elastic.co)
  - [ ] Elastic Cloud Enterprise (https://www.elastic.co/products/ece)
  - [ ] Elastic Cloud on Kubernetes (https://www.elastic.co/elastic-cloud-kubernetes)
• get elastic APM agent devs involved to check if the fix breaks any of the current agents.

How to test these changes

Sending payloads where metadata.service.agent.name,metadata.service.agent.version,metadata.service.runtime.name,metadata.serice.runtime.version,metadata.service.language.name are missing or set to null and ensure payload is invalid.

Related issues

fixes #4141

[apmmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #4142 updated]

• Start Time: 2020-09-14T07:51:15.906+0000

• Duration: 42 min 20 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3543
Skipped	145
Total	3688

Steps errors

Expand to view the steps failures

• Name: Compress

  • Description: tar --exclude=coverage-files.tgz -czf coverage-files.tgz coverage

  • Duration: 0 min 0 sec

  • Start Time: 2020-09-14T08:06:11.247+0000

  • log

• Name: Compress

  • Description: tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests

  • Duration: 0 min 0 sec

  • Start Time: 2020-09-14T08:19:16.521+0000

  • log

• Name: Test Sync

  • Description: ./script/jenkins/sync.sh

  • Duration: 3 min 53 sec

  • Start Time: 2020-09-14T08:00:34.840+0000

  • log

[axw]

LGTM, but we should probably wait until we've got a resolution on #4141 (comment)

[simitt]

Update: I added a minLen=1 requirement for the metadata.service.name and metadata.service.agent.name - since sending an empty string should also be considered a bug.

[simitt]

@elastic/apm-agent-devs as just discussed please check the requirements for the current agent versions and let me know if this bugfix causes issues, otherwise I'll merge by the end of the week.

[axw]

LGTM, but I'm wondering why we make version required but don't also require it to be non-empty. Seems like all fields should be either optional or required and non-empty.

[simitt]

I didn't want to introduce too many changes with this as it will be deployed as bugfix; the reason I went for adding minLen to service.name and service.agent.name is that they are more crucial to the APM solution.

[eyalkoren]

Java agent should be OK with this change.

• metadata.service.name - there should be fallback always, including when setting an empty string.
• metadata.service.agent.name: hardcoded "java"
• metadata.service.agent.version: either the valid version or "unknown"
• metadata.service.runtime.name: hardcoded "Java"
• metadata.service.runtime.version: the value of the java.version system property, should never be null
• metadata.service.language.name: hardcoded "Java"

As far as I can see, it is the case since GA.
Since the only configurable option is service.name, I think it shouldn't break anything.

[vigneshshanmugam]

Looks good from RUM agent side.