add --apm-server-enable-data-streams option

[graphaelli]

What does this PR do?

adds --apm-server-enable-data-streams option for instructing APM Server to write to data streams. This is valid in 7.16 and 8.0 (and maybe 7.15? it's not important to prevent it from being set in earlier versions)

Related issues

related to elastic/apm-server#5949 - apm-server remains unhealthy until the integration is installed, not ideal but not blocked on that.

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-08-22T06:26:30.045+0000

• Duration: 30 min 1 sec

• Commit: c4baaf3

Test stats 🧪

Test	Results
Failed	0
Passed	498
Skipped	1
Total	499

Trends 🧪

[simitt]

Why not allowing from 7.15 on?

[graphaelli]

I've removed any restriction on version number, this is just the one I picked for the test case.

[sorenlouv]

Running ./scripts/compose.py start master --apm-server-enable-data-streams I get:

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "unable to authenticate user [elastic] for REST request [/_index_template/traces-apm]",
        "header": {
          "WWW-Authenticate": [
            "Basic realm=\"security\" charset=\"UTF-8\"",
            "Bearer realm=\"security\"",
            "ApiKey"
          ]
        }
      }
    ],
    "type": "security_exception",
    "reason": "unable to authenticate user [elastic] for REST request [/_index_template/traces-apm]",
    "header": {
      "WWW-Authenticate": [
        "Basic realm=\"security\" charset=\"UTF-8\"",
        "Bearer realm=\"security\"",
        "ApiKey"
      ]
    }
  },
  "status": 401
}

Anything I'm missing?

[sorenlouv]

Hmm.. This might be a general problem on master caused by Kibana actually

[graphaelli]

Yeah, #1232 opened for that

[graphaelli]

That issue is resolved. Other things I've run into that are preventing this option from really working for now:

• apm-server will not start listening for conns until the integration is installed, there are a few competing options that can resolve this including queuing until installation, installation of the integration by default, installation of the integration as part of apm-it start. For now, installing the integration manually works until ...
• the apm-server kibana user currently configured is apm_server_user which is not a superuser. There is no built in option to override that at the moment but changing it to admin manually works until...
• apm-server can't index events using the apm pipeline, reporting: pipeline with id [apm] does not exist. Changing that to output.elasticsearch.pipelines=[{pipeline: 'traces-apm-0.3.0'}] doesn't seem to have any impact

[graphaelli]

That last pipeline step is incorrect, that does actually work when you make sure it applies 🤦 .

The next issue is:

(status=403): {\"type\":\"security_exception\",\"reason\":\"action [indices:admin/auto_create] is unauthorized for user [apm_server_user] with roles [apm_server,apm_system,apm_user,ingest_admin,kibana_user] on indices [traces-apm-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]\"}, dropping event!","service.name":"apm-server","event.dataset":"apm-server","ecs.version":"1.6.0"}

This makes sense, we'll need a roles.yml update for that