Backwards incompatible changes to consider for 7.0

[roncohen]

This is a list to track things we'd like to change which cannot be changed in a minor:

Documents in Elasticsearch:

• Move transaction.span_count.total.dropped to transaction.span_count.dropped so it's at the same level as transaction.span_count.started and aligns with the v2 intake protocol. Investigate to move transaction.span_count.dropped.total to transaction.span_count.dropped #1675
• Drop span.start and timestamp.us in favor of just using microsecond precision @timestamp. Use @timestamp instead of timestamp.us and switch to type date_nanos #1678
• Remove Kibana Index Pattern and Dashboards from shipping with APM Server
• Proposal: Change duration to be microseconds apm#2 Intake API: Change duration to be microseconds
• Get rid of the span.hex_id fields in favor of just span.id. Moving span.hex_id to span.id #1677

Refactoring APM Server:

• Remove v1 intake protocol endpoints, related code, tests, testdata and documentation. Remove v1 intake protocol endpoints, related code, tests, testdata and documentation #1671

[simitt]

related: planned breaking changes in beats

[simitt]

Refactoring APM Server:

• Remove config options for Dashboards and _Kibana. Remove dashboards, remove config options and docs for index patterns, #1673

To discuss:

• Change default indices and retention policy. Investigate changing default indices and retention policy #1680
• Remove shipping dashboards and Kibana index pattern with 7.x, as they are shipped with Kibana. This probably requires additional flexibility for loading them in Kibana.
• Change config settings structure for setup.template
• Remove concurrent_requests option, Remove concurrent_requestoption #1674
• Remove user facing config options when they can be calculated based on a formula, e.g. mem.queue.flush.min_events, output.elasticsearch.max_bulk_size. Calculate values before initializing the beat and passing in the values. Investigate auto-setting config options related to ES output based on other config values.  #1676

[graphaelli]

To discuss: #1679

• Change / response format. The ok doesn't match the new v2 response format so now it seems strange. For reference, es 7 looks like:

HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 513

{
  "name" : "06b42f641b07",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "u58I5cGVRgyJm1uRigWJjg",
  "version" : {
    "number" : "7.0.0-SNAPSHOT",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "5eb7040",
    "build_date" : "2018-11-29T17:32:22.811477Z",
    "build_snapshot" : true,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.6.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
```  changing this may impact the java agent.

[graphaelli]

Change apm-server.host to an array of hosts and allow listening on multiple interfaces

[graphaelli]

To discuss: #1681

Switch to user-friendly configuration. For example, replace:

output.elasticsearch.indices:
    - index: "apm-%{[beat.version]}-error-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "error"

with something like:

apm-server.elasticsearch.indices:
  - error: "apm-%{[beat.version]}-error-%{+yyyy.MM.dd}"

[jalvz]

Switch to user-friendly configuration

Moreover, consider only allowing changes that don't break the UI (eg adding a string after apm) and not any arbitrary change such us removing {beat.version}.

[jalvz]

Consider #1401

[simitt]

Closing this, as we created issues for every single suggestion that we will consider.