[APM] Move impact calculation to Elasticsearch

[sorenlouv]

Closes #26428

This moves the calculation of impact from the Nodejs to Elasticsearch.
Impact used to be calculated as transactionPerMinute * avgResponseTime. The equivalent in ES is to sum avg response times.

Calculating and sorting by impact in ES ensures that the top 100 buckets returned are actually the top 100 most impactful endpoints - and not just the top 100 slowest endpoints.

This will be backported to 6.5.

cc @roncohen

[elasticmachine]

Pinging @elastic/apm-ui

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[sorenlouv]

And here is the manual backport #26443

[sorenlouv]

I find the one-liners much more readable.

[sorenlouv]

No, these sum values are not just random numbers. They are doc_count * avg.value for each bucket. Was hoping the test snapshot would stay consistent but if you look at the changed impact values the differences are very small.

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[roncohen]

would be cool to try it out on some real data to make sure it works before merging?

[sorenlouv]

@roncohen I already verified that it produces the same impact score as before (for opbeans data). Would be great to try it against elastic.co data but not sure how I can access that locally.

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[sorenlouv]

retest

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[sorenlouv]

retest

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[ogupte]

why is as unknown necessary when it get interpreted as ESResponse later on?

[sorenlouv]

Without first casting it to unknown, it complains because the object contains properties that are not defined on ESResponse. I think this is due to @types/elasticsearch not having everything defined properly.

I wish we didn't have to do this, because it's definitely defeating the purpose of a type system, but I think it's the most pragmatic solution for now.

[sorenlouv]

This is the tsc error I get without unknown:

Conversion of type '{ took: number; timed_out: false; _shards: { total: number; successful: number; skipped: number; failed: number; }; hits: { total: number; max_score: null; hits: never[]; }; aggregations: { transactions: { ...; }; }; }' to type 'AggregationSearchResponse<void, Aggs>' may be a mistake because neither type sufficiently overlaps with the other. If this was intentional, convert the expression to 'unknown' first.
  Types of property 'aggregations' are incompatible.
    Type '{ transactions: { doc_count_error_upper_bound: number; sum_other_doc_count: number; buckets: ({ key: string; doc_count: number; avg: { value: number; }; p95: { values: { '95.0': number; }; }; sample: { hits: { ...; }; }; } | ... 12 more ... | { ...; })[]; }; }' is not comparable to type 'Aggs'.
      Types of property 'transactions' are incompatible.
        Type '{ doc_count_error_upper_bound: number; sum_other_doc_count: number; buckets: ({ key: string; doc_count: number; avg: { value: number; }; p95: { values: { '95.0': number; }; }; sample: { hits: { total: number; max_score: null; hits: { ...; }[]; }; }; } | ... 12 more ... | { ...; })[]; }' is not comparable to type '{ buckets: Bucket[]; }'.
          Types of property 'buckets' are incompatible.
            Type '({ key: string; doc_count: number; avg: { value: number; }; p95: { values: { '95.0': number; }; }; sample: { hits: { total: number; max_score: null; hits: { _index: string; _type: string; _id: string; _score: null; _source: { ...; }; sort: number[]; }[]; }; }; } | ... 12 more ... | { ...; })[]' is not comparable to type 'Bucket[]'.
              Type '{ key: string; doc_count: number; avg: { value: number; }; p95: { values: { '95.0': number; }; }; sample: { hits: { total: number; max_score: null; hits: { _index: string; _type: string; _id: string; _score: null; _source: { ...; }; sort: number[]; }[]; }; }; } | ... 12 more ... | { ...; }' is not comparable to type 'Bucket'.
                Type '{ key: string; doc_count: number; avg: { value: number; }; p95: { values: { '95.0': number; }; }; sample: { hits: { total: number; max_score: null; hits: { _index: string; _type: string; _id: string; _score: null; _source: { ...; }; sort: number[]; }[]; }; }; }' is not comparable to type 'Bucket'.
                  Types of property 'sample' are incompatible.
                    Type '{ hits: { total: number; max_score: null; hits: { _index: string; _type: string; _id: string; _score: null; _source: { '@timestamp': string; processor: { name: "transaction"; event: "transaction"; }; transaction: { ...; }; ... 4 more ...; host: { ...; }; }; sort: number[]; }[]; }; }' is not comparable to type '{ hits: { total: number; max_score: number | null; hits: { _source: Transaction; }[]; }; }'.
                      Types of property 'hits' are incompatible.
                        Type '{ total: number; max_score: null; hits: { _index: string; _type: string; _id: string; _score: null; _source: { '@timestamp': string; processor: { name: "transaction"; event: "transaction"; }; transaction: { ...; }; ... 4 more ...; host: { ...; }; }; sort: number[]; }[]; }' is not comparable to type '{ total: number; max_score: number | null; hits: { _source: Transaction; }[]; }'.
                          Types of property 'hits' are incompatible.
                            Type '{ _index: string; _type: string; _id: string; _score: null; _source: { '@timestamp': string; processor: { name: "transaction"; event: "transaction"; }; transaction: { name: string; duration: { us: number; }; ... 4 more ...; id: string; }; ... 4 more ...; host: { ...; }; }; sort: number[]; }[]' is not comparable to type '{ _source: Transaction; }[]'.
                              Type '{ _index: string; _type: string; _id: string; _score: null; _source: { '@timestamp': string; processor: { name: "transaction"; event: "transaction"; }; transaction: { name: string; duration: { us: number; }; ... 4 more ...; id: string; }; ... 4 more ...; host: { ...; }; }; sort: number[]; }' is not comparable to type '{ _source: Transaction; }'.
                                Object literal may only specify known properties, and '_index' does not exist in type '{ _source: Transaction; }'. [2352]
(property) _index: string