[APM] POC: Metric-based service maps #114468

dgieselaar · 2021-10-11T11:51:55Z

Summary

POC for a metrics-based service maps.

For the agent:

For each exit span, generate a hash based on span.destination.service.resource that is passed as tracestate. Store this hash as span.destination.service.hash on the span event.
For the resulting transaction, store the hash as transaction.upstream.hash on the transaction event and propagate to child spans.
For each exit span that is a child of this transaction, hash both transaction.upstream.hash and span.destination.service.resource, and store it as span.destination.service.hash

On APM Server:

aggregate span destination metrics and transaction metrics additionally on span.destination.service.hash and transaction.upstream.hash

In Kibana:

Use two separate composite aggregations on span destination metrics and transaction metrics to reconstruct the service map.

Simple distributed trace:

Messaging systems:

Obs test cluster's load balancer/round robin setup:

…kdown-metrics

kibanamachine · 2021-10-13T15:31:45Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: b397b87
Storybooks Preview
Pipeline Steps (look for red circles / failed steps)
Interpreting CI Failures

Failed CI Steps

Test Failures

Kibana Pipeline / jest / Jest Tests.packages/elastic-apm-generator/src/test/scenarios.simple trace generates the same data every time

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

Error: expect(received).toMatchSnapshot()

Snapshot name: `simple trace generates the same data every time 1`

- Snapshot  -  0
+ Received  + 15

@@ -32,10 +32,11 @@
      "span.name": "GET apm-*/_search",
      "span.subtype": "elasticsearch",
      "span.type": "db",
      "trace.id": "b1c6c04a9ac15b138f716d383cc85e6b",
      "transaction.id": "36c16f18e75058f8",
+     "transaction.upstream.hash": undefined,
    },
    Object {
      "@timestamp": 1609459260000,
      "agent.name": "java",
      "container.id": "instance-1",
@@ -68,10 +69,11 @@
      "span.name": "GET apm-*/_search",
      "span.subtype": "elasticsearch",
      "span.type": "db",
      "trace.id": "53c6c37bd4c85f4fbc880cd80704a9cd",
      "transaction.id": "65ce74106eb050be",
+     "transaction.upstream.hash": undefined,
    },
    Object {
      "@timestamp": 1609459320000,
      "agent.name": "java",
      "container.id": "instance-1",
@@ -104,10 +106,11 @@
      "span.name": "GET apm-*/_search",
      "span.subtype": "elasticsearch",
      "span.type": "db",
      "trace.id": "5eebf2e8d8cc5f85be8c573a1b501c7d",
      "transaction.id": "91fa709d90625fff",
+     "transaction.upstream.hash": undefined,
    },
    Object {
      "@timestamp": 1609459380000,
      "agent.name": "java",
      "container.id": "instance-1",
@@ -140,10 +143,11 @@
      "span.name": "GET apm-*/_search",
      "span.subtype": "elasticsearch",
      "span.type": "db",
      "trace.id": "6e8da4beb752589a86d53287c9d902de",
      "transaction.id": "6c500d1d19835e68",
+     "transaction.upstream.hash": undefined,
    },
    Object {
      "@timestamp": 1609459440000,
      "agent.name": "java",
      "container.id": "instance-1",
@@ -176,10 +180,11 @@
      "span.name": "GET apm-*/_search",
      "span.subtype": "elasticsearch",
      "span.type": "db",
      "trace.id": "0aaa92bd91df543c8fd10b662051d9e8",
      "transaction.id": "1b3246cc83595869",
+     "transaction.upstream.hash": undefined,
    },
    Object {
      "@timestamp": 1609459500000,
      "agent.name": "java",
      "container.id": "instance-1",
@@ -212,10 +217,11 @@
      "span.name": "GET apm-*/_search",
      "span.subtype": "elasticsearch",
      "span.type": "db",
      "trace.id": "26be5f0e2c16576ebf5f39c505eb1ff2",
      "transaction.id": "12b49e3c83fe58d5",
+     "transaction.upstream.hash": undefined,
    },
    Object {
      "@timestamp": 1609459560000,
      "agent.name": "java",
      "container.id": "instance-1",
@@ -248,10 +254,11 @@
      "span.name": "GET apm-*/_search",
      "span.subtype": "elasticsearch",
      "span.type": "db",
      "trace.id": "c17c414c0b51564ca30e2ad839393180",
      "transaction.id": "d9272009dd4354a1",
+     "transaction.upstream.hash": undefined,
    },
    Object {
      "@timestamp": 1609459620000,
      "agent.name": "java",
      "container.id": "instance-1",
@@ -284,10 +291,11 @@
      "span.name": "GET apm-*/_search",
      "span.subtype": "elasticsearch",
      "span.type": "db",
      "trace.id": "0280b1ffaae75e7ab097c0b52c3b3e6a",
      "transaction.id": "bc52ca08063c505b",
+     "transaction.upstream.hash": undefined,
    },
    Object {
      "@timestamp": 1609459680000,
      "agent.name": "java",
      "container.id": "instance-1",
@@ -320,10 +328,11 @@
      "span.name": "GET apm-*/_search",
      "span.subtype": "elasticsearch",
      "span.type": "db",
      "trace.id": "6fb5191297fb59cebdb6a0196e273676",
      "transaction.id": "186858dd88b75d59",
+     "transaction.upstream.hash": undefined,
    },
    Object {
      "@timestamp": 1609459740000,
      "agent.name": "java",
      "container.id": "instance-1",
@@ -356,10 +365,11 @@
      "span.name": "GET apm-*/_search",
      "span.subtype": "elasticsearch",
      "span.type": "db",
      "trace.id": "77b5ffe303ae59b49f9b0e5d5270c16a",
      "transaction.id": "0d5f44d48189546c",
+     "transaction.upstream.hash": undefined,
    },
    Object {
      "@timestamp": 1609459800000,
      "agent.name": "java",
      "container.id": "instance-1",
@@ -392,10 +402,11 @@
      "span.name": "GET apm-*/_search",
      "span.subtype": "elasticsearch",
      "span.type": "db",
      "trace.id": "51c6b70db4dc5cf89b690de45c0c7b71",
      "transaction.id": "7483e0606e435c83",
+     "transaction.upstream.hash": undefined,
    },
    Object {
      "@timestamp": 1609459860000,
      "agent.name": "java",
      "container.id": "instance-1",
@@ -428,10 +439,11 @@
      "span.name": "GET apm-*/_search",
      "span.subtype": "elasticsearch",
      "span.type": "db",
      "trace.id": "5d91a6cde6015897935e413bc500f211",
      "transaction.id": "f142c4cbc7f3568e",
+     "transaction.upstream.hash": undefined,
    },
    Object {
      "@timestamp": 1609459920000,
      "agent.name": "java",
      "container.id": "instance-1",
@@ -464,10 +476,11 @@
      "span.name": "GET apm-*/_search",
      "span.subtype": "elasticsearch",
      "span.type": "db",
      "trace.id": "c097c19d884d52579bb11a601b8a98b3",
      "transaction.id": "2e3a47fa2d905519",
+     "transaction.upstream.hash": undefined,
    },
    Object {
      "@timestamp": 1609459980000,
      "agent.name": "java",
      "container.id": "instance-1",
@@ -500,10 +513,11 @@
      "span.name": "GET apm-*/_search",
      "span.subtype": "elasticsearch",
      "span.type": "db",
      "trace.id": "4591e57f4d7f5986bdd7892561224e0f",
      "transaction.id": "de5eaa1e47dc56b1",
+     "transaction.upstream.hash": undefined,
    },
    Object {
      "@timestamp": 1609460040000,
      "agent.name": "java",
      "container.id": "instance-1",
@@ -536,7 +550,8 @@
      "span.name": "GET apm-*/_search",
      "span.subtype": "elasticsearch",
      "span.type": "db",
      "trace.id": "85ee8e618433577b9316a1e14961aa89",
      "transaction.id": "af7eac7ae61e576a",
+     "transaction.upstream.hash": undefined,
    },
  ]
    at Object.<anonymous> (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/packages/elastic-apm-generator/src/test/scenarios/01_simple_trace.test.ts:45:20)
    at Promise.then.completed (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/utils.js:276:28)
    at new Promise (<anonymous>)
    at callAsyncCircusFn (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/utils.js:216:10)
    at _callCircusTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:212:40)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at _runTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:149:3)
    at _runTestsForDescribeBlock (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:63:9)
    at _runTestsForDescribeBlock (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:57:9)
    at run (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:25:3)
    at runAndTransformResultsToJestFormat (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapterInit.js:176:21)
    at jestAdapter (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapter.js:109:19)
    at runTestInternal (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/runTest.js:380:16)
    at runTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/runTest.js:472:34)
    at Object.worker (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/testWorker.js:133:12)

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`apm`	2.7MB	2.7MB	+354.0B

History

💔 Build #159542 failed 1d97421
💔 Build #159469 failed 108b667

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

dgieselaar added 3 commits October 8, 2021 16:01

[APM] Generate breakdown metrics

f7ca882

Merge branch 'master' of github.com:elastic/kibana into generate-brea…

042ad7b

…kdown-metrics

[APM] POC for metric-based service maps

108b667

dgieselaar added Team:APM All issues that need APM UI Team support release_note:skip Skip the PR/issue when compiling release notes v7.16.0 labels Oct 11, 2021

dgieselaar added 2 commits October 11, 2021 16:03

Add messaging/opbeans lb examples

1d97421

Merge branch 'master' of github.com:elastic/kibana into service-map-poc

b397b87

dgieselaar mentioned this pull request Oct 25, 2021

Store upstream paths in transactions/spans for service maps elastic/apm#364

Open

spalger added v7.17.0 and removed v7.16.0 labels Dec 7, 2021

dgieselaar closed this Mar 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM] POC: Metric-based service maps #114468

[APM] POC: Metric-based service maps #114468

dgieselaar commented Oct 11, 2021 •

edited

Loading

kibanamachine commented Oct 13, 2021 •

edited

Loading

Standard Out

Stack Trace

[APM] POC: Metric-based service maps #114468

[APM] POC: Metric-based service maps #114468

Conversation

dgieselaar commented Oct 11, 2021 • edited Loading

Summary

kibanamachine commented Oct 13, 2021 • edited Loading

💔 Build Failed

Failed CI Steps

Test Failures

Standard Out

Stack Trace

Metrics [docs]

Async chunks

History

dgieselaar commented Oct 11, 2021 •

edited

Loading

kibanamachine commented Oct 13, 2021 •

edited

Loading