Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't drop RUM unsampled transactions #6669

Merged
merged 8 commits into from
Nov 22, 2021
Merged

Conversation

axw
Copy link
Member

@axw axw commented Nov 19, 2021

Motivation/summary

The User Experience app in Kibana currently relies on transaction events only, and not transaction metrics. Store all RUM transaction events so the UX app can calculate statistics correctly.

Checklist

How to test these changes

  1. Run apm-server, send unsampled RUM transactions and unsampled backend transactions
  2. Check that only the RUM transactions are stored

Related issues

Closes #6260

@axw axw added v8.0.0 backport-8.0 Automated backport with mergify labels Nov 19, 2021
@axw axw force-pushed the keep-rum-unsampled branch from e9e03eb to d896bce Compare November 19, 2021 08:11
@axw axw requested a review from a team November 19, 2021 08:21
@axw axw marked this pull request as ready for review November 19, 2021 08:21
@apmmachine
Copy link
Contributor

apmmachine commented Nov 19, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-11-22T10:52:51.097+0000

  • Duration: 39 min 22 sec

  • Commit: 49e38eb

Test stats 🧪

Test Results
Failed 0
Passed 5622
Skipped 19
Total 5641

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /hey-apm : Run the hey-apm benchmark.

  • /package : Generate and publish the docker images.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@axw
Copy link
Member Author

axw commented Nov 19, 2021

This is broken, moving back into draft for now.

@axw axw removed the request for review from a team November 19, 2021 08:58
@axw axw marked this pull request as draft November 19, 2021 08:58
Copy link
Contributor

@simitt simitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good so far

beater/beater.go Show resolved Hide resolved
@axw axw marked this pull request as ready for review November 20, 2021 01:02
@axw axw requested a review from simitt November 20, 2021 01:02
changelogs/head.asciidoc Outdated Show resolved Hide resolved
@axw axw enabled auto-merge (squash) November 22, 2021 07:47
@axw axw merged commit 8e63576 into elastic:master Nov 22, 2021
mergify bot pushed a commit that referenced this pull request Nov 22, 2021
* Don't drop RUM unsampled transactions

* Revert to using global monitoring registry

* Fix systemtest

* Update comment

* Fix systemtest even more

* Update changelog

(cherry picked from commit 8e63576)
axw added a commit that referenced this pull request Nov 22, 2021
* Don't drop RUM unsampled transactions

* Revert to using global monitoring registry

* Fix systemtest

* Update comment

* Fix systemtest even more

* Update changelog

(cherry picked from commit 8e63576)

Co-authored-by: Andrew Wilkins <[email protected]>
@marclop marclop self-assigned this Dec 21, 2021
@marclop
Copy link
Contributor

marclop commented Dec 21, 2021

Verified with 9bb5e4c:

Ingested transactions that are both sampled and upsampled, the "normal" unsampled transactions are dropped:

$ curl -H "Content-type: application/x-ndjson" --data-binary @testdata/intake-v2/transactions_spans.ndjson http://localhost:8200/intake/v2/events
$ grep 'sampled":false' testdata/intake-v2/transactions_spans.ndjson
{"transaction":{"id":"85925e55b43f4341","name":"GET /api/types","type":"request","duration":13.980558,"result":"200","timestamp":1496170422000000,"sampled":false,"trace_id":"85925e55b43f4341aaaaaaaaaaaaaaaa","span_count":{"started":0},"context":{"service":null}}}
GET traces-apm*/_search
{
  "query": {
    "match": {
      "transaction.id": "85925e55b43f4341"
    }
  }
}

// ...returns no results.

Ingested upsampled rum transactions:

{"metadata":{"service":{"name":"apm-agent-js","version":"1.0.0","agent":{"name":"rum-js","version":"0.0.0"}},"network":{"connection":{"type":"5G"}}}}
{"transaction":{"sampled":false,"id":"611f4fa950f04632","type":"page-load","duration":643,"context":{"page":{"referer":"http://localhost:8000/test/e2e/","url":"http://localhost:8000/test/e2e/general-usecase/"}},"trace_id":"611f4fa950f04631aaaaaaaaaaaaaaaa","span_count":{"started":1},"experience":{"cls":1,"fid":2.0,"tbt":3.4,"ignored":5,"also":"ignored"}}}
{"span":{"name":"transaction","type":"transaction","start":0,"duration":643,"stacktrace":[{"abs_path":"http://localhost:8000/test/e2e/general-usecase/bundle.js.map","filename":"test/e2e/general-usecase/bundle.js.map","function":"<anonymous>","lineno":1,"colno":18},{"abs_path":"http://localhost:8000/test/e2e/general-usecase/bundle.js.map","filename":"~/test/e2e/general-usecase/bundle.js.map","function":"<anonymous>","lineno":1,"colno":18}],"context":{"http":{"url":"http://localhost:8000/test/e2e/general-usecase/span"}},"transaction_id":"611f4fa950f04631","parent_id":"611f4fa950f04631","trace_id":"611f4fa950f04631aaaaaaaaaaaaaaaa","id":"aaaaaaaaaaaaaaaa"}}
GET traces-apm.rum*/_search
{
  "query": {
    "match": {
      "transaction.id": "611f4fa950f04632"
    }
  }
}
{
  "took" : 127,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.7917595,
    "hits" : [
      {
        "_index" : ".ds-traces-apm.rum-default-2021.12.21-000001",
        "_id" : "rDP6230ByDpadNqDuU9C",
        "_score" : 1.7917595,
        "_source" : {
          "agent" : {
            "name" : "rum-js",
            "version" : "0.0.0"
          },
          "data_stream.namespace" : "default",
          "source" : {
            "port" : 53042,
            "ip" : "127.0.0.1"
          },
          "data_stream.type" : "traces",
          "processor" : {
            "name" : "transaction",
            "event" : "transaction"
          },
          "url" : {
            "path" : "/test/e2e/general-usecase/",
            "original" : "http://localhost:8000/test/e2e/general-usecase/",
            "scheme" : "http",
            "port" : 8000,
            "domain" : "localhost",
            "full" : "http://localhost:8000/test/e2e/general-usecase/"
          },
          "network" : {
            "connection" : {
              "type" : "5G"
            }
          },
          "observer" : {
            "hostname" : "Marcs-MacBook-Pro-2.local",
            "id" : "bf2a6d26-c5ed-4b41-a507-feab5b75c095",
            "type" : "apm-server",
            "ephemeral_id" : "fdeb69ec-dad9-4791-b23f-022c3948ec37",
            "version" : "8.0.0",
            "version_major" : 8
          },
          "trace" : {
            "id" : "611f4fa950f04631aaaaaaaaaaaaaaaa"
          },
          "@timestamp" : "2021-12-21T15:53:00.481+08:00",
          "ecs" : {
            "version" : "1.12.0"
          },
          "service" : {
            "name" : "apm-agent-js",
            "version" : "1.0.0"
          },
          "data_stream.dataset" : "apm.rum",
          "http" : {
            "request" : {
              "referrer" : "http://localhost:8000/test/e2e/"
            }
          },
          "client" : {
            "ip" : "127.0.0.1"
          },
          "event" : {
            "agent_id_status" : "missing",
            "ingested" : "2021-12-21T07:53:01Z",
            "outcome" : "unknown"
          },
          "user_agent" : {
            "original" : "curl/7.64.1",
            "name" : "curl",
            "device" : {
              "name" : "Other"
            },
            "version" : "7.64.1"
          },
          "transaction" : {
            "duration" : {
              "us" : 643000
            },
            "id" : "611f4fa950f04632",
            "span_count" : {
              "started" : 1
            },
            "type" : "page-load",
            "experience" : {
              "fid" : 2,
              "tbt" : 3.4,
              "cls" : 1
            }
          },
          "timestamp" : {
            "us" : 1640073180481075
          }
        }
      }
    ]
  }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[RUM] Keep unsampled transactions for the rum intake
5 participants