Introduce model.Processor #5984

axw · 2021-08-20T08:59:08Z

Motivation/summary

Introduce type model.Processor and field model.APMEvent.Processor, which controls the processor.name and processor.event fields. These fields are used for identifying the event type, used for routing to the appropriate index/data stream and also for monitoring purposes.

Processed event metrics have been moved to a dedicated model.BatchProcessor, and out of the model package; they use the Processor field to form the metric name.

Note: Several unused metrics have been removed: stack trace and stack frame counts.

Checklist

Update CHANGELOG.asciidoc
~~- [ ] Documentation has been updated~~

How to test these changes

Enable monitoring, ingest some events, check that apm-server.processor.<event-type>.transformations metrics are recorded.

Related issues

#4410
#4120
#4715

apmmachine · 2021-08-20T09:45:41Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-24T02:50:06.266+0000
Duration: 42 min 5 sec
Commit: 876955f

Test stats 🧪

Test	Results
Failed	0
Passed	5940
Skipped	14
Total	5954

Trends 🧪

Processor identifies the event kind: transaction, span, metricset, error, or profile. This is used for routing events to the appropriate index or data stream, and in queries for fetching events of a given kind.

Also, remove unused metrics.

stuartnelson3

Big fan of the move to explicitly setting the datastream based on Processor. One question where it looks like processor might be getting double set, but that's all.

stuartnelson3 · 2021-08-23T11:31:30Z

model/modeldecoder/rumv3/decoder.go

-		*batch = append(*batch, metricset)
+		event := input.Base
+		mapToMetricsetModel(&m, &event)
+		event.Processor = model.MetricsetProcessor


It appears this event.Processor = model.MetricsetProcessor is also being set in mapToMetricsetModel? Is that intentional?

good catch, I've removed these ones and consolidated on setting it in mapToFooModel

stuartnelson3 · 2021-08-23T11:43:28Z

model/modelprocessor/datastream.go

-	switch {
-	case event.Transaction != nil || event.Span != nil:
+	switch event.Processor {
+	case model.SpanProcessor, model.TransactionProcessor:


mergify · 2021-08-24T00:08:30Z

This pull request is now in conflicts. Could you fix it @axw? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b model-event-processor upstream/model-event-processor
git merge upstream/master
git push upstream model-event-processor

…ent-processor

* Introduce model.Processor Processor identifies the event kind: transaction, span, metricset, error, or profile. This is used for routing events to the appropriate index or data stream, and in queries for fetching events of a given kind. * Update SetDataStream to use APMEvent.Processor * Move monitoring event counters to model processor Also, remove unused metrics. * Update aggregators to use Processor * sampling: use Processor * Update changelog * Remove redundant setting of event.Processor Co-authored-by: stuart nelson <[email protected]> (cherry picked from commit cb6b2da) # Conflicts: # changelogs/head.asciidoc # model/apmevent.go # model/apmevent_test.go # model/modeldecoder/rumv3/decoder.go # model/span.go # model/transaction.go # processor/otel/exceptions_test.go # x-pack/apm-server/sampling/groups_test.go # x-pack/apm-server/sampling/processor.go # x-pack/apm-server/sampling/processor_test.go

* Introduce model.Processor Processor identifies the event kind: transaction, span, metricset, error, or profile. This is used for routing events to the appropriate index or data stream, and in queries for fetching events of a given kind. * Update SetDataStream to use APMEvent.Processor * Move monitoring event counters to model processor Also, remove unused metrics. * Update aggregators to use Processor * sampling: use Processor * Update changelog * Remove redundant setting of event.Processor Co-authored-by: stuart nelson <[email protected]> # Conflicts: # changelogs/head.asciidoc

marclop · 2021-10-28T06:56:53Z

Verified this running 7.16.0 BC1 locally:

./apm-server -E instrumentation.enabled=true -E output.elasticsearch.username=admin -E output.elasticsearch.password=changeme -E apm-server.expvar.enabled=true
curl -sL http://localhost:8200/debug/vars | grep transformations:

"apm-server.processor.span.transformations": 58,
"apm-server.processor.transaction.transformations": 40,
"apm-server.processor.metric.transformations": 59,

axw added backport-7.x v7.16.0 labels Aug 20, 2021

zube bot added the [zube]: In Review label Aug 20, 2021

axw added 5 commits August 21, 2021 10:35

Introduce model.Processor

7f47042

Processor identifies the event kind: transaction, span, metricset, error, or profile. This is used for routing events to the appropriate index or data stream, and in queries for fetching events of a given kind.

Update SetDataStream to use APMEvent.Processor

dee6a25

Move monitoring event counters to model processor

d2ee7d2

Also, remove unused metrics.

Update aggregators to use Processor

b8c703b

sampling: use Processor

9a20129

axw force-pushed the model-event-processor branch from b09a075 to 9a20129 Compare August 21, 2021 02:41

axw requested a review from a team August 23, 2021 03:56

axw marked this pull request as ready for review August 23, 2021 03:56

stuartnelson3 self-assigned this Aug 23, 2021

stuartnelson3 approved these changes Aug 23, 2021

View reviewed changes

Merge branch 'master' into model-event-processor

9c8dd8d

stuartnelson3 removed their assignment Aug 23, 2021

simitt approved these changes Aug 23, 2021

View reviewed changes

axw added 5 commits August 24, 2021 08:17

Merge branch 'master' into model-event-processor

674c73e

Update changelog

5de2075

Merge commit '9c8dd8d289cbb392938fff1368af9434c5b54e12' into model-ev…

4ff8136

…ent-processor

Remove redundant setting of event.Processor

69dbf45

Merge branch 'master' into model-event-processor

876955f

axw enabled auto-merge (squash) August 24, 2021 02:49

axw merged commit cb6b2da into elastic:master Aug 24, 2021

zube bot added [zube]: Done and removed [zube]: In Review labels Aug 24, 2021

mergify bot mentioned this pull request Aug 24, 2021

[7.x] Introduce model.Processor (backport #5984) #6028

Closed

axw mentioned this pull request Aug 24, 2021

[7.x] Introduce model.Processor (#5984) #6033

Merged

marclop added backport-skip Skip notification from the automated backport with mergify test-plan labels Oct 25, 2021

marclop self-assigned this Oct 28, 2021

marclop added the test-plan-ok label Oct 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce model.Processor #5984

Introduce model.Processor #5984

axw commented Aug 20, 2021 •

edited

Loading

apmmachine commented Aug 20, 2021 •

edited

Loading

Build stats

Test stats 🧪

Trends 🧪

stuartnelson3 left a comment

stuartnelson3 Aug 23, 2021

axw Aug 24, 2021

stuartnelson3 Aug 23, 2021

mergify bot commented Aug 24, 2021

marclop commented Oct 28, 2021

Introduce model.Processor #5984

Introduce model.Processor #5984

Conversation

axw commented Aug 20, 2021 • edited Loading

Motivation/summary

Checklist

How to test these changes

Related issues

apmmachine commented Aug 20, 2021 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

stuartnelson3 left a comment

Choose a reason for hiding this comment

stuartnelson3 Aug 23, 2021

Choose a reason for hiding this comment

axw Aug 24, 2021

Choose a reason for hiding this comment

stuartnelson3 Aug 23, 2021

Choose a reason for hiding this comment

mergify bot commented Aug 24, 2021

marclop commented Oct 28, 2021

axw commented Aug 20, 2021 •

edited

Loading

apmmachine commented Aug 20, 2021 •

edited

Loading