Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sampling: introduce event processor #4241

Merged
merged 3 commits into from
Oct 7, 2020
Merged

sampling: introduce event processor #4241

merged 3 commits into from
Oct 7, 2020

Conversation

axw
Copy link
Member

@axw axw commented Sep 29, 2020

Motivation/summary

Implement a tail-based sampling event processor.

The processor passes root transactions through a reservoir sampler, and periodically "finalizes" the reservoirs to make local sampling decisions. The resulting sampled trace events are then indexed into Elasticsearch; sampled trace IDs are published for other APM Servers to index related events.

To be done in follow-ups:

  • introduce config and wire up the processor in x-pack/apm-server/main.go
  • add system tests

Checklist

I have considered changes for:
- [ ] documentation

How to test these changes

make test

Related issues

Part of #4185

@apmmachine
Copy link
Contributor

apmmachine commented Sep 29, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #4241 updated]

  • Start Time: 2020-10-07T10:36:04.694+0000

  • Duration: 43 min 41 sec

Test stats 🧪

Test Results
Failed 0
Passed 4278
Skipped 145
Total 4423

Steps errors 2

Expand to view the steps failures

  • Name: Compress

    • Description: tar --exclude=coverage-files.tgz -czf coverage-files.tgz coverage

    • Duration: 0 min 0 sec

    • Start Time: 2020-10-07T10:52:05.039+0000

    • log

  • Name: Compress

    • Description: tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests

    • Duration: 0 min 0 sec

    • Start Time: 2020-10-07T11:04:08.125+0000

    • log

@axw axw force-pushed the tbs-processor branch 4 times, most recently from 3529d05 to f6401c3 Compare September 30, 2020 09:48
@axw axw marked this pull request as ready for review October 5, 2020 00:06
@axw axw requested a review from a team October 5, 2020 00:06
Copy link
Contributor

@simitt simitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only minor comments

x-pack/apm-server/sampling/config.go Show resolved Hide resolved
x-pack/apm-server/sampling/processor.go Outdated Show resolved Hide resolved
if err == errTooManyTraceGroups {
// Too many trace groups, drop the transaction.
//
// TODO(axw) log a warning with a rate limit.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on logging

x-pack/apm-server/sampling/processor.go Show resolved Hide resolved
@axw axw merged commit 6ab9ddb into elastic:master Oct 7, 2020
@axw axw deleted the tbs-processor branch October 7, 2020 11:28
@simitt simitt self-assigned this Oct 15, 2020
@simitt
Copy link
Contributor

simitt commented Oct 16, 2020

nothing to be tested here manually as the processor is not used yet.

axw added a commit to axw/apm-server that referenced this pull request Nov 18, 2020
axw added a commit that referenced this pull request Nov 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants