Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SVLS-4238] Add warning log when too many tags #21480

Closed
wants to merge 2 commits into from
Closed

[SVLS-4238] Add warning log when too many tags #21480

wants to merge 2 commits into from

Conversation

joeyzhao2018
Copy link
Contributor

What does this PR do?

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Reviewer's Checklist

  • If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
  • Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
  • A release note has been added or the changelog/no-changelog label has been applied.
  • Changed code has automated tests for its functionality.
  • Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
  • At least one team/.. label has been applied, indicating the team(s) that should QA this change.
  • If applicable, docs team has been notified or an issue has been opened on the documentation repo.
  • If applicable, the need-change/operator and need-change/helm labels have been applied.
  • If applicable, the k8s/<min-version> label, indicating the lowest Kubernetes version compatible with this feature.
  • If applicable, the config template has been updated.

@joeyzhao2018 joeyzhao2018 requested a review from a team as a code owner December 11, 2023 18:59
@joeyzhao2018 joeyzhao2018 added this to the Triage milestone Dec 11, 2023
@pr-commenter
Copy link

pr-commenter bot commented Dec 11, 2023

Bloop Bleep... Dogbot Here

Regression Detector Results

Run ID: c68a3f6d-aea5-47e0-8737-8045905bfb65
Baseline: 8e3aa2f
Comparison: c46f75b
Total CPUs: 7

Performance changes are noted in the perf column of each table:

  • ✅ denotes better comparison variant performance
  • ❌ denotes worse comparison variant performance

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

  1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

  2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

  3. Its configuration does not mark it "erratic".

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Declared stable experiments that are now erratic

An experiment is erratic (i.e., not stable) if its coefficient of variation is at least 0.10.

perf experiment goal Δ mean % Δ mean % CI confidence
✅ otel_to_otel_logs ingress throughput +1.11 [+0.40, +1.81] 99.05%

Experiments with missing or malformed data

  • idle

Usually, this warning means that there is no usable optimization goal data for that experiment, which could be a result of misconfiguration.

Fine details of change detection per experiment

perf experiment goal Δ mean % Δ mean % CI confidence
✅ otel_to_otel_logs ingress throughput +1.11 [+0.40, +1.81] 99.05%
✅ file_tree egress throughput +0.05 [-1.74, +1.84] 3.80%
❌ tcp_dd_logs_filter_exclude ingress throughput +0.00 [-0.00, +0.00] 0.00%
❌ trace_agent_msgpack ingress throughput +0.00 [-0.00, +0.00] 0.00%
❌ dogstatsd_string_interner_8MiB_100 ingress throughput +0.00 [-0.00, +0.00] 0.00%
❌ dogstatsd_string_interner_8MiB_1k ingress throughput +0.00 [-0.00, +0.00] 0.00%
❌ dogstatsd_string_interner_64MiB_100 ingress throughput +0.00 [-0.00, +0.00] 0.00%
❌ uds_dogstatsd_to_api ingress throughput +0.00 [-0.00, +0.00] 0.00%
❌ dogstatsd_string_interner_8MiB_50k ingress throughput +0.00 [-0.00, +0.00] 0.00%
❌ trace_agent_json ingress throughput +0.00 [-0.01, +0.01] 0.00%
❌ dogstatsd_string_interner_64MiB_1k ingress throughput +0.00 [-0.00, +0.00] 0.00%
❌ dogstatsd_string_interner_8MiB_10k ingress throughput +0.00 [-0.00, +0.00] 0.00%
❌ dogstatsd_string_interner_128MiB_100 ingress throughput +0.00 [-0.00, +0.00] 0.00%
❌ dogstatsd_string_interner_128MiB_1k ingress throughput +0.00 [-0.02, +0.02] 0.00%
❌ dogstatsd_string_interner_8MiB_100k ingress throughput -0.00 [-0.02, +0.02] 1.11%
❌ file_to_blackhole egress throughput -0.02 [-6.19, +6.14] 0.50%
❌ tcp_syslog_to_blackhole ingress throughput -0.15 [-0.21, -0.08] 99.97%

@joeyzhao2018 joeyzhao2018 changed the title add warning log when too many tags [SVLS-4238] Add warning log when too many tags Dec 12, 2023
purple4reina
purple4reina approved these changes Dec 12, 2023
View reviewed changes
Copy link
Contributor

@purple4reina purple4reina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💜

purple4reina
purple4reina reviewed Dec 12, 2023
View reviewed changes
Copy link
Contributor

@purple4reina purple4reina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I know I just approved this, but can we move this to the span modifier? This check doesn't include the fact that the trace agent will later add the global tags to the span.

@joeyzhao2018 joeyzhao2018 reopened this Dec 15, 2023
@github-actions GitHub Actions
Copy link
Contributor

Serverless Benchmark Results

BenchmarkStartEndInvocation comparison between 81dd765 and f4ee62d.

tl;dr

  1. Skim down the vs base column in each chart. If there is a ~, then there was no statistically significant change to the benchmark. Otherwise, ensure the estimated percent change is either negative or very small.

  2. The last row of each chart is the geomean. Ensure this percentage is either negative or very small.

What is this benchmarking?

The BenchmarkStartEndInvocation compares the amount of time it takes to call the start-invocation and end-invocation endpoints. For universal instrumentation languages (Dotnet, Golang, Java, Ruby), this represents the majority of the duration overhead added by our tracing layer.

The benchmark is run using a large variety of lambda request payloads. In the charts below, there is one row for each event payload type.

How do I interpret these charts?

The charts below comes from benchstat. They represent the statistical change in duration (sec/op), memory overhead (B/op), and allocations (allocs/op).

The benchstat docs explain how to interpret these charts.

Before the comparison table, we see common file-level configuration. If there are benchmarks with different configuration (for example, from different packages), benchstat will print separate tables for each configuration.

The table then compares the two input files for each benchmark. It shows the median and 95% confidence interval summaries for each benchmark before and after the change, and an A/B comparison under "vs base". ... The p-value measures how likely it is that any differences were due to random chance (i.e., noise). The "~" means benchstat did not detect a statistically significant difference between the two inputs. ...

Note that "statistically significant" is not the same as "large": with enough low-noise data, even very small changes can be distinguished from noise and considered statistically significant. It is, of course, generally easier to distinguish large changes from noise.

Finally, the last row of the table shows the geometric mean of each column, giving an overall picture of how the benchmarks changed. Proportional changes in the geomean reflect proportional changes in the benchmarks. For example, given n benchmarks, if sec/op for one of them increases by a factor of 2, then the sec/op geomean will increase by a factor of ⁿ√2.

Benchmark stats 
goos: linux
goarch: amd64
pkg: github.com/DataDog/datadog-agent/pkg/serverless/daemon
cpu: AMD EPYC 7763 64-Core Processor                
                                      │   previous    │               current                │
                                      │    sec/op     │    sec/op     vs base                │
api-gateway-appsec.json                  159.5µ ±  2%   158.9µ ±  1%        ~ (p=0.853 n=10)
api-gateway-kong-appsec.json             105.5µ ±  1%   106.4µ ±  2%        ~ (p=0.190 n=10)
api-gateway-kong.json                    102.0µ ±  3%   104.0µ ±  3%        ~ (p=0.105 n=10)
api-gateway-non-proxy-async.json         160.0µ ±  2%   158.0µ ±  2%        ~ (p=0.353 n=10)
api-gateway-non-proxy.json               158.3µ ±  1%   156.8µ ±  2%        ~ (p=0.393 n=10)
api-gateway-websocket-connect.json       101.3µ ±  4%   101.3µ ±  2%        ~ (p=1.000 n=10)
api-gateway-websocket-default.json       86.79µ ±  3%   86.41µ ±  4%        ~ (p=0.684 n=10)
api-gateway-websocket-disconnect.json    84.54µ ±  3%   84.38µ ±  4%        ~ (p=0.971 n=10)
api-gateway.json                         183.1µ ±  2%   182.2µ ±  2%        ~ (p=0.280 n=10)
application-load-balancer.json           80.45µ ±  4%   79.26µ ±  4%        ~ (p=0.481 n=10)
cloudfront.json                          64.70µ ±  3%   64.18µ ±  5%        ~ (p=0.529 n=10)
cloudwatch-events.json                   49.17µ ±  3%   49.82µ ±  2%        ~ (p=0.239 n=10)
cloudwatch-logs.json                     66.66µ ±  4%   66.68µ ±  3%        ~ (p=1.000 n=10)
custom.json                              34.13µ ±  8%   33.30µ ±  8%        ~ (p=0.190 n=10)
dynamodb.json                            183.2µ ±  5%   176.9µ ±  4%   -3.41% (p=0.019 n=10)
empty.json                               33.80µ ± 11%   30.50µ ±  5%        ~ (p=0.063 n=10)
eventbridge-custom.json                  57.11µ ±  8%   53.03µ ±  3%   -7.15% (p=0.023 n=10)
http-api.json                            99.06µ ±  8%   89.18µ ±  4%   -9.97% (p=0.000 n=10)
kinesis-batch.json                       116.6µ ±  4%   109.6µ ±  5%   -6.04% (p=0.000 n=10)
kinesis.json                             87.70µ ±  6%   76.41µ ±  3%  -12.87% (p=0.000 n=10)
s3.json                                 109.29µ ±  9%   96.55µ ±  5%  -11.65% (p=0.011 n=10)
sns-batch.json                           160.3µ ± 16%   153.9µ ±  4%        ~ (p=0.218 n=10)
sns.json                                 106.9µ ± 12%   102.4µ ±  5%        ~ (p=0.165 n=10)
snssqs.json                              164.9µ ± 17%   161.0µ ±  4%        ~ (p=0.165 n=10)
snssqs_no_dd_context.json                161.9µ ± 11%   149.5µ ±  9%   -7.70% (p=0.029 n=10)
sqs-aws-header.json                      96.38µ ±  9%   95.57µ ±  9%        ~ (p=0.393 n=10)
sqs-batch.json                           158.4µ ± 18%   166.9µ ± 33%        ~ (p=0.315 n=10)
sqs.json                                 110.0µ ± 17%   119.9µ ± 14%        ~ (p=0.631 n=10)
sqs_no_dd_context.json                   109.5µ ± 12%   102.3µ ± 16%        ~ (p=0.912 n=10)
geomean                                  100.4µ         97.67µ         -2.75%

                                      │   previous    │               current                │
                                      │     B/op      │     B/op       vs base               │
api-gateway-appsec.json                 47.11Ki ±  2%   47.07Ki ±  2%       ~ (p=0.912 n=10)
api-gateway-kong-appsec.json            30.99Ki ± 10%   30.99Ki ± 10%       ~ (p=0.813 n=10)
api-gateway-kong.json                   28.31Ki ± 17%   28.31Ki ± 22%       ~ (p=0.147 n=10)
api-gateway-non-proxy-async.json        54.89Ki ±  0%   54.89Ki ±  0%       ~ (p=0.868 n=10)
api-gateway-non-proxy.json              53.46Ki ±  0%   53.47Ki ±  0%       ~ (p=0.138 n=10)
api-gateway-websocket-connect.json      31.12Ki ±  0%   31.12Ki ±  0%       ~ (p=0.781 n=10)
api-gateway-websocket-default.json      25.02Ki ±  0%   25.02Ki ±  0%       ~ (p=0.540 n=10)
api-gateway-websocket-disconnect.json   24.56Ki ±  0%   24.56Ki ±  0%       ~ (p=0.722 n=10)
api-gateway.json                        59.24Ki ±  0%   59.24Ki ±  0%       ~ (p=0.739 n=10)
application-load-balancer.json          23.85Ki ±  0%   23.84Ki ±  0%       ~ (p=0.724 n=10)
cloudfront.json                         18.58Ki ±  0%   18.58Ki ±  0%       ~ (p=0.920 n=10)
cloudwatch-events.json                  12.57Ki ±  0%   12.57Ki ±  0%       ~ (p=0.150 n=10)
cloudwatch-logs.json                    54.57Ki ±  0%   54.57Ki ±  0%       ~ (p=0.541 n=10)
custom.json                             9.694Ki ±  0%   9.694Ki ±  0%       ~ (p=0.897 n=10)
dynamodb.json                           52.27Ki ±  0%   52.26Ki ±  0%       ~ (p=0.259 n=10)
empty.json                              9.146Ki ±  0%   9.146Ki ±  0%       ~ (p=0.446 n=10)
eventbridge-custom.json                 13.75Ki ±  0%   13.75Ki ±  0%       ~ (p=0.320 n=10)
http-api.json                           24.86Ki ±  0%   24.86Ki ±  0%       ~ (p=0.181 n=10)
kinesis-batch.json                      28.67Ki ±  0%   28.67Ki ±  0%       ~ (p=0.867 n=10)
kinesis.json                            18.66Ki ±  0%   18.67Ki ±  0%       ~ (p=0.839 n=10)
s3.json                                 21.86Ki ±  0%   21.86Ki ±  0%       ~ (p=0.897 n=10)
sns-batch.json                          42.60Ki ±  0%   42.60Ki ±  0%       ~ (p=0.985 n=10)
sns.json                                25.67Ki ±  0%   25.67Ki ±  0%       ~ (p=0.360 n=10)
snssqs.json                             44.84Ki ±  0%   44.83Ki ±  0%       ~ (p=0.381 n=10)
snssqs_no_dd_context.json               38.03Ki ±  0%   38.02Ki ±  0%  -0.02% (p=0.029 n=10)
sqs-aws-header.json                     19.64Ki ±  0%   19.63Ki ±  0%       ~ (p=0.809 n=10)
sqs-batch.json                          43.72Ki ±  0%   43.71Ki ±  0%       ~ (p=0.928 n=10)
sqs.json                                26.70Ki ±  0%   26.70Ki ±  0%  +0.02% (p=0.014 n=10)
sqs_no_dd_context.json                  20.97Ki ±  0%   20.97Ki ±  0%       ~ (p=0.589 n=10)
geomean                                 27.72Ki         27.72Ki        -0.00%

                                      │  previous  │               current               │
                                      │ allocs/op  │ allocs/op   vs base                 │
api-gateway-appsec.json                 597.0 ± 0%   597.0 ± 0%       ~ (p=1.000 n=10)
api-gateway-kong-appsec.json            495.0 ± 0%   495.0 ± 0%       ~ (p=1.000 n=10)
api-gateway-kong.json                   473.0 ± 0%   473.0 ± 0%       ~ (p=1.000 n=10) ¹
api-gateway-non-proxy-async.json        679.0 ± 0%   679.0 ± 0%       ~ (p=1.000 n=10)
api-gateway-non-proxy.json              670.0 ± 0%   670.0 ± 0%       ~ (p=1.000 n=10)
api-gateway-websocket-connect.json      436.0 ± 0%   436.0 ± 0%       ~ (p=1.000 n=10) ¹
api-gateway-websocket-default.json      347.0 ± 0%   347.0 ± 0%       ~ (p=0.582 n=10)
api-gateway-websocket-disconnect.json   338.0 ± 0%   338.0 ± 0%       ~ (p=1.000 n=10)
api-gateway.json                        773.0 ± 0%   772.0 ± 0%       ~ (p=0.370 n=10)
application-load-balancer.json          336.0 ± 0%   336.0 ± 0%       ~ (p=1.000 n=10) ¹
cloudfront.json                         246.0 ± 0%   246.0 ± 0%       ~ (p=1.000 n=10) ¹
cloudwatch-events.json                  227.0 ± 0%   227.0 ± 0%       ~ (p=1.000 n=10) ¹
cloudwatch-logs.json                    225.0 ± 0%   225.0 ± 0%       ~ (p=1.000 n=10) ¹
custom.json                             172.0 ± 0%   172.0 ± 0%       ~ (p=1.000 n=10) ¹
dynamodb.json                           597.0 ± 0%   597.0 ± 0%       ~ (p=1.000 n=10) ¹
empty.json                              161.0 ± 0%   161.0 ± 0%       ~ (p=1.000 n=10) ¹
eventbridge-custom.json                 242.0 ± 0%   242.0 ± 0%       ~ (p=1.000 n=10) ¹
http-api.json                           385.0 ± 0%   385.0 ± 0%       ~ (p=0.303 n=10)
kinesis-batch.json                      309.0 ± 0%   309.0 ± 0%       ~ (p=1.000 n=10)
kinesis.json                            250.0 ± 0%   250.0 ± 0%       ~ (p=1.000 n=10) ¹
s3.json                                 306.0 ± 0%   306.0 ± 0%       ~ (p=1.000 n=10) ¹
sns-batch.json                          363.0 ± 0%   363.0 ± 0%       ~ (p=1.000 n=10)
sns.json                                282.0 ± 0%   282.0 ± 0%       ~ (p=1.000 n=10) ¹
snssqs.json                             338.0 ± 0%   337.0 ± 0%       ~ (p=0.370 n=10)
snssqs_no_dd_context.json               298.0 ± 0%   298.0 ± 0%       ~ (p=1.000 n=10)
sqs-aws-header.json                     244.0 ± 0%   244.0 ± 0%       ~ (p=1.000 n=10) ¹
sqs-batch.json                          406.0 ± 0%   405.5 ± 0%       ~ (p=1.000 n=10)
sqs.json                                305.0 ± 0%   305.0 ± 0%       ~ (p=1.000 n=10)
sqs_no_dd_context.json                  281.0 ± 0%   281.0 ± 0%       ~ (p=1.000 n=10)
geomean                                 343.2        343.1       -0.02%
¹ all samples are equal

@dd-devflow dd-devflow bot closed this Jun 12, 2024
@dd-devflow dd-devflow bot deleted the joey/add-warning branch June 12, 2024 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@purple4reina purple4reina purple4reina approved these changes

Assignees
No one assigned
Labels
changelog/no-changelog team/serverless
Projects
None yet
Milestone
Triage
Development

Successfully merging this pull request may close these issues.
2 participants
@joeyzhao2018 @purple4reina