[EBPF-474] Improve kernel information in platforms.json #24097

gjulianm · 2024-03-26T11:24:02Z

What does this PR do?

Change the format of the platforms.json file, with more extended version information that gets generated automatically, including kernel version, using the manifest files generated in the ami-builder repo (https://github.com/DataDog/ami-builder/pull/171)

Motivation

Ensure that the list of available platforms is updated
Ensure that VM tags are always constructed in a standard, predictable way

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

pr-commenter · 2024-03-26T12:12:56Z

Regression Detector

Regression Detector Results

Run ID: 394ec212-3470-4904-a5da-5bbe49528c04
Baseline: 7d2aebf
Comparison: 008eb28

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI
❌	file_to_blackhole	% cpu utilization	+17.52	[+10.06, +24.98]

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
❌	file_to_blackhole	% cpu utilization	+17.52	[+10.06, +24.98]
➖	process_agent_standard_check	memory utilization	+3.66	[+3.43, +3.89]
➖	tcp_syslog_to_blackhole	ingress throughput	+3.27	[-17.82, +24.36]
➖	process_agent_standard_check_with_stats	memory utilization	+0.75	[+0.70, +0.80]
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	+0.56	[-2.43, +3.55]
➖	pycheck_1000_100byte_tags	% cpu utilization	+0.42	[-4.02, +4.86]
➖	basic_py_check	% cpu utilization	+0.09	[-2.42, +2.61]
➖	otel_to_otel_logs	ingress throughput	+0.08	[-0.27, +0.43]
➖	process_agent_real_time_mode	memory utilization	+0.02	[-0.04, +0.07]
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.00	[-0.04, +0.04]
➖	trace_agent_msgpack	ingress throughput	-0.00	[-0.00, +0.00]
➖	uds_dogstatsd_to_api	ingress throughput	-0.01	[-0.21, +0.20]
➖	trace_agent_json	ingress throughput	-0.01	[-0.03, +0.01]
➖	idle	memory utilization	-0.49	[-0.53, -0.46]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

tasks/kernel_matrix_testing/platforms.py

tasks/kernel_matrix_testing/download.py

usamasaqib

Minor comments. Otherwise looks good.

usamasaqib · 2024-04-23T09:46:48Z

tasks/kmt.py

+    res = ctx.run(
+        "aws-vault exec sso-staging-engineering -- aws s3 ls --recursive s3://dd-agent-omnibus/kernel-version-testing/rootfs"
+    )
+    if res is None or not res.ok:


I think if you want to check the error condition explicitly then you need to specify warn=True in the call above. Otherwise ctx.run will throw an exception.

usamasaqib · 2024-04-23T09:48:38Z

tasks/kmt.py

+                image_name = keyvals['IMAGE_NAME']
+                image_filename = keyvals['IMAGE_FILENAME']
+            except KeyError:
+                warn(f"[!] Invalid manifest {manifest}, skipping")


I think we should consider this version corrupted if any manifest file is invalid, and refuse to update the platforms file.

usamasaqib · 2024-04-23T10:03:45Z

Can we also filter out the custom kernels in kmt.ls. At the moment they just add noise.

gjulianm · 2024-04-23T16:50:32Z

/trigger-ci --variable RUN_ALL_BUILDS=true --variable RUN_KITCHEN_TESTS=true --variable RUN_E2E_TESTS=on --variable RUN_UNIT_TESTS=on

gjulianm · 2024-04-23T16:50:48Z

/trigger-ci --variable RUN_KMT_TESTS=on

dd-devflow · 2024-04-23T16:51:07Z

🚂 Gitlab pipeline started

Started pipeline #32725041

dd-devflow · 2024-04-23T16:51:18Z

🚂 Gitlab pipeline started

Started pipeline #32725059

agent-platform-auto-pr · 2024-04-24T09:02:45Z

[Fast Unit Tests Report]

On pipeline 32888982 (CI Visibility). The following jobs did not run any unit tests:

Jobs:

tests_deb-arm64-py3
tests_deb-x64-py3
tests_flavor_dogstatsd_deb-x64
tests_flavor_heroku_deb-x64
tests_flavor_iot_deb-x64
tests_rpm-arm64-py3
tests_rpm-x64-py3
tests_windows-x64

If you modified Go files and expected unit tests to run in these jobs, please double check the job logs. If you think tests should have been executed reach out to #agent-developer-experience

paulcacheux · 2024-04-24T09:25:30Z

.gitlab/kernel_matrix_testing/security_agent.yml

-          - "ubuntu_18.04"
-          - "ubuntu_20.04"
-          - "ubuntu_22.04"
-          - "ubuntu_23.10"


I don't understand this change, ubuntu releases 2 versions per year 2x.04 and 2x.10, how do you differentiate between the 2 ?

Before, we were using custom names in the platform.json as that was edited manually. Now we switch to automatic generation, so we use the same names there are in the ami-builder repo. There, the Ubuntu versions are defined just with the release year. As we only have one image per release year there's no risk of using different images in the CI. However I agree that it can be confusing for the devs, so I'll make a followup PR in ami-builder to use unambigous names and then update them here.

Honestly I still disagree, in december of 2024 we will most likely want to have both 24.04 (the latest LTS) and 24.10 (the latest ubuntu, playing the role of 23.10 currently), I don't see how you represent that here

So yeah to me the real issue is https://github.com/DataDog/ami-builder/blame/7cfb8eda153a86bf46d70bd98908639342927529/.gitlab/distro.yml#L105, which really does not make any sense

Indeed, I'll make a PR changing those names and then update them here.

gjulianm · 2024-04-25T09:04:28Z

/merge

dd-devflow · 2024-04-25T09:04:36Z

🚂 MergeQueue

Pull request added to the queue.

This build is next! (estimated merge in less than 47m)

Use /merge -c to cancel this operation!

* Allow downloading images for all architectures * Add update-platform-info task * Improve output of kmt.ls * Adapt to new platforms.json structure * Use guestfs python package * Ignore rescue images when detecting kernel * Ignore empty version names * Fix kernel detection * Use dynamic distribution mappings * Update platform info * Remove empty alternative names * Better help * Fix archs for custom kernels * Fix import order * Fix format * Use manifest files * Improve sorting in kmt.ls * Update platforms.json * Warn when some images have not been updated * Update platforms.json * Update to latest images * Raise an error when image is missing * Fix platforms.json file * Add kmt.validate-platform-info task * review comments * Improve kmt.ls * Show an error when kernels are missing * Fix image names in CI * Switch Amazon 2023 version * Use master version for AL2023 * test with updated names * Update Amazon 2023 name * Add --exclude-matching argument to update-platform-info task * Update platforms.json

gjulianm self-assigned this Mar 26, 2024

gjulianm changed the title ~~Allow downloading images for all architectures~~ [EBPF] Improve kernel information in platforms.json Mar 27, 2024

gjulianm added changelog/no-changelog team/ebpf-platform qa/done QA done before merge and regressions are covered by tests labels Mar 27, 2024

gjulianm added this to the 7.54.0 milestone Mar 27, 2024

Base automatically changed from guillermo.julian/fix-kmt-image-suffix to main April 1, 2024 09:00

gjulianm added 10 commits April 2, 2024 11:46

Allow downloading images for all architectures

ba9b0ff

Add update-platform-info task

fb12333

Improve output of kmt.ls

f3d7df0

Adapt to new platforms.json structure

d2e4ead

Use guestfs python package

695e0a1

Ignore rescue images when detecting kernel

895b2b6

Ignore empty version names

d835886

Fix kernel detection

9dabe97

Use dynamic distribution mappings

70e6e1c

Update platform info

b73d346

gjulianm force-pushed the guillermo.julian/improved-kmt-kernel-info branch from 4174f53 to b73d346 Compare April 2, 2024 09:47

gjulianm added 4 commits April 2, 2024 11:55

Remove empty alternative names

474bbbc

Better help

33d7d1f

Fix archs for custom kernels

2b7fe93

Fix import order

f509678

gjulianm marked this pull request as ready for review April 2, 2024 10:37

gjulianm requested a review from a team as a code owner April 2, 2024 10:37

usamasaqib reviewed Apr 2, 2024

View reviewed changes

tasks/kernel_matrix_testing/platforms.py Outdated Show resolved Hide resolved

usamasaqib reviewed Apr 2, 2024

View reviewed changes

tasks/kernel_matrix_testing/download.py Show resolved Hide resolved

tasks/kernel_matrix_testing/download.py Show resolved Hide resolved

gjulianm added 3 commits April 9, 2024 17:32

Merge branch 'main' into guillermo.julian/improved-kmt-kernel-info

3e3c9bf

Fix format

e8409f6

Use manifest files

9a78fcc

usamasaqib approved these changes Apr 23, 2024

View reviewed changes

gjulianm added 6 commits April 23, 2024 12:40

review comments

1b6bf91

Merge branch 'main' into guillermo.julian/improved-kmt-kernel-info

c62c3f3

Merge branch 'main' into guillermo.julian/improved-kmt-kernel-info

61ef0cf

Improve kmt.ls

4a23d83

Show an error when kernels are missing

4451e47

Fix image names in CI

b248670

gjulianm requested a review from a team as a code owner April 23, 2024 12:36

Switch Amazon 2023 version

008eb28

paulcacheux requested changes Apr 24, 2024

View reviewed changes

paulcacheux approved these changes Apr 24, 2024

View reviewed changes

gjulianm added 6 commits April 24, 2024 12:08

Use master version for AL2023

d21ee51

test with updated names

bd18087

Update Amazon 2023 name

876aafb

Add --exclude-matching argument to update-platform-info task

cd7f259

Update platforms.json

b508fa6

Merge branch 'main' into guillermo.julian/improved-kmt-kernel-info

2e0c04e

dd-mergequeue bot merged commit db828d1 into main Apr 25, 2024
273 of 274 checks passed

dd-mergequeue bot deleted the guillermo.julian/improved-kmt-kernel-info branch April 25, 2024 10:14

gjulianm mentioned this pull request Dec 4, 2024

[EBPF] Backport KMT platform files to 6.53.x #31699

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[EBPF-474] Improve kernel information in platforms.json #24097

[EBPF-474] Improve kernel information in platforms.json #24097

gjulianm commented Mar 26, 2024 •

edited

Loading

pr-commenter bot commented Mar 26, 2024 •

edited

Loading

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

usamasaqib left a comment

usamasaqib Apr 23, 2024

usamasaqib Apr 23, 2024

usamasaqib commented Apr 23, 2024

gjulianm commented Apr 23, 2024

gjulianm commented Apr 23, 2024

dd-devflow bot commented Apr 23, 2024

dd-devflow bot commented Apr 23, 2024

agent-platform-auto-pr bot commented Apr 24, 2024 •

edited

Loading

paulcacheux Apr 24, 2024

gjulianm Apr 24, 2024

paulcacheux Apr 24, 2024

paulcacheux Apr 24, 2024

gjulianm Apr 24, 2024

gjulianm commented Apr 25, 2024

dd-devflow bot commented Apr 25, 2024

[EBPF-474] Improve kernel information in platforms.json #24097

[EBPF-474] Improve kernel information in platforms.json #24097

Conversation

gjulianm commented Mar 26, 2024 • edited Loading

What does this PR do?

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

pr-commenter bot commented Mar 26, 2024 • edited Loading

Regression Detector

Regression Detector Results

No significant changes in experiment optimization goals

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

usamasaqib left a comment

Choose a reason for hiding this comment

usamasaqib Apr 23, 2024

Choose a reason for hiding this comment

usamasaqib Apr 23, 2024

Choose a reason for hiding this comment

usamasaqib commented Apr 23, 2024

gjulianm commented Apr 23, 2024

gjulianm commented Apr 23, 2024

dd-devflow bot commented Apr 23, 2024

dd-devflow bot commented Apr 23, 2024

agent-platform-auto-pr bot commented Apr 24, 2024 • edited Loading

paulcacheux Apr 24, 2024

Choose a reason for hiding this comment

gjulianm Apr 24, 2024

Choose a reason for hiding this comment

paulcacheux Apr 24, 2024

Choose a reason for hiding this comment

paulcacheux Apr 24, 2024

Choose a reason for hiding this comment

gjulianm Apr 24, 2024

Choose a reason for hiding this comment

gjulianm commented Apr 25, 2024

dd-devflow bot commented Apr 25, 2024

gjulianm commented Mar 26, 2024 •

edited

Loading

pr-commenter bot commented Mar 26, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Apr 24, 2024 •

edited

Loading