Create agent internal API component #21028

Kaderinho · 2023-11-22T10:24:53Z

What does this PR do?

This PR creates the API component
The code is not yet moved so the component acts as a "proxy" for now (will be done in a follow up)

Motivation

Component framework

Also, because there are a lot of conflicts with this part of the code, this PR focused on the component creation and usage. Other PR will follow up to move code from cmd to comp + use dependencies and lifecycle hooks

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Configure the Agent

Create a JMX config file (copy paste as it)

# conf.d/jmx.d/conf.yaml

init_config:
  is_jmx: true
  collect_default_metrics: true

instances:
  - host: localhost
    port: port
    user: username
    password: password
    name: jmx_instance_name

And add logs_enabled: true to datadog.yaml

Testing

Run datadog-agent check jmx and datadog-agent jmx list everything and check that Config jmx was loaded is logged (command is expected to fail after).

Verify that command such as status / diagnose / config are corectly working

Lastly, run this script and verify that it outputs no error. (you might need to modify AUTH_TOKEN_PATH or the port on which we send request)

#!/bin/bash

AUTH_TOKEN_PATH=/opt/datadog-agent/etc/auth_token
AUTH_TOKEN=$(cat ${AUTH_TOKEN_PATH}) # or replace directly by the value

endpoints=(
        "agent/version"
        "agent/hostname"
        "agent/flare"
        "agent/stop"
        "agent/status"
        "agent/stream-event-platform"
        "agent/status/formatted"
        "agent/status/health"
        "agent/py/status"
        "agent/jmx/configs"
        "agent/gui/csrf-token"
        "agent/config-check"
        "agent/config"
        "agent/config/list-runtime"
        "agent/config/log_level"
        "agent/tagger-list"
        "agent/workload-list"
        "agent/secrets"
        "agent/metadata/v5"
        "agent/diagnose"
        "agent/dogstatsd-stats"
        "agent/stream-logs"
        "check/"
        "check/disk"
        "check/disk/reload"
)


for endpoint in ${endpoints[@]}; do
    echo "Testing endpoint '${endpoint}'"
    
    if ! curl -s --output /dev/null  "https://localhost:5001/${endpoint}" -k -H "Authorization: Bearer ${AUTH_TOKEN}"  ; then
        echo "Could not reach endpoint localhost:5001/${endpoint}"
        exit 1
    fi
    resp=$(curl --write-out '%{http_code}' --silent --output /dev/null -k -H "Authorization: Bearer ${AUTH_TOKEN}" "https://localhost:5001/${endpoint}")
    if [[ ${resp} == "404" ]]; then
        echo "Was able to reach the Agent but endpoint ${endpoint} was not configured"
    fi
done

Reviewer's Checklist

Signed-off-by: Nicolas Guerguadj <[email protected]>

cit-pr-commenter · 2023-11-22T10:27:48Z

Go Package Import Differences

Baseline: b710e3d
Comparison: 7a4ecb4

binary	os	arch	change
agent	linux	amd64	+2, -0 +github.com/DataDog/datadog-agent/comp/api/api +github.com/DataDog/datadog-agent/comp/api/api/apiimpl
agent	linux	arm64	+2, -0 +github.com/DataDog/datadog-agent/comp/api/api +github.com/DataDog/datadog-agent/comp/api/api/apiimpl
agent	windows	amd64	+2, -0 +github.com/DataDog/datadog-agent/comp/api/api +github.com/DataDog/datadog-agent/comp/api/api/apiimpl
agent	windows	386	+2, -0 +github.com/DataDog/datadog-agent/comp/api/api +github.com/DataDog/datadog-agent/comp/api/api/apiimpl
agent	darwin	amd64	+2, -0 +github.com/DataDog/datadog-agent/comp/api/api +github.com/DataDog/datadog-agent/comp/api/api/apiimpl
agent	darwin	arm64	+2, -0 +github.com/DataDog/datadog-agent/comp/api/api +github.com/DataDog/datadog-agent/comp/api/api/apiimpl
iot-agent	linux	amd64	+2, -0 +github.com/DataDog/datadog-agent/comp/api/api +github.com/DataDog/datadog-agent/comp/api/api/apiimpl
iot-agent	linux	arm64	+2, -0 +github.com/DataDog/datadog-agent/comp/api/api +github.com/DataDog/datadog-agent/comp/api/api/apiimpl
heroku-agent	linux	amd64	+2, -0 +github.com/DataDog/datadog-agent/comp/api/api +github.com/DataDog/datadog-agent/comp/api/api/apiimpl
cluster-agent	linux	amd64	+51, -0 +github.com/DataDog/datadog-agent/cmd/agent/api +github.com/DataDog/datadog-agent/cmd/agent/api/internal/agent +github.com/DataDog/datadog-agent/cmd/agent/api/internal/check +github.com/DataDog/datadog-agent/cmd/agent/gui +github.com/DataDog/datadog-agent/comp/api/api +github.com/DataDog/datadog-agent/comp/api/api/apiimpl +github.com/DataDog/datadog-agent/comp/core/flare +github.com/DataDog/datadog-agent/comp/core/workloadmeta/server +github.com/DataDog/datadog-agent/comp/logs/agent +github.com/DataDog/datadog-agent/comp/metadata/host +github.com/DataDog/datadog-agent/comp/metadata/inventoryhost +github.com/DataDog/datadog-agent/comp/metadata/resources +github.com/DataDog/datadog-agent/comp/remote-config/rcclient +github.com/DataDog/datadog-agent/pkg/gohai +github.com/DataDog/datadog-agent/pkg/gohai/cpu +github.com/DataDog/datadog-agent/pkg/gohai/filesystem +github.com/DataDog/datadog-agent/pkg/gohai/memory +github.com/DataDog/datadog-agent/pkg/gohai/network +github.com/DataDog/datadog-agent/pkg/gohai/platform +github.com/DataDog/datadog-agent/pkg/gohai/processes +github.com/DataDog/datadog-agent/pkg/gohai/processes/gops +github.com/DataDog/datadog-agent/pkg/gohai/utils +github.com/DataDog/datadog-agent/pkg/logs/internal/decoder +github.com/DataDog/datadog-agent/pkg/logs/internal/framer +github.com/DataDog/datadog-agent/pkg/logs/internal/parsers +github.com/DataDog/datadog-agent/pkg/logs/internal/parsers/dockerfile +github.com/DataDog/datadog-agent/pkg/logs/internal/parsers/encodedtext +github.com/DataDog/datadog-agent/pkg/logs/internal/parsers/kubernetes +github.com/DataDog/datadog-agent/pkg/logs/internal/parsers/noop +github.com/DataDog/datadog-agent/pkg/logs/internal/tag +github.com/DataDog/datadog-agent/pkg/logs/internal/util +github.com/DataDog/datadog-agent/pkg/logs/launchers +github.com/DataDog/datadog-agent/pkg/logs/launchers/container +github.com/DataDog/datadog-agent/pkg/logs/launchers/file +github.com/DataDog/datadog-agent/pkg/logs/launchers/file/provider +github.com/DataDog/datadog-agent/pkg/logs/launchers/journald +github.com/DataDog/datadog-agent/pkg/logs/launchers/listener +github.com/DataDog/datadog-agent/pkg/logs/launchers/windowsevent +github.com/DataDog/datadog-agent/pkg/logs/schedulers +github.com/DataDog/datadog-agent/pkg/logs/service +github.com/DataDog/datadog-agent/pkg/logs/tailers/file +github.com/DataDog/datadog-agent/pkg/logs/tailers/socket +github.com/DataDog/datadog-agent/pkg/tagger/replay +github.com/cyphar/filepath-securejoin +github.com/urfave/negroni +golang.org/x/text/encoding +golang.org/x/text/encoding/internal +golang.org/x/text/encoding/internal/identifier +golang.org/x/text/encoding/japanese +golang.org/x/text/encoding/unicode +golang.org/x/text/internal/utf8internal
cluster-agent	linux	arm64	+51, -0 +github.com/DataDog/datadog-agent/cmd/agent/api +github.com/DataDog/datadog-agent/cmd/agent/api/internal/agent +github.com/DataDog/datadog-agent/cmd/agent/api/internal/check +github.com/DataDog/datadog-agent/cmd/agent/gui +github.com/DataDog/datadog-agent/comp/api/api +github.com/DataDog/datadog-agent/comp/api/api/apiimpl +github.com/DataDog/datadog-agent/comp/core/flare +github.com/DataDog/datadog-agent/comp/core/workloadmeta/server +github.com/DataDog/datadog-agent/comp/logs/agent +github.com/DataDog/datadog-agent/comp/metadata/host +github.com/DataDog/datadog-agent/comp/metadata/inventoryhost +github.com/DataDog/datadog-agent/comp/metadata/resources +github.com/DataDog/datadog-agent/comp/remote-config/rcclient +github.com/DataDog/datadog-agent/pkg/gohai +github.com/DataDog/datadog-agent/pkg/gohai/cpu +github.com/DataDog/datadog-agent/pkg/gohai/filesystem +github.com/DataDog/datadog-agent/pkg/gohai/memory +github.com/DataDog/datadog-agent/pkg/gohai/network +github.com/DataDog/datadog-agent/pkg/gohai/platform +github.com/DataDog/datadog-agent/pkg/gohai/processes +github.com/DataDog/datadog-agent/pkg/gohai/processes/gops +github.com/DataDog/datadog-agent/pkg/gohai/utils +github.com/DataDog/datadog-agent/pkg/logs/internal/decoder +github.com/DataDog/datadog-agent/pkg/logs/internal/framer +github.com/DataDog/datadog-agent/pkg/logs/internal/parsers +github.com/DataDog/datadog-agent/pkg/logs/internal/parsers/dockerfile +github.com/DataDog/datadog-agent/pkg/logs/internal/parsers/encodedtext +github.com/DataDog/datadog-agent/pkg/logs/internal/parsers/kubernetes +github.com/DataDog/datadog-agent/pkg/logs/internal/parsers/noop +github.com/DataDog/datadog-agent/pkg/logs/internal/tag +github.com/DataDog/datadog-agent/pkg/logs/internal/util +github.com/DataDog/datadog-agent/pkg/logs/launchers +github.com/DataDog/datadog-agent/pkg/logs/launchers/container +github.com/DataDog/datadog-agent/pkg/logs/launchers/file +github.com/DataDog/datadog-agent/pkg/logs/launchers/file/provider +github.com/DataDog/datadog-agent/pkg/logs/launchers/journald +github.com/DataDog/datadog-agent/pkg/logs/launchers/listener +github.com/DataDog/datadog-agent/pkg/logs/launchers/windowsevent +github.com/DataDog/datadog-agent/pkg/logs/schedulers +github.com/DataDog/datadog-agent/pkg/logs/service +github.com/DataDog/datadog-agent/pkg/logs/tailers/file +github.com/DataDog/datadog-agent/pkg/logs/tailers/socket +github.com/DataDog/datadog-agent/pkg/tagger/replay +github.com/cyphar/filepath-securejoin +github.com/urfave/negroni +golang.org/x/text/encoding +golang.org/x/text/encoding/internal +golang.org/x/text/encoding/internal/identifier +golang.org/x/text/encoding/japanese +golang.org/x/text/encoding/unicode +golang.org/x/text/internal/utf8internal

Signed-off-by: Nicolas Guerguadj <[email protected]>

pr-commenter · 2023-11-22T15:51:48Z

Bloop Bleep... Dogbot Here

Regression Detector Results

Run ID: 6a12304f-1f60-457a-9f78-c2ca4224f046
Baseline: b710e3d
Comparison: 7a4ecb4
Total datadog-agent CPUs: 7

Explanation

A regression test is an integrated performance test for datadog-agent in a repeatable rig, with varying configuration for datadog-agent. What follows is a statistical summary of a brief datadog-agent run for each configuration across SHAs given above. The goal of these tests are to determine quickly if datadog-agent performance is changed and to what degree by a pull request.

Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval.

We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:

The estimated |Δ mean %| ≥ 5.00%. This criterion intends to answer the question "Does the estimated change in mean optimization goal performance have a meaningful impact on your customers?". We assume that when |Δ mean %| < 5.00%, the impact on your customers is not meaningful. We also assume that a performance change in optimization goal is worth investigating whether it is an increase or decrease, so long as the magnitude of the change is sufficiently large.
Zero is not in the 90.00% confidence interval "Δ mean % CI" about "Δ mean %". This statement is equivalent to saying that there is at least a 90.00% chance that the mean difference in optimization goal is not zero. This criterion intends to answer the question, "Is there a statistically significant difference in mean optimization goal performance?". It also means there is no more than a 10.00% chance this criterion reports a statistically significant difference when the true difference in mean optimization goal is zero -- a "false positive". We assume you are willing to accept a 10.00% chance of inaccurately detecting a change in performance when no true difference exists.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.

experiment	goal	Δ mean %	Δ mean % CI	confidence
tcp_syslog_to_blackhole	ingress throughput	+0.68	[+0.54, +0.81]	100.00%
file_tree	egress throughput	+0.29	[-1.51, +2.09]	20.70%
trace_agent_json	ingress throughput	+0.02	[-0.12, +0.15]	16.40%
file_to_blackhole	egress throughput	+0.01	[-1.00, +1.02]	0.82%
tcp_dd_logs_filter_exclude	ingress throughput	+0.01	[-0.15, +0.16]	4.33%
dogstatsd_string_interner_8MiB_50k	ingress throughput	+0.00	[-0.06, +0.07]	9.78%
dogstatsd_string_interner_64MiB_100	ingress throughput	+0.00	[-0.14, +0.14]	0.29%
idle	egress throughput	+0.00	[-2.38, +2.38]	0.02%
dogstatsd_string_interner_128MiB_100	ingress throughput	-0.00	[-0.14, +0.14]	0.35%
dogstatsd_string_interner_64MiB_1k	ingress throughput	-0.00	[-0.13, +0.13]	0.38%
dogstatsd_string_interner_8MiB_100	ingress throughput	-0.01	[-0.13, +0.12]	5.61%
dogstatsd_string_interner_8MiB_1k	ingress throughput	-0.01	[-0.11, +0.09]	10.01%
dogstatsd_string_interner_128MiB_1k	ingress throughput	-0.01	[-0.15, +0.13]	8.81%
dogstatsd_string_interner_8MiB_10k	ingress throughput	-0.01	[-0.02, -0.00]	90.16%
dogstatsd_string_interner_8MiB_100k	ingress throughput	-0.02	[-0.06, +0.03]	47.67%
trace_agent_msgpack	ingress throughput	-0.03	[-0.16, +0.10]	32.59%
uds_dogstatsd_to_api	ingress throughput	-0.03	[-0.18, +0.11]	29.33%
otel_to_otel_logs	ingress throughput	-0.33	[-1.92, +1.25]	27.17%

Signed-off-by: Nicolas Guerguadj <[email protected]>

ogaca-dd · 2023-11-24T16:08:55Z

comp/core/api/component_mock.go

@@ -0,0 +1,13 @@
+// Unless explicitly stated otherwise all files in this repository are licensed


I don't think the component should be in core folder as it is specific to the core agent (The core is for components used by most agents and yes the naming can be misleading).

Done here: bdfd985

ogaca-dd · 2023-11-24T16:10:45Z

comp/core/api/component.go

+
+// Component is the component type.
+type Component interface {
+	StartServer(


Can you add a comment prefixed by TODO(components): to not forget to remove this function?

Done here: bdfd985

…update imports Signed-off-by: Nicolas Guerguadj <[email protected]>

Signed-off-by: Nicolas Guerguadj <[email protected]>

Kaderinho added 6 commits November 21, 2023 17:19

feat: create api component which simply call functions from api package

2c07c49

Signed-off-by: Nicolas Guerguadj <[email protected]>

feat: use api component instead of api package

4246ffe

Signed-off-by: Nicolas Guerguadj <[email protected]>

feat: add api Component everywhere needed

af3ea64

Signed-off-by: Nicolas Guerguadj <[email protected]>

feat: inject api Component

941c825

Signed-off-by: Nicolas Guerguadj <[email protected]>

fix: api mock prototype

48f1c6d

Signed-off-by: Nicolas Guerguadj <[email protected]>

fix: add missing import on windows

e5a85c7

Signed-off-by: Nicolas Guerguadj <[email protected]>

Kaderinho added team/agent-shared-components changelog/no-changelog labels Nov 22, 2023

Kaderinho added this to the 7.51.0 milestone Nov 22, 2023

chore: lint-components

aafa8aa

Signed-off-by: Nicolas Guerguadj <[email protected]>

Kaderinho marked this pull request as ready for review November 22, 2023 13:04

Kaderinho requested a review from a team as a code owner November 22, 2023 13:04

Kaderinho added the team/windows-agent label Nov 22, 2023

ogaca-dd added the component framework label Nov 22, 2023

Kaderinho added 2 commits November 27, 2023 14:29

Merge branch 'main' into nicolas.guerguadj/create-agent-api-component

2d8a9cc

Signed-off-by: Nicolas Guerguadj <[email protected]>

fix: update StartServer prototype in component

cdac881

Signed-off-by: Nicolas Guerguadj <[email protected]>

hush-hush approved these changes Nov 28, 2023

View reviewed changes

Kaderinho added 2 commits November 28, 2023 13:50

fix: add missing import in test file

9b330b7

Signed-off-by: Nicolas Guerguadj <[email protected]>

chore: add comments around the current state of the API component

357de63

Signed-off-by: Nicolas Guerguadj <[email protected]>

ogaca-dd reviewed Nov 28, 2023

View reviewed changes

Kaderinho added 4 commits November 29, 2023 11:53

chore: create empty bundle and move api component into this bundle + …

bdfd985

…update imports Signed-off-by: Nicolas Guerguadj <[email protected]>

fix: lint components

605dacf

Signed-off-by: Nicolas Guerguadj <[email protected]>

Merge branch 'main' into nicolas.guerguadj/create-agent-api-component

4bba941

Merge branch 'main' into nicolas.guerguadj/create-agent-api-component

7a4ecb4

Kaderinho merged commit 0da2a3c into main Nov 30, 2023
143 checks passed

Kaderinho deleted the nicolas.guerguadj/create-agent-api-component branch November 30, 2023 10:57

Kaderinho mentioned this pull request Dec 6, 2023

Move api code from cmd to comp #21359

Merged

10 tasks

mx-psi mentioned this pull request Dec 13, 2023

Update opentelemetry-mapping-go to v0.9.0 #21524

Merged

10 tasks

Kaderinho mentioned this pull request Dec 13, 2023

Move args from StartAgent to Api component dependencies #21530

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create agent internal API component #21028

Create agent internal API component #21028

Kaderinho commented Nov 22, 2023 •

edited

Loading

cit-pr-commenter bot commented Nov 22, 2023 •

edited

Loading

pr-commenter bot commented Nov 22, 2023 •

edited

Loading

ogaca-dd Nov 24, 2023

Kaderinho Nov 29, 2023

ogaca-dd Nov 24, 2023

Kaderinho Nov 29, 2023

		@@ -0,0 +1,13 @@
		// Unless explicitly stated otherwise all files in this repository are licensed

Create agent internal API component #21028

Create agent internal API component #21028

Conversation

Kaderinho commented Nov 22, 2023 • edited Loading

What does this PR do?

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Configure the Agent

Testing

Reviewer's Checklist

cit-pr-commenter bot commented Nov 22, 2023 • edited Loading

Go Package Import Differences

pr-commenter bot commented Nov 22, 2023 • edited Loading

Bloop Bleep... Dogbot Here

Regression Detector Results

ogaca-dd Nov 24, 2023

Choose a reason for hiding this comment

Kaderinho Nov 29, 2023

Choose a reason for hiding this comment

ogaca-dd Nov 24, 2023

Choose a reason for hiding this comment

Kaderinho Nov 29, 2023

Choose a reason for hiding this comment

Kaderinho commented Nov 22, 2023 •

edited

Loading

cit-pr-commenter bot commented Nov 22, 2023 •

edited

Loading

pr-commenter bot commented Nov 22, 2023 •

edited

Loading