Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ASCII-145] Update pkg/status/render to only handle rendering logic #20824

Merged
merged 22 commits into from
Nov 23, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
9616a2f
use collector package to just collect check information
GustavoCaso Nov 8, 2023
2ee0a06
apply feedback
GustavoCaso Nov 15, 2023
6055861
fix unit tests
GustavoCaso Nov 13, 2023
aae22c9
move fictires inside the pkg/status/render
GustavoCaso Nov 15, 2023
fca2adb
use collector package to just collect check information
GustavoCaso Nov 8, 2023
48e9e0f
apply feedback
GustavoCaso Nov 13, 2023
f2e164e
unify forwared status information
GustavoCaso Nov 13, 2023
373769b
unify dogstatsd status information
GustavoCaso Nov 13, 2023
175e59d
unify logsStats status information
GustavoCaso Nov 13, 2023
8838c5e
unify clusterAgentStatus status information
GustavoCaso Nov 13, 2023
f032730
unify endpointsInfos status information
GustavoCaso Nov 13, 2023
3f8c310
fix logStats temaplte
GustavoCaso Nov 13, 2023
2b5b4ae
unify systemProbeStats and processAgentStatus status information
GustavoCaso Nov 13, 2023
dbf2346
unify snmpTrapsStats status information
GustavoCaso Nov 13, 2023
a064b90
unify netflowStats and otlp status information
GustavoCaso Nov 13, 2023
089acad
unify apmStats status information
GustavoCaso Nov 13, 2023
0db4d4d
unify autodiscovery status information
GustavoCaso Nov 13, 2023
31b5927
unify aggregatorStats status information and avoid calling TranslateE…
GustavoCaso Nov 13, 2023
c797208
omve cluster agent check value to template
GustavoCaso Nov 13, 2023
dffa6e8
update FormatDCAStatus to unify stats access withint the templates an…
GustavoCaso Nov 13, 2023
6c8847a
remove inderection for security agent status
GustavoCaso Nov 13, 2023
05d8059
make sure templates preserve the same amount of white space
GustavoCaso Nov 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 15 additions & 16 deletions cmd/agent/gui/views/templates/generalStatus.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -268,24 +268,23 @@
</span>
{{- end}}
{{- end -}}
{{/* The subsection `On-disk storage` is not inside `{{- with .forwarderStats -}}` as it need to access `.config` */}}
<span class="stat_subtitle">On-disk storage</span>
<span class="stat_subdata">
{{- if .config.forwarder_storage_max_size_in_bytes }}
{{- if .forwarderStats.FileStorage.CurrentSizeInBytes }}
Disk usage in bytes: {{ .forwarderStats.FileStorage.CurrentSizeInBytes }}<br>
Number of files: {{ .forwarderStats.FileStorage.FilesCount }}<br>
Number of files dropped: {{ .forwarderStats.FileStorage.FilesRemovedCount }}<br>
Deserialization errors count: {{ .forwarderStats.FileStorage.DeserializeErrorsCount }}<br>
Outdated files removed at startup: {{ .forwarderStats.RemovalPolicy.OutdatedFilesCount }}<br>
{{- with .forwarderStats -}}
<span class="stat_subtitle">On-disk storage</span>
<span class="stat_subdata">
{{- if .forwarder_storage_max_size_in_bytes }}
{{- if .FileStorage.CurrentSizeInBytes }}
Disk usage in bytes: {{ .FileStorage.CurrentSizeInBytes }}<br>
Number of files: {{ .FileStorage.FilesCount }}<br>
Number of files dropped: {{ .FileStorage.FilesRemovedCount }}<br>
Deserialization errors count: {{ .FileStorage.DeserializeErrorsCount }}<br>
Outdated files removed at startup: {{ .RemovalPolicy.OutdatedFilesCount }}<br>
{{- else }}
Enabled, not in-use.<br>
{{- end}}
{{- else }}
Enabled, not in-use.<br>
On-disk storage is disabled. Configure `forwarder_storage_max_size_in_bytes` to enable it.<br>
{{- end}}
{{- else }}
On-disk storage is disabled. Configure `forwarder_storage_max_size_in_bytes` to enable it.<br>
{{- end}}
</span>
{{- with .forwarderStats -}}
</span>
{{- if .APIKeyStatus}}
<span class="stat_subtitle">API Keys Status</span>
<span class="stat_subdata">
Expand Down
144 changes: 26 additions & 118 deletions pkg/status/render/render.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,7 @@ import (
"path"
"text/template"

"github.com/DataDog/datadog-agent/comp/netflow/server"
"github.com/DataDog/datadog-agent/comp/otelcol/otlp"
checkstats "github.com/DataDog/datadog-agent/pkg/collector/check/stats"
"github.com/DataDog/datadog-agent/pkg/config"
"github.com/DataDog/datadog-agent/pkg/snmp/traps"
"github.com/DataDog/datadog-agent/pkg/util/log"
)

var fmap = Textfmap()
Expand All @@ -31,28 +26,6 @@ func FormatStatus(data []byte) (string, error) {
if renderError != "" || err != nil {
return renderError, err
}
forwarderStats := stats["forwarderStats"]
if forwarderStatsMap, ok := forwarderStats.(map[string]interface{}); ok {
forwarderStatsMap["config"] = stats["config"]
} else {
log.Warn("The Forwarder status format is invalid. Some parts of the `Forwarder` section may be missing.")
}
Comment on lines -35 to -39
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I aggregated all forwarded stats within the same function when fetching the status.

forwarderStatsJSON := []byte(expvar.Get("forwarder").String())
forwarderStats := make(map[string]interface{})
json.Unmarshal(forwarderStatsJSON, &forwarderStats) //nolint:errcheck
forwarderStorageMaxSizeInBytes := config.Datadog.GetInt("forwarder_storage_max_size_in_bytes")
if forwarderStorageMaxSizeInBytes > 0 {
forwarderStats["forwarder_storage_max_size_in_bytes"] = strconv.Itoa(forwarderStorageMaxSizeInBytes)
}
stats["forwarderStats"] = forwarderStats


aggregatorStats := stats["aggregatorStats"]
s, err := checkstats.TranslateEventPlatformEventTypes(aggregatorStats)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already call this function when fetching the status in the pkg/status package.

s, err := checkstats.TranslateEventPlatformEventTypes(stats["aggregatorStats"])

if err != nil {
log.Debugf("failed to translate event platform event types in aggregatorStats: %s", err.Error())
} else {
aggregatorStats = s
}
dogstatsdStats := stats["dogstatsdStats"]
logsStats := stats["logsStats"]
dcaStats := stats["clusterAgentStatus"]
endpointsInfos := stats["endpointsInfos"]
systemProbeStats := stats["systemProbeStats"]
processAgentStatus := stats["processAgentStatus"]
snmpTrapsStats := stats["snmpTrapsStats"]
netflowStats := stats["netflowStats"]
title := fmt.Sprintf("Agent (v%s)", stats["version"])
stats["title"] = title

Expand All @@ -62,55 +35,20 @@ func FormatStatus(data []byte) (string, error) {
return renderStatusTemplate(b, "/collector.tmpl", stats)
}
jmxFetchFunc := func() error { return renderStatusTemplate(b, "/jmxfetch.tmpl", stats) }
forwarderFunc := func() error { return renderStatusTemplate(b, "/forwarder.tmpl", forwarderStats) }
endpointsFunc := func() error { return renderStatusTemplate(b, "/endpoints.tmpl", endpointsInfos) }
logsAgentFunc := func() error { return renderStatusTemplate(b, "/logsagent.tmpl", logsStats) }
systemProbeFunc := func() error {
if systemProbeStats != nil {
return renderStatusTemplate(b, "/systemprobe.tmpl", systemProbeStats)
}
return nil
}
processAgentFunc := func() error { return renderStatusTemplate(b, "/process-agent.tmpl", processAgentStatus) }
traceAgentFunc := func() error { return renderStatusTemplate(b, "/trace-agent.tmpl", stats["apmStats"]) }
aggregatorFunc := func() error { return renderStatusTemplate(b, "/aggregator.tmpl", aggregatorStats) }
dogstatsdFunc := func() error { return renderStatusTemplate(b, "/dogstatsd.tmpl", dogstatsdStats) }
clusterAgentFunc := func() error {
if config.Datadog.GetBool("cluster_agent.enabled") || config.Datadog.GetBool("cluster_checks.enabled") {
return renderStatusTemplate(b, "/clusteragent.tmpl", dcaStats)
}
return nil
}
snmpTrapFunc := func() error {
if traps.IsEnabled(config.Datadog) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move that check to pkg/status GetStatus() function

return renderStatusTemplate(b, "/snmp-traps.tmpl", snmpTrapsStats)
}
return nil
}

netflowFunc := func() error {
if server.IsEnabled() {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move that check to pkg/status GetStatus() function

return renderStatusTemplate(b, "/netflow.tmpl", netflowStats)
}
return nil
}

autodiscoveryFunc := func() error {
if config.IsContainerized() {
return renderAutodiscoveryStats(b, stats["adEnabledFeatures"], stats["adConfigErrors"],
stats["filterErrors"])
}
return nil
}
remoteConfigFunc := func() error {
return renderStatusTemplate(b, "/remoteconfig.tmpl", stats)
}
otlpFunc := func() error {
if otlp.IsDisplayed() {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move that check to pkg/status GetStatus() function

return renderStatusTemplate(b, "/otlp.tmpl", stats)
}
return nil
}
forwarderFunc := func() error { return renderStatusTemplate(b, "/forwarder.tmpl", stats) }
endpointsFunc := func() error { return renderStatusTemplate(b, "/endpoints.tmpl", stats) }
logsAgentFunc := func() error { return renderStatusTemplate(b, "/logsagent.tmpl", stats) }
systemProbeFunc := func() error { return renderStatusTemplate(b, "/systemprobe.tmpl", stats) }
processAgentFunc := func() error { return renderStatusTemplate(b, "/process-agent.tmpl", stats) }
traceAgentFunc := func() error { return renderStatusTemplate(b, "/trace-agent.tmpl", stats) }
aggregatorFunc := func() error { return renderStatusTemplate(b, "/aggregator.tmpl", stats) }
dogstatsdFunc := func() error { return renderStatusTemplate(b, "/dogstatsd.tmpl", stats) }
clusterAgentFunc := func() error { return renderStatusTemplate(b, "/clusteragent.tmpl", stats) }
snmpTrapFunc := func() error { return renderStatusTemplate(b, "/snmp-traps.tmpl", stats) }
netflowFunc := func() error { return renderStatusTemplate(b, "/netflow.tmpl", stats) }
autodiscoveryFunc := func() error { return renderStatusTemplate(b, "/autodiscovery.tmpl", stats) }
remoteConfigFunc := func() error { return renderStatusTemplate(b, "/remoteconfig.tmpl", stats) }
otlpFunc := func() error { return renderStatusTemplate(b, "/otlp.tmpl", stats) }

var renderFuncs []func() error
if config.IsCLCRunner() {
Expand Down Expand Up @@ -141,14 +79,11 @@ func FormatDCAStatus(data []byte) (string, error) {
return renderError, err
}

forwarderStats := stats["forwarderStats"]
// We nil these keys because we do not want to display that information in the collector template
stats["pyLoaderStats"] = nil
stats["pythonInit"] = nil
stats["inventories"] = nil
endpointsInfos := stats["endpointsInfos"]
logsStats := stats["logsStats"]
orchestratorStats := stats["orchestrator"]

title := fmt.Sprintf("Datadog Cluster Agent (v%s)", stats["version"])
stats["title"] = title

Expand All @@ -160,24 +95,21 @@ func FormatDCAStatus(data []byte) (string, error) {
if err := renderStatusTemplate(b, "/collector.tmpl", stats); err != nil {
errs = append(errs, err)
}
if err := renderStatusTemplate(b, "/forwarder.tmpl", forwarderStats); err != nil {
if err := renderStatusTemplate(b, "/forwarder.tmpl", stats); err != nil {
errs = append(errs, err)
}
if err := renderStatusTemplate(b, "/endpoints.tmpl", endpointsInfos); err != nil {
if err := renderStatusTemplate(b, "/endpoints.tmpl", stats); err != nil {
errs = append(errs, err)
}
if config.Datadog.GetBool("compliance_config.enabled") {
if err := renderStatusTemplate(b, "/logsagent.tmpl", logsStats); err != nil {
errs = append(errs, err)
}
if err := renderStatusTemplate(b, "/logsagent.tmpl", stats); err != nil {
errs = append(errs, err)
}
if err := renderAutodiscoveryStats(b, stats["adEnabledFeatures"], stats["adConfigErrors"], stats["filterErrors"]); err != nil {
if err := renderStatusTemplate(b, "/autodiscovery.tmpl", stats); err != nil {
errs = append(errs, err)
}
if config.Datadog.GetBool("orchestrator_explorer.enabled") {
if err := renderStatusTemplate(b, "/orchestrator.tmpl", orchestratorStats); err != nil {
errs = append(errs, err)
}

if err := renderStatusTemplate(b, "/orchestrator.tmpl", stats); err != nil {
errs = append(errs, err)
}
if err := renderErrors(b, errs); err != nil {
fmt.Println(err)
Expand Down Expand Up @@ -209,9 +141,7 @@ func FormatSecurityAgentStatus(data []byte) (string, error) {
if renderError != "" || err != nil {
return renderError, err
}
runnerStats := stats["runnerStats"]
complianceChecks := stats["complianceChecks"]
complianceStatus := stats["complianceStatus"]

title := fmt.Sprintf("Datadog Security Agent (v%s)", stats["version"])
stats["title"] = title

Expand All @@ -220,10 +150,10 @@ func FormatSecurityAgentStatus(data []byte) (string, error) {
if err := renderStatusTemplate(b, "/header.tmpl", stats); err != nil {
errs = append(errs, err)
}
if err := renderRuntimeSecurityStats(b, stats["runtimeSecurityStatus"]); err != nil {
if err := renderStatusTemplate(b, "/runtimesecurity.tmpl", stats); err != nil {
errs = append(errs, err)
}
if err := renderComplianceChecksStats(b, runnerStats, complianceChecks, complianceStatus); err != nil {
if err := renderStatusTemplate(b, "/compliance.tmpl", stats); err != nil {
errs = append(errs, err)
}
if err := renderErrors(b, errs); err != nil {
Expand Down Expand Up @@ -287,28 +217,6 @@ func FormatCheckStats(data []byte) (string, error) {
return b.String(), nil
}

func renderComplianceChecksStats(w io.Writer, runnerStats interface{}, complianceChecks, complianceStatus interface{}) error {
checkStats := make(map[string]interface{})
checkStats["RunnerStats"] = runnerStats
checkStats["ComplianceStatus"] = complianceStatus
checkStats["ComplianceChecks"] = complianceChecks
return renderStatusTemplate(w, "/compliance.tmpl", checkStats)
}

func renderRuntimeSecurityStats(w io.Writer, runtimeSecurityStatus interface{}) error {
status := make(map[string]interface{})
status["RuntimeSecurityStatus"] = runtimeSecurityStatus
return renderStatusTemplate(w, "/runtimesecurity.tmpl", status)
}

func renderAutodiscoveryStats(w io.Writer, adEnabledFeatures interface{}, adConfigErrors interface{}, filterErrors interface{}) error {
autodiscoveryStats := make(map[string]interface{})
autodiscoveryStats["adEnabledFeatures"] = adEnabledFeatures
autodiscoveryStats["adConfigErrors"] = adConfigErrors
autodiscoveryStats["filterErrors"] = filterErrors
return renderStatusTemplate(w, "/autodiscovery.tmpl", autodiscoveryStats)
}

//go:embed templates
var templatesFS embed.FS

Expand Down
2 changes: 2 additions & 0 deletions pkg/status/render/templates/aggregator.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ NOTE: Changes made to this template should be reflected on the following templat
==========
Aggregator
==========
{{- with .aggregatorStats }}
GustavoCaso marked this conversation as resolved.
Show resolved Hide resolved
{{- if .ChecksMetricSample }}
Checks Metric Sample: {{humanize .ChecksMetricSample}}
{{- end }}
Expand Down Expand Up @@ -55,3 +56,4 @@ Aggregator
{{- if .HostnameUpdate}}
Hostname Update: {{humanize .HostnameUpdate}}
{{- end }}
{{- end }}
2 changes: 1 addition & 1 deletion pkg/status/render/templates/autodiscovery.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ NOTE: Changes made to this template should be reflected on the following templat
=============
Autodiscovery
=============
{{- with .adEnabledFeatures}}
{{- with .adEnabledFeatures -}}
Enabled Features
================
{{- range $feature, $empty := . }}
Expand Down
31 changes: 16 additions & 15 deletions pkg/status/render/templates/clusteragent.tmpl
Original file line number Diff line number Diff line change
@@ -1,18 +1,19 @@

{{- with .clusterAgentStatus -}}
=====================
Datadog Cluster Agent
=====================
{{ if .DetectionError }}
- Could not detect the Datadog Cluster Agent's endpoint: {{ .DetectionError }}
{{ else }}
- Datadog Cluster Agent endpoint detected: {{ .Endpoint }}
{{- end }}
{{- if .ConnectionError }}
- Could not reach the Datadog Cluster Agent: {{ .ConnectionError }}
{{- end }}
{{- if not .Version }}
- Could not retrieve the version of the Datadog Cluster Agent.
{{ else }}
Successfully connected to the Datadog Cluster Agent.
- Running: {{ .Version }}
{{- end }}
{{ if .DetectionError }}
- Could not detect the Datadog Cluster Agent's endpoint: {{ .DetectionError }}
{{ else }}
- Datadog Cluster Agent endpoint detected: {{ .Endpoint }}
{{- end }}
{{- if .ConnectionError }}
- Could not reach the Datadog Cluster Agent: {{ .ConnectionError }}
{{- end }}
{{- if not .Version }}
- Could not retrieve the version of the Datadog Cluster Agent.
{{ else }}
Successfully connected to the Datadog Cluster Agent.
- Running: {{ .Version }}
{{- end }}
{{- end -}}
8 changes: 4 additions & 4 deletions pkg/status/render/templates/compliance.tmpl
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
==========
Compliance
==========
{{ if not .ComplianceStatus}}
{{ if not .complianceStatus}}
Not enabled
{{- else}}
{{- with .ComplianceStatus}}
{{- with .complianceStatus}}
{{ if .endpoints }}
{{- range $endpoint := .endpoints }}
{{ $endpoint }}
Expand All @@ -14,8 +14,8 @@ Compliance

Checks
======
{{ $runnerStats := .RunnerStats }}
{{- range $Check := .ComplianceChecks }}
{{ $runnerStats := .runnerStats }}
{{- range $Check := .complianceChecks }}
{{ $Check.Name }}
{{printDashes $Check.Name "-"}}
Framework: {{ $Check.Framework }} ({{ $Check.Version }})
Expand Down
2 changes: 2 additions & 0 deletions pkg/status/render/templates/dogstatsd.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,10 @@ NOTE: Changes made to this template should be reflected on the following templat
=========
DogStatsD
=========
{{- with .dogstatsdStats -}}
{{- range $key, $value := .}}
{{formatTitle $key}}: {{humanize $value}}
{{- end }}
{{- end }}

Tip: For troubleshooting, enable 'dogstatsd_metrics_stats_enable' in the main datadog.yaml file to generate Dogstatsd logs. Once 'dogstatsd_metrics_stats_enable' is enabled, users can also use 'dogstatsd-stats' command to get visibility of the latest collected metrics.
12 changes: 5 additions & 7 deletions pkg/status/render/templates/endpoints.tmpl
Original file line number Diff line number Diff line change
@@ -1,20 +1,18 @@
{{/*
NOTE: Changes made to this template should be reflected on the following templates, if applicable:
* cmd/agent/gui/views/templates/generalStatus.tmpl
* Dockerfiles/cluster-agent/dist/templates/forwarder.tmpl
hush-hush marked this conversation as resolved.
Show resolved Hide resolved
*/}}==========
Endpoints
==========

{{- with .}}
{{- range $key, $value := .}}
{{$key}} - API Key{{ if gt (len $value) 1}}s{{end}} ending with:
{{- with .endpointsInfos -}}
{{ range $key, $value := .}}
{{$key}} - API Key{{ if gt (len $value) 1}}s{{end}} ending with:
{{- range $idx, $apikey := $value }}
- {{$apikey}}
{{- end}}
{{- end}}
{{- else }}

No endpoints information. The agent may be misconfigured.
{{- end }}
No endpoints information. The agent may be misconfigured.
{{end }}

4 changes: 3 additions & 1 deletion pkg/status/render/templates/forwarder.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ NOTE: Changes made to this template should be reflected on the following templat
*/}}=========
Forwarder
=========
{{with .forwarderStats -}}
{{ if .Transactions }}
Transactions
============
Expand Down Expand Up @@ -57,7 +58,7 @@ Forwarder

On-disk storage
===============
{{- if .config.forwarder_storage_max_size_in_bytes }}
{{- if .forwarder_storage_max_size_in_bytes }}
{{- if .FileStorage.CurrentSizeInBytes }}
Disk usage in bytes: {{ .FileStorage.CurrentSizeInBytes }}
Current number of files: {{ .FileStorage.FilesCount }}
Expand Down Expand Up @@ -88,4 +89,5 @@ Forwarder
{{yellowText $key}}{{yellowText ":"}} {{yellowText $value}}
{{- end }}
{{- end}}
{{- end}}

Loading