feat: add Nginx Ingress Controller Otel integration

[rogercoll]

Proposed commit message

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Unit testing

1. Create a fake k8s pods logs path, for example:

$ mkdir -p /tmp/var/log/pods/kube-system_kube-controller-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d6/kube-controller/

2. Modify the configuration to read from the /tmp directory instead:

  filelog:
    include_file_path: true
    include: [/tmp/var/log/pods/*/*/*.log]

3. Update your Elasticsearch endpoint/api variables
4. Append some faked Nginx Ingress Controller access or error logs to the log file, for example:

$ echo '2024-10-07T07:48:23.966071760Z stdout F 81.2.69.140 - - [25/Jun/2024:15:16:56 +0000] "POST /flagservice/flagd.evaluation.v1.Service/EventStream HTTP/1.1" 400 47 "http://1.2.3.4.5/" "Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0" 522 12.065 [default-my-otel-demo-frontendproxy-8080] [] 10.80.0.67:8080 47 12.065 200 797ee0551e7c23b541757e13b5b6b4e6'  >> /tmp/var/log/pods/kube-system_kube-controller-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d6/kube-controller/1.log

$ echo '2024-10-07T07:48:23.137838690Z stdout F F0930 10:39:59.102914 8 config.go:659] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.' >> /tmp/var/log/pods/kube-system_kube-controller-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d6/kube-controller/1.log

Related issues

Screenshots

[rogercoll]

Is there a way to remove the "Add Nginx Ingress Controller OpenTelemetry Logs integration" button? (deployment mode)

[ChrsMark]

This is a bit fuzzy. The non-otel integration is elastic/obs-cloudnative-monitoring but this is obs-infraobs-integrations?

[lalit-satapathy]

Adding @bturquet for any suggestions, I am ok with either.

[rogercoll]

I am ok with both too. If it is helpful for you, I could additionally add myself as a codowner until Otel integrations become more stable.

[ishleenk17]

@rogercoll : It has build failures:

  | 1. item [collector-config.yaml] is not allowed in folder [/opt/buildkite-agent/builds/bk-agent-prod-gcp-1728919549244574433/elastic/integrations/packages/nginx_ingress_controller_otel]
  | 2. item [search] is not allowed in folder [/opt/buildkite-agent/builds/bk-agent-prod-gcp-1728919549244574433/elastic/integrations/packages/nginx_ingress_controller_otel/kibana]

I think since elastic package doesn't expect collector-config.yaml to be present. It is throwing an error. We will have to see how we can place this collector config. Or we should change the elastic package spec to have this addition.

Please check for the 2nd error as well.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 08431b4

History

• 💔 Build #17133 failed 08b78a0
• 💔 Build #17108 failed 56d2ad0
• 💔 Build #17022 failed 20421fc
• 💔 Build #17012 failed ba38192
• 💚 Build #16966 succeeded 754452b

[ChrsMark]

LGTM

[rogercoll]

@rogercoll : It has build failures:

| 1. item [collector-config.yaml] is not allowed in folder [/opt/buildkite-agent/builds/bk-agent-prod-gcp-1728919549244574433/elastic/integrations/packages/nginx_ingress_controller_otel] | 2. item [search] is not allowed in folder [/opt/buildkite-agent/builds/bk-agent-prod-gcp-1728919549244574433/elastic/integrations/packages/nginx_ingress_controller_otel/kibana]

I think since elastic package doesn't expect collector-config.yaml to be present. It is throwing an error. We will have to see how we can place this collector config. Or we should change the elastic package spec to have this addition.

Please check for the 2nd error as well.

Thanks @ishleenk17, I think those were fixed on the latest comment

[ishleenk17]

Looks good!
Curious, how did the search issue get resolved? Are we no longer having the saved search in the dashboard ?

[ishleenk17]

Also, for now I see we are moving the configuration to the README.
It would make the README look quite big. Hope that is discussed before releasing this Integration.
As soon as its merged it will be out for usage to the customers in Tech Preview.
cc: @AlexanderWert

[rogercoll]

Curious, how did the search issue get resolved? Are we no longer having the saved search in the dashboard ?

Correct, I was not able to add the search object, but the ES/QL visualization alternative seems to work fine (similar results).

[ishleenk17]

Looks good!

[elastic-vault-github-plugin-prod]

Package nginx_ingress_controller_otel - 0.0.1 containing this change is available at https://epr.elastic.co/search?package=nginx_ingress_controller_otel