Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Operator doesn't delete components when removed from configuration #3454

Open
sfc-gh-dguy opened this issue Nov 13, 2024 · 3 comments · May be fixed by #3494
Open

Operator doesn't delete components when removed from configuration #3454

sfc-gh-dguy opened this issue Nov 13, 2024 · 3 comments · May be fixed by #3494
Assignees
@swiatekm
Labels
bug Something isn't working

Comments

@sfc-gh-dguy
Copy link

Component(s)

collector, target allocator

What happened?

Description

I have an otel operator that creates otel collectors as daemonset and an otel target allocator based on an OpenTelemetryCollector CR.
I removed the target allocator part from the OpenTelemetryCollector CR but the target allocator wasn't deleted.
Only when I manually delete it, it's removed and is not recreated as it was removed from the config.
I also deleted the whole OpenTelemetryCollector CR and both the daemonset and the target allocator weren't deleted.

Steps to Reproduce

  1. Create an OpenTelemetryCollector CR that defines collectors daemonset and atarget allocator deployment
  2. Let the operator create the components
  3. Delete the target allocator part from the CR

Expected Result

Target allocator should be deleted when removed from the CR

Actual Result

Target allocator wasn't deleted when removed from the CR

Kubernetes Version

1.29.8

Operator version

0.103.0

Collector version

0.111.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

Log output

These are the only prints I see in the operator logs:
{"level":"INFO","timestamp":"2024-11-13T16:43:17.143921523Z","logger":"controllers.OpenTelemetryCollector","message":"pdb field is unset in Spec, skipping podDisruptionBudget creation"}
{"level":"INFO","timestamp":"2024-11-13T16:43:17.362086635Z","logger":"controllers.OpenTelemetryCollector","message":"pdb field is unset in Spec, skipping podDisruptionBudget creation"}
{"level":"INFO","timestamp":"2024-11-13T16:43:17.543764697Z","logger":"controllers.OpenTelemetryCollector","message":"pdb field is unset in Spec, skipping podDisruptionBudget creation"}
{"level":"INFO","timestamp":"2024-11-13T16:47:27.3789637Z","logger":"controllers.OpenTelemetryCollector","message":"pdb field is unset in Spec, skipping podDisruptionBudget creation"}
{"level":"INFO","timestamp":"2024-11-13T16:47:27.554856277Z","logger":"controllers.OpenTelemetryCollector","message":"pdb field is unset in Spec, skipping podDisruptionBudget creation"}
{"level":"INFO","timestamp":"2024-11-13T16:47:27.761659569Z","logger":"controllers.OpenTelemetryCollector","message":"pdb field is unset in Spec, skipping podDisruptionBudget creation"}

Additional context

No response
@sfc-gh-dguy sfc-gh-dguy added bug Something isn't working needs triage labels Nov 13, 2024
@swiatekm swiatekm self-assigned this Nov 14, 2024
@swiatekm
Copy link
Contributor

I can reproduce the part where the target allocator resources are not deleted if you disable it. This is a more general problem with how we handle these kinds of resources, whose lifecycle is not the same as the OpenTelemetryCollector CR lifecycle.

I can't reproduce the issue where deleting the OpenTelemetryCollector CR leaves orphaned resources behind. That's actually not something the operator does directly - we simply set an owner reference, and rely on Kubernetes garbage collection. Are you able to reproduce this?

@sfc-gh-dguy
Copy link
Author

Yes. When I completely deleted the OpenTelemetryCollector CR, both the daemonset and the target collector remained. I had to manually delete them

@swiatekm
Copy link
Contributor

Yes. When I completely deleted the OpenTelemetryCollector CR, both the daemonset and the target collector remained. I had to manually delete them

Are you able to consistently reproduce this? Ideally in a kind cluster, but I'll take what I can get.

@swiatekm swiatekm linked a pull request Nov 25, 2024 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@swiatekm swiatekm

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix optional resource deletion for collector CR swiatekm/opentelemetry-operator
2 participants
@swiatekm @sfc-gh-dguy