Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Telegraf crashing in 1.26 version #13219

Closed
samskan opened this issue May 2, 2023 · 17 comments · Fixed by #13231
Closed

Telegraf crashing in 1.26 version #13219

samskan opened this issue May 2, 2023 · 17 comments · Fixed by #13231
Labels
bug unexpected problem or unintended behavior

Comments

@samskan
Copy link

samskan commented May 2, 2023

Relevant telegraf.conf

[[inputs.prometheus]]
      metric_version = 2
      monitor_kubernetes_pods = false
      pod_scrape_scope = "cluster"
      bearer_token = "/var/run/secrets/kubernetes.io/serviceaccount/token"
      kubernetes_services = ["https://kubernetes.default.svc/api/v1/nodes/${MY_NODE_NAME}/proxy/metrics/cadvisor"]

Logs from Telegraf

fatal error: concurrent map read and map write

goroutine 30 [running]:
github.com/influxdata/telegraf/plugins/inputs/prometheus.shouldScrapePod(0xc0017c4ee0, 0xc000618d80)
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:110 +0x1be
github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).watchPod.func1({0x62c3c60?, 0xc0017c4ee0?})
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:150 +0x56
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/controller.go:232
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:818 +0x134
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0008c8738?, {0x6fa7c80, 0xc000813620}, 0x1, 0xc0011a43c0)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0xc0008c8788?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000645b80?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:812 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc000b9bef0?)

System info

TelegrafVersion:1.26, OS:Mariner,2.0.20230407

Docker

No response

Steps to reproduce

  1. Install telegraf with 1.26 version
    2.add the input kubernetes-service for cadvisor

...

Expected behavior

Telegraf should not panic. It should be up and running

Actual behavior

Telegraf is crashing

Additional info

No response
@samskan samskan added the bug unexpected problem or unintended behavior label May 2, 2023
@powersj
Copy link
Contributor

powersj commented May 2, 2023

Hi,

1.26

Can you please update to v1.26.2, reproduce, and provide the full trace please.

Are you building a custom version of telegraf? Are you applying any patch to the prometheus plugin itself?

@powersj powersj added the waiting for response waiting for response from contributor label May 2, 2023
@samskan
Copy link
Author

samskan commented May 2, 2023 •
edited by powersj
Loading

We are building custom version with telegraf.star and telegraf_processor.star. No we are not applying patch to prometeus plugin.
On 1.23 version with same config we dont see any issue. When I change to 1.26 its crashing
fatal error: concurrent map read and map write

goroutine 30 [running]:
github.com/influxdata/telegraf/plugins/inputs/prometheus.shouldScrapePod(0xc0017c4ee0, 0xc000618d80)
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:110 +0x1be
github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).watchPod.func1({0x62c3c60?, 0xc0017c4ee0?})
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:150 +0x56
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/controller.go:232
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:818 +0x134
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0008c8738?, {0x6fa7c80, 0xc000813620}, 0x1, 0xc0011a43c0)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0xc0008c8788?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000645b80?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:812 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc000b9bef0?)
        /usr/lib/golang/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0x6433a98?)
        /usr/lib/golang/src/sync/waitgroup.go:139 +0x52
github.com/influxdata/telegraf/agent.(*Agent).Run(0xc000709990, {0x6ffb870?, 0xc000e79380})
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:197 +0x957
main.(*Telegraf).runAgent(0xc000a897c0, {0x6ffb870, 0xc000e79380}, 0xa356340?, 0xb0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/cmd/telegraf/telegraf.go:357 +0xedc
main.(*Telegraf).reloadLoop(0xc000a897c0)
        /usr/src/mariner/BUILD/telegraf-1.26.0/cmd/telegraf/telegraf.go:161 +0x285
main.(*Telegraf).Run(0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/cmd/telegraf/telegraf_posix.go:9 +0x4f
main.runApp.func1(0xc000c07440)
        /usr/src/mariner/BUILD/telegraf-1.26.0/cmd/telegraf/main.go:244 +0xa90
github.com/urfave/cli/v2.(*Command).Run(0xc0010c8280, 0xc000c07440, {0xc0001901f0, 0x1, 0x1})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/github.com/urfave/cli/v2/command.go:271 +0xaab
github.com/urfave/cli/v2.(*App).RunContext(0xc000abd680, {0x6ffb8a8?, 0xc0001a4000}, {0xc0001901f0, 0x1, 0x1})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/github.com/urfave/cli/v2/app.go:329 +0x665
github.com/urfave/cli/v2.(*App).Run(...)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/github.com/urfave/cli/v2/app.go:306
main.runApp({0xc0001901f0, 0x1, 0x1}, {0x6fa8ba0?, 0xc0001a0008}, {0x6fdfb68?, 0xc0001a0fc8}, {0x6fd2fd0?, 0xc000485080}, {0x70004c8, ...})
        /usr/src/mariner/BUILD/telegraf-1.26.0/cmd/telegraf/main.go:372 +0x1012
main.main()
        /usr/src/mariner/BUILD/telegraf-1.26.0/cmd/telegraf/main.go:379 +0xde

goroutine 35 [chan receive]:
github.com/awnumar/memguard/core.NewCoffer.func1(0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/github.com/awnumar/memguard/core/coffer.go:39 +0x4d
created by github.com/awnumar/memguard/core.NewCoffer
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/github.com/awnumar/memguard/core/coffer.go:36 +0x10f

goroutine 25 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00016c000)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/go.opencensus.io/stats/view/worker.go:292 +0xad
created by go.opencensus.io/stats/view.init.0
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/go.opencensus.io/stats/view/worker.go:34 +0x8d

goroutine 27 [syscall]:
os/signal.signal_recv()
        /usr/lib/golang/src/runtime/sigqueue.go:152 +0x2f
os/signal.loop()
        /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x19
created by os/signal.Notify.func1.1
        /usr/lib/golang/src/os/signal/signal.go:151 +0x2a

goroutine 7 [chan receive]:
github.com/ClickHouse/clickhouse-go.init.0.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/github.com/ClickHouse/clickhouse-go/bootstrap.go:48 +0x2d
created by github.com/ClickHouse/clickhouse-go.init.0
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/github.com/ClickHouse/clickhouse-go/bootstrap.go:45 +0x3f

goroutine 11 [select]:
main.(*Telegraf).reloadLoop.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/cmd/telegraf/telegraf.go:145 +0xc5
created by main.(*Telegraf).reloadLoop
        /usr/src/mariner/BUILD/telegraf-1.26.0/cmd/telegraf/telegraf.go:144 +0x25b

goroutine 12 [select]:
k8s.io/apimachinery/pkg/util/wait.WaitForWithContext({0x6ffb870, 0xc000c2c240}, 0xc000636108, 0x23d790a?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:660 +0xe5
k8s.io/apimachinery/pkg/util/wait.poll({0x6ffb870, 0xc000c2c240}, 0xf8?, 0x23d6a65?, 0xc000b40390?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:596 +0x9a
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x6ffb870, 0xc000c2c240}, 0x20?, 0xa356340?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:547 +0x49
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0xc000c1c138?, 0x0?, 0xc000aaa320?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:538 +0x7c
k8s.io/client-go/tools/cache.WaitForCacheSync(0xc00032fe30?, {0xc00063c0b8, 0x1, 0x1})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:270 +0x88
k8s.io/client-go/informers.(*sharedInformerFactory).WaitForCacheSync(0xc000bb4640?, 0xc0011a4240?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/informers/factory.go:157 +0x85
github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).watchPod(0xc000618480, {0x6ffb870, 0xc000b51dc0}, 0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:187 +0x279
github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).startK8s.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:92 +0x24f
created by github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).startK8s
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:79 +0x42a

goroutine 13 [select]:
k8s.io/apimachinery/pkg/util/wait.WaitForWithContext({0x6ffb870, 0xc000bad580}, 0xc0008b9ce0, 0x23d790a?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:660 +0xe5
k8s.io/apimachinery/pkg/util/wait.poll({0x6ffb870, 0xc000bad580}, 0xf8?, 0x23d6a65?, 0xc000a74fa0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:596 +0x9a
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x6ffb870, 0xc000bad580}, 0x20?, 0xc000714000?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:547 +0x49
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0xc000963278?, 0x0?, 0xc000989720?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:538 +0x7c
k8s.io/client-go/tools/cache.WaitForCacheSync(0xc0000cde30?, {0xc000709c60, 0x1, 0x1})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:270 +0x88
k8s.io/client-go/informers.(*sharedInformerFactory).WaitForCacheSync(0xc000bb4640?, 0xc000180120?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/informers/factory.go:157 +0x85
github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).watchPod(0xc000618d80, {0x6ffb870, 0xc000b52a40}, 0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:187 +0x279
github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).startK8s.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:92 +0x24f
created by github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).startK8s
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:79 +0x42a

goroutine 14 [select]:
k8s.io/apimachinery/pkg/util/wait.WaitForWithContext({0x6ffb870, 0xc0008d8bc0}, 0xc000c19e18, 0x23d790a?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:660 +0xe5
k8s.io/apimachinery/pkg/util/wait.poll({0x6ffb870, 0xc0008d8bc0}, 0xf8?, 0x23d6a65?, 0xc000b71190?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:596 +0x9a
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x6ffb870, 0xc0008d8bc0}, 0x20?, 0xc0003d8000?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:547 +0x49
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0xc000caad08?, 0x0?, 0xc000a5fb80?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:538 +0x7c
k8s.io/client-go/tools/cache.WaitForCacheSync(0xc000c0de30?, {0xc000164820, 0x1, 0x1})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:270 +0x88
k8s.io/client-go/informers.(*sharedInformerFactory).WaitForCacheSync(0xc000bb4640?, 0xc0011a42a0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/informers/factory.go:157 +0x85
github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).watchPod(0xc000619680, {0x6ffb870, 0xc000b53680}, 0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:187 +0x279
github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).startK8s.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:92 +0x24f
created by github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).startK8s
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:79 +0x42a

goroutine 15 [select]:
k8s.io/apimachinery/pkg/util/wait.WaitForWithContext({0x6ffb870, 0xc0002640c0}, 0xc00027e000, 0x23d790a?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:660 +0xe5
k8s.io/apimachinery/pkg/util/wait.poll({0x6ffb870, 0xc0002640c0}, 0xf8?, 0x23d6a65?, 0xc0007e0890?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:596 +0x9a
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x6ffb870, 0xc0002640c0}, 0x20?, 0xa356340?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:547 +0x49
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0xc00019af60?, 0x0?, 0xc000266000?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:538 +0x7c
k8s.io/client-go/tools/cache.WaitForCacheSync(0xc000c0ae30?, {0xc00027c000, 0x1, 0x1})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:270 +0x88
k8s.io/client-go/informers.(*sharedInformerFactory).WaitForCacheSync(0xc000bb4640?, 0xc000181200?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/informers/factory.go:157 +0x85
github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).watchPod(0xc0011bd200, {0x6ffb870, 0xc000bac2c0}, 0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:187 +0x279
github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).startK8s.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:92 +0x24f
created by github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).startK8s
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:79 +0x42a

goroutine 16 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runOutputs(0xc000709990, 0xc0007dd120)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:861 +0x2c5
github.com/influxdata/telegraf/agent.(*Agent).Run.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:166 +0x5c
created by github.com/influxdata/telegraf/agent.(*Agent).Run
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:164 +0x5fb

goroutine 66 [semacquire]:
sync.runtime_Semacquire(0xc000162270?)
        /usr/lib/golang/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0x0?)
        /usr/lib/golang/src/sync/waitgroup.go:139 +0x52
github.com/influxdata/telegraf/agent.(*Agent).runProcessors(0x0?, {0xc000709b88, 0x1, 0x0?})
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:660 +0x115
github.com/influxdata/telegraf/agent.(*Agent).Run.func2()
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:173 +0x68
created by github.com/influxdata/telegraf/agent.(*Agent).Run
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:171 +0x6b8

goroutine 67 [semacquire]:
sync.runtime_Semacquire(0xc000962ea0?)
        /usr/lib/golang/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0x62ac360?)
        /usr/lib/golang/src/sync/waitgroup.go:139 +0x52
github.com/influxdata/telegraf/agent.(*Agent).runAggregators(0xc000709990, {0x6fa4280?, 0x6f97480?, 0xa354520?}, 0xc00088a9c0)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:729 +0x51f
github.com/influxdata/telegraf/agent.(*Agent).Run.func3()
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:179 +0x6c
created by github.com/influxdata/telegraf/agent.(*Agent).Run
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:177 +0x785

goroutine 68 [semacquire]:
sync.runtime_Semacquire(0xc0002fc0d8?)
        /usr/lib/golang/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0x6fd0938?)
        /usr/lib/golang/src/sync/waitgroup.go:139 +0x52
github.com/influxdata/telegraf/agent.(*Agent).runProcessors(0x6fa8cc0?, {0xc000709b90, 0x1, 0x6fc3a58?})
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:660 +0x115
github.com/influxdata/telegraf/agent.(*Agent).Run.func4()
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:187 +0x68
created by github.com/influxdata/telegraf/agent.(*Agent).Run
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:185 +0x84a

goroutine 69 [semacquire]:
sync.runtime_Semacquire(0xc000962e58?)
        /usr/lib/golang/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0x0?)
        /usr/lib/golang/src/sync/waitgroup.go:139 +0x52
github.com/influxdata/telegraf/agent.(*Agent).runInputs(0xc000709990, {0x6ffb870?, 0xc000e79380}, {0x6ff4130?, 0x0?, 0xa354520?}, 0xc0007dd1e0)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:416 +0x597
github.com/influxdata/telegraf/agent.(*Agent).Run.func5()
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:194 +0x74
created by github.com/influxdata/telegraf/agent.(*Agent).Run
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:192 +0x94a

goroutine 70 [select]:
github.com/influxdata/telegraf/agent.(*AlignedTicker).run(0xc000bacfc0, {0x6ffb870, 0xc000bad000}, 0xc000b9bf90)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:87 +0xf8
github.com/influxdata/telegraf/agent.(*AlignedTicker).start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:65 +0x6a
created by github.com/influxdata/telegraf/agent.(*AlignedTicker).start
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:63 +0x18a

goroutine 71 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherLoop(0x700afe0?, {0x6ffb870, 0xc000e79380}, {0x7019140, 0xc000b592e0}, 0xc00063e050, {0x6fd2f58, 0xc000bacfc0}, 0x6fdc2b0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:554 +0x11c
github.com/influxdata/telegraf/agent.(*Agent).runInputs.func1(0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:412 +0x7e
created by github.com/influxdata/telegraf/agent.(*Agent).runInputs
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:410 +0xec

goroutine 72 [select]:
github.com/influxdata/telegraf/agent.(*AlignedTicker).run(0xc000bad040, {0x6ffb870, 0xc000bad080}, 0xc000bb40a0)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:87 +0xf8
github.com/influxdata/telegraf/agent.(*AlignedTicker).start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:65 +0x6a
created by github.com/influxdata/telegraf/agent.(*AlignedTicker).start
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:63 +0x18a

goroutine 73 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherLoop(0x6fc4a20?, {0x6ffb870, 0xc000e79380}, {0x7019140, 0xc000b59300}, 0xc0009782d0, {0x6fd2f58, 0xc000bad040}, 0x6fefed0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:554 +0x11c
github.com/influxdata/telegraf/agent.(*Agent).runInputs.func1(0x6fc62a8?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:412 +0x7e
created by github.com/influxdata/telegraf/agent.(*Agent).runInputs
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:410 +0xec

goroutine 74 [select]:
github.com/influxdata/telegraf/agent.(*AlignedTicker).run(0xc000bad100, {0x6ffb870, 0xc000bad140}, 0xc000bb4190)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:87 +0xf8
github.com/influxdata/telegraf/agent.(*AlignedTicker).start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:65 +0x6a
created by github.com/influxdata/telegraf/agent.(*AlignedTicker).start
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:63 +0x18a

goroutine 75 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherLoop(0x6f971e0?, {0x6ffb870, 0xc000e79380}, {0x7019140, 0xc000b59320}, 0xc000978500, {0x6fd2f58, 0xc000bad100}, 0x6feb160?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:554 +0x11c
github.com/influxdata/telegraf/agent.(*Agent).runInputs.func1(0x6fd06e0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:412 +0x7e
created by github.com/influxdata/telegraf/agent.(*Agent).runInputs
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:410 +0xec

goroutine 76 [select]:
github.com/influxdata/telegraf/agent.(*AlignedTicker).run(0xc000bad180, {0x6ffb870, 0xc000bad1c0}, 0xc000bb4280)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:87 +0xf8
github.com/influxdata/telegraf/agent.(*AlignedTicker).start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:65 +0x6a
created by github.com/influxdata/telegraf/agent.(*AlignedTicker).start
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:63 +0x18a

goroutine 77 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherLoop(0x0?, {0x6ffb870, 0xc000e79380}, {0x7019140, 0xc000b59340}, 0xc0009785f0, {0x6fd2f58, 0xc000bad180}, 0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:554 +0x11c
github.com/influxdata/telegraf/agent.(*Agent).runInputs.func1(0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:412 +0x7e
created by github.com/influxdata/telegraf/agent.(*Agent).runInputs
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:410 +0xec

goroutine 78 [select]:
github.com/influxdata/telegraf/agent.(*AlignedTicker).run(0xc000bad200, {0x6ffb870, 0xc000bad240}, 0xc000bb4370)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:87 +0xf8
github.com/influxdata/telegraf/agent.(*AlignedTicker).start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:65 +0x6a
created by github.com/influxdata/telegraf/agent.(*AlignedTicker).start
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:63 +0x18a

goroutine 79 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherLoop(0x0?, {0x6ffb870, 0xc000e79380}, {0x7019140, 0xc000b59360}, 0xc000978c30, {0x6fd2f58, 0xc000bad200}, 0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:554 +0x11c
github.com/influxdata/telegraf/agent.(*Agent).runInputs.func1(0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:412 +0x7e
created by github.com/influxdata/telegraf/agent.(*Agent).runInputs
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:410 +0xec

goroutine 80 [select]:
github.com/influxdata/telegraf/agent.(*AlignedTicker).run(0xc000bad280, {0x6ffb870, 0xc000bad2c0}, 0xc000bb4460)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:87 +0xf8
github.com/influxdata/telegraf/agent.(*AlignedTicker).start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:65 +0x6a
created by github.com/influxdata/telegraf/agent.(*AlignedTicker).start
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:63 +0x18a

goroutine 81 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherLoop(0x0?, {0x6ffb870, 0xc000e79380}, {0x7019140, 0xc000b593a0}, 0xc000978cd0, {0x6fd2f58, 0xc000bad280}, 0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:554 +0x11c
github.com/influxdata/telegraf/agent.(*Agent).runInputs.func1(0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:412 +0x7e
created by github.com/influxdata/telegraf/agent.(*Agent).runInputs
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:410 +0xec

goroutine 82 [select]:
github.com/influxdata/telegraf/agent.(*AlignedTicker).run(0xc000bad300, {0x6ffb870, 0xc000bad340}, 0xc000bb4550)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:87 +0xf8
github.com/influxdata/telegraf/agent.(*AlignedTicker).start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:65 +0x6a
created by github.com/influxdata/telegraf/agent.(*AlignedTicker).start
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:63 +0x18a

goroutine 83 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherLoop(0x0?, {0x6ffb870, 0xc000e79380}, {0x7019140, 0xc000b59400}, 0xc000978dc0, {0x6fd2f58, 0xc000bad300}, 0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:554 +0x11c
github.com/influxdata/telegraf/agent.(*Agent).runInputs.func1(0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:412 +0x7e
created by github.com/influxdata/telegraf/agent.(*Agent).runInputs
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:410 +0xec

goroutine 43 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc000962828)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:649 +0x107
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:645 +0x3c

goroutine 28 [select]:
github.com/influxdata/telegraf/agent.(*Agent).flushLoop(0xc000813320?, {0x6ffb870, 0xc000264000}, 0xc0011a6600, {0x6fd2f80, 0xc000813320})
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:906 +0x1d4
github.com/influxdata/telegraf/agent.(*Agent).runOutputs.func1(0x6fab2a0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:857 +0x125
created by github.com/influxdata/telegraf/agent.(*Agent).runOutputs
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:851 +0xc7

goroutine 29 [select]:
github.com/influxdata/telegraf/agent.(*RollingTicker).run(0xc000813320, {0x6ffb870, 0xc000264080}, 0xc0002620f0)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:258 +0xe6
github.com/influxdata/telegraf/agent.(*RollingTicker).start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:248 +0x6a
created by github.com/influxdata/telegraf/agent.(*RollingTicker).start
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/tick.go:246 +0x16c

goroutine 98 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc0009627f8)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:649 +0x107
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:645 +0x3c

goroutine 84 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runAggregators.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:698 +0xa7
created by github.com/influxdata/telegraf/agent.(*Agent).runAggregators
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:696 +0x39a

goroutine 85 [select]:
github.com/influxdata/telegraf/agent.(*Agent).push(0x0?, {0x6ffb870, 0xc000bad380}, 0xc0011d2000, {0x7019140, 0xc000b59440})
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:767 +0xce
github.com/influxdata/telegraf/agent.(*Agent).runAggregators.func2(0xc0011d2000)
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:725 +0x18d
created by github.com/influxdata/telegraf/agent.(*Agent).runAggregators
        /usr/src/mariner/BUILD/telegraf-1.26.0/agent/agent.go:717 +0x3c7

goroutine 86 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0009896a8, 0x1)
        /usr/lib/golang/src/runtime/sema.go:517 +0x14c
sync.(*Cond).Wait(0xc000bf4980?)
        /usr/lib/golang/src/sync/cond.go:70 +0x8c
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc000989680, 0xc000967410)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:527 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000bb2870)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/controller.go:184 +0x36
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x40f95f?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x23d6585?, {0x6fa7c80, 0xc000f110e0}, 0x1, 0xc000180120)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000bb28d8?, 0x3b9aca00, 0x0, 0x80?, 0x7f2774329af8?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*controller).Run(0xc000bb2870, 0xc000180120)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/controller.go:155 +0x2c6
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0007bb900, 0xc0002db798?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:443 +0x49c
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 87 [chan send]:
k8s.io/client-go/tools/cache.(*processorListener).add(...)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:773
k8s.io/client-go/tools/cache.(*sharedProcessor).distribute(0x5a73460?, {0x5a73460, 0xc000a751d0}, 0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:654 +0x107
k8s.io/client-go/tools/cache.(*sharedIndexInformer).OnAdd(0xc0009888c0, {0x62c3c60, 0xc0017c4ee0})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:581 +0x85
k8s.io/client-go/tools/cache.processDeltas({0x6ff8a80, 0xc0009888c0}, {0x7f2774329a70, 0xc000962fa8}, 0x0, {0xc001705760?, 0x1, 0x70?})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/controller.go:439 +0x2a7
k8s.io/client-go/tools/cache.(*sharedIndexInformer).HandleDeltas(0xc0009888c0, {0x593fac0?, 0xc000963530?})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:571 +0x167
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0009bc1e0, 0xc000615570)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:554 +0x596
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc00032a000)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/controller.go:184 +0x36
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x40f95f?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x23d6585?, {0x6fa7c80, 0xc0008ad3e0}, 0x1, 0xc000180120)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00032a068?, 0x3b9aca00, 0x0, 0x0?, 0x7f2774329af8?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*controller).Run(0xc00032a000, 0xc000180120)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/controller.go:155 +0x2c6
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0009888c0, 0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:443 +0x49c
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 91 [chan receive]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc000f06230, 0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:669 +0x45
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:58 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 92 [chan receive]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0x28
created by k8s.io/client-go/tools/cache.(*controller).Run
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/controller.go:130 +0xbe

goroutine 45 [chan receive]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc000f062a0, 0x675f6363130a3512?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:669 +0x45
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:58 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 46 [chan receive]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0x28
created by k8s.io/client-go/tools/cache.(*controller).Run
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/controller.go:130 +0xbe

goroutine 31 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc000645b80)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:783 +0x119
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 32 [runnable]:
time.appendInt({0xc000c0e8d8?, 0x0?, 0x40?}, 0x7e7?, 0x4)
        /usr/lib/golang/src/time/format.go:396 +0x265
time.Time.AppendFormat({0xa396900?, 0x0?, 0x0?}, {0xc000c0e8d8, 0x0, 0x40}, {0x640c843, 0x19})
        /usr/lib/golang/src/time/format.go:656 +0x1225
time.Time.Format({0x0?, 0x0?, 0x0?}, {0x640c843?, 0x203000?})
        /usr/lib/golang/src/time/format.go:608 +0xe5
github.com/influxdata/telegraf/logger.(*telegrafLog).Write(0xc0008bf050, {0xc001194800, 0x52, 0x400})
        /usr/src/mariner/BUILD/telegraf-1.26.0/logger/logger.go:73 +0x165
log.(*Logger).Output(0xc0001ba230, 0x32?, {0xc00016baa0, 0x51})
        /usr/lib/golang/src/log/log.go:198 +0x368
log.Printf({0xc0009f5d40?, 0x637ed7c?}, {0xc000a75430?, 0xc0008b8180?, 0xc000100400?})
        /usr/lib/golang/src/log/log.go:354 +0x55
github.com/influxdata/telegraf/models.(*Logger).Debugf(0xc000a94db0?, {0x6428a2c?, 0xc000bc58e0?}, {0xc000a75430, 0x1, 0x1})
        /usr/src/mariner/BUILD/telegraf-1.26.0/models/log.go:47 +0x8c
github.com/influxdata/telegraf/plugins/inputs/prometheus.registerPod(0xc0017c4ee0, 0xc000619680)
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:371 +0x15c
github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).watchPod.func1({0x62c3c60?, 0xc0017c4ee0?})
        /usr/src/mariner/BUILD/telegraf-1.26.0/plugins/inputs/prometheus/kubernetes.go:151 +0x69
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/controller.go:232
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:818 +0x134
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x1277446efb8?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000057f38?, {0x6fa7c80, 0xc000702900}, 0x1, 0xc0008d21e0)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x6faabc0?, 0x3b9aca00, 0x0, 0x10?, 0x70055c0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0011a6880?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:812 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 33 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0011a6880)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:783 +0x119
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 114 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:813 +0x57
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x6ffad00?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0002daf38?, {0x6fa7c80, 0xc0005d44e0}, 0x1, 0xc00097c1e0)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0xc0?, 0xc000eda900?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0010a6500?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:812 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 115 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0010a6500)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:783 +0x119
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 93 [select]:
k8s.io/client-go/tools/cache.watchHandler({0x6?, 0x0?, 0xa354520?}, {0x6fdd4f8?, 0xc0008d8a00}, {0x7f2774329af8, 0xc000989680}, {0x70372e0?, 0x62c2de0}, 0x0, ...)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/reflector.go:493 +0x1b8
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0010a9e00, 0xc000180120)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/reflector.go:329 +0x62c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/reflector.go:223 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000c63a20?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000a94a50?, {0x6fa7c60, 0xc000bb46e0}, 0x1, 0xc000180120)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0010a9e00, 0xc000180120)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/reflector.go:222 +0x17d
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:58 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 106 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/reflector.go:273 +0x11d
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/reflector.go:267 +0x25f

goroutine 47 [runnable]:
golang.org/x/net/http2.(*ClientConn).RoundTrip(0xc000bf6900, 0xc000c38900)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/transport.go:1269 +0x451
golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc000c437a0, 0xc000c38900, {0x80?})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/transport.go:561 +0x1c5
golang.org/x/net/http2.(*Transport).RoundTrip(...)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/transport.go:513
golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip({0xc0003fd540?}, 0xc000c38900?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/transport.go:3085 +0x1b
net/http.(*Transport).roundTrip(0xc0003fd540, 0xc000c38900)
        /usr/lib/golang/src/net/http/transport.go:540 +0x38a
net/http.(*Transport).RoundTrip(0x5dd6560?, 0xc00179d560?)
        /usr/lib/golang/src/net/http/roundtrip.go:17 +0x19
k8s.io/client-go/transport.(*bearerAuthRoundTripper).RoundTrip(0xc00096b320, 0xc000c38800)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/transport/round_trippers.go:317 +0x3e5
k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0xc000896d40, 0xc000c38700)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/transport/round_trippers.go:168 +0x350
net/http.send(0xc000c38700, {0x6fa8320, 0xc000896d40}, {0x61f30c0?, 0x1?, 0x0?})
        /usr/lib/golang/src/net/http/client.go:251 +0x5f7
net/http.(*Client).send(0xc00096b350, 0xc000c38700, {0x7f279fc92a68?, 0x100?, 0x0?})
        /usr/lib/golang/src/net/http/client.go:175 +0x9b
net/http.(*Client).do(0xc00096b350, 0xc000c38700)
        /usr/lib/golang/src/net/http/client.go:715 +0x8fc
net/http.(*Client).Do(...)
        /usr/lib/golang/src/net/http/client.go:581
k8s.io/client-go/rest.(*Request).Watch(0xc000c38500, {0x6ffb8a8, 0xc0001a4008})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/rest/request.go:639 +0x154
k8s.io/client-go/kubernetes/typed/core/v1.(*pods).Watch(0xc001705e80, {0x6ffb8a8, 0xc0001a4008}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, ...}, ...})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/kubernetes/typed/core/v1/pod.go:116 +0x146
k8s.io/client-go/informers/core/v1.NewFilteredPodInformer.func2({{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x1, ...})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/informers/core/v1/pod.go:71 +0x16a
k8s.io/client-go/tools/cache.(*ListWatch).Watch(0x7f2774583698?, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, ...})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/listwatch.go:111 +0x57
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0002cfa40, 0xc000180120)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/reflector.go:315 +0x49b
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/reflector.go:223 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000844600?, {0x6fa7c60, 0xc0003260a0}, 0x1, 0xc000180120)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0002cfa40, 0xc000180120)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/reflector.go:222 +0x17d
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:58 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 133 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/reflector.go:273 +0x11d
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/reflector.go:267 +0x25f

goroutine 48 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:813 +0x57
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x6ffad00?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0002d8738?, {0x6fa7c80, 0xc000b441b0}, 0x1, 0xc0003481e0)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x13?, 0x3b9aca00, 0x0, 0xc0?, 0xc000edafc0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000c20000?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:812 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 49 [runnable]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc000c20000)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/tools/cache/shared_informer.go:783 +0x119
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 107 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc000cad080, 0xc0010a4d00)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/transport.go:1437 +0xa87
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/transport.go:1299 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/transport.go:1228 +0x30a

goroutine 111 [select]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:710 +0x1c5
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:693 +0xca

goroutine 108 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000cad0c8, 0x0)
        /usr/lib/golang/src/runtime/sema.go:517 +0x14c
sync.(*Cond).Wait(0xc000cad0b8?)
        /usr/lib/golang/src/sync/cond.go:70 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc000cad0b0, {0xc0008cea00, 0x200, 0x200})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0xc0008cb4b4?}, {0xc0008cea00?, 0x0?, 0x3?})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/transport.go:2507 +0x75
encoding/json.(*Decoder).refill(0xc0011d8b40)
        /usr/lib/golang/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc0011d8b40)
        /usr/lib/golang/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc0011d8b40, {0x58640a0, 0xc000caa870})
        /usr/lib/golang/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000bd6ab0, {0xc000c80c00, 0x400, 0x400})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0008d0410, 0x0?, {0x6fdd340, 0xc0008d8a40})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000bf4c40)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0008d8a00)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdc
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x130

goroutine 110 [select]:
k8s.io/apimachinery/pkg/util/wait.ContextForChannel.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:303 +0x71
created by k8s.io/apimachinery/pkg/util/wait.ContextForChannel
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:302 +0xc5

goroutine 104 [IO wait]:
internal/poll.runtime_pollWait(0x7f27766320e8, 0x72)
        /usr/lib/golang/src/runtime/netpoll.go:305 +0x89
internal/poll.(*pollDesc).wait(0xc0010a6c00?, 0xc000a50500?, 0x0)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0010a6c00, {0xc000a50500, 0x2500, 0x2500})
        /usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc0010a6c00, {0xc000a50500?, 0xc000bf42c0?, 0xc000a50505?})
        /usr/lib/golang/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc00063c0b0, {0xc000a50500?, 0x2a?, 0xc000bf42c0?})
        /usr/lib/golang/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc000caad38, {0xc000a50500?, 0x0?, 0x518e68?})
        /usr/lib/golang/src/crypto/tls/conn.go:787 +0x3d
bytes.(*Buffer).ReadFrom(0xc000b8e5f8, {0x6f99900, 0xc000caad38})
        /usr/lib/golang/src/bytes/buffer.go:202 +0x98
crypto/tls.(*Conn).readFromUntil(0xc000b8e380, {0x6fa8640?, 0xc00063c0b0}, 0x203000?)
        /usr/lib/golang/src/crypto/tls/conn.go:809 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc000b8e380, 0x0)
        /usr/lib/golang/src/crypto/tls/conn.go:616 +0x116
crypto/tls.(*Conn).readRecord(...)
        /usr/lib/golang/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc000b8e380, {0xc0008fd000, 0x1000, 0xc0007010d0?})
        /usr/lib/golang/src/crypto/tls/conn.go:1315 +0x16f
bufio.(*Reader).Read(0xc0000ebe00, {0xc00026cd60, 0x9, 0xc0010c48c4?})
        /usr/lib/golang/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x6f96a80, 0xc0000ebe00}, {0xc00026cd60, 0x9, 0x9}, 0x9)
        /usr/lib/golang/src/io/io.go:332 +0x9a
io.ReadFull(...)
        /usr/lib/golang/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc00026cd60?, 0x9?, 0xc000c71da0?}, {0x6f96a80?, 0xc0000ebe00?})
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00026cd20)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/frame.go:498 +0x95
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000c71f98)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/transport.go:2224 +0x130
golang.org/x/net/http2.(*ClientConn).readLoop(0xc000bf6900)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/transport.go:2119 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/transport.go:817 +0xc5c

goroutine 134 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0017b1800, 0xc000c38900)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/transport.go:1437 +0xa87
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0x0?)
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/transport.go:1299 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/golang.org/x/net/http2/transport.go:1228 +0x30a

goroutine 97 [select]:
k8s.io/apimachinery/pkg/util/wait.ContextForChannel.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:303 +0x71
created by k8s.io/apimachinery/pkg/util/wait.ContextForChannel
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:302 +0xc5

goroutine 146 [select]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:710 +0x1c5
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:693 +0xca

goroutine 122 [select]:
k8s.io/apimachinery/pkg/util/wait.ContextForChannel.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:303 +0x71
created by k8s.io/apimachinery/pkg/util/wait.ContextForChannel
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:302 +0xc5

goroutine 123 [select]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:710 +0x1c5
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:693 +0xca

goroutine 162 [select]:
k8s.io/apimachinery/pkg/util/wait.ContextForChannel.func1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:303 +0x71
created by k8s.io/apimachinery/pkg/util/wait.ContextForChannel
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:302 +0xc5

goroutine 163 [select]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
        /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:710 +0x1c5
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
      /usr/src/mariner/BUILD/telegraf-1.26.0/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:693 +0xca

@telegraf-tiger telegraf-tiger bot removed the waiting for response waiting for response from contributor label May 2, 2023
@powersj
Copy link
Contributor

powersj commented May 3, 2023

@redbaron based on the above crash it looks like we may need to add some mutex around access to the pods. Does it make sense to do this in the UpdateFunc when registering a pod?

@redbaron
Copy link
Contributor

redbaron commented May 3, 2023

@samskan , your sample config shows monitor_kubernetes_pods = false, but it disables code path to watchPod where problem arises. From the stacktraces, there should be multiple instances of prometheus plugin, could you provide full config please?

@redbaron redbaron mentioned this issue May 3, 2023
Merged
3 tasks
@powersj powersj added the waiting for response waiting for response from contributor label May 3, 2023
@samskan
Copy link
Author

samskan commented May 4, 2023

@redbaron we do have multiple prometheus plugins. But I am sure that only the above mentioned plugin is causing the issue. When we comment this input plugin we dont see the crash.
[[inputs.prometheus]]
metric_version = 2
name_override = "packetcoremonitor"

  monitor_kubernetes_pods = false
  kubernetes_services = ["http://core-kube-state-metrics.core:8080/metrics"]
[[inputs.prometheus]]
  metric_version = 2
  monitor_kubernetes_pods = true
  pod_scrape_scope = "cluster"
  monitor_kubernetes_pods_namespace = "core"
[[inputs.prometheus]]
  metric_version = 2
  monitor_kubernetes_pods = true
  pod_scrape_scope = "cluster"
  monitor_kubernetes_pods_namespace = "core"
[[inputs.prometheus]]
  metric_version = 2
  name_override = "packetcoremonitor"
  monitor_kubernetes_pods = true
  pod_scrape_scope = "cluster"
  monitor_kubernetes_pods_namespace = "core"
[[inputs.prometheus]]
  metric_version = 2
  monitor_kubernetes_pods = true
  pod_scrape_scope = "cluster"
  monitor_kubernetes_pods_namespace = "core"
[[inputs.prometheus]]
  metric_version = 2
  monitor_kubernetes_pods = false
  pod_scrape_scope = "cluster"
  bearer_token = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  kubernetes_services = ["https://kubernetes.default.svc/api/v1/nodes/${MY_NODE_NAME}/proxy/metrics/cadvisor"]

@telegraf-tiger telegraf-tiger bot removed the waiting for response waiting for response from contributor label May 4, 2023
@redbaron
Copy link
Contributor

redbaron commented May 4, 2023

@samskan , thanks. I suspect adding extra plugin instance just highlights existing race.

I prepared PR which streamlines Prometheus plugin initialization, which might or might not help with your case: #13231 . If you don't mind building it with go build -race flag then it can help detecting the root cause.

@samskan
Copy link
Author

samskan commented May 5, 2023

@redbaron I think previously i mentioned that we are building custom version of telegraf. But we are not. We are just taking the released version and using our own telegraf.star and telegraf_processor.star.
So I cant do that "go build -race"

@powersj
Copy link
Contributor

powersj commented May 5, 2023

@samskan you can try out the artifact in @13231 though and confirm if it fixes the issue. Telegraf binaries are built with -race already.

@powersj powersj added the waiting for response waiting for response from contributor label May 5, 2023
@redbaron
Copy link
Contributor

redbaron commented May 5, 2023 •
edited
Loading

@powersj , is it possible to add docker image to the artifact list built for each PR?

@telegraf-tiger telegraf-tiger bot removed the waiting for response waiting for response from contributor label May 5, 2023
@powersj
Copy link
Contributor

powersj commented May 5, 2023

@powersj , is it possible to add docker image to the artifact list built for each PR?

Unfortunately, this is not something we plan on doing. The amount of artifacts generated are already fairly costly.

@samskan
Copy link
Author

samskan commented May 8, 2023

With the provided artifacts, there was no panic observed.Please let me know when we can get this in the official release

@powersj
Copy link
Contributor

powersj commented May 8, 2023

Thank you for confirming. It will go into v1.26.3 in a couple weeks.

@samskan
Copy link
Author

samskan commented May 26, 2023

Looks like this fix has not gone in 1.26.3 release. I dont see that in release notes as well.
Also when tested with 1.26.3 version i see the issue which was seen previously.
@powersj u please confirm if fix #13231 is part of 1.26.3 release?

@powersj
Copy link
Contributor

powersj commented May 26, 2023

Looks like this fix has not gone in 1.26.3 release.

correct, if you want to try this out please use a nightly build: https://github.com/influxdata/telegraf/blob/master/docs/NIGHTLIES.md

@samskan
Copy link
Author

samskan commented May 29, 2023

When can I expect this fix to be present in the official build?

@samskan
Copy link
Author

samskan commented May 30, 2023

@powersj Please let me know when this fix will be part of official build?

@powersj
Copy link
Contributor

powersj commented May 30, 2023

You can look at the very first question in the FAQ: https://github.com/influxdata/telegraf/blob/master/docs/FAQ.md

As June is in 2 days it will be part of the next minor release, v1.27

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug unexpected problem or unintended behavior
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(inputs.prometheus): avoid race when creating informer factory redbaron/telegraf
3 participants
@redbaron @powersj @samskan