inputs.gnmi with dynamic tagging won't work with more than one CPUs #11006

kb70 · 2022-04-20T14:07:37Z

Relevant telegraf.conf

...
[[inputs.gnmi]]
   addresses = ["device1:6030", "device2:6030", "device3:6030", "device4:6030"]
   username = "xxx"
   password = "yyy"

   redial = "15s"
   encoding = "json"

   tagexclude = ["path", "network_instances_network_instance_protocols_protocol_name"]

   ## Build dynamic tag map for Interface descriptions
   [[inputs.gnmi.subscription]]
      name = "if"
      path = "/interfaces/interface/state/description"
      subscription_mode = "on_change"
      tag_only = true

   [[inputs.gnmi.subscription]]
      name = "if_state"
      path = "/interfaces/interface/state"
      subscription_mode = "sample"
      sample_interval = "15s"
      fielddrop = ["description", "mtu", "name", "last_change", "hardware_port", "type"]

Logs from Telegraf

2022-04-20T12:01:41Z I! Using config file: /etc/telegraf/telegraf.conf
2022-04-20T12:01:41Z I! Starting Telegraf 1.22.1
2022-04-20T12:01:41Z I! Loaded inputs: gnmi internal
2022-04-20T12:01:41Z I! Loaded aggregators: 
2022-04-20T12:01:41Z I! Loaded processors: enum (3x) execd regex
2022-04-20T12:01:41Z I! Loaded outputs: file prometheus_client
2022-04-20T12:01:41Z I! Tags enabled: host=xxx
2022-04-20T12:01:41Z I! [agent] Config: Interval:2m0s, Quiet:false, Hostname:"odin02.arista", Flush Interval:10s
2022-04-20T12:01:41Z D! [agent] Initializing plugins
2022-04-20T12:01:41Z D! [agent] Connecting outputs
2022-04-20T12:01:41Z D! [agent] Attempting connection to [outputs.prometheus_client]
2022-04-20T12:01:41Z I! [outputs.prometheus_client] Listening on http://[::]:9273/metrics
2022-04-20T12:01:41Z D! [agent] Successfully connected to outputs.prometheus_client
2022-04-20T12:01:41Z D! [agent] Attempting connection to [outputs.file]
2022-04-20T12:01:41Z D! [agent] Successfully connected to outputs.file
2022-04-20T12:01:41Z I! [processors.execd] Starting process: /etc/telegraf/execd/base64ToFloat []
2022-04-20T12:01:41Z D! [agent] Starting service inputs
2022-04-20T12:01:41Z E! [processors.execd] stderr: "base64ToFloat here: starting up"
2022-04-20T12:01:41Z D! [inputs.gnmi] Connection to gNMI device device1:6030 established
2022-04-20T12:01:41Z D! [inputs.gnmi] Connection to gNMI device device2:6030 established
2022-04-20T12:01:41Z D! [inputs.gnmi] Connection to gNMI device device3:6030 established
2022-04-20T12:01:41Z D! [inputs.gnmi] Connection to gNMI device device4:6030 established
fatal error: concurrent map read and map write

goroutine 79 [running]:
runtime.throw({0x5413263, 0x0})
	/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc0005a5768 sp=0xc0005a5738 pc=0x439091
runtime.mapaccess2_faststr(0x49a6320, 0xc000075699, {0xc001480dc0, 0x19})
	/usr/local/go/src/runtime/map_faststr.go:116 +0x3d4 fp=0xc0005a57d0 sp=0xc0005a5768 pc=0x414e14
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).handleSubscribeResponseUpdate(0xc0003fea80, {0xc000075699, 0x12}, 0xc00064b8a8)
	/go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:327 +0xb27 fp=0xc0005a5d70 sp=0xc0005a57d0 pc=0x17be4e7
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).handleSubscribeResponse(0xc000691570, {0xc000075699, 0xc001069e60}, 0xc00065f7b0)
	/go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:264 +0x4e fp=0xc0005a5dc0 sp=0xc0005a5d70 pc=0x17bd88e
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).subscribeGNMI(0xc0003fea80, {0x5e68590, 0xc001069e60}, {0xc000075699, 0x12}, 0x0, 0x0)
	/go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:256 +0x536 fp=0xc0005a5ee8 sp=0xc0005a5dc0 pc=0x17bd476
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start.func1({0xc000075699, 0x12})
	/go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:158 +0x125 fp=0xc0005a5fc0 sp=0xc0005a5ee8 pc=0x17bc8a5
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start·dwrap·1()
	/go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:167 +0x31 fp=0xc0005a5fe0 sp=0xc0005a5fc0 pc=0x17bc751
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005a5fe8 sp=0xc0005a5fe0 pc=0x46d8e1
created by github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start
	/go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:155 +0x9d3

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0x0)
	/usr/local/go/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0x53ecaae)
	/usr/local/go/src/sync/waitgroup.go:130 +0x71
github.com/influxdata/telegraf/agent.(*Agent).Run(0xc000b8e0d0, {0x5e684e8, 0xc00108c700})
	/go/src/github.com/influxdata/telegraf/agent/agent.go:181 +0x89c
main.runAgent({0x5e684e8, 0xc00108c700}, {0x8d08ae8, 0x0, 0x0}, {0x8d08ae8, 0x0, 0x0})
	/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:328 +0xc57
main.reloadLoop({0x8d08ae8, 0x0, 0x0}, {0x8d08ae8, 0x0, 0x0})
	/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:158 +0x28a
main.run(...)
	/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf_posix.go:8
main.main()
	/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:501 +0xa9a

goroutine 69 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000228000)
	/go/pkg/mod/[email protected]/stats/view/worker.go:276 +0xb9
created by go.opencensus.io/stats/view.init.0
	/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x92

goroutine 70 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)
	/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1181 +0x6a
created by k8s.io/klog/v2.init.0
	/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:420 +0xfb

goroutine 71 [chan receive]:
github.com/ClickHouse/clickhouse-go.init.0.func1()
	/go/pkg/mod/github.com/!click!house/[email protected]/bootstrap.go:48 +0x2d
created by github.com/ClickHouse/clickhouse-go.init.0
	/go/pkg/mod/github.com/!click!house/[email protected]/bootstrap.go:45 +0x45

goroutine 83 [syscall]:
os/signal.signal_recv()
	/usr/local/go/src/runtime/sigqueue.go:169 +0x98
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:24 +0x19
created by os/signal.Notify.func1.1
	/usr/local/go/src/os/signal/signal.go:151 +0x2c

goroutine 84 [select]:
main.reloadLoop.func1()
	/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:145 +0x85
created by main.reloadLoop
	/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:144 +0x245

goroutine 77 [IO wait]:
internal/poll.runtime_pollWait(0x7f098c750a18, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000320980, 0x4, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000320980)
	/usr/local/go/src/internal/poll/fd_unix.go:402 +0x22c
net.(*netFD).accept(0xc000320980)
	/usr/local/go/src/net/fd_unix.go:173 +0x35
net.(*TCPListener).accept(0xc000aaf818)
	/usr/local/go/src/net/tcpsock_posix.go:140 +0x28
net.(*TCPListener).Accept(0xc000aaf818)
	/usr/local/go/src/net/tcpsock.go:262 +0x3d
net/http.(*Server).Serve(0xc000abe1c0, {0x5e0fa78, 0xc000aaf818})
	/usr/local/go/src/net/http/server.go:3002 +0x394
github.com/influxdata/telegraf/plugins/outputs/prometheus_client.(*PrometheusClient).Connect.func1()
	/go/src/github.com/influxdata/telegraf/plugins/outputs/prometheus_client/prometheus_client.go:227 +0x75
created by github.com/influxdata/telegraf/plugins/outputs/prometheus_client.(*PrometheusClient).Connect
	/go/src/github.com/influxdata/telegraf/plugins/outputs/prometheus_client/prometheus_client.go:225 +0x252

goroutine 78 [syscall]:
syscall.Syscall6(0xf7, 0x1, 0x1f, 0xc0004d4d08, 0x1000004, 0x0, 0x0)
	/usr/local/go/src/syscall/asm_linux_amd64.s:43 +0x5
os.(*Process).blockUntilWaitable(0xc000fed950)
	/usr/local/go/src/os/wait_waitid.go:33 +0x9c
os.(*Process).wait(0xc000fed950)
	/usr/local/go/src/os/exec_unix.go:23 +0x28
os.(*Process).Wait(...)
	/usr/local/go/src/os/exec.go:132
os/exec.(*Cmd).Wait(0xc000aba580)
	/usr/local/go/src/os/exec/exec.go:507 +0x54
github.com/influxdata/telegraf/internal/process.(*Process).cmdWait(0xc0004ee630, {0x5e684e8, 0xc000aa8680})
	/go/src/github.com/influxdata/telegraf/internal/process/process.go:178 +0x27c
github.com/influxdata/telegraf/internal/process.(*Process).cmdLoop(0xc0004ee630, {0x5e684e8, 0xc000aa8680})
	/go/src/github.com/influxdata/telegraf/internal/process/process.go:121 +0x5f
github.com/influxdata/telegraf/internal/process.(*Process).Start.func1()
	/go/src/github.com/influxdata/telegraf/internal/process/process.go:65 +0x2e
created by github.com/influxdata/telegraf/internal/process.(*Process).Start
	/go/src/github.com/influxdata/telegraf/internal/process/process.go:64 +0xf3

goroutine 98 [IO wait]:
internal/poll.runtime_pollWait(0x7f098c750760, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc001083440, 0xc00032e000, 0x1)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc001083440, {0xc00032e000, 0x400, 0x400})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
os.(*File).read(...)
	/usr/local/go/src/os/file_posix.go:32
os.(*File).Read(0xc000b8e148, {0xc00032e000, 0x7f098c76ffc0, 0xc001083318})
	/usr/local/go/src/os/file.go:119 +0x5e
github.com/influxdata/telegraf/plugins/parsers/influx.(*streamMachine).Next(0xc0004a6048)
	plugins/parsers/influx/machine.go.rl:519 +0x214
github.com/influxdata/telegraf/plugins/parsers/influx.(*StreamParser).Next(0xc00037c3f0)
	/go/src/github.com/influxdata/telegraf/plugins/parsers/influx/parser.go:192 +0x28
github.com/influxdata/telegraf/plugins/processors/execd.(*Execd).cmdReadOutStream(0xc000320780, {0x5db2760, 0xc000b8e148})
	/go/src/github.com/influxdata/telegraf/plugins/processors/execd/execd.go:151 +0x45
github.com/influxdata/telegraf/plugins/processors/execd.(*Execd).cmdReadOut(0x5255420, {0x5db2760, 0xc000b8e148})
	/go/src/github.com/influxdata/telegraf/plugins/processors/execd/execd.go:123 +0x4b
github.com/influxdata/telegraf/internal/process.(*Process).cmdWait.func1()
	/go/src/github.com/influxdata/telegraf/internal/process/process.go:158 +0x48
created by github.com/influxdata/telegraf/internal/process.(*Process).cmdWait
	/go/src/github.com/influxdata/telegraf/internal/process/process.go:157 +0x148

goroutine 99 [IO wait]:
internal/poll.runtime_pollWait(0x7f098c750590, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc001083500, 0xc000048020, 0x1)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc001083500, {0xc000048020, 0xfe0, 0xfe0})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
os.(*File).read(...)
	/usr/local/go/src/os/file_posix.go:32
os.(*File).Read(0xc000b8e158, {0xc000048020, 0xc000194050, 0x20})
	/usr/local/go/src/os/file.go:119 +0x5e
bufio.(*Scanner).Scan(0xc0007c6ef0)
	/usr/local/go/src/bufio/scan.go:215 +0x865
github.com/influxdata/telegraf/plugins/processors/execd.(*Execd).cmdReadErr(0xc000320780, {0x5db2760, 0xc000b8e158})
	/go/src/github.com/influxdata/telegraf/plugins/processors/execd/execd.go:177 +0xad
github.com/influxdata/telegraf/internal/process.(*Process).cmdWait.func2()
	/go/src/github.com/influxdata/telegraf/internal/process/process.go:164 +0x48
created by github.com/influxdata/telegraf/internal/process.(*Process).cmdWait
	/go/src/github.com/influxdata/telegraf/internal/process/process.go:163 +0x1b2

goroutine 100 [select]:
github.com/influxdata/telegraf/internal/process.(*Process).cmdWait.func3()
	/go/src/github.com/influxdata/telegraf/internal/process/process.go:170 +0xac
created by github.com/influxdata/telegraf/internal/process.(*Process).cmdWait
	/go/src/github.com/influxdata/telegraf/internal/process/process.go:169 +0x26f

goroutine 80 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc000178fc0)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:323 +0x85
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:338
google.golang.org/grpc.(*csAttempt).recvMsg(0xc00061a370, {0x50d54a0, 0xc0011905f0}, 0x0)
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:961 +0xbb
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x203000)
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:826 +0x25
google.golang.org/grpc.(*clientStream).withRetry(0xc000375c00, 0xc0006bdd60, 0xc0006bdd30)
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:684 +0xd3
google.golang.org/grpc.(*clientStream).RecvMsg(0xc000375c00, {0x50d54a0, 0xc0011905f0})
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:825 +0x11f
github.com/openconfig/gnmi/proto/gnmi.(*gNMISubscribeClient).Recv(0xc0008d4e90)
	/go/pkg/mod/github.com/openconfig/[email protected]/proto/gnmi/gnmi.pb.go:2591 +0x4c
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).subscribeGNMI(0xc0003fea80, {0x5e68590, 0xc001069e60}, {0xc0000756e1, 0x12}, 0x0, 0x0)
	/go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:249 +0x565
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start.func1({0xc0000756e1, 0x12})
	/go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:158 +0x125
created by github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start
	/go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:155 +0x9d3

goroutine 81 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc000178d80)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:323 +0x85
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:338
google.golang.org/grpc.(*csAttempt).recvMsg(0xc000372160, {0x50d54a0, 0xc000ffe2d0}, 0x0)
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:961 +0xbb
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x203000)
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:826 +0x25
google.golang.org/grpc.(*clientStream).withRetry(0xc0001fe000, 0xc0005a9d60, 0xc0005a9d30)
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:684 +0xd3
google.golang.org/grpc.(*clientStream).RecvMsg(0xc0001fe000, {0x50d54a0, 0xc000ffe2d0})
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:825 +0x11f
github.com/openconfig/gnmi/proto/gnmi.(*gNMISubscribeClient).Recv(0xc0002d84f0)
	/go/pkg/mod/github.com/openconfig/[email protected]/proto/gnmi/gnmi.pb.go:2591 +0x4c
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).subscribeGNMI(0xc0003fea80, {0x5e68590, 0xc001069e60}, {0xc000075729, 0x12}, 0x0, 0x0)
	/go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:249 +0x565
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start.func1({0xc000075729, 0x12})
	/go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:158 +0x125
created by github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start
	/go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:155 +0x9d3

goroutine 114 [runnable]:
reflect.Value.Set({0x51db260, 0xc00108cc50, 0x196}, {0x51db260, 0xc000acb620, 0x16})
	/usr/local/go/src/reflect/value.go:1911 +0x172
google.golang.org/protobuf/internal/impl.makeMessageFieldCoder.func3({0xc000ad735a, 0x1e, 0x1e}, {0xc000628ab0}, 0x0, 0x655, {0x6c, {0x5df4708, 0xc0001a3b00}})
	/go/pkg/mod/google.golang.org/[email protected]/internal/impl/codec_field.go:218 +0x133
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer(0xc0002d7600, {0xc000ad730e, 0x6a, 0x5093f00}, {0xc00108cc40}, 0x0, {0xb0, {0x5df4708, 0xc0001a3b00}})
	/go/pkg/mod/google.golang.org/[email protected]/internal/impl/decode.go:139 +0x3ad
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshal(0x5093f00, {{}, {0x5f14ce0, 0xc001088d10}, {0xc000ad730e, 0x6a, 0x6a}, 0x0, {0x5df4708, 0xc0001a3b00}})
	/go/pkg/mod/google.golang.org/[email protected]/internal/impl/decode.go:62 +0xb6
google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal({{}, 0x1, 0x1, 0x0, {0x5df4708, 0xc0001a3b00}}, {0xc000ad730e, 0x6a, 0x6a}, {0x5f14ce0, ...})
	/go/pkg/mod/google.golang.org/[email protected]/proto/decode.go:93 +0x1d9
google.golang.org/protobuf/proto.UnmarshalOptions.UnmarshalState(...)
	/go/pkg/mod/google.golang.org/[email protected]/proto/decode.go:66
google.golang.org/protobuf/internal/impl.consumeMessageSlice({0xc000ad730d, 0xc000276300, 0x83ac69}, {0xc000ad7303}, {0x5f2f930, 0x5093f00}, 0x0, {0x0, {0x5df4708, 0xc0001a3b00}})
	/go/pkg/mod/google.golang.org/[email protected]/internal/impl/codec_field.go:558 +0x1ee
google.golang.org/protobuf/internal/impl.makeMessageSliceFieldCoder.func3({0xc000ad730d, 0xc000276300, 0x57e}, {0x66c}, 0x40, 0xc000628990, {0xe0, {0x5df4708, 0xc0001a3b00}})
	/go/pkg/mod/google.golang.org/[email protected]/internal/impl/codec_field.go:460 +0x4a
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer(0xc0002d74a0, {0xc000ad7302, 0x76, 0x4042d2}, {0xc0005e1378}, 0x0, {0x0, {0x5df4708, 0xc0001a3b00}})
	/go/pkg/mod/google.golang.org/[email protected]/internal/impl/decode.go:139 +0x3ad
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshal(0x8b14480, {{}, {0x5f14ce0, 0xc001088d00}, {0xc000ad7302, 0x76, 0x76}, 0x0, {0x5df4708, 0xc0001a3b00}})
	/go/pkg/mod/google.golang.org/[email protected]/internal/impl/decode.go:62 +0xb6
google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal({{}, 0x1, 0x1, 0x0, {0x5df4708, 0xc0001a3b00}}, {0xc000ad7302, 0x76, 0x76}, {0x5f14ce0, ...})
	/go/pkg/mod/google.golang.org/[email protected]/proto/decode.go:93 +0x1d9
google.golang.org/protobuf/proto.UnmarshalOptions.UnmarshalState(...)
	/go/pkg/mod/google.golang.org/[email protected]/proto/decode.go:66
google.golang.org/protobuf/internal/impl.consumeMessage({0xc000ad7301, 0xc00078a598, 0x50d5220}, {0x5db0f40, 0xc001088d00}, 0x34, {0xc8, {0x5df4708, 0xc0001a3b00}})
	/go/pkg/mod/google.golang.org/[email protected]/internal/impl/codec_field.go:283 +0x1c5
google.golang.org/protobuf/internal/impl.makeMessageFieldCoder.func3({0xc000ad7301, 0x77, 0x77}, {0x7f098c40f420}, 0x3e, 0x1c, {0x0, {0x5df4708, 0xc0001a3b00}})
	/go/pkg/mod/google.golang.org/[email protected]/internal/impl/codec_field.go:220 +0x186
google.golang.org/protobuf/internal/impl.(*MessageInfo).initOneofFieldCoders.func1({0xc000ad7301, 0x77, 0x77}, {0x2}, 0x20, 0xc001500008, {0x0, {0x5df4708, 0xc0001a3b00}})
	/go/pkg/mod/google.golang.org/[email protected]/internal/impl/codec_field.go:64 +0x2a5
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer(0xc0001e4420, {0xc000ad7300, 0x78, 0xc0003065f0}, {0xc0005e1860}, 0x0, {0xa0, {0x5df4708, 0xc0001a3b00}})
	/go/pkg/mod/google.golang.org/[email protected]/internal/impl/decode.go:139 +0x3ad
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshal(0xc0003065f0, {{}, {0x5f14ce0, 0xc001088cf0}, {0xc000ad7300, 0x78, 0x78}, 0x0, {0x5df4708, 0xc0001a3b00}})
	/go/pkg/mod/google.golang.org/[email protected]/internal/impl/decode.go:62 +0xb6
google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal({{}, 0x1, 0x1, 0x0, {0x5df4708, 0xc0001a3b00}}, {0xc000ad7300, 0x78, 0x78}, {0x5f14ce0, ...})
	/go/pkg/mod/google.golang.org/[email protected]/proto/decode.go:93 +0x1d9
google.golang.org/protobuf/proto.UnmarshalOptions.UnmarshalState(...)
	/go/pkg/mod/google.golang.org/[email protected]/proto/decode.go:66
github.com/golang/protobuf/proto.UnmarshalMerge({0xc000ad7300, 0x78, 0x78}, {0x5e02878, 0xc0003065f0})
	/go/pkg/mod/github.com/golang/[email protected]/proto/wire.go:67 +0x14c
github.com/golang/protobuf/proto.Unmarshal({0xc000ad7300, 0x78, 0x78}, {0x5e02878, 0xc0003065f0})
	/go/pkg/mod/github.com/golang/[email protected]/proto/wire.go:58 +0x65
google.golang.org/grpc/encoding/proto.codec.Unmarshal({}, {0xc000ad7300, 0x78, 0x78}, {0x50d54a0, 0xc0003065f0})
	/go/pkg/mod/google.golang.org/[email protected]/encoding/proto/proto.go:53 +0x65
google.golang.org/grpc.recv(0xc0004e896c, {0x7f098c68dd98, 0x8d08ae8}, 0x0, {0x0, 0x0}, {0x50d54a0, 0xc0003065f0}, 0x0, 0x0, ...)
	/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:760 +0xb5
google.golang.org/grpc.(*csAttempt).recvMsg(0xc0009b80b0, {0x50d54a0, 0xc0003065f0}, 0x4)
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:975 +0x2b0
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x203000)
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:826 +0x25
google.golang.org/grpc.(*clientStream).withRetry(0xc000332100, 0xc0005e1d60, 0xc0005e1d30)
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:680 +0x2f6
google.golang.org/grpc.(*clientStream).RecvMsg(0xc000332100, {0x50d54a0, 0xc0003065f0})
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:825 +0x11f
github.com/openconfig/gnmi/proto/gnmi.(*gNMISubscribeClient).Recv(0xc000322b20)
	/go/pkg/mod/github.com/openconfig/[email protected]/proto/gnmi/gnmi.pb.go:2591 +0x4c
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).subscribeGNMI(0xc0003fea80, {0x5e68590, 0xc001069e60}, {0xc00118a019, 0x12}, 0x0, 0x0)
	/go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:249 +0x565
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start.func1({0xc00118a019, 0x12})
	/go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:158 +0x125
created by github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start
	/go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:155 +0x9d3

goroutine 115 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runOutputs(0xc000b8e0d0, 0xc000805400)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:773 +0x2b7
github.com/influxdata/telegraf/agent.(*Agent).Run.func1()
	/go/src/github.com/influxdata/telegraf/agent/agent.go:150 +0x5c
created by github.com/influxdata/telegraf/agent.(*Agent).Run
	/go/src/github.com/influxdata/telegraf/agent/agent.go:148 +0x525

goroutine 116 [semacquire]:
sync.runtime_Semacquire(0x0)
	/usr/local/go/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0x0)
	/usr/local/go/src/sync/waitgroup.go:130 +0x71
github.com/influxdata/telegraf/agent.(*Agent).runProcessors(0x0, {0xc000aa8780, 0x5, 0x2000})
	/go/src/github.com/influxdata/telegraf/agent/agent.go:572 +0x125
github.com/influxdata/telegraf/agent.(*Agent).Run.func4()
	/go/src/github.com/influxdata/telegraf/agent/agent.go:171 +0x68
created by github.com/influxdata/telegraf/agent.(*Agent).Run
	/go/src/github.com/influxdata/telegraf/agent/agent.go:169 +0x785

goroutine 117 [semacquire]:
sync.runtime_Semacquire(0x0)
	/usr/local/go/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0x0)
	/usr/local/go/src/sync/waitgroup.go:130 +0x71
github.com/influxdata/telegraf/agent.(*Agent).runInputs(0xc000b8e0d0, {0x5e684e8, 0xc00108c700}, {0x0, 0x0, 0x8cc64c0}, 0xc0008056e0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:330 +0x4a6
github.com/influxdata/telegraf/agent.(*Agent).Run.func5()
	/go/src/github.com/influxdata/telegraf/agent/agent.go:178 +0x74
created by github.com/influxdata/telegraf/agent.(*Agent).Run
	/go/src/github.com/influxdata/telegraf/agent/agent.go:176 +0x88f

goroutine 118 [select]:
github.com/influxdata/telegraf/agent.(*AlignedTicker).run(0xc000fbc900, {0x5e684e8, 0xc000fbc940}, 0xc000ab0f00)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:86 +0xfe
github.com/influxdata/telegraf/agent.(*AlignedTicker).start.func1()
	/go/src/github.com/influxdata/telegraf/agent/tick.go:64 +0x6a
created by github.com/influxdata/telegraf/agent.(*AlignedTicker).start
	/go/src/github.com/influxdata/telegraf/agent/tick.go:62 +0x185

goroutine 119 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherLoop(0x0, {0x5e684e8, 0xc00108c700}, {0x5edcb20, 0xc0004c69e0}, 0xc000ab0550, {0x5df0478, 0xc000fbc900}, 0x0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:465 +0x125
github.com/influxdata/telegraf/agent.(*Agent).runInputs.func1(0x0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:326 +0x7e
created by github.com/influxdata/telegraf/agent.(*Agent).runInputs
	/go/src/github.com/influxdata/telegraf/agent/agent.go:324 +0xa7

goroutine 120 [select]:
github.com/influxdata/telegraf/agent.(*AlignedTicker).run(0xc000fbc9c0, {0x5e684e8, 0xc000fbca00}, 0xc000ab0ff0)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:86 +0xfe
github.com/influxdata/telegraf/agent.(*AlignedTicker).start.func1()
	/go/src/github.com/influxdata/telegraf/agent/tick.go:64 +0x6a
created by github.com/influxdata/telegraf/agent.(*AlignedTicker).start
	/go/src/github.com/influxdata/telegraf/agent/tick.go:62 +0x185

goroutine 121 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherLoop(0x0, {0x5e684e8, 0xc00108c700}, {0x5edcb20, 0xc0004c6a00}, 0xc000ab05f0, {0x5df0478, 0xc000fbc9c0}, 0x0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:465 +0x125
github.com/influxdata/telegraf/agent.(*Agent).runInputs.func1(0x0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:326 +0x7e
created by github.com/influxdata/telegraf/agent.(*Agent).runInputs
	/go/src/github.com/influxdata/telegraf/agent/agent.go:324 +0xa7

goroutine 130 [select]:
github.com/influxdata/telegraf/agent.(*Agent).flushLoop(0xc00078fb90, {0x5e684e8, 0xc000190800}, 0xc000228380, {0x5df04a0, 0xc00078fb90})
	/go/src/github.com/influxdata/telegraf/agent/agent.go:818 +0x1e5
github.com/influxdata/telegraf/agent.(*Agent).runOutputs.func1(0x0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:769 +0x125
created by github.com/influxdata/telegraf/agent.(*Agent).runOutputs
	/go/src/github.com/influxdata/telegraf/agent/agent.go:763 +0xc5

goroutine 146 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc000aaf860)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:561 +0x107
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
	/go/src/github.com/influxdata/telegraf/agent/agent.go:557 +0x45

goroutine 147 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc000aaf8a8)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:561 +0x107
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
	/go/src/github.com/influxdata/telegraf/agent/agent.go:557 +0x45

goroutine 148 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc000aaf8c0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:561 +0x107
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
	/go/src/github.com/influxdata/telegraf/agent/agent.go:557 +0x45

goroutine 149 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc000aaf8d8)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:561 +0x107
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
	/go/src/github.com/influxdata/telegraf/agent/agent.go:557 +0x45

goroutine 150 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc000aaf8f0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:561 +0x107
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
	/go/src/github.com/influxdata/telegraf/agent/agent.go:557 +0x45

goroutine 131 [select]:
github.com/influxdata/telegraf/agent.(*Agent).flushLoop(0xc000b808a0, {0x5e684e8, 0xc000190800}, 0xc000228500, {0x5df04a0, 0xc000b808a0})
	/go/src/github.com/influxdata/telegraf/agent/agent.go:818 +0x1e5
github.com/influxdata/telegraf/agent.(*Agent).runOutputs.func1(0x0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:769 +0x125
created by github.com/influxdata/telegraf/agent.(*Agent).runOutputs
	/go/src/github.com/influxdata/telegraf/agent/agent.go:763 +0xc5

goroutine 132 [select]:
github.com/influxdata/telegraf/agent.(*RollingTicker).run(0xc000b808a0, {0x5e684e8, 0xc000640040}, 0xc0011903c0)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:261 +0xec
github.com/influxdata/telegraf/agent.(*RollingTicker).start.func1()
	/go/src/github.com/influxdata/telegraf/agent/tick.go:249 +0x6a
created by github.com/influxdata/telegraf/agent.(*RollingTicker).start
	/go/src/github.com/influxdata/telegraf/agent/tick.go:247 +0x165

goroutine 151 [select]:
github.com/influxdata/telegraf/agent.(*RollingTicker).run(0xc00078fb90, {0x5e684e8, 0xc00069e000}, 0xc000b880f0)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:261 +0xec
github.com/influxdata/telegraf/agent.(*RollingTicker).start.func1()
	/go/src/github.com/influxdata/telegraf/agent/tick.go:249 +0x6a
created by github.com/influxdata/telegraf/agent.(*RollingTicker).start
	/go/src/github.com/influxdata/telegraf/agent/tick.go:247 +0x165

goroutine 162 [select]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0007960a0)
	/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:77 +0xa7
created by google.golang.org/grpc.newCCBalancerWrapper
	/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:67 +0x246

goroutine 198 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:333 +0x98
created by google.golang.org/grpc.newClientStreamWithParams
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:332 +0xb45

goroutine 161 [IO wait]:
internal/poll.runtime_pollWait(0x7f098c750020, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000338880, 0xc0006ea000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000338880, {0xc0006ea000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000338880, {0xc0006ea000, 0x60100000000, 0x8})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc0006a0030, {0xc0006ea000, 0x43bc20, 0xc00051c1a0})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0000724e0, {0xc00106e580, 0x9, 0xc000643d18})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x5da4680, 0xc0000724e0}, {0xc00106e580, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
golang.org/x/net/http2.readFrameHeader({0xc00106e580, 0x9, 0x18}, {0x5da4680, 0xc0000724e0})
	/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00106e540)
	/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0002db860)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1499 +0x41f
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:364 +0x192f

goroutine 122 [select]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000ab1130)
	/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:77 +0xa7
created by google.golang.org/grpc.newCCBalancerWrapper
	/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:67 +0x246

goroutine 133 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:333 +0x98
created by google.golang.org/grpc.newClientStreamWithParams
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:332 +0xb45

goroutine 194 [runnable]:
bytes.(*Buffer).Write(0xc00077d8f0, {0xc0009e01c0, 0xc4, 0x1a5})
	/usr/local/go/src/bytes/buffer.go:168 +0xff
google.golang.org/grpc/internal/transport.(*http2Client).handleData(0xc000750000, 0xc0009e4060)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1079 +0x352
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000750000)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1534 +0x91c
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:364 +0x192f

goroutine 56 [select]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0003140a0)
	/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:77 +0xa7
created by google.golang.org/grpc.newCCBalancerWrapper
	/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:67 +0x246

goroutine 180 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:333 +0x98
created by google.golang.org/grpc.newClientStreamWithParams
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:332 +0xb45

goroutine 152 [select]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000b88230)
	/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:77 +0xa7
created by google.golang.org/grpc.newCCBalancerWrapper
	/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:67 +0x246

goroutine 159 [IO wait]:
internal/poll.runtime_pollWait(0x7f098c7502d8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000379580, 0xc0006d2000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000379580, {0xc0006d2000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000379580, {0xc0006d2000, 0xc00070f400, 0xc000f144b0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc0006a0028, {0xc0006d2000, 0xc0007b7c78, 0x50e005})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0000723c0, {0xc00106e4a0, 0x9, 0xc0007b7cb8})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x5da4680, 0xc0000723c0}, {0xc00106e4a0, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
golang.org/x/net/http2.readFrameHeader({0xc00106e4a0, 0x9, 0xc000fb44e0}, {0x5da4680, 0xc0000723c0})
	/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00106e460)
	/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0002db680)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1499 +0x41f
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:364 +0x192f

goroutine 210 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000b88640, 0x1)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x11b
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000072540)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:414 +0x65
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:412 +0x1f85

goroutine 199 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:333 +0x98
created by google.golang.org/grpc.newClientStreamWithParams
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:332 +0xb45

goroutine 196 [IO wait]:
internal/poll.runtime_pollWait(0x7f098c750930, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000379800, 0xc000756000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000379800, {0xc000756000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000379800, {0xc000756000, 0x60100000000, 0x8})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000736008, {0xc000756000, 0x43bc20, 0xc0007521a0})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00025b2c0, {0xc0011a8200, 0x9, 0xc000646d18})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x5da4680, 0xc00025b2c0}, {0xc0011a8200, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
golang.org/x/net/http2.readFrameHeader({0xc0011a8200, 0x9, 0x18}, {0x5da4680, 0xc00025b2c0})
	/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0011a81c0)
	/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0007501e0)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1499 +0x41f
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:364 +0x192f

goroutine 160 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000b88550, 0x1)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x11b
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000072420)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:414 +0x65
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:412 +0x1f85

goroutine 195 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000ffe050, 0x1)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x11b
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00025afc0)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:414 +0x65
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:412 +0x1f85

goroutine 197 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000ffe140, 0x1)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x11b
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00025b320)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:414 +0x65
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:412 +0x1f85

System info

Telegraf 1.22.1, Debian Bullseye, Docker 20.10.14

Docker

Docker container is provisioned with ansible, only 'special' setting is memory_limit 512MB

Steps to reproduce

-) run Telegraf docker container with dynamic tagging configuration and more than 4 subscribed addresses.

When there's only one CPU available (1 socket, 1 core) everything work's as expected.

As soon as ther's more than one CPU available (eg. 1 socket, 2 cores) a fatal error arises:
fatal error: concurrent map read and map write

Expected behavior

working telegraf container

Actual behavior

fatal error: concurrent map read and map write

Additional info

No response

The text was updated successfully, but these errors were encountered:

powersj · 2022-04-20T16:06:39Z

Hi,

fatal error: concurrent map read and map write

This is probably from this recent addition of gnmi subscriptions. Telegraf may need to either sync the lookup map or put some mutexes in place.

When there's only one CPU available (1 socket, 1 core) everything work's as expected.
As soon as ther's more than one CPU available (eg. 1 socket, 2 cores) a fatal error arises:

Out of curiosity, how are you controlling this? You said the only special setting is the memory_limit.

kb70 · 2022-04-20T16:09:29Z

Hi,

fatal error: concurrent map read and map write

This is probably from this recent addition of gnmi subscriptions. Telegraf may need to either sync the lookup map or put some mutexes in place.

When there's only one CPU available (1 socket, 1 core) everything work's as expected.
As soon as ther's more than one CPU available (eg. 1 socket, 2 cores) a fatal error arises:

Out of curiosity, how are you controlling this? You said the only special setting is the memory_limit.

In a test environment, where the docker container runs on a KVM virtualized host, where i can arbitrarily configure the number of sockets and cores.

powersj · 2022-04-20T16:15:06Z

@bewing as you are the author FYI - curious if you have seen this or if you are interested in looking to resolve this.

bewing · 2022-04-20T16:39:08Z

Thanks for the heads up -- My primary application was running telegraf directly on the device we were monitoring with gNMI, so I never ran into this.

I believe I just have to add a sync.Mutex to update or read from gnmi.GNMI.lookup ?

bewing · 2022-04-20T17:50:17Z

@kb70 are you easily able to checkout the PR and see if the change resolves this issue?

Or download one of the build artifacts? #11008 (comment)

kb70 · 2022-04-21T06:27:35Z

@bewing Thanks for the quick fix.
I tested the PR (via PR build artifacts and 'quick & dirty' approach, ie. replacing the telegraf binary in the docker container(Telegraf 1.23.0-dfd54ceb)) and as far as I can tell it works now.
Independent of the number of sockets or cores telegraf starts up without errors.

Thanks again for your work on this very valuable feature.

PS.: I'll test it on one of our production monitoring machines, ie. real hardware, later on, but won't be able to report about the outcome before tomorrow afternoon.

powersj · 2022-04-21T13:19:56Z

Thank you both for the fix and for testing this! I want to get this in our bugfix release Monday, so please do let us know the outcome, but I may merge anyway before then.

kb70 · 2022-04-22T10:24:21Z

The fixed version runs without problems since about a day now on the virtualized testsystem as well as on bare-metal, both with multiple socket CPUs and cores.

Again thanks for the quick response and fix.

powersj · 2022-04-22T13:39:52Z

@kb70 thank you for testing!

kb70 added the bug unexpected problem or unintended behavior label Apr 20, 2022

telegraf-tiger bot added the area/clickhouse label Apr 20, 2022

powersj added help wanted Request for community participation, code, contribution area/gnmi and removed area/clickhouse labels Apr 20, 2022

bewing added a commit to bewing/telegraf that referenced this issue Apr 20, 2022

fix: add mutex to gnmi lookup map (influxdata#11006)

63b278f

bewing mentioned this issue Apr 20, 2022

fix: add mutex to gnmi lookup map (#11006) #11008

Merged

3 tasks

bewing added a commit to bewing/telegraf that referenced this issue Apr 20, 2022

fix: add mutex to gnmi lookup map (influxdata#11006)

dfd54ce

powersj closed this as completed in #11008 Apr 22, 2022

madd2008 mentioned this issue Sep 8, 2022

inputs.gnmi with dynamic tagging won't work #11778

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

inputs.gnmi with dynamic tagging won't work with more than one CPUs #11006

inputs.gnmi with dynamic tagging won't work with more than one CPUs #11006

kb70 commented Apr 20, 2022

powersj commented Apr 20, 2022

kb70 commented Apr 20, 2022

powersj commented Apr 20, 2022

bewing commented Apr 20, 2022 •

edited

Loading

bewing commented Apr 20, 2022 •

edited

Loading

kb70 commented Apr 21, 2022

powersj commented Apr 21, 2022

kb70 commented Apr 22, 2022

powersj commented Apr 22, 2022

inputs.gnmi with dynamic tagging won't work with more than one CPUs #11006

inputs.gnmi with dynamic tagging won't work with more than one CPUs #11006

Comments

kb70 commented Apr 20, 2022

Relevant telegraf.conf

Logs from Telegraf

System info

Docker

Steps to reproduce

Expected behavior

Actual behavior

Additional info

powersj commented Apr 20, 2022

kb70 commented Apr 20, 2022

powersj commented Apr 20, 2022

bewing commented Apr 20, 2022 • edited Loading

bewing commented Apr 20, 2022 • edited Loading

kb70 commented Apr 21, 2022

powersj commented Apr 21, 2022

kb70 commented Apr 22, 2022

powersj commented Apr 22, 2022

bewing commented Apr 20, 2022 •

edited

Loading

bewing commented Apr 20, 2022 •

edited

Loading