Segmentation fault with GOMAXPROCS >1

[rshriram]

In order to handle higher loads with just AF_PACKET, I tried to run packetbeat with GOMAXPROCS > 1. With runtime.GOMAXPROCS(2) or higher, it results in a segmentation fault. It seems some part of the code is not threadsafe.

Error output:

fatal error: bad map state

goroutine 10 [running]:
runtime.gothrow(0x8fb1d0, 0xd)
    /usr/lib/go/src/runtime/panic.go:503 +0x8e fp=0xc208039388 sp=0xc208039370
runtime.evacuate(0x775280, 0xc2090d5ef0, 0xa5e)
    /usr/lib/go/src/runtime/hashmap.go:799 +0x36a fp=0xc208039448 sp=0xc208039388
runtime.growWork(0x775280, 0xc2090d5ef0, 0x578)
    /usr/lib/go/src/runtime/hashmap.go:769 +0x8a fp=0xc208039468 sp=0xc208039448
runtime.mapassign1(0x775280, 0xc2090d5ef0, 0xc2080395b0, 0xc2080395e8)
    /usr/lib/go/src/runtime/hashmap.go:408 +0x18f fp=0xc208039508 sp=0xc208039468
github.com/elastic/packetbeat/protos/http.(*Http).receivedHttpRequest(0xc208010a80, 0xc22d094960)
    /home/shriram/go/src/github.com/elastic/packetbeat/protos/http/http.go:741 +0x13a1 fp=0xc2080397b8 sp=0xc208039508
github.com/elastic/packetbeat/protos/http.(*Http).handleHttp(0xc208010a80, 0xc22d094960, 0xc22519c390, 0x0, 0xc20b67efb2, 0x83, 0x83)
    /home/shriram/go/src/github.com/elastic/packetbeat/protos/http/http.go:725 +0x14f fp=0xc208039830 sp=0xc2080397b8
github.com/elastic/packetbeat/protos/http.(*Http).messageComplete(0xc208010a80, 0xc22519c390, 0xecd670100, 0xc225c34500)
    /home/shriram/go/src/github.com/elastic/packetbeat/protos/http/http.go:585 +0xf8 fp=0xc2080398a8 sp=0xc208039830
github.com/elastic/packetbeat/protos/http.(*Http).Parse(0xc208010a80, 0xc228cb7200, 0xc22519c390, 0xc208010a00, 0x0, 0x0, 0x0, 0x0)
    /home/shriram/go/src/github.com/elastic/packetbeat/protos/http/http.go:638 +0x500 fp=0xc2080399a8 sp=0xc2080398a8
github.com/elastic/packetbeat/protos/tcp.(*TcpStream).AddPacket(0xc22519c370, 0xc228cb7200, 0xc209105bf8, 0x0)
    /home/shriram/go/src/github.com/elastic/packetbeat/protos/tcp/tcp.go:83 +0x2e1 fp=0xc208039a68 sp=0xc2080399a8
github.com/elastic/packetbeat/protos/tcp.(*Tcp).Process(0xc209238000, 0xc209105bf8, 0xc228cb7200)
    /home/shriram/go/src/github.com/elastic/packetbeat/protos/tcp/tcp.go:174 +0xc76 fp=0xc208039ce0 sp=0xc208039a68
github.com/elastic/packetbeat/decoder.(*DecoderStruct).DecodePacketData(0xc209105900, 0xc20b67ef70, 0xc5, 0xc5, 0xc21c1bbef0)
    /home/shriram/go/src/github.com/elastic/packetbeat/decoder/decoder.go:136 +0x44a fp=0xc208039dd0 sp=0xc208039ce0
github.com/elastic/packetbeat/sniffer.(*SnifferSetup).Run(0xc208064730, 0x0, 0x0)
    /home/shriram/go/src/github.com/elastic/packetbeat/sniffer/sniffer.go:322 +0xc4a fp=0xc208039f58 sp=0xc208039dd0
main.func·002()
    /home/shriram/go/src/github.com/elastic/packetbeat/main.go:166 +0x36 fp=0xc208039fe0 sp=0xc208039f58
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208039fe8 sp=0xc208039fe0
created by main.main
    /home/shriram/go/src/github.com/elastic/packetbeat/main.go:172 +0x12ff

goroutine 1 [chan receive]:
main.main()
    /home/shriram/go/src/github.com/elastic/packetbeat/main.go:183 +0x139d

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 5 [syscall]:
os/signal.loop()
    /usr/lib/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
    /usr/lib/go/src/os/signal/signal_unix.go:27 +0x35

goroutine 7 [select]:
github.com/elastic/libbeat/outputs/redis.(*RedisOutput).SendMessagesGoroutine(0xc2080e0280)
    /home/shriram/go/src/github.com/elastic/packetbeat/Godeps/_workspace/src/github.com/elastic/libbeat/outputs/redis/redis.go:184 +0x98d
created by github.com/elastic/libbeat/outputs/redis.(*RedisOutput).Init
    /home/shriram/go/src/github.com/elastic/packetbeat/Godeps/_workspace/src/github.com/elastic/libbeat/outputs/redis/redis.go:123 +0xc75

goroutine 8 [chan receive]:
github.com/elastic/libbeat/publisher.(*PublisherType).publishFromQueue(0xd82dc0)
    /home/shriram/go/src/github.com/elastic/packetbeat/Godeps/_workspace/src/github.com/elastic/libbeat/publisher/publish.go:92 +0x5e
created by github.com/elastic/libbeat/publisher.(*PublisherType).Init
    /home/shriram/go/src/github.com/elastic/packetbeat/Godeps/_workspace/src/github.com/elastic/libbeat/publisher/publish.go:299 +0xf07

goroutine 9 [chan receive]:
github.com/elastic/packetbeat/protos/thrift.(*Thrift).publishTransactions(0xc208062180)
    /home/shriram/go/src/github.com/elastic/packetbeat/protos/thrift/thrift.go:1078 +0x63
created by github.com/elastic/packetbeat/protos/thrift.(*Thrift).Init
    /home/shriram/go/src/github.com/elastic/packetbeat/protos/thrift/thrift.go:256 +0x18f

goroutine 11 [chan receive]:
github.com/elastic/libbeat/service.func·001()
    /home/shriram/go/src/github.com/elastic/packetbeat/Godeps/_workspace/src/github.com/elastic/libbeat/service/service.go:23 +0x48
created by github.com/elastic/libbeat/service.HandleSignals
    /home/shriram/go/src/github.com/elastic/packetbeat/Godeps/_workspace/src/github.com/elastic/libbeat/service/service.go:26 +0x253

goroutine 1227 [runnable]:
github.com/elastic/packetbeat/protos/http.(*Http).expireTransaction(0xc208010a80, 0xc230791a00)
    /home/shriram/go/src/github.com/elastic/packetbeat/protos/http/http.go:809 +0x8c
github.com/elastic/packetbeat/protos/http.func·001()
    /home/shriram/go/src/github.com/elastic/packetbeat/protos/http/http.go:803 +0x36
created by time.goFunc
    /usr/lib/go/src/time/sleep.go:129 +0x4b

[tsg]

Thanks for the report, we know there are several places in which the code is not thread safe. We'll start patching them, I suspect this access to the transaction map is the biggest issue.

[urso]

Duplicate for more general ticket #204

[urso]

#203 I mean