Concurrent map read/writes while running beacon-chain

[AlexSSD7]

🐞 Bug Report

Description

Runtime panic caused by concurrent map read/writes. This issue can be resolved by using mutex.

Has this worked before in a previous version?

Can't gather information. It is pretty rare issue.

🔬 Minimal Reproduction

I was just very lucky to have two goroutines reading/writing values from single map. Would be very hard to reproduce.

🔥 Error

[2020-04-22 08:48:30]  INFO blockchain: Finished applying state transition attestations=19 attesterSlashings=0 deposits=0 proposerSlashings=0 slot=31440 voluntaryExits=0
[2020-04-22 08:48:30]  INFO sync: Processed pending block and cleared it in cache blockRoot=8149d9eb3741 slot=31440
[2020-04-22 08:48:30]  INFO blockchain: Executing state transition on block root=0x9ca85155... slot=31441
[2020-04-22 08:48:30]  INFO blockchain: Finished applying state transition attestations=10 attesterSlashings=0 deposits=0 proposerSlashings=0 slot=31441 voluntaryExits=0
[2020-04-22 08:48:30]  INFO sync: Processed pending block and cleared it in cache blockRoot=9ca85155ebee slot=31441
[2020-04-22 08:48:33]  INFO sync: Verified and saved pending attestations to pool blockRoot=9ca85155ebee pendingAttsCount=4
fatal error: concurrent map read and map write

// Panic (https://pastebin.com/ffJQzRrL)

🌍 Your Environment

Operating System:

  
        Ubuntu 18.04 Server (5.3.0-46-generic)
  

What version of Prysm are you running? (Which release)

  
        beacon-chain version Prysm/Git commit: 1224e75ba1af59d5ed074db766d0bd4e9b77ea64. Built at: 2020-04-18 10:50:26+00:00 (Docker image)
  

Anything else relevant (validator index / public key)?

Not neccesary.

[AlexSSD7]

Duplicate of #5513, still not fixed.

[nisdas]

Hey @AlexSSD7 , from your report the commit hash is 4 days old so it is an old image. This was fixed here, #5554, which was only merged in yesterday. Could you pull the latest and try again ?

[AlexSSD7]

Oh, apparently I didn't notice it the date. Anyway I've upgraded the prysm version on my server. Left it running for couple days, will see if it is fixed.

[terencechain]

Thanks @AlexSSD7 , please give us an update

[prestonvanloon]

There have been two releases since 1224e75. This has almost certainly been fixed. I'll close but please reopen if you have continued issues.

[stefa2k]

I might have the same issue, running docker tag HEAD-1dfeb6 with this setup:

• https://github.com/stefa2k/prysm-docker-compose/tree/ha
• commit 517c193ebeb8bcd1d771b572a8cc870ea36ff432

and log (Kibana export):
panic.LOG
2nd occasion:
panic2.LOG

I got "fatal error: concurrent map read and map write" 17 times in my logs after I started using haproxy as reverse proxy between the components beacon-chain, validator and geth.

[prestonvanloon]

Thanks @stefa2k. The problem in your log is this bit of code:

prysm/beacon-chain/rpc/service.go

Lines 403 to 408 in 78465e2

	if !s.connectedRPCClients[clientInfo.Addr] {
	log.WithFields(logrus.Fields{
	"addr": clientInfo.Addr.String(),
	}).Infof("New gRPC client connected to beacon node")
	s.connectedRPCClients[clientInfo.Addr] = true
	}

connectedRPCClients map is accessed without a lock. I'll file a new issue for this, rather than resurrect this old one.

[prestonvanloon]

New issue filed #6434