Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fatal error: concurrent map iteration and map write #6418

Closed
bonedaddy opened this issue Jun 6, 2019 · 4 comments · Fixed by #6419
Closed

fatal error: concurrent map iteration and map write #6418

bonedaddy opened this issue Jun 6, 2019 · 4 comments · Fixed by #6419
Labels
kind/bug A bug in existing code (including security flaws)

Comments

@bonedaddy
Copy link
Contributor

Version information:

go-ipfs version: 0.4.21-
Repo version: 7
System version: amd64/linux
Golang version: go1.12.5

Description:

Regular operation of an IPFS node crashed due to what would appear to be a data race?

Output from the logs can be viewed on IPFS via Qmb39D6U7YWx49kNSTVFb8iXpZ5zAT8qfdiF2bRraEo9eC
Stebalien added a commit that referenced this issue Jun 6, 2019
@Stebalien
pin: fix concurrent map access race
486f013 
Not sure why this didn't show up sooner.

fixes #6418
@Stebalien Stebalien mentioned this issue Jun 6, 2019
Merged
@Stebalien Stebalien added the kind/bug A bug in existing code (including security flaws) label Jun 6, 2019
@bonedaddy bonedaddy mentioned this issue Jun 23, 2019
Closed
@hareeshnagaraj
Copy link

hareeshnagaraj commented Nov 21, 2019 •
edited
Loading

@Stebalien - seeing this on several nodes on 0.4.22, with the following stack trace:

^[[0;37m01:22:26.593 ^[[36mDEBUG ^[[0;34m    swarm2: ^[[0m[limiter] freeing FD token; waiting: 0; consuming: 120 ^[[0;37mlimiter.go:81^[[0m
fatal error: ^[[0;37m01:22:26.593 ^[[36mDEBUG ^[[0;34m    swarm2: ^[[0m[limiter] freeing peer token; peer QmU75VHxfkbuC9UomGphXWwUgFXmFDUVAcT9f9aBfjTdPv; addr: /ip4/40.122.130.171/tcp/15296; active for peer: 5; waiting on peer limit: 0 ^[[0;37mlimiter.go:108^[[0m
concurrent map iteration and map write

goroutine 1246165 [running]:
runtime.throw(0x17eb015, 0x26)
        /usr/lib/go/src/runtime/panic.go:617 +0x72 fp=0xc001f167b8 sp=0xc001f16788 pc=0x99d052
runtime.mapiternext(0xc012ef1aa0)
        /usr/lib/go/src/runtime/map.go:860 +0x597 fp=0xc001f16840 sp=0xc001f167b8 pc=0x97e707
reflect.mapiternext(0xc012ef1aa0)
        /usr/lib/go/src/runtime/map.go:1349 +0x2b fp=0xc001f16858 sp=0xc001f16840 pc=0x97f75b
reflect.(*MapIter).Next(0xc001f168e8, 0x1615560)
        /usr/lib/go/src/reflect/value.go:1200 +0xe0 fp=0xc001f16880 sp=0xc001f16858 pc=0xa07490
internal/fmtsort.Sort(0x160c080, 0xc0000d5760, 0x1b5, 0x1b2)
        /usr/lib/go/src/internal/fmtsort/sort.go:59 +0x17a fp=0xc001f16918 sp=0xc001f16880 pc=0xa24bea
fmt.(*pp).printValue(0xc010ea2a80, 0x160c080, 0xc0000d5760, 0x1b5, 0x76, 0x2)
        /usr/lib/go/src/fmt/print.go:759 +0xd20 fp=0xc001f16af0 sp=0xc001f16918 pc=0xa5c730
fmt.(*pp).printValue(0xc010ea2a80, 0x17368c0, 0xc0000d5740, 0x1b9, 0xc000000076, 0x1)
        /usr/lib/go/src/fmt/print.go:796 +0x1b52 fp=0xc001f16cc8 sp=0xc001f16af0 pc=0xa5d562
fmt.(*pp).printValue(0xc010ea2a80, 0x16d0ec0, 0xc0000d5740, 0x36, 0x76, 0x0)
        /usr/lib/go/src/fmt/print.go:866 +0x1956 fp=0xc001f16ea0 sp=0xc001f16cc8 pc=0xa5d366
fmt.(*pp).badVerb(0xc010ea2a80, 0xc000000073)
        /usr/lib/go/src/fmt/print.go:344 +0x4fb fp=0xc001f16f40 sp=0xc001f16ea0 pc=0xa58acb
fmt.(*pp).fmtPointer(0xc010ea2a80, 0x16d0ec0, 0xc0000d5740, 0x36, 0xc000000073)
        /usr/lib/go/src/fmt/print.go:537 +0xc7 fp=0xc001f16fc8 sp=0xc001f16f40 pc=0xa59f87
fmt.(*pp).printValue(0xc010ea2a80, 0x16d0ec0, 0xc0000d5740, 0x36, 0x73, 0x2)
        /usr/lib/go/src/fmt/print.go:872 +0xc9c fp=0xc001f171a0 sp=0xc001f16fc8 pc=0xa5c6ac
fmt.(*pp).printValue(0xc010ea2a80, 0x1778c60, 0xc00019e540, 0x199, 0xc000000073, 0x1)
        /usr/lib/go/src/fmt/print.go:796 +0x1b52 fp=0xc001f17378 sp=0xc001f171a0 pc=0xa5d562
fmt.(*pp).printValue(0xc010ea2a80, 0x16d3080, 0xc00019e540, 0x16, 0xc000000073, 0x0)
        /usr/lib/go/src/fmt/print.go:866 +0x1956 fp=0xc001f17550 sp=0xc001f17378 pc=0xa5d366
fmt.(*pp).printArg(0xc010ea2a80, 0x16d3080, 0xc00019e540, 0x73)
        /usr/lib/go/src/fmt/print.go:702 +0x2ba fp=0xc001f175e8 sp=0xc001f17550 pc=0xa5b29a
fmt.(*pp).doPrintf(0xc010ea2a80, 0x17e7f20, 0x24, 0xc001f17810, 0x5, 0x5)
        /usr/lib/go/src/fmt/print.go:1016 +0x158 fp=0xc001f176d0 sp=0xc001f175e8 pc=0xa5ebe8
fmt.Sprintf(0x17e7f20, 0x24, 0xc001f17810, 0x5, 0x5, 0x2e, 0xc00ddc3798)
        /usr/lib/go/src/fmt/print.go:214 +0x66 fp=0xc001f17728 sp=0xc001f176d0 pc=0xa57bc6
github.com/libp2p/go-libp2p-swarm.(*Conn).String(0xc013f93b60, 0x19438d0, 0xc014b463c0)
        pkg/mod/github.com/libp2p/[email protected]/swarm_conn.go:123 +0x318 fp=0xc001f17870 sp=0xc001f17728 pc=0xf3afc8
fmt.(*pp).handleMethods(0xc014b463c0, 0xc000000073, 0x17c8601)
        /usr/lib/go/src/fmt/print.go:616 +0x29b fp=0xc001f17910 sp=0xc001f17870 pc=0xa5ad6b
fmt.(*pp).printArg(0xc014b463c0, 0x1761460, 0xc013f93b60, 0x73)
        /usr/lib/go/src/fmt/print.go:699 +0x20a fp=0xc001f179a8 sp=0xc001f17910 pc=0xa5b1ea
fmt.(*pp).doPrintf(0xc014b463c0, 0x17e1f7f, 0x20, 0xc00c8eb3a0, 0x1, 0x1)
        /usr/lib/go/src/fmt/print.go:1016 +0x158 fp=0xc001f17a90 sp=0xc001f179a8 pc=0xa5ebe8
fmt.Sprintf(0x17e1f7f, 0x20, 0xc00c8eb3a0, 0x1, 0x1, 0xc00ddc3b00, 0x9e1a61)
        /usr/lib/go/src/fmt/print.go:214 +0x66 fp=0xc001f17ae8 sp=0xc001f17a90 pc=0xa57bc6
github.com/whyrusleeping/go-logging.(*Record).Message(0xc003d56c60, 0xc00bdbba00, 0x4)
        pkg/mod/github.com/whyrusleeping/[email protected]/logger.go:78 +0xf5 fp=0xc001f17b50 sp=0xc001f17ae8 pc=0xd14fa5
github.com/whyrusleeping/go-logging.(*stringFormatter).Format(0xc00013e180, 0x5, 0xc003d56c60, 0x1a99780, 0xc015533710, 0xc003d56c60, 0x5ea)
        pkg/mod/github.com/whyrusleeping/[email protected]/format.go:294 +0x3cd fp=0xc001f17c28 sp=0xc001f17b50 pc=0xd1376d
github.com/whyrusleeping/go-logging.(*Record).Formatted(0xc003d56c60, 0x4, 0x3, 0xc00004e570)
        pkg/mod/github.com/whyrusleeping/[email protected]/logger.go:62 +0xa3 fp=0xc001f17c78 sp=0xc001f17c28 pc=0xd14e03
github.com/whyrusleeping/go-logging.(*LogBackend).Log(0xc00013a290, 0x5, 0x3, 0xc003d56c60, 0x23598f01, 0xc003d56c60)
        pkg/mod/github.com/whyrusleeping/[email protected]/log.go:52 +0x279 fp=0xc001f17d50 sp=0xc001f17c78 pc=0xd14489
github.com/whyrusleeping/go-logging.(*moduleLeveled).Log(0xc00015a140, 0x5, 0x2, 0xc003d56c60, 0x1, 0x10)
        pkg/mod/github.com/whyrusleeping/[email protected]/level.go:117 +0xcc fp=0xc001f17d90 sp=0xc001f17d50 pc=0xd1415c
github.com/whyrusleeping/go-logging.(*Logger).log(0xc0001a1bf0, 0x5, 0x17e1f7f, 0x20, 0xc00c8eb3a0, 0x1, 0x1)
        pkg/mod/github.com/whyrusleeping/[email protected]/logger.go:179 +0x1ca fp=0xc001f17df0 sp=0xc001f17d90 pc=0xd1554a
github.com/whyrusleeping/go-logging.(*Logger).Debugf(0xc0001a1bf0, 0x17e1f7f, 0x20, 0xc00c8eb3a0, 0x1, 0x1)
        pkg/mod/github.com/whyrusleeping/[email protected]/logger.go:266 +0x66 fp=0xc001f17e38 sp=0xc001f17df0 pc=0xd16166
github.com/ipfs/go-log.(*eventLogger).Debugf(0xc0001a9380, 0x17e1f7f, 0x20, 0xc00c8eb3a0, 0x1, 0x1)
        <autogenerated>:1 +0x6e fp=0xc001f17e78 sp=0xc001f17e38 pc=0xd99d0e
github.com/libp2p/go-libp2p/p2p/protocol/identify.(*IDService).IdentifyConn(0xc00015ba40, 0x1ad7c40, 0xc013f93b60)
        pkg/mod/github.com/libp2p/[email protected]/p2p/protocol/identify/id.go:92 +0x157 fp=0xc001f17f48 sp=0xc001f17e78 pc=0xf01587
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).newConnHandler(0xc0001c3ad0, 0x1ad7c40, 0xc013f93b60)
        pkg/mod/github.com/libp2p/[email protected]/p2p/host/basic/basic_host.go:228 +0xb2 fp=0xc001f17fa8 sp=0xc001f17f48 pc=0xf0b4c2
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).newConnHandler-fm(0x1ad7c40, 0xc013f93b60)
        pkg/mod/github.com/libp2p/[email protected]/p2p/host/basic/basic_host.go:224 +0x3e fp=0xc001f17fd0 sp=0xc001f17fa8 pc=0xf115fe
runtime.goexit()
        /usr/lib/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc001f17fd8 sp=0xc001f17fd0 pc=0x9cd641
created by github.com/libp2p/go-libp2p-swarm.(*Swarm).addConn
        pkg/mod/github.com/libp2p/[email protected]/swarm.go:231 +0x384

is there anything we can do to prevent encountering this scenario?

@Stebalien
Copy link
Member

This is a different issue that was fixed in libp2p/go-libp2p-swarm#134. This change was not included in go-ipfs 0.4.22, you'll have to either cherry-pick this change, use go-ipfs master, or wait for go-ipfs 0.5.0 (which should be released by the new year).

Stebalien added a commit that referenced this issue Jan 28, 2020
@Stebalien
chore(dep): update go-libp2p-swarm
29400bc 
Fixes a concurrent map access issue: #6418. The fix for this somehow didn't make
it into 0.4.22 either.
@Stebalien Stebalien mentioned this issue Jan 28, 2020
Closed
51 tasks
ralendor pushed a commit to ralendor/go-ipfs that referenced this issue Jun 6, 2020
@Stebalien
chore(dep): update go-libp2p-swarm
1e91cf0 
Fixes a concurrent map access issue: ipfs#6418. The fix for this somehow didn't make
it into 0.4.22 either.
ralendor pushed a commit to ralendor/go-ipfs that referenced this issue Jun 8, 2020
@Stebalien
chore(dep): update go-libp2p-swarm
eeb7d27 
Fixes a concurrent map access issue: ipfs#6418. The fix for this somehow didn't make
it into 0.4.22 either.
ralendor pushed a commit to ralendor/go-ipfs that referenced this issue Jun 8, 2020
@Stebalien
chore(dep): update go-libp2p-swarm
3bb7655 
Fixes a concurrent map access issue: ipfs#6418. The fix for this somehow didn't make
it into 0.4.22 either.
@0xsarvesh
Copy link

Hey,
I see this is close. But I get this error.

Version details

Dec 06 16:54:35 ip-172-31-3-129 ipfs[485]: Kubo version: 0.16.0
Dec 06 16:54:35 ip-172-31-3-129 ipfs[485]: Repo version: 12
Dec 06 16:54:35 ip-172-31-3-129 ipfs[485]: System version: amd64/linux
Dec 06 16:54:35 ip-172-31-3-129 ipfs[485]: Golang version: go1.19.1

Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]: fatal error: concurrent map iteration and map write
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]: goroutine 186 [running]:
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]: github.com/libp2p/go-libp2p/p2p/net/connmgr.peerInfos.SortByValueAndStreams.func1.1(0xc02a8a38c0?)
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]:         github.com/libp2p/[email protected]/p2p/net/connmgr/connmgr.go:267 +0x6a
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]: github.com/libp2p/go-libp2p/p2p/net/connmgr.peerInfos.SortByValueAndStreams.func1(0x156?, 0x25d?)
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]:         github.com/libp2p/[email protected]/p2p/net/connmgr/connmgr.go:277 +0x127
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]: sort.partition_func({0xc02b33b370?, 0xc016cf8570?}, 0x0, 0x2ad, 0x32?)
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]:         sort/zsortfunc.go:157 +0x191
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]: sort.pdqsort_func({0xc02b33b370?, 0xc016cf8570?}, 0xc00c47e0a8?, 0x0?, 0x14a50?)
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]:         sort/zsortfunc.go:114 +0x254
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]: sort.Slice({0x22ec480, 0xc00c47e0a8}, 0x421?)
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]:         sort/slice.go:23 +0x97
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]: github.com/libp2p/go-libp2p/p2p/net/connmgr.peerInfos.SortByValueAndStreams({0xc031acc000, 0x2ad, 0x421}, 0x0)
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]:         github.com/libp2p/[email protected]/p2p/net/connmgr/connmgr.go:256 +0x8a
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]: github.com/libp2p/go-libp2p/p2p/net/connmgr.(*BasicConnMgr).getConnsToClose(0xc0011ba000)
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]:         github.com/libp2p/[email protected]/p2p/net/connmgr/connmgr.go:468 +0x365
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]: github.com/libp2p/go-libp2p/p2p/net/connmgr.(*BasicConnMgr).trim(0xc02b33bf88?)
Dec 07 01:55:06 ip-172-31-3-129 ipfs[485]:         github.com/libp2p/[email protected]/p2p/net/connmgr/connmgr.go:353 +0x2b

@Jorropo
Copy link
Contributor

Jorropo commented Dec 7, 2022

@0xsarvesh this is a known issue, please update to 0.17

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug A bug in existing code (including security flaws)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

pin: fix concurrent map access race ipfs/kubo
5 participants
@Stebalien @hareeshnagaraj @bonedaddy @Jorropo @0xsarvesh