Cdn restart with fatal error

[likunbyl]

Bug report:

When some images being pulled, one of cdn instances restart, with the following errors:

fatal error: concurrent map iteration and map write

goroutine 56237 [running]:
runtime.throw({0xd2b9d5, 0x3})
        /usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc0005d31d0 sp=0xc0005d31a0 pc=0x435871
runtime.mapiternext(0xbfd7e0)
        /usr/local/go/src/runtime/map.go:858 +0x4eb fp=0xc0005d3240 sp=0xc0005d31d0 pc=0x40f84b
d7y.io/dragonfly/v2/cdn/supervisor.(*cdnService).GetSeedPieces(0x0, {0xc00032c1c0, 0x0})
        /go/src/d7y.io/dragonfly/v2/cdn/supervisor/service.go:132 +0xe5 fp=0xc0005d3340 sp=0xc0005d3240 pc=0xa38405
d7y.io/dragonfly/v2/cdn/rpcserver.(*Server).GetPieceTasks(0xc00001c400, {0xef8878, 0xc0004c7830}, 0xc000c4f800)
        /go/src/d7y.io/dragonfly/v2/cdn/rpcserver/rpcserver.go:199 +0x7bc fp=0xc0005d3700 sp=0xc0005d3340 pc=0xa3d47c
d7y.io/dragonfly/v2/pkg/rpc/cdnsystem/server.(*proxy).GetPieceTasks(0xc00032ae10, {0xef8878, 0xc0004c7830}, 0xca6400)
        /go/src/d7y.io/dragonfly/v2/pkg/rpc/cdnsystem/server/server.go:91 +0x2b fp=0xc0005d3730 sp=0xc0005d3700 pc=0xa2c10b
d7y.io/dragonfly/v2/pkg/rpc/cdnsystem._Seeder_GetPieceTasks_Handler.func1({0xef8878, 0xc0004c7830}, {0xca6400, 0xc000c4f800})
        /go/src/d7y.io/dragonfly/v2/pkg/rpc/cdnsystem/cdnsystem_grpc.pb.go:145 +0x78 fp=0xc0005d3770 sp=0xc0005d3730 pc=0xa291d8
github.com/grpc-ecosystem/go-grpc-middleware/validator.UnaryServerInterceptor.func1({0xef8878, 0xc0004c7830}, {0xca6400, 0xc000c4f800}, 0x0, 0xc00000de60)
        /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/validator/validator.go:47 +0x6f fp=0xc0005d37a0 sp=0xc0005d3770 pc=0xa23d8f
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0xef8878, 0xc0004c7830}, {0xca6400, 0xc000c4f800})
        /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a fp=0xc0005d37e0 sp=0xc0005d37a0 pc=0xa1fbfa
github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.UnaryServerInterceptor.func1({0xef8878, 0xc0004c7770}, {0xca6400, 0xc000c4f800}, 0xc00014e7c0, 0xc00014e7e0)
        /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/logging/zap/server_interceptors.go:31 +0x115 fp=0xc0005d3948 sp=0xc0005d37e0 pc=0xa22675
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0xef8878, 0xc0004c7770}, {0xca6400, 0xc000c4f800})
        /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a fp=0xc0005d3988 sp=0xc0005d3948 pc=0xa1fbfa
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1({0xef8878, 0xc0004c7770}, {0xca6400, 0xc000c4f800}, 0xc000c4f800, 0xc00014e840)
        /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:107 +0x87 fp=0xc0005d39e8 sp=0xc0005d3988 pc=0x9fd907
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0xef8878, 0xc0004c7770}, {0xca6400, 0xc000c4f800})
        /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a fp=0xc0005d3a28 sp=0xc0005d39e8 pc=0xa1fbfa
d7y.io/dragonfly/v2/pkg/rpc.unaryServerInterceptor({0xef8878, 0xc0004c7770}, {0xca6400, 0xc000c4f800}, 0xc00014e7c0, 0xc00014e7c0)
        /go/src/d7y.io/dragonfly/v2/pkg/rpc/server.go:72 +0x57 fp=0xc0005d3ad8 sp=0xc0005d3a28 pc=0xa25a97
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0xef8878, 0xc0004c7770}, {0xca6400, 0xc000c4f800})
        /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a fp=0xc0005d3b18 sp=0xc0005d3ad8 pc=0xa1fbfa
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0xef8878, 0xc0004c7770}, {0xca6400, 0xc000c4f800}, 0xc000829bb8, 0xc04700)
        /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34 +0xbf fp=0xc0005d3b70 sp=0xc0005d3b18 pc=0xa1fa9f
d7y.io/dragonfly/v2/pkg/rpc/cdnsystem._Seeder_GetPieceTasks_Handler({0xc1ca80, 0xc0003740c0}, {0xef8878, 0xc0004c7770}, 0xc000c4f740, 0xc000218d50)
        /go/src/d7y.io/dragonfly/v2/pkg/rpc/cdnsystem/cdnsystem_grpc.pb.go:147 +0x138 fp=0xc0005d3bc8 sp=0xc0005d3b70 pc=0xa29098
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00036f180, {0xf088d0, 0xc00072c180}, 0xc00027e480, 0xc00032ab70, 0x14aa150, 0x0)
        /go/pkg/mod/google.golang.org/[email protected]/server.go:1292 +0xc6f fp=0xc0005d3e48 sp=0xc0005d3bc8 pc=0x9bdaef
google.golang.org/grpc.(*Server).handleStream(0xc00036f180, {0xf088d0, 0xc00072c180}, 0xc00027e480, 0x0)
        /go/pkg/mod/google.golang.org/[email protected]/server.go:1617 +0xa2a fp=0xc0005d3f68 sp=0xc0005d3e48 pc=0x9c182a
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /go/pkg/mod/google.golang.org/[email protected]/server.go:940 +0x98 fp=0xc0005d3fe0 sp=0xc0005d3f68 pc=0x9bb5d8
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005d3fe8 sp=0xc0005d3fe0 pc=0x468ae1
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /go/pkg/mod/google.golang.org/[email protected]/server.go:938 +0x294

More errors see the attached log.

Expected behavior:

Cdn run stably.

cdn-error.log

How to reproduce it:

Deploy dragonfly with Helm chart 0.5.38, then pull some images, sometimes, one of the cdns restart.

Environment:

• Dragonfly version: v2.0.2-rc.4
• OS: Centos 7
• Kernel (e.g. uname -a): Linux 3.10.0-1160.31.1.el7.x86_64
• Others: Helm chart 0.5.38

[244372610]

Thanks for reporting the bug. I'll take care of it later

[likunbyl]

Any update about this issue? @244372610

[244372610]

I'll deal with the problem today