Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash while benchmarking #153

Closed
buraksezer opened this issue Apr 10, 2022 · 1 comment
Closed

Crash while benchmarking #153

buraksezer opened this issue Apr 10, 2022 · 1 comment
Assignees
Labels
bug Something isn't working release:v0.5.x Issues related with 0.5.x
Milestone

Comments

@buraksezer
Copy link
Owner

Reproduction steps:

  1. Create a 10 nodes Olric cluster with docker-compose: docker-compose up --scale olric=10 olric
  2. Run memtier_benchmark: memtier_benchmark -p 3320 --command="dm.put bench __key__ brk"

Some of the nodes crash during the test:

olric_8      | 2022/04/10 10:43:13 [DEBUG] memberlist: Stream connection from=172.20.0.2:37290
olric_2      | 2022/04/10 10:43:14 [DEBUG] memberlist: Stream connection from=172.20.0.2:39128
olric_7      | panic: interface conversion: interface {} is int, not uint64
olric_7      |
olric_7      | goroutine 3609 [running]:
olric_7      | github.com/buraksezer/olric/internal/kvstore.(*KVStore).makeTable(0xc000502540)
olric_7      | 	/src/internal/kvstore/kvstore.go:85 +0x265
olric_7      | github.com/buraksezer/olric/internal/kvstore.(*KVStore).Put(0xc000502540, 0xbbf97c?, {0xdc1098, 0xc0005011c0})
olric_7      | 	/src/internal/kvstore/kvstore.go:210 +0xba
olric_7      | github.com/buraksezer/olric/internal/dmap.(*DMap).putEntryOnFragment(0xc000500180?, 0xc0005af380?, {0xdc1098?, 0xc0005011c0?})
olric_7      | 	/src/internal/dmap/put.go:78 +0xb6
olric_7      | github.com/buraksezer/olric/internal/dmap.(*DMap).putOnCluster(0xc000500180, 0xc0005af380)
olric_7      | 	/src/internal/dmap/put.go:321 +0x3c5
olric_7      | github.com/buraksezer/olric/internal/dmap.(*DMap).put(0xc000500180, 0xc0005af380)
olric_7      | 	/src/internal/dmap/put.go:354 +0x2f2
olric_7      | github.com/buraksezer/olric/internal/dmap.(*Service).putCommandHandler(0xc000171320, {0xdc1cf0, 0xc000592c40}, {{0xc0000e37c0, 0x3a, 0x40}, {0xc0008474a0, 0x4, 0x4}})
olric_7      | 	/src/internal/dmap/put_handlers.go:62 +0x225
olric_7      | github.com/buraksezer/olric/internal/server.Handler.ServeRESP({0xc000195830?, 0xc0001956a0?}, {0xdc1cf0, 0xc000592c40}, {{0xc0000e37c0, 0x3a, 0x40}, {0xc0008474a0, 0x4, 0x4}})
olric_7      | 	/src/internal/server/handler.go:68 +0x264
olric_7      | github.com/buraksezer/olric/internal/server.(*ServeMux).ServeRESP(0xc0000105b8, {0xdc1cf0, 0xc000592c40}, {{0xc0000e37c0, 0x3a, 0x40}, {0xc0008474a0, 0x4, 0x4}})
olric_7      | 	/src/internal/server/mux.go:73 +0x368
olric_7      | github.com/tidwall/redcon.handle.func2(0xc000592c40, 0xc0001c1570)
olric_7      | 	/go/pkg/mod/github.com/tidwall/[email protected]/redcon.go:421 +0x212
olric_7      | github.com/tidwall/redcon.handle(0x0?, 0x0?)
olric_7      | 	/go/pkg/mod/github.com/tidwall/[email protected]/redcon.go:434 +0x78
olric_7      | created by github.com/tidwall/redcon.serve
olric_7      | 	/go/pkg/mod/github.com/tidwall/[email protected]/redcon.go:370 +0x5e5
olric_4      | 2022/04/10 10:43:14 [DEBUG] memberlist: Stream connection from=172.20.0.2:59996
olric_1      | 2022/04/10 10:43:14 [DEBUG] memberlist: Stream connection from=172.20.0.2:57542
olric_10     | 2022/04/10 10:43:14 [DEBUG] memberlist: Stream connection from=172.20.0.2:59050
docker_olric_7 exited with code 2


olric_10     | 2022/04/10 10:43:24 [DEBUG] memberlist: Stream connection from=172.20.0.2:59072
olric_8      | panic: interface conversion: interface {} is int, not uint64
olric_8      |
olric_8      | goroutine 180 [running]:
olric_8      | github.com/buraksezer/olric/internal/kvstore.(*KVStore).makeTable(0xc000a72b70)
olric_8      | 	/src/internal/kvstore/kvstore.go:85 +0x265
olric_8      | github.com/buraksezer/olric/internal/kvstore.(*KVStore).Put(0xc000a72b70, 0xbbf97c?, {0xdc1098, 0xc000a5ddc0})
olric_8      | 	/src/internal/kvstore/kvstore.go:210 +0xba
olric_8      | github.com/buraksezer/olric/internal/dmap.(*DMap).putEntryOnFragment(0xc000480000?, 0xc000ae3080?, {0xdc1098?, 0xc000a5ddc0?})
olric_8      | 	/src/internal/dmap/put.go:78 +0xb6
olric_8      | github.com/buraksezer/olric/internal/dmap.(*DMap).putOnCluster(0xc000480000, 0xc000ae3080)
olric_8      | 	/src/internal/dmap/put.go:321 +0x3c5
olric_8      | github.com/buraksezer/olric/internal/dmap.(*DMap).put(0xc000480000, 0xc000ae3080)
olric_8      | 	/src/internal/dmap/put.go:354 +0x2f2     
olric_8      | github.com/buraksezer/olric/internal/dmap.(*Service).putCommandHandler(0xc000276870, {0xdc1cf0, 0xc000522460}, {{0xc000b2d240, 0x3a, 0x40}, {0xc0018ee840, 0x4, 0x4}})
olric_8      | 	/src/internal/dmap/put_handlers.go:62 +0x225
olric_8      | github.com/buraksezer/olric/internal/server.Handler.ServeRESP({0xc0002059a0?, 0xc000205810?}, {0xdc1cf0, 0xc000522460}, {{0xc000b2d240, 0x3a, 0x40}, {0xc0018ee840, 0x4, 0x4}})
olric_8      | 	/src/internal/server/handler.go:68 +0x264              
olric_8      | github.com/buraksezer/olric/internal/server.(*ServeMux).ServeRESP(0xc000208060, {0xdc1cf0, 0xc000522460}, {{0xc000b2d240, 0x3a, 0x40}, {0xc0018ee840, 0x4, 0x4}})
olric_8      | 	/src/internal/server/mux.go:73 +0x368
olric_8      | github.com/tidwall/redcon.handle.func2(0xc000522460, 0xc0002d8150)
olric_8      | 	/go/pkg/mod/github.com/tidwall/[email protected]/redcon.go:421 +0x212
olric_8      | github.com/tidwall/redcon.handle(0xc0004a6000?, 0x0?)
olric_8      | 	/go/pkg/mod/github.com/tidwall/[email protected]/redcon.go:434 +0x78
olric_8      | created by github.com/tidwall/redcon.serve
olric_8      | 	/go/pkg/mod/github.com/tidwall/[email protected]/redcon.go:370 +0x5e5
olric_3      | 2022/04/10 10:43:24 [DEBUG] memberlist: Stream connection from=172.20.0.2:53638
olric_5      | 2022/04/10 10:43:24 [DEBUG] memberlist: Stream connection from=172.20.0.2:41720
olric_9      | 2022/04/10 10:43:24 [DEBUG] memberlist: Stream connection from=172.20.0.2:35484
docker_olric_8 exited with code 2
@buraksezer buraksezer added the bug Something isn't working label Apr 10, 2022
@buraksezer buraksezer self-assigned this Apr 10, 2022
@buraksezer buraksezer added the release:v0.5.x Issues related with 0.5.x label Apr 10, 2022
@buraksezer buraksezer added this to the 0.5.0 milestone Apr 10, 2022
@buraksezer
Copy link
Owner Author

Done 888bb14

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working release:v0.5.x Issues related with 0.5.x
Projects
None yet
Development

No branches or pull requests

1 participant