Error handling while DTopic publish in embeded mode

[TerrySolar]

hi, I use embeded olric in my project, some DTopic will be create dynamicly. After the cluster initialed, some DTopic just created in a few instances, when publish message at this time will cause an error(topic not found) in some instance which not create this DTopic.

Should I add a fucntion like "NewDTopicToCluster" or change the error in function publishDTopicMessageToAddr to a warning log?

[buraksezer]

Hello @TerrySolar, I'm sorry for my late response. I was on vacation.

Could you please tell me what Olric version you use? I also need some logs from the cluster members. It seems that you are having problems with forming a healthy cluster.

Should I add a fucntion like "NewDTopicToCluster" or change the error in function publishDTopicMessageToAddr to a warning log?

There is no need for such a function. Everything is clustered in Olric and it handles all complexity.

[TerrySolar]

thanks for response. I am using olric v0.4.5, I think I created a healthy cluster, get or set data works normally in my cluster. I did not create DTopic while cluster initial, the creation trigger by some request, and a random node will call NewDTopic function, I read the code in NewDTopic, I think if I just call NewDTopic in single node, the others in cluster did not know this creation, If I use publish this time, the others nodes will form an error(topic not found)

[buraksezer]

Hello, could you please upgrade Olric to v0.4.9? It's the latest version. BTW, I'm planning to release v0.5.0 soon. The API has changed considerably. I'll continue maintaining v0.4.x but it will only receive security updates and bug fixes. I recommend using v0.5.0-rc.1 for new projects.

Could you please share some logs if you don't mind? I'll try to reproduce the problem on my side.

[TerrySolar]

Hi, I was formed a three nodes cluster with olric v0.4.9

• node1: 9.134.43.216:10002
• node2: 9.134.43.216:20002
• node3: 9.134.43.216:30002

It seems like a healthy cluster from logs

2022/11/25 16:47:57 [INFO] Storage engine has been loaded: kvstore => service.go:129
2022/11/25 16:47:58 [WARN] memberlist: Binding to public address without encryption!
2022/11/25 16:47:59 [DEBUG] memberlist: Initiating push/pull sync with: 9.134.43.216:20002
2022/11/25 16:47:59 [DEBUG] memberlist: Initiating push/pull sync with: 9.134.43.216:30002
2022/11/25 16:47:59 [INFO] Join completed. Synced with 2 initial nodes => discovery.go:59
2022/11/25 16:47:59 [INFO] Memberlist bindAddr: 9.134.43.216, bindPort: 10002 => routingtable.go:399
2022/11/25 16:47:59 [INFO] Cluster coordinator: 9.134.43.216:30001 => routingtable.go:400
2022/11/25 16:47:59 [INFO] Node name in the cluster: 9.134.43.216:10001 => olric.go:392
2022/11/25 16:47:59 [INFO] Olric bindAddr: 9.134.43.216, bindPort: 10001 => olric.go:398
2022/11/25 16:47:59 [INFO] Replication count is 1 => olric.go:400
2022/11/25 16:47:59 [INFO] Node joined: 9.134.43.216:20001 => routingtable.go:254
2022/11/25 16:47:59 [INFO] Node joined: 9.134.43.216:30001 => routingtable.go:254
2022/11/25 16:47:59 [INFO] Olric is ready to accept connections
2022/11/25 16:47:59 [INFO] Routing table has been pushed by 9.134.43.216:30001 => operations.go:87
2022/11/25 16:48:38 [DEBUG] memberlist: Stream connection from=9.134.43.216:56034
2022/11/25 16:48:39 [DEBUG] memberlist: Initiating push/pull sync with: 9.134.43.216:20002

Got some error logs After I call NewDTopic and publish in node1

2022/11/25 16:48:59 [ERROR] Failed to publish message to 9.134.43.216:20001: invalid argument: invalid argument: topic not found: test => publish.go:73
2022/11/25 16:48:59 [ERROR] Failed to publish message to 9.134.43.216:30001: invalid argument: invalid argument: topic not found: test => publish.go:73
2022/11/25 16:48:59 Publish error: invalid argument

BTW, I notice DTopic has been deprecated while v0.5.x，so I choosed to stay in v0.4.x before

[buraksezer]

@TerrySolar thank you very much. I'll try to reproduce it.

BTW, I notice DTopic has been deprecated while v0.5.x，so I choosed to stay in v0.4.x before

Olric v0.5.0 implements a drop-in replacement of the Redis Publish/Subscribe feature. The official client (in both deployment modes) provides a thin layer around the commands. See this: https://pkg.go.dev/github.com/buraksezer/[email protected]#PubSub

[buraksezer]

Hey @TerrySolar, the problem has been fixed and I just released Olric v0.4.10. https://github.com/buraksezer/olric/releases/tag/v0.4.10

Thank you for reporting the problem! Please don't hesitate to open an issue or join our Discord channel if you have further problems or questions.

[TerrySolar]

thank you very much, and I'll try v0.5.x recently

[buraksezer]

Would you mind explaining your use case, if it's possible? It's good to know about the user's problems.

[TerrySolar]

I have a lightweight long-connection cluster, openresty as a broker, and store connection meta in olric. Some connection will subscribe to a topic(add a listener in olric)， And I use "Publish" to push message, every listener will get metadata of the connection, and push the message to the right broker instance which hold the WebSocket