Technology selection and design for real-time communication among players in this game

[metalalive]

For real-time communication,
I'll start an experiment with web socket or server-sent event (SSE)
to see how these technologies can fit into the game application.

If you'd like to learn with me please let me know, thanks.

[teds-lin]

個人淺見

• port最好分開，也就是restful API port與即時通訊的API port分開，這可以避免一些神奇問題
• 因為遊戲對即時性要求較高
  • 傳送data format不要用JSON，使用簡單格式，讓傳輸和解析簡單格式更有效率
  • 考慮只傳送差異更新而不是完整遊戲狀態
• 可以考慮用通訊框架
  • websocket 有像Socket.io這種框架,它會處理底層的複雜工作。
  • 如果是用SSE,可以考慮EventSource
  • 進行斷線和重連處理 - 確保伺服器和client端可以平穩處理突然中斷又重建的連線

[metalalive]

I'll start with websocket first , currently I have very little knowledge on it,
I am reading following documents , and meanwhile try the websocket feature in FastAPI :

• RFC6455
  • operate on top of TCP
  • websocket favours frame-based mechanism over stream-based mechanism
    • ref 1, ref 2
  • the initial handshake is as the same as typical HTTP protocol
    • after that, subsequent communication relies on sequences of frames transmitting between the peers
  • a data frame could be textual (UTF-8 encoded, such as JSON, XML) , or binary (custom structured format)
    • size of a frame is specified by payload-length section in a frame
  • the control frames ping and pong are viewed as heartbeat of a client or server, which ensures connection is still opened.
• wiki
• websocket vs. REST

[metalalive]

像Socket.io這種框架,它會處理底層的複雜工作

Allow me to study (and try) this framework for a bit ,

I'll reply later with more detail

[metalalive]

I found this python websocket library websockets supports some of features (e.g. heartbeat, broadcast events) required in this game application,
these features are also supported by socket.io ,I will also take a look .

To me, both of socket.io and websocket are good,

• but websocket is an internet standard , while socket.io is not, I might be kinda concerned about the future of socket.io,
  • the document describes that socket.io is not websocket implementation , it is just that part of transport process is the same as websocket.
• Google Trends shows that number of searches world-wide for websocket is greater than socket.io in past 5 years

but anyway, let me try both first

[metalalive]

The built-in websocket feature in starlette and FastAPI only provides low-level API functions such as send(), recv(), accept() ...etc. I don't recommend them in the game development.

[metalalive]

port最好分開，也就是restful API port與即時通訊的API port分開，這可以避免一些神奇問題

OK, HTTP and websocket are 2 different protocols , for potential scaling issues (in the future) it is also recommended to separate them to different servers bound to different ports .

• Could you please provide some Google keywords about what problems or difficulty do other people encounter when they mix websocket and HTTP in one single server port ? this is only for expanding my knowledge because I don't know what are the possible problems.
• The only problem I can see is that , for Linux-based server, each TCP connection occupies an opened file
  • for example , the system call socket() , see the function code sock_alloc_file() and struct socket .
  • websocket requires long-lived connection will make the file resource stay opened for a long time,
  • it is a challenge to manage large number of websocket connections meanwhile support normal HTTP connections for REST endpoint services.

傳送data format不要用JSON，使用簡單格式，讓傳輸和解析簡單格式更有效率

Protocol buffer is the only option coming to my mind

• GPT introduces few other alternatives e.g. MessagePack, Apache Thrift, Apache Avro , BSON (Binary JSON), will take a look then

考慮只傳送差異更新而不是完整遊戲狀態

Absolutely , that relies on HTTP (RESTful) server emits each update event to websocket server

• remind both of the servers run with different ports
• pub / sub pattern implementation
• the message queue could be Redis, RabbitMQ, Kafka

[teds-lin]

Could you please provide some Google keywords about what problems or difficulty do other people encounter when they mix websocket and HTTP in one single server port ?

我沒辦法給出適當的keyword，因為這句話只是根據過去開發經驗給出的建議
以下是Google到的可能相關文章

• Unable to run WebSocket and HTTP Server on same port
• WebSockets and HTTP sharing the same port using request headers
• RESTful HTTP and websocket in the same application?

[metalalive]

So far all the comments come from backend people.
What are frontend people's opinions about socket.io ?
@wingtkw , please let us know your development experience , thanks .

[wingtkw]

我之前沒有用過socket io
但socket io 有npm library 可以使用
也有教程
要跟着教程試一下才能評論

[metalalive]

OK, 因為需要考慮 前端開發人員 socket.io 的使用經驗是好是壞,
話雖這麼說，這已經發布好幾年了，應該不會有什麼嚴重問題 lol
Wing 請先 socket io 讀文件 然後回覆我們, 看你是否可以接受在遊戲開發中使用它, 謝謝

[update]
the team Hunter x Gungi also used socket.io and they gave positive feedback about it .

[metalalive]

@wingtkw , @teds-lin ,
If nobody leaves comment at here by the end of Friday afternoon (17:00:00 , 17th Nov. 2023),
I will randomly choose one of them for the app development.
If you have any concern please leave your comment as early as possible,
thanks

[wingtkw]

socket.io 感覺還好
但我覺得可能要注意以下問題：

• connection

• • socket io by default 使用 at most once messaging semantics (如果connection 中斷, 有機會收不到消息)
• • 把它弄成exactly once 的話, 官方做法是要要把event msg 儲存起來, 要看一下如何儲存

• security

• • socket io 沒有end to end encryption, 要看一下怎樣弄
• • authentication 問題 => 怎樣renew token 之類的

• scaling

• • 因為socket.io 是designed to work in single region，所以要想一下如何scale 或應乎outage
• • 看到官方有tutorial 是說scale horizontally, 要配合 adapter 例如redis to keep things in sync

[metalalive]

@wingtkw

For socket.io over secure connection, I found something interesting in the client API doc :

• you can specify certificate to io.connect(), see the options , I dunno about node.js and typescript , I will try to understand whether python library supports similar feature for our backend server .
• It looks like you can put authorization token at here , but I have not tried it yet
• We should check whether socket.io supports TLS v1.3 , which is the latest (also recommended) version of secure connection protocol announced since 2018.

For other issues

• I read adapter in doc , there could be Redis, RabbitMQ, or PostgreSQL.
  • for PostgreSQL and RabbitMQ , events can be stored at somewhere permanently before broadcasting to all other clients (players in a game),
  • Redis is an in-memory data-store, the data in Redis will be lost once outage happened.
• the scaling issue can be postponed until we decide which cloud platform to deploy .

[metalalive]

Let us determine which one would you like to apply to the game development ?
Now I am OK with both of them, please leave your comment if you prefer one of them. thanks

Here is summary table about socket.io vs. python library websockets :

	socket.io	websockets	required in the game app
internet standard	no	yes, RFC6455	N/A
auto long-polling	yes	no	N/A, do we really need this in the game app ?
heartbeat (ping/pong frame)	yes	yes	yes
reconnect	yes	no, but see code sample in doc	yes
broadcast	yes	no, see workaround in doc	yes, but broadcast happens ONLY to the players of a specific game, NOT all players of all existing games, it may not be difficult to implement in my opinion
acknowledgments	yes	no	no, event subscribers don't need to respond in this game
multiplexing	yes	no	N/A

[metalalive]

As described in my previous comment :

If nobody leaves comment at here by the end of Friday afternoon (17:00:00 , 17th Nov. 2023),
I will randomly choose one of them for the app development.
If you have any concern please leave your comment as early as possible,
thanks

@wingtkw , @teds-lin ,
It has been one week since we started discussing this,
so far nobody would like to make decision.
Let's try socket.io in the game development.

If you encounter difficulty on socket.io implementation and integration,
please, let us know as early as possible , thank you

[teds-lin]

大方向來說，用socket.io我沒有意見
話說後端是用python-socketio嗎

[metalalive]

話說後端是用python-socketio嗎 ?

是的, python-socketio 看起來是唯一的選擇

[metalalive]

@teds-lin , @wingtkw

Another design issue to consider :

websocket server works with HTTP RESTful server, each server occupies different port,

• do both servers run in the same process at operating-system level ?
  • pros : use internal queue for pub/sub pattern
  • cons : may cause scaling issue if the game server will be split to websocket part and normal HTTP part
• or would you like to run them in separate processes ?
  • pros : would be easier to scale the game server
  • cons : use external queue for pub/sub pattern, external tools to install, such as Redis, RabbitMQ, Kafka (if the game needs the replay feature) ...etc,

Ted, please lead the discussion , thank you.

[wingtkw]

For servers,

• implement socket.io server and HTTP RESTful server separately
• • if using the same server, it will pose some difficulty in managing and scaling them separately

For processes,

• I would actually prefer running them in separate processes and utilize external queue for pub/sub pattern
• putting them in same os level is hard to scale
• I am concerning if the server is down, will the queue be gone

btw, we can consider putting the external queue tool & config that we are using in docker, so it can be run more easily

[metalalive]

Yes, that makes sense when it comes to scalability,
Next question would be which message queue to choose for this game project .

[metalalive]

Ted 我只有看到你按讚, 但不確定你的意思,
你是否也同意 HTTP REST server 和 socket.io server 執行 在不同的 OS processes 上 ?

那我就不用等到明天下午了 ??
我們直接討論決定下一個問題 --> 選擇哪個 message queue ? (看下面的留言)

[teds-lin]

你是否也同意 HTTP REST server 和 socket.io server 執行 在不同的 OS processes 上 ?

agree

選擇哪個 message queue ?

沒有特別偏好的話, Redis吧

[metalalive]

沒有特別偏好的話, Redis吧

can you move the reply to my comment below, I just finishing editing it thanks

[metalalive]

Next question : which message queue to work with our socket.io server ?

According to the socket.io document, following adapters are supported :

• Redis , in-memory data store, data will lost once outage happened
• Kombu , which supports AMQP protocol, to communicate with other AMQP message broker
  • currently the only option is RabbitMQ
  • AMQP support message persistence in the queue at protocol level , to minimize chance of data lost
• Kafka , which developed its own binary protocol
  • I have not studied that , I heard its primary feature is event replay
  • I also heard big tech corp may use that , but that depends on usage scenario / application context

@wingtkw , Ted, we can choose any of them to learn and integrate to the game app,
leave your comments below to describe the technology you are interested the most,
thanks

[metalalive]

Personally, I would not suggest using redis for message queue
Instead it is better to use redis for caching purposes

To avoid data lost (event from HTTP server to socket.io server) , this makes sense.

OK, Let's apply Kafka to the game development , is it ok for both of you ?

[wingtkw]

I'm ok with that

[teds-lin]

Kafka

• another question, when it comes to deployment, the cloud platform options (free tier) might be limited

沒問題，不過我想了解一下Kafka在雲平台上可能會有什麼問題？
因為就我所知，有許多free hostng平台可供開發者使用

• upstash
• cloudkarafka

[metalalive]

the cloud platform options (free tier) might be limited

Ted, allow me to edit the description above , it should be :
"there might NOT be many options for cloud platforms which provide Kafka "

如果用 Kafka ,（免費d）雲端平台可能沒有太多選擇
當然我可能是錯的，因為我沒有花時間了解 每個平台支援其免費套餐中的功能
我並不是說雲端平台 d 免費方案有任何技術限制

[teds-lin]

如果用 Kafka ,（免費d）雲端平台可能沒有太多選擇

喔喔，了解！

[metalalive]

I guess this one should be the final question (of the entire discussion thread) .
According Ted's previous comment :

傳送data format不要用JSON，使用簡單格式，讓傳輸和解析簡單格式更有效率

I guess what you mean by 簡單格式 is like binary format,
The python libraries below claims that they support binary format and data serialization,
I am not sure whether they can fit the requirement of this game app well

• Protocol buffer
• MessagePack
• Apache Thrift
• Apache Avro
• BSON (Binary JSON)
  • might not recommend, I don't know whether there are still maintainers in these python libraries.

Wing, Ted, please leave your comments below to describe the tool / format you are interested the most,
to apply to the game app,
thanks

[teds-lin]

補充一個聽過但我沒用過的

• Flatbuffers

這邊的話，我稍微碰過Protocol Buffers，不過選擇其他的我也ok

[metalalive]

Quickly go through the documentations of ProtoBuf and FlatBuffers , the usage is similar :

• write a schema for the message structure in the application
• compile the schema to modules of specific programming language(s)
• modify the value under the scope of the message structure
• once value update is done, serialize it to byte string

Schema styling :

• ProtoBuf schema looks kinda like C language
• FlatBuffers schema looks like mix of C++ and Rust

Community thoughts

• ProtoBuf vs. FlatBuffers - StackOverflow

I will choose one among the 2 above.

[metalalive]

For now we can postpone this decision until we start the socket.io implementation on HTTP server side.
I will start GitHub issue that create a new socket.io API endpoint , for each player who just joined the game.

[metalalive]

Recap

For real-time communication and game state synchronization among players

• protocol : socket.io
  • protocol version : v4 or v5 (not library version)
    • let's try v5 first, if it is problematic then we can rollback to v4
    • IMPORTANT NOTE --> backend and frontend have to communicate in the same protocol version
  • backend library : python-socketio
  • frontend library: N/A <-- Wing , please help to update this
  • Adapter to external message queue : Kafka
  • payload format : FlatBuffers
    • might switch to ProtoBuf if any significant error found, so far FlatBuffers works well
• Software Design
  • implement socket.io server and HTTP RESTful server separately , bound to different ports

c.c. @teds-lin , @wingtkw

[metalalive]

I will keep this discussion open until we decide :

• what payload format to use

[metalalive]

I tried FlatBuffers yesterday, it is fairly easy to use.
I'll choose FlatBuffers as payload format of socket.io events in this project