Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to serialize/deserialize Schema on the server side #115

Open
jrabek opened this issue Sep 23, 2021 · 14 comments
Open

Ability to serialize/deserialize Schema on the server side #115

jrabek opened this issue Sep 23, 2021 · 14 comments
Assignees

Comments

@jrabek
Copy link

jrabek commented Sep 23, 2021

Thanks again @endel for the great multiplayer game framework! If you would provide some basic guidance on this enhancement I would be happy to implement it in a PR.

Summary

This is a request for a Schema method for completely serializing/deserializing the server Schema (i.e. game state) so that it can be persisted in a database.

Motivation

In the current colyseus server architecture, the game state (i.e. some Schema derived class) persists in memory in one specific server.

One of the shortcomings is that it prevents seamless deploys and crash recovery since the game state is in memory and therefore lost when the server process is killed. For certain game types (e.g. long running games, games involving money) it is critical that game state is not lost.

To support a more robust game state, the ability to fully serialize the server Schema in a way that it could be stored in a database is needed.

Currently there are two ways to serialize the server Schema neither of which are meant for deserializing it later.

  1. toJSON: This method is meant for logging / debugging
  2. encodeAll / decode: These are meant to send state updates to the clients and not for recreating the entire server Schema.

For (2) this is what happens when the server state is serialized/deserialized using encodeAll / decode: https://gist.github.com/jrabek/6303d3f94b82c692de7721f0fc331f8c.

@jrabek
Copy link
Author

jrabek commented Sep 23, 2021

Some related discord discussions:

Original conversation: https://discord.com/channels/525739117951320081/526083188108296202/880867521824170005

Follow up when encodeAll / decode wasn't working: https://discord.com/channels/525739117951320081/526083188108296202/889878308983046215.

@endel endel self-assigned this Sep 27, 2021
@Wenish
Copy link

Wenish commented Feb 24, 2022

as a workaround:

in the onCreate function of the room call a loadState function which gets the room state from a database and maps it to the state (mapping has to be manual)

and in the onDispose function of the room call a saveState function which save the room in a database (mapping has to be manual)

at least thats how i do it :)

and maybe its worth for your game to auto save the state every 5 minutes in case server crashes. so only 5 minutes of game gets lost.

@jrabek
Copy link
Author

jrabek commented Mar 1, 2022

@Wenish so I am not using the rest of colyseus, just the schema. Also I don't think what you are proposing will work because not all the internal values are saved (most importantly the ids of the fields meaning the diff that gets sent after reloading the data breaks). Check out the gist I had sent before: https://gist.github.com/jrabek/6303d3f94b82c692de7721f0fc331f8c

@Wenish
Copy link

Wenish commented Mar 1, 2022

@jrabek im talking about a workaround where you write your own mapper from the schema to your database model and visa versa. not using any built in function you try to use.
and it does work. i load hole maps from a json file

lpsandaruwan added a commit to lpsandaruwan/colyseus-devmode-test that referenced this issue Mar 8, 2022
@lpsandaruwan
Copy link
Contributor

lpsandaruwan commented Apr 21, 2022

Hi @jrabek

Still investigating this issue. Hopefully plan to fix this in future. You will be able to use this workaround for now.

 // Simulate a serialization to a database
  if (doDecode) {
    state = new State();
    state.decode(encodedState);
    state = state.clone();
  }

cc @endel

@jrabek
Copy link
Author

jrabek commented Apr 21, 2022

Thanks for the update @lpsandaruwan! Definitely would love to try out colyseus again!

I must have missed clone before. Was that added more recently?

With this approach will the cloned state work with per client filters?

@lpsandaruwan
Copy link
Contributor

lpsandaruwan commented Apr 22, 2022

Hi @jrabek
Cloning the state and reassigning, was a shot in the dark :-D. It resolved the undefined property exception when decoding state back.
It should work without any interruption. I'm still testing client's behavior against this. But if you have any inputs/help to offer on this please feel free to mention.

cc @endel

@jrabek
Copy link
Author

jrabek commented May 10, 2022

@lpsandaruwan I had a chance to play with your suggestion and while I don't see the exception I also don't see the correct behavior when decoding from an encoded state and then cloning.

You can check out my experiment here https://github.com/jrabek/colyseus-schema-experiments/tree/main. I put a few notes in the README.md.

@lpsandaruwan
Copy link
Contributor

Thank you @jrabek. Will look into this.

@lpsandaruwan
Copy link
Contributor

Hi @jrabek
From your provided sample, I think after cloning the schema you might have to set the schema inside the serializer object again since the serializer object is referenced to the old schema object, not the cloned one. Please let us know if that worked for you. In, https://github.com/jrabek/colyseus-schema-experiments/blob/def4ed67724ce2da52e5bd6491fb4a7fc41136fd/server.ts#L23

    if (this.doDecode) {
      this.state = new GameState();
      this.state.decode(encodedState);
      this.state = this.state.clone();
      this.serializer = new Serializer(this.state);
    }

@jrabek
Copy link
Author

jrabek commented Aug 11, 2022

Thanks for the follow-up! I'll check when I have a second this week.

@jrabek
Copy link
Author

jrabek commented Sep 1, 2022

@lpsandaruwan sorry for the delay and thanks again for the follow up!

I tried this out locally and the initial results look really promising! I'll need to spend some more time with it but this seems like an acceptable approach to the original problem.

@jrabek
Copy link
Author

jrabek commented Sep 1, 2022

Hmmm, I may have spoken too soon. It seems that any modification to the state followed by a save/load cycle result in the full state getting sent to all clients. Per client filtering is working as expected but the full state is always sent.

So it seems like the data required to know what the last state update was is lost when doing a serialization/deserialization.

To be extra sure I added a small test that simple sets a string in the state to "a". The resulting patch sent to each client is the entire state.

console.log("--- minor update ---");
server.smallUpdateTest();
updateClientsStates();

If doDecode is false then the update sent to each client is of length 5.

============= smallUpdateTest =============
==== saveLoadState ==== (length:179)
Server state {
  currentPlayerUserId: 'player2',
  players: {
    player1: { userId: 'player1', isVip: false },
    player2: { userId: 'player2', isVip: false },
    player3: { userId: 'player3', isVip: false },
    player4: { userId: 'player4', isVip: false },
    player5: { userId: 'player5', isVip: false }
  },
  words: [ 'hey', 'there' ],
  secret: 'secretWord for player 2',
  title: 'a'
}
Getting updates for  [ 'player1', 'player2', 'player3', 'player4', 'player5' ]
--- Updating client state ---
Client updateState (length:5) before (player2) {
  currentPlayerUserId: 'player2',
  players: {
    player1: { userId: 'player1', isVip: false },
    player2: { userId: 'player2', isVip: false },
    player3: { userId: 'player3', isVip: false },
    player4: { userId: 'player4', isVip: false },
    player5: { userId: 'player5', isVip: false }
  },
  words: [ 'hey', 'there' ],
  secret: 'secretWord for player 2'
}
Client updateState (length:5) after (player2) {
  currentPlayerUserId: 'player2',
  players: {
    player1: { userId: 'player1', isVip: false },
    player2: { userId: 'player2', isVip: false },
    player3: { userId: 'player3', isVip: false },
    player4: { userId: 'player4', isVip: false },
    player5: { userId: 'player5', isVip: false }
  },
  words: [ 'hey', 'there' ],
  secret: 'secretWord for player 2',
  title: 'a'
}
Client updateState (length:5) before (player1) {
  currentPlayerUserId: 'player2',
  players: {
    player1: { userId: 'player1', isVip: false },
    player2: { userId: 'player2', isVip: false },
    player3: { userId: 'player3', isVip: false },
    player4: { userId: 'player4', isVip: false },
    player5: { userId: 'player5', isVip: false }
  },
  words: [ 'hey', 'there' ]
}
Client updateState (length:5) after (player1) {
  currentPlayerUserId: 'player2',
  players: {
    player1: { userId: 'player1', isVip: false },
    player2: { userId: 'player2', isVip: false },
    player3: { userId: 'player3', isVip: false },
    player4: { userId: 'player4', isVip: false },
    player5: { userId: 'player5', isVip: false }
  },
  words: [ 'hey', 'there' ],
  title: 'a'
}

If doDecode is true then the length of the client update is 206 or 156 (depending on filtering).

@jrabek
Copy link
Author

jrabek commented Sep 1, 2022

So I guess the option at this point would be to serialize the state to the database continually, but only as a backup. If the server crashes it can reread the state and would send the full update to the clients to ensure they are back in sync. Which is probably what you want anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants