replay.server.battlelobby parsing

[EugenPolyakov]

To parse replay.server.battlelobby try using https://github.com/EugenPolyakov/HOTS_battlelobby/blob/master/battlelobby.py

Sorry for te spam.

[koliva8245]

Just when I started looking at the battlelobby again... Thank you. It will take me some time to validate this and convert to C#.

[barrett777]

That's great, thank you! That's been a long standing mystery.

Do you know what the unnamed fields are for? Any more information is appreciated

[EugenPolyakov]

This script was written by reverse engineering replay files. I gave all the names to the fields, based on the statistics of their content and changes in patches. Not all names may be true. Some versions of the structure may not match, because I investigated mostly my replays while playing the game.

[koliva8245]

Some things as I'm going through.

('_bitarray', [(32, 0)]), #6198 -> this is a string (or blob) of 4 bytes. It is an attribute value.

('_struct', [[('f32', 6200, 0), ('NewField6', 6201, 0), ('word', 6202, 0)]]), #6203 -> 'word' I believe is an id that matches with the id in some of the s2ml xml files.

example:

    <e id="34">No Match History</e>
    <e id="35">Participant Role</e>
    <e id="36">Participant</e>
    <e id="37">Observer</e>
    <e id="38">Observer Type</e>
    <e id="39">Spectator</e>
    <e id="40">Referee</e>

[koliva8245]

I should not have said, blob, that implies a byte aligned string, the 4 bytes are unaligned.

1. ('_bitarray', [(32, 0)]), #6226 -> is also a 4 unaligned byte string

2. ('_bitarray', [(256, 0)]), #6393 -> its an array of 16 (total player slots) with 2 bytes each

3. ('_struct', [[('static1', 6248, 0), ('Attributes', 6264, 0), ('static2', 6285, 0), ('ModParams', 6385, 0), ('s2mh', 6390, 0), ('skins', 6392, 0), ('hasSkins', 6394, 0), ('filler', 6476, 0), ('playersInfo', 6435, 0), ('_end', 6474, 0)]]), #6477 -> some of these could be rename better

   static1 -> Attributes
   Attributes -> PlayerSelectedAttributes
   static2 -> Locales
   filler -> InitData??? ( I suspect that's what this data is)

4. Not sure if you looked at my decoded code but it will help fill in some missing data, especially for the playersInfo section.

   • The filler section has the m_disabledHeroList and m_randomSeed data (from the replay.initData file)
   • The playersInfo section is misaligned
     ('_array', [(0, 8), 6434]), #6435 -> should be an array of 5
     ('_bitarray', [(29, 0)]), #6396 -> should be 32 bits
     ('_bitarray', [(3, 0)]), #6433 -> should be 1 bit (btw is m_hasActiveBoost)

5. I suspect you didn't try any AI games, I know that has some extra data and might help with decoding more data.

[koliva8245]

1. ('_bitarray', [(62, 0)]), #6436 -> should be 64
2. ('_bitarray', [(87, 0)]), #6444 -> should be 72. Then after that the next 4 bytes (32 bits) is an int32 which the // m_ammId (the game mode)

[MGatner]

Thanks all! Great content.

@Zemill making sure you see this too.

[koliva8245]

('_bitarray', [(198, 0)]), #6421 -> after 5 additional bits, the next 4 bytes (32 bits) is the build number of the replay (either m_build, m_baseBuild, m_dataBuildNum; it should all be the same)

[EugenPolyakov]

I'll take a look at the weekend. As for playerInfo, it looks strange, I have not seen any parsing errors.

either m_build, m_baseBuild, m_dataBuildNum; it should all be the same

There was at least a patch in which these versions differed from each other

[koliva8245]

It parses fine, but the data is wrong.

The first player in the loop is fine, but after that the data is incorrect. The m_programId (Hero) int value should always be 1214607983, the m_realm should be 1 and the m_region should be.. the region your in. The m_SlotId is wrong too. Should increment up from 0.

// first two

'playersInfo': [{'m_PartyInfo': None,
                  'm_PlayerLevel': 1119,
                  'm_TagInfo': b'<battletag>',
                  'm_Unk1': {'m_Name': {'m_Flags': (1, 0),
                                        'm_stringToon': {'m_globalName': b'<A T:ID>',
                                                         'm_programId': (32, 1214607983),
                                                         'm_realm': 1,
                                                         'm_region': 1}},
                             'm_SecName': None,
                             'm_SlotId': 0,
                             'm_Unk1': (29, 447904),
                             'm_Unk4': (2, 0),
                             'm_toon': {'m_id': 10897, 'm_programId': (32, 1214607983), 'm_realm': 1, 'm_region': 1}},
                  'm_Unk2': (198, 98440051050568283746666135255451843401172051619116744704),
                  'm_Unk3': None,
                  'm_Unk4': {'m_Skins2': (4, 0), 'm_Unk1': (35, 8590114202), 'm_UnkFlags1': (2, 0)},
                  'm_UnkFlags': (3, 0)},
                 {'m_PartyInfo': None,
                  'm_PlayerLevel': 292,
                  'm_TagInfo': b'<battletag>
                  'm_Unk1': {'m_Name': {'m_Flags': (1, 0),
                                        'm_stringToon': {'m_globalName': b'<A T:ID>',
                                                         'm_programId': (32, 1144174895),
                                                         'm_realm': 536870913,
                                                         'm_region': 33}},
                             'm_SecName': None,
                             'm_SlotId': 3,
                             'm_Unk1': (29, 223952),
                             'm_Unk4': (2, 0),
                             'm_toon': {'m_id': 5375653,
                                        'm_programId': (32, 1144174895),
                                        'm_realm': 536870913,
                                        'm_region': 1}},
                  'm_Unk2': (198, 98440051781319102412117594287793508808726942214002835456),
                  'm_Unk3': None,
                  'm_Unk4': {'m_Skins2': (4, 0), 'm_Unk1': (35, 8590114202), 'm_UnkFlags1': (2, 0)},
                  'm_UnkFlags': (3, 0)},
       ...

There was at least a patch in which these versions differed from each other

I think I remembered that too. If someone has a replay where those numbers differ, that would be great.

[EugenPolyakov]

For the past few years I've been making edits for new versions in quick mode and seem to have broken PlayersInfo.

Added a description of the structure to the readme, it's better to refer to it (instead of python script) pointing out inaccuracies.

[koliva8245]

That description structure is much easier to read through.

So an update on the build and m_ammId

// for the build
m_Unk4: record[:0 + 0]
  m_UnkFlags1: array [:0 + 2] of bit
  m_Unk1: array [:0 + 35] of bit
  m_hasSilencePenalty: array [:0 + 1] of bit
  m_Unk2: array [:0 + 1] of bit
  m_hasVoiceSilencePenalty: array [:0 + 1] of bit
  m_isBlizzardStaff: array [:0 + 1] of bit
end

for m_Unk1 its just 2 bits, then the next 32 bits is the build number, then with 1 bit left over. Since this build is part of the player loop, I suspect it's the base build of the player's client.

// m_ammId
_end: record[:0 + 0]
  sss: array [:0 + 64] of bit
  m_toon: case (:1 + 0) of
    filler(0): array [:0 + 16] of bit
    m_toon(1): record[:0 + 0]
      m_region: Integer(:8 + 0)
      m_programId: array [:0 + 32] of bit
      m_realm: Integer(:32 + 0)
      m_id: Integer(:64 + 0)
    end
  end

for m_id: Integer(:64 + 0), its actually 32 bits for the m_ammId. Then with 32 bits left over. Interesting they made it a toon.

[Zemill]

@koliva8245 Requested files

Files4.zip
Files3.zip
Files2.zip
Files1.zip

[EugenPolyakov]

I fixed the parsing bugs of old versions and posted the structures of versions known to me. Some version structures may not match, because I don't have PTR replays and I arranged them according to the major version.

[koliva8245]

m_Unk4: record[:0 + 0]
  m_UnkFlags: array [:0 + 4] of bit
  m_build: Integer(:32 + 0)
  m_Skins: case (:1 + 0) of
    m_Skins(0): record[:0 + 0]
      m_Skins: array [:12 + 0] of bit
      unk: Boolean:1
    end
    none(1): array [:0 + 0] of bit
  end
  m_hasSilencePenalty: Boolean:1
  unk2: Boolean:1
  m_hasVoiceSilencePenalty: Boolean:1
  m_isBlizzardStaff: Boolean:1
end

m_build does not return the build in 68509 or older but does in 70200 and newer. So somewhere between those two builds it changed.

@barrett777 FYI, I will not be updating this repo's battlelobby anymore as I created my own parser at https://github.com/HeroesToolChest/Heroes.StormReplayParser and I don't really want to maintain two repos.

[EugenPolyakov]

m_build does not return the build in 68509 or older but does in 70200 and newer. So somewhere between those two builds it changed.

m_build became a build from version 2.39.0.69228
in 2.38.69264 it is not a buld