Support bit perfect reading and writing LDB, LMT, and LMU

[mateofio]

This series of changes supports bitperfect reading and writing of RPG_RT.ldb binary databases. It supports 2k and 2k3 mode.

Closes #234 and closes #229.

This one depends on all my other PR's.

I've tested it with the following:

• An empty 2k3 database
• The default New project database for 2k3
• An empty 2k database
• The default new project database for 2k
• HH3 (2k game ~ 16MB RPG_RT.ldb)

Please if anyone has time. Checkout this branch, download some rm2k/rm2k3 games and test it against their databases. Let me know if any game doesn't come through perfectly.

[Ghabry]

Great. I will shoot this at my local game list and the rmarchiv.tk database (>1000 games) later.

[mateofio]

.flow would likely fail to convert cleanly as it has a 2k3 database but we detect it as 2k. I don't think there is anything we can do in situations like that.

Regarding Makefile.am, is maintaining autotools support really still needed now that we have Cmake?

[carstene1ns]

As said, we currently use the autotools for the distribution archives and some packagers will also depend on them (i.e. macOS homebrew, some GNU/Linux distributions, platforms without cmake support).

[Ghabry]

but what we will remove soon are the Visual Studio project files which are by far the worst to maintain with there XML bloat.

[Ghabry]

@fmatthew5876
Question about LDB read-write: Did you test this with LDB->XML->LDB or LDB read->LDB write?

[mateofio]

@fmatthew5876
Question about LDB read-write: Did you test this with LDB->XML->LDB or LDB read->LDB write?

Not yet. I'll look into this later today.

[Ghabry]

Okay then I maybe misunderstood what I'm supposed to test.
Because you wrote "bit perfect reading/writing" I expect that a file loaded with LDB_Reader::Load followed by LDB_Reader::Save matches but when I diff e.g. your HH LDB-Database I get "binary files differ".

[mateofio]

Sorry I didn't answer that well. I've only tested LDB -> LDB. I haven't done xml yet.

Which version of HH do you have? I tested it on the latest one in my github.

[Ghabry]

Okay, what I did: Compiled liblcf on branch "ldb_enums2", then compiled the following program:

#include <cstring>
#include <string>

#include "ldb_reader.h"

int main(int argc, char** argv)
{
        std::string infile = argv[1];
        std::string outfile = argv[2];
        LDB_Reader::Load(infile, "1252");
        LDB_Reader::Save(outfile, "1252");

        return 0;
}

Then I convert via this tool and run "diff RPG_RT.ldb RPG_RT2.ldb"

binary files dffer

Also tested it with your HH3 ldb

[mateofio]

My test driver is this:

https://github.com/fmatthew5876/hh3-rm2k/blob/fixes-2.1/dev/autodb/main.C

without the call to doWeaponCopies()

One difference I see is I'm passing "" as the encoding, where you're using "1252"

EDIT: also I forgot to mention. The hh3 version I tested on is in the fixes-2.1 branch. Not master.

[Ghabry]

okay it doesn't matter if I do LDB->XML->LDB or LDB->LDB, both give the same results.

With your hh3 version I get "identity" but for anything else I fire at the converter the result differs.

[Ghabry]

For my test data I get a perfect match for:

For 2k: 2 out of 1110
For 2k3: 42 out of 435

Which is a good start imo

[mateofio]

I found a bug in how we read RPG::Parameter when actor.final_level is not default.

With this fix I've tested:

• hh3 v2.03
• hh3 v2.04
• Alter AILA Genesis
• Dungeoneer

@Ghabry That test setup is awesome. Could you run your test suite again and tell me a couple sample games that still fail?

[mateofio]

I've rebased this onto master. Also fixed the excel related differences.

This one is a huge pain to rebase. Basically have to redo fields.csv manually each time.

I verified rebased fields.csv against the old version using vimdiff, but this needs a full retest to make sure I didn't break it.

[Ghabry]

codewise it looks fine to me but I will do a retest with LDB and also add LMT and LMU files. (maybe not all LMU files though because there are so many...)

[Ghabry]

Going to bed now, therefore only dumping you the raw output of my improved test script: (omitted LSD as this is part of another PR) and havn't filtered the bogus games you already reported...

For Maps I only used 0001 because I noticed that when the 1st map fails usually all other maps fail, too and using all maps far too long.

http://ix.io/1qkp

[mateofio]

I don't see any new regressions for LDB from your list. All the ones there have some corruption or have terms issues.

Sweet Dreams (Demo) all the files (LDB,LMT,LMU) have 9 or 10 extra zero bytes at the end. I don't know what these are supposed to be for but liblcf doesn't write them back out.

This is the only one failing LMT testing, so we can sign off on LMT being done.

For LMU, are you sure your test is correct? I tested some of the failing games in your list and they came out clean, all LMU files.

Do you load the database and the treemap into memory before loading and saving maps? I do that in my test as they can rely on those data structures to parse correctly.

I'm using a hacked up version of the below test driver to do maps. The call to loadRPG() in there loads the LDB and LMT.

• https://github.com/fmatthew5876/hh3-rm2k/blob/fixes-2.1/dev/copyrpg/main.C

[Ghabry]

My C++ test program is a bit less sophisticated but I'm indeed not loading the LDB or LMT which means the version field is not initialized... my fault. Won't have access to my PC before the evening, then I rerun it. I wonder if it will fix the 2k games, too (and then I wonder why).

I confirm that there are no new regressions which means I will remove all the non-fixable games from my test data.

[Ghabry]

hm I messed something up, now I get for LMU

Summary:
2k:   1072/1092 (98.2%)
2k3:  402/416 (96.6%)
2k3e: 44/45 (97.8%)

but for LDB

2k:   14/1092 (1.3%)
2k3:  404/416 (97.1%)
2k3e: 45/46 (97.8%)

investigating...

[Ghabry]

Found another flaw in my test code but fixing this requires going through all ZIPs again, but I had rates of 95% now, giving the real values when I finished fixing this.