Optimize memory usage of liblcf

[mateofio]

liblcf uses a lot of memory.

I wrote a program that reads RPG_RT.ldb and then exits. I measured it's memory usage with valgrind massif.

Here are some quick stats:

Game	RPG_RT.ldb	Loaded Database Mem Usage	overhead
Heros Realm	27 MB	300 MB	11x
HH3	16M	136.4 MB	8.3x
BloodmoonEnt. & Hangman	88KB	543KB	6x
Fate Hunter Orleon	368KB	2.1MB	6x
First Fantasy	1.3MB	5.2MB	4x
MKs Quest 3 (Demo)	2.3MB	11.1MB	4.8x
Siara	172KB	1.2MB	7x

[carstene1ns]

Heros Realm has the biggest database we know of so far, so is a good candidate for testing. We have tried to optimize this a bit before, but never went far. I suspect the usage of STL (i.e. std::string) and generally template code may have some overhead.

[mateofio]

Here is some data on the breakup per section of the database. I generated this by hacking liblcf to only load a single database chunk.

Game	Actor	Skills	Items	Enemies	Troops	Terrains	Attributes	States	Animations	Chipsets	Terms	System	Switches	Variables	CommonEvents	Version	CommonEventD2	CommonEventD3	BattleCommands	Classes	ClassD1	BattlerAnimations
HH3	201.8KB	454.5KB	1MB	203.3KB	121.6MB	130.7KB	120.8KB	203KB	1.9MB	161.8KB	125.6KB	121.8KB	199.7KB	177.4KB	6.2MB	118.6KB	118.6KB	118.6KB	118.6KB	118.6KB	118.6KB	118.6KB
HeroesRealm	158.2KB	577KB	507KB	328.4KB	294.6MB	132.6KB	120.7KB	134.7KB	2.4MB	130.4KB	125.6KB	121.8KB	205.9KB	136.3KB	3.2MB	118.6KB	118.6KB	118.6KB	120.9KB	224.1KB	118.6KB	415.7KB
Violated Heroine	243.1	215KB	2.2MB	179.6KB	2.4MB	132.3KB	120.6KB	130.9KB	2.3MB	349.8KB	125.9KB	121.8KB	245.6KB	342.6KB	68.5MB	118.6KB	116.8KB	118.6KB	118.6KB	118.6KB	118.6KB	118.6KB

[mateofio]

If I remove EventCommand::string by making it static, HH3 troop usage goes from 121.6MB to 69.8MB which is a 42% savings in memory.

On my amd64 linux machine, sizeof(EventCommand) == 64 and sizeof(std::string) == 32.

From this we can conclude that sizeof(EventCommand) is paramount. We need to make this structure as small as possible.

Furthermore, C++11 std::string is not optimized for us. std::string uses a small buffer optimization to store small strings inline without allocating memory. The problem is we have a lot of smaller or empty strings, but all of them consume 32 bytes.

Some kind of replacement for std::string would likely get us large memory savings all around.

[mateofio]

Here is a WIP branch that implements an RPG::String and uses it for EventCommands::string. This string type stores a single pointer and behaves like a dynamically allocated const char*.

https://github.com/fmatthew5876/liblcf/tree/string

With this, sizeof(EventCommand) goes from 64 down to 40.

With this branch:

Game	Master LDB Mem	branch LDB Mem	Savings
HH3	136.4MB	94.1MB	31%
Heroes Realm	300MB	202.3MB	33%
Violated Heroine	75.9MB	54.8MB	28%

[mateofio]

One more test before bed. Here is liblcf master build with -D_GLIBCXX_USE_CXX11_ABI=0. This is effectively like using my RPG::String everywhere. This macro enables the old pre-C++11 std::string which used copy on write. In this version, sizeof(EventCommand) == 40.

Game	LDB mem usage	Savings
HH3	97.3MB	29%
Heros Realm	205.3MB	32%
Violated Heroine	58.1MB	23%

Interestingly, using this string everywhere actually increases memory usage compared to using it only in EventCommand.

[Ghabry]

Interesting savings you have there just by replacing one class, wow.

As you have my test set of game files now:
I'm curious if there is any other game which has a high LDB-memory usage that is not caused by event commands which is worth taking a look at.