Load wallet file very slow

[EWIT521]

The process of serializing the contents of the wallet file into memory objects during the wallet file loading process at program startup is very slow, and a 600M file takes two hours.
Why is this taking so long? Is this normal? If not, what is causing the delay?

[selsta]

What kind of hardware do you have? I don't have a 600MB wallet cache file but a 60MB one takes less than a second to load. Two hours is definitely not normal.

[selsta]

I have tried a 600MB wallet file now, doesn't take more than 2-3 seconds. Are you sure that you have enough free RAM?

[EWIT521]

The machine has 32 GB of memory, and during the serialization process there seems to be an issue. Can you provide any insight or suggestion as to what might be causing this?
log:
2023-02-12 02:18:32.202 7f1bda7e7640 INFO wallet.wallet2 src/wallet/wallet2.cpp:5474 Trying to decrypt cache data
2023-02-12 02:18:32.497 7f1bda7e7640 INFO wallet.wallet2 src/wallet/wallet2.cpp:5478 wallet2::load----serialization::parse_binary end size :626166864
2023-02-12 02:18:43.950 7f1bda7e7640 INFO wallet.wallet2 src/wallet/wallet2.cpp:5482 wallet2::load---- cache_data--decrypt end
2023-02-12 02:18:43.951 7f1bda7e7640 INFO wallet.wallet2 src/wallet/wallet2.cpp:5489 wallet2::load---- cache_data-- serialize 111

[selsta]

Which wallet are you using? Which version are you using? Where did you get the binaries?

[EWIT521]

Problems with slow file loading:
The loaded file is a data wallet file containing transaction information;
monero-wallet-cli,The version is monero-x86_64-linux-gnu-v0.18.0.0;
The file source is the monero-wallet-cli export

[selsta]

Does the same issue exist with v0.18.1.2?

[EWIT521]

Not tried v0.18.1.2

[EWIT521]

Using version is v0.18.0.0 ,Have not tried v0.18.1.2

[EWIT521]

The reason for the slow loading of the wallet has been found. It is because m_subaddresses and m_subaddress_labels have 10.7 million records each, while the actual subaddress is only more than 50,000. How should it be optimized?

[selsta]

Is this an exchange wallet?

[EWIT521]

Only one wallet file that contains all wallet addresses, self-created and exchange

[moneromooo-monero]

#5370 is likely to speedup loads, it uses a flat vector to store subaddresses. Doesn't do anything for labels, but since most labels are probably empty, I'll make another patch that omits those altogether.
5370 is years old, doesn't apply anymore, I'll push a fresh version soon.

[moneromooo-monero]

5370 is now rebased to current master. Let us know if you get any speedup. I'll look at what we can do for labels now.

[moneromooo-monero]

BTW, this will make your wallet cache incompatible with current master or release, so only use this on a COPY of your wallet cache.

[moneromooo-monero]

Oh, and you want to try loading it with this code after it's been saved with this code, so it then loads the new format, not the old one plus conversion.

[moneromooo-monero]

Pushed again with serialization fixes, serialization changed to use monero instead of boost since 2019.

[rbrunner7]

The reason for the slow loading of the wallet has been found. It is because m_subaddresses and m_subaddress_labels have 10.7 million records each, while the actual subaddress is only more than 50,000. How should it be optimized?

I don't understand. Do you mean you definitely do not have 10 million subaddresses, not even remotely that many, only somewhat over 50,000, yet if you look at the map and the vector, they have 10 million elements each?

If yes, something does not square here, if you ask me.

[moneromooo-monero]

Actually, it doesn't work yet, can't load old wallet caches. I'll fix.

[moneromooo-monero]

Fixed.

[moneromooo-monero]

That said, I tried with a 900 MB cache (5k accounts, 5k subaddresses per account), and it loads in less than 30 seconds with current master. So that doesn't seem to be your problem.
Run "sudo perf top -a" while loading, paste the first dozen lines after waiting for it to settle.

[moneromooo-monero]

In fact, on that 5k/5k wallet, loading the subaddress data takes very little time in the first place, after adding timing code.

[moneromooo-monero]

I tried a similar thing for labels, but it turns out it gains nothing, as labels are already implicitely optimized out.

[EWIT521]

Thanks for all the replies above，I'll run the test again and see the result

[EWIT521]

Using Java to invoke an xmr so library to load a wallet file and serialize 100 million sub-addresses is very slow.

[moneromooo-monero]

Wait. Is it our code you're using or someone else's ? Just making sure :D

[EWIT521]

you have a Discord account for convenient Do chatting?

[moneromooo-monero]

No, IRC on the libera.chat network in #monero is appropriate. It's accessible via matrix.

[EWIT521]

Modify the monero code compilation output a Library ,then java integration that ,loading monero wallet very slow

[EWIT521]

When the Java calls monero library of SO to load the wallet file, the prompt loading failed malloc. Memory allocation failed, then through ulimit to modify the max lockmemory size, change 64 to 102400. After modification, the wallet can be loaded normally, but it is very slow when serializing 10 million sub-addresses.

[EWIT521]

system's The memory is adequate.

[EWIT521]

The program started running, and the process of serializing 10 million subaddresses while loading the wallet was very slow, taking 2 hours and 9 minutes
Here is the code and log
When serializing 10 million sub-addresses, print the first ten logs and the last ten logs
The first and last ten times take 1 millisecond respectively
Why is serialization so slow?
Is there a problem with the serialization process?
Is there a problem with the stored data of the subaddress?

[moneromooo-monero]

You were asked two things (try with the patch linked, and post the output of perf top), which you ignored. Asking the same thing again will not help. The perf top output is likely the one most likely to help determine where the issue lies.