Executing query leadership-schedule is killing the node because of the lack of RAM

[dorin100]

Internal/External
Internal if an IOHK staff member.

Area
Other Any other topic (Delegation, Ranking, ...).

Summary
When running the query leadership-schedule command, on some machines (with 16GB of RAM) synced to Mainnet, the processing of the CLI command is consuming all the RAM and the node is killed.

Even more, after restarting the node, it needs to validate the DB and it is also a longer restart.
Is there anything we can do to limit the RAM consumption by this CLI command and avoid eating all the RAM (that is killing the node)?

Steps to reproduce
Steps to reproduce the behavior:

1. Start and sync the node on Mainnet using a Linux machine with 16GB of RAM
2. Execute the query leadership-schedule command --> RAM usage goes to 100% and the node is killed by the OS because of the lack of RAM

Expected behavior
Processing the CLI command should not kill the node/other processes.

System info (please complete the following information):

• OS Name: [e.g. Ubuntu]
• OS Version [e.g. 20.04]

cardano-cli --version
cardano-cli 1.33.0 - linux-x86_64 - ghc-8.10
git rev 54155a28dbdb9a17e4ffe52b68f6c58af54ff83b

Screenshots and attachments

Additional context

[CyberCyclone]

I experienced the same issue. 24GB of RAM was enough to run the leader-log. An equivalent total of 16GB RAM plus 8GB of Swap should avoid the issue. This isn't new as it happened with third party leader-log tools.

[rdlrt]

I think originally Kevin mentioned they can "expose more things through the consensus interface" as long-term solution, but I do not believe that was realised?
(PS: I refer to this note so that we're not looking at a simply decodeFull reference being changed to a map, but look at what can be done better for the original issue - being able to use this readily without parsing large amount maps on client)

All commonly known solutions Ogmios/DbSync/cncli/and now cardano-cli... have to go through extracting information indirectly from same source - resulting in lot of clientside processing and resources, not to actually get schedule information, but to extract details like pool's active stake from ledger).

IMO - Such hacks are not sustainable on client-side (not when processing full ledgerState, and not for too long when processing a sub-map) - unless we start using external dependencies where information is pre-processed (which a lot of us have already been doing).

Also, might help if QA/testing prior releases is done against mainnet data, so that such issues could be identified before release is marked (not necessarily to hold release back, but so that release notes can capture full story).

[adrem1]

Can confirm that query leadership-schedule is a resource hog. But as said above, this was true for third party tools achieving the same result.

At this stage 24GiB RAM is enough to run this command (testing has shown 20.8GiB average used while also running the node).

Maybe slightly off-topic here, but 3rd party tools for this purpose allow flagging specific time zones, making the output more readily useful to the SPO. Perhaps a feature to be added for future versions?

[regel]

Lower memory consumption, and getting the information right away via the IPC socket would be very much appreciated- if it’s technically feasible.

I’ve been maintaining open source Kubernetes helm charts for Cardano for the past nine months. Changing minimum RAM requirements make capacity planning more difficult, and also more costly.

[Jimbo4350]

Closing this. If this is still relevant please reopen.

[dorin100]

this is still relevant

[newhoggy]

Did this fix the problem? #4250

[SteffenKeller]

Still facing this problem with 16GB RAM and 8GB swap

[newhoggy]

What does cardano-cli version and cardano-node version say?

[SteffenKeller]

Using node version 1.35.5. Increasing the swap solves the problem.