High CPU Consumed

[BawdyAnarchist]

Description

I have been using Bisq for a couple years, on Qubes exclusively. I have always had problems with very high CPU usage (60-99%) to run the app. Very delayed entry of password. Long delays between clicking a button and the popup window. Even longer for the text in popup windows to show. Every 10 to 60 seconds it spins itself up to nearly max CPU for 10-30 seconds.

Synced to my own node. Bisq DAO turned off. 4GB RAM allocated to a dedicated Virtual Machine

Version

All .deb
1.3.2
1.3.4
1.3.6
And all versions since Jan 2019

Steps to reproduce

Start the application, watch it consume nearly max CPU for 2-4 minutes just to start, allow it to sync, once synced, it still continuously cycles near max CPU. Even my Bitcoin Node doesn't consume these resources.

Expected behaviour

Not spin up any CPU if not doing anything. Low delay time

Device or machine

Purism 15v4, Qubes 4.0.3, Whonix-15 virtual machine

[boring-cyborg]

Thanks for opening your first issue here!

Be sure to follow the issue template. Your issue will be reviewed by a maintainer and labeled for further action.

[ghost]

I still have problems with Firefox crashing regularly, and graphical heavy applications consuming like 50-90% of my CPU. I think this is unavoidable because of the lack of OpenGL (graphics libraries) in dom0, which Qubes devs say is a big security vulnerability, please let me know if you have a solution for this.

Sounds like you have a handle on the problem already! Seriously though, memory problems are the biggest issue in Bisq UI at the moment and unfortunately seem to be inherent to the underlying (GUI) JavaFX library. For instance if you run the command line version of Bisq (seed_node) it consumes only 10% of the memory than the one that includes a GUI.

The most promising solution would be to use Bisq as a daemon running on the same box as your fullnode, and connect to its API via a lightweight client. This is not ready, but coming soon! ^(TM)

[BawdyAnarchist]

Firefox as a web browser is significantly more complex than Bisq, yet consumes far less CPU, unless I have half a dozen charting tabs open, or streaming videos. And graphically speaking, Bisq, by comparison is far less complicated. I can also regularly run Firefox with a couple tabs of say, Github, reddit, or Facebook, rarely breaking 30% CPU, and only when I'm loading a page.

I say all of that to prevent anyone from just hand waving this away. Just sitting there doing nothing in a totally separate workspace should not be consuming 90% of my CPU every 20 seconds. I had a similar problem with the Monero wallet, and they replied with a solution related to QML_SCENE within 15 minutes, which totally fixed the issue.

I just now tried allocating 12GB of memory to my BTC Node / Bisq VM, bu that did very little. My password, from the moment the popup arrived, to actually accepting the input, was about 2-3 minutes total. Typed the password and hit enter, and watched for 60 more seconds for Bisq to catch up ... character .... by ......... character ..... by ......[you get the idea].

To sum up: Yes it's inconvenient not having OpenGL, but this is not a showstopper for other apps I use that are far more graphically intensive than Bisq. I personally would find it preferable to fix, I'm guessing a not insignificant number of people use Bisq on Qubes. But I also admit I have no idea how difficult this problem would be understand root cause and fix.

I suppose I can abandon the GUI and go CLI if the GUI ever becomes totally unusable for me.

[chimp1984]

I still have problems with Firefox crashing regularly, and graphical heavy applications consuming like 50-90% of my CPU. I think this is unavoidable because of the lack of OpenGL (graphics libraries) in dom0, which Qubes devs say is a big security vulnerability, please let me know if you have a solution for this.

Sounds like you have a handle on the problem already! Seriously though, memory problems are the biggest issue in Bisq UI at the moment and unfortunately seem to be inherent to the underlying (GUI) JavaFX library. For instance if you run the command line version of Bisq (seed_node) it consumes only 10% of the memory than the one that includes a GUI.

The most promising solution would be to use Bisq as a daemon running on the same box as your fullnode, and connect to its API via a lightweight client. This is not ready, but coming soon! (TM)

Those issues are all related to Linux versions. The RAM usage is likely a JVM setting issue, I have not followed the discussions but I think some people have found already something. I dont know about Qubes but might be that the JVM/JFX implementation has issues there.

[cd2357]

Did you setup the qube as described in #4386 (or similar)? Particularly relevant are the RAM and CPU settings.

[crocket]

I also see a lot of CPU consumption of bisq GUI on gentoo linux after manually building bisq with JDK 11.
It's so slow that it's not usable. I ran bisq 1.5.1

Perhaps, using java GUI was a mistake?

[cd2357]

Make sure you have enough free RAM available (about 4GB). Also have a look at https://bisq.wiki/Performance_Tips

[crocket]

I have 32GB RAM.

[chimp1984]

@crocket If you cpu is a more or less decent one there should not be a problem. I can run several bisq nodes without issues (OSX). I guess it some local config issue on linux. Also 1 GB ram is normal that bisq consumed, not much more. Max. 2 GB. Java takes what it gets from the OS so if u have 32GB it takes more, but that if there is not enough ram it can run even with 500 MB (as i saw when i started 10-20 apps) and it still worked just getting slower when switching between apps.

There have been a GH issue with recommended settings for Linux. Should be added to the wiki page...

[chimp1984]

Sounds like you have a handle on the problem already! Seriously though, memory problems are the biggest issue in Bisq UI at the moment

I think that is mainly issue on Linux. We really should collect the information gathered to solve that. I think it was some JVM setting which has on Linux a weird default value.
On OSX with a 4 years old Imac i can run easily 3 apps in parallel and all are responsive and RAM is usually 1-2 GB.

[chimp1984]

@ghubstan has investigated it while back can here was his conclusion:
#3918 (comment) (see earlier posts for more background)

@ghubstan: Are those recommended settings confirmed/tested? If so @ripcurlx can we get those added to the linux builds? If not, we should add it to the https://bisq.wiki/Performance_Tips page.

Here was some other investigation:
#3430 (comment)

[crocket]

A few minutes after startup, bisq-desktop stops consuming a lot of CPU resource and its GUI becomes useable.
But, the GUI is not nearly as responsive as firefox or brave web browser.

[chimp1984]

Have you tried the settings described in above links?

[crocket]

Are you telling me to assign more RAM to JVM?

[chimp1984]

No, its about limiting RAM.
@ghubstan Could you make a summary of the recommeneded settings from your investigations?

[ghubstan]

From @crocket

A few minutes after startup, bisq-desktop stops consuming a lot of CPU resource and its GUI becomes useable.
But, the GUI is not nearly as responsive as firefox or brave web browser.

A long chain sync at startup locks up the UI thread until it's done, gobbles off heap memory, and doesn't give it back to the OS until Bisq is shutdown. The most impolite off heap memory usage happens when you start Bisq on a new, completely empty data directory, or when starting Bisq after an extended period of time of not running it -- it has to perform another long chain sync.

This is a problem I have not been able to solve.

The only way I know of giving the large amount of off heap memory Bisq used to sync the chain back to the OS:
Shut Bisq down after a long sync is complete, then restart it. The UI will be more responsive after starting it the 2nd time.

From @crocket

Are you telling me to assign more RAM to JVM?

Don't fiddle with the jvm ram options. I'll expain more in some of the following comments.

[ghubstan]

I spent a lot of time in Feb 2020 profiling Bisq's RAM usage on Linux (Ubuntu), followed by some conclusions detailed here.

Long story short, I found three types of problems, but no perfect solutions yet.

1. Uncontrollable off heap memory use by Java / JavaFX based app.
   Any Java/JFX based UI is a greedy beast, and if you don't configure the jvm's maximum RAM limit -- how much the jvm 'thinks' it has -- it may bring your machine to a halt as it tries to use more ram than your machine actually has. This is not a generalization about all Java UIs, but does seem to be generally true for JavaFX on Linux, regardless of whether you use mesa or gpu drivers. On the other hand, reducing your JVMs max ram limit too much risks an earlier Java app OOME than would normally happen. This leaves me with having to choose between two options: which do I want to see run out of memory first, my Bisq instance or my computer?
   To control Bisq's off heap memory use, we decided to start Bisq on Linux and OSX with this jvm option: -XX:MaxRAM=4g". That means Bisq's JVM normally uses about 25% of the maximum RAM limit, or about 1 Gb. The upside is Bisq users' computers are not liable to crash because of Bisq, but it increases the risk Bisq may run out of memory, and throw an OutOfMemoryException.
   See the Setting JVM's MaxRAM section of this comment for a few details.

2. A long chain sync at startup locks up the UI thread until it's done, gobbles off heap memory, and doesn't give it back to the OS.
   This is unavoidable (so far). I work around it like so: wait a few minutes (don't touch the UI) while my Bisq installation finishes the chain download (check log), then shut down and restart Bisq. The UI will become responsive more quickly after the restart.

3. There are numerous and serious memory leak problems in JFX 11, and Bisq is stuck with it until we can upgrade it with the migration to JDK 15 (in progress). See the section below about JavaFX Memory Leaks .

Setting JVM's MaxRAM

My OpenJDK 11 JVM is configured with a default MaxRAM setting of 128 Gb, while my machine has only 32 Gb.

To check yours...

java -XX:+PrintFlagsFinal -version | grep MaxRAM
 uint64_t MaxRAM                                   = 137438953472                           {pd product} {default}
    uintx MaxRAMFraction                           = 4                                         {product} {default}
   double MaxRAMPercentage                         = 25.000000                                 {product} {default}

I show how adjusting (lowering) the jvm's MaxRAM can affect memory usage here.

Today, the bisq-desktop startup script sets this value to 4Gb by including the setting on the java command: DEFAULT_JVM_OPTS="-XX:MaxRAM=4g"

JavaFX Memory Leaks

One of the major sources of these problems is the JFX 11 dependency. The upgrade to JDK 15 will allow us to also upgrade Bisq's JFX dependency, which I do hope will not be as leaky. The newer JFX will have many leak related bug fixes -- and hopefully not too many new leak related bugs.

Pasted from an earlier comment on the subject.

Btw, this means Bisq (or any JavaFX app) could behave differently on different systems, depending on that system's video settings (graphic card hardware? graphics drivers? amount of shared RAM between video and system? etc -- don't know what could play a role, but smth to keep in mind when looking at performance).

JavaFX does behave differently with different GPUs. It does use native GPU drivers, and there have been many leak problems in JFX related to GPU and mesa drivers.

Here is a list of some jfx leak related issues -- old and new. You have to hunt around for those related to GPUs, and for issues about leak related bugs found and fixed after JFX 11 was released.

Issue 8188094 shows how to make JFX leak from a simple app running on Linux. The unresolved issue reports the problem in JDK/JFX 8/9, and I reproduced it running JFX 11 on Ubuntu + NVIDIA GPU in Feb. 2020, when I started working on this problem.

JFX 14 has some leak bugfixes, but Bisq is holding back on upgrading to OpenJDK 14 until the new java packaging tool is released from incubation to GA.

I did a lot of profiling to find the leaks (maybe you saw some of my comments in issue 3918) using JMC/JFR, JProfiler, async-profiler, bcc and bpftrace tools, under the assumption the problem was/is in Bisq and JFX. I'm sure there are leaks to find and fix, but the only firm conclusion my analysis supported was that Bisq's JVM was reserving far more resident and virtual memory at startup than the Linux JVM needs. (I could not find problems with Bisq's on-heap management.) The problem was more serious when it starts with an empty data dir, syncs the entire chain, and the JVM never gives back any memory (to the OS).

Well, that sucks.

Yes, it does. If anyone can advise on how solve these difficult problems, do tell! I would get right to work, but moving off Java is not an option in the near or medium term. I'm not going to spend time here defending or bashing Java because I'm more worried about my nice, fast GPU's native GPU driver that crashes my machine at random times, sometimes right after clicking a browser tab.

[chimp1984]

A long chain sync at startup locks up the UI thread until it's done

BitcoinJ is woing its work on a dedicated thread but as it takes 100% of available CPU it also slows down the userthread.

The DAO sync is running on userthread but the BSQ tx verification itself is super fast and the the problem. I think the problme is some client code reacting on state changes and doing expensive operations (e.g. lookups). There have been done some improvements already but I think there is more needed, but not a trivial task...

To isolate the problem to BitcoinJ sync one can add --daoActivated=false, so the DAO sync will not be done (whole DAO domain is not activated).

[chimp1984]

A long chain sync at startup locks up the UI thread until it's done, gobbles off heap memory, and doesn't give it back to the OS until Bisq is shutdown. The most impolite off heap memory usage happens when you start Bisq on a new, completely empty data directory, or when starting Bisq after an extended period of time of not running it -- it has to perform another long chain sync.

@oscarguindzberg Are you aware of any memory issues from BitcoinJ side at block download? Might be that it is only an issue on Linux (and maybe only at some distributions).

[chimp1984]

1. Uncontrollable off heap memory use by Java / JavaFX based app.

Have you tried with a headless app to see if it related to JavaFx?

[chimp1984]

3\. There are numerous and serious memory leak problems in JFX 11

Is that Linux only?
The ones I saw have been Linux related, but not sure if there are others affecting Windows (also common that ppl complain about low performance).
On OSX I do not remember to observe any leak. Tested various times with long running seednodes (headless) to find out what causes ram increase on seed nodes (running on linux OS). But never could reproduce growing RAM. In fact when I started a lot of instances (about 20) RAM goes down to about 500 MB - also with GUI apps, thought its a while since I done those tests, so not sure if its still the same...

[chimp1984]

@ripcurlx How long do you think it takes for the Java upgrade? Maybe we should increase priority and work force to get that out asap as those problems are repeating since too long and we lose for sure a lot of users by that.

[cd2357]

The DAO sync is running on userthread but the BSQ tx verification itself is super fast and the the problem. I think the problme is some client code reacting on state changes and doing expensive operations (e.g. lookups). There have been done some improvements already but I think there is more needed, but not a trivial task...

I suspect its the concurrent seednode connections via tor, that happen during "Synchronizing DAO". Last time I checked, tor connections go via some sort of io-blocking bottleneck. I noticed similar effects with the connections to the bitcoin nodes - connecting to them via clearnet caused less CPU overhead and was generally faster. Similar effect if I chose to use 1 tor bitcoin node, vs multiple (1 node was faster, smaller CPU spike). This was a few months ago, so not sure if it's still the case. But it might indicate that highly-active concurrent tor connections may cause/contribute to the CPU spike due to how multi-threading/IO is managed there.

[oscarguindzberg]

A long chain sync at startup locks up the UI thread until it's done, gobbles off heap memory, and doesn't give it back to the OS until Bisq is shutdown. The most impolite off heap memory usage happens when you start Bisq on a new, completely empty data directory, or when starting Bisq after an extended period of time of not running it -- it has to perform another long chain sync.

@oscarguindzberg Are you aware of any memory issues from BitcoinJ side at block download? Might be that it is only an issue on Linux (and maybe only at some distributions).

I am not.

[chimp1984]

The DAO sync is running on userthread but the BSQ tx verification itself is super fast and the the problem. I think the problme is some client code reacting on state changes and doing expensive operations (e.g. lookups). There have been done some improvements already but I think there is more needed, but not a trivial task...

I suspect its the concurrent seednode connections via tor, that happen during "Synchronizing DAO". Last time I checked, tor connections go via some sort of io-blocking bottleneck. I noticed similar effects with the connections to the bitcoin nodes - connecting to them via clearnet caused less CPU overhead and was generally faster. Similar effect if I chose to use 1 tor bitcoin node, vs multiple (1 node was faster, smaller CPU spike). This was a few months ago, so not sure if it's still the case. But it might indicate that highly-active concurrent tor connections may cause/contribute to the CPU spike due to how multi-threading/IO is managed there.

This is expected as Tor is slower and as the connections get clogged by multiple requests.
We have those connections streams:

• BitcoinJ block download (starts early and can finish any time after startup and app init)
• Initial data requests from multiple seeds (usually 2 if one does not respond more). After first response we start DAO block request, but other responses might come then parallel to the DAO block response
• Get BSQ blocks

So in the worst case we do 3 data requests at the same time: Btc blocks, Bisq data, BSQ blocks. Additional there are price feed requests (I think those are quite lite compared to the rest).

The initial data requests have been improved by 50% in 1.4.2 and in 1.5.2. we get anoute 25% less data (AccountAgeWitness data will be handled via HistoricalDataStore). So that will help overall to reduce bandwith requirements.

[ghubstan]

@chimp1984

1. Uncontrollable off heap memory use by Java / JavaFX based app.

Have you tried with a headless app to see if it related to JavaFx?

Yes, the headless daemon needs far less heap and off-heap memory. I've not seen a single OOME using the api daemon to sync the entire chain. But, I have never let the api daemon run for days or weeks.

I want to add (edit) here... if you remove that new MaxRAM=4g setting from the Linux startup and using Java Mission Control when starting the Bisq UI on an empty data dir, you will see very large memory allocations to the JFX main application thread that cannot be released until Bisq shutdown. Look at this screen shot showing the 66 Gb allocated to the jfx main app thread.

The virtual memory files without that jvm setting are also much larger.

Using the MaxRAM=4g helps control this, but does not solve JFX leak problem.

[ghubstan]

3\. There are numerous and serious memory leak problems in JFX 11

Is that Linux only?
The ones I saw have been Linux related, but not sure if there are others affecting Windows (also common that ppl complain about low performance).
On OSX I do not remember to observe any leak. Tested various times with long running seednodes (headless) to find out what causes ram increase on seed nodes (running on linux OS). But never could reproduce growing RAM. In fact when I started a lot of instances (about 20) RAM goes down to about 500 MB - also with GUI apps, thought its a while since I done those tests, so not sure if its still the same...

I think most of the complaints about JFX leaks are Linux related, and I can only talk about the problems I've seen while profiling on Linux. (I don't have a Windows machine.)

I have seen problems on OSX, but I have a mac-mini with 8Gb of ram and a slow processor; it is not a fair comparison.

[crocket]

Is there a way to make the thread run in the background? The priority of the thread should be low so that it doesn't slow down other programs.

[cd2357]

Could it be related to this? #4649 (comment)

From what I've seen, "Synchronizing DAO" (which shows the progress bar) is dirrectly correlated with UI sluggishness. Whether it happens on startup, or periodically while Bisq is running:

• as long as "Synchronizing DAO" is running, the UI is sluggish + CPU spikes
• as soon as it finishes, UI and CPU are back to normal

[ghubstan]

Is there a way to make the thread run in the background? The priority of the thread should be low so that it doesn't slow down other programs.

Doing the chain download off the main jfx application thread was attempted, but didn't work. (Not sure when it was tried -- a long time before I started working on Bisq.)

[chimp1984]

Is there a way to make the thread run in the background? The priority of the thread should be low so that it doesn't slow down other programs.

Doing the chain download off the main jfx application thread was attempted, but didn't work. (Not sure when it was tried -- a long time before I started working on Bisq.)

You refer to the BTC block download? That is threaded...

The DAO parsing is not the bottleneck, that is very fast and the risk and complexity to do that threaded was not worth the performance difference, in fact it was slower due threading overhead.

The problem is likely client code which gets executed on DAO state change, and some expensive methods (iteration, lookups). There have been done some improvements already but it requires more.

[ghost]

@BawdyAnarchist we disabled some CPU-intensive animations in the latest release - the "Synchronizing DAO" progress bar in the footer and the spinner shown when querying if an offer can be taken. Since you reported performance issues with Bisq, I wonder if you could try v1.5.5 and see if there is any improvement?
https://github.com/bisq-network/bisq/releases/tag/v1.5.5
#4649

[ghost]

I also see a lot of CPU consumption of bisq GUI on gentoo linux after manually building bisq with JDK 11.
It's so slow that it's not usable. I ran bisq 1.5.1
A few minutes after startup, bisq-desktop stops consuming a lot of CPU resource and its GUI becomes useable.
But, the GUI is not nearly as responsive as firefox or brave web browser.

@crocket we disabled some CPU-intensive animations in the latest release - the "Synchronizing DAO" progress bar in the footer and the spinner shown when querying if an offer can be taken. Since you reported performance issues with Bisq, I wonder if you could try v1.5.5 and see if there is any improvement?
https://github.com/bisq-network/bisq/releases/tag/v1.5.5
#4649

[crocket]

I just tested 1.5.5. CPU consumption is still high during synchronization, but other GUI programs don't stutter while bisq-desktop is synchronizing.

Detach synchronization from bisq GUI. You will see smoother bisq GUI. I think you can prevent synchronization thread from using the same CPU cores that bisq GUI uses.

[cd2357]

FYI: This should be further reduced in the most recent release (v1.6.4) which brings several UI performance improvements and generally reduces system resource consumption.