Massive performance issue with new JabRef version

[AEgit]

JabRef 5.0-alpha--2019-12-11--c3b7ee7
Windows 10 10.0 amd64
Java 13.0.1

The current developer version of JabRef suddenly has a massive performance problem. Just starting JabRef with a database of ca. 20,000 entries leads to a CPU usage of 60 to 70% on a 6-core machine with hyperthreading. Indeed, while JabRef is running 5 out of 12 threads are running with 100% workload.
CPU load does not drop after a while but stays high all the time.

This might due to some changes that have been recently introduced.

Edit: CPU load does actually drop after a while - but it takes about 5 to 10 mins to drop.

[AEgit]

Note, that this differs from the performance problems reported herein:
#5071 (comment)

The CPU load in this case increases immediately when the JabRef database is loaded (without requiring further user interaction).

[AEgit]

Ok, update to the problem: The CPU loads drops to zero if you wait for 5 to 10 min after loading your JabRef database. Once the CPU load has dropped, the entry preview is visible again. The performance problem here is therefore likely related to the issue reported here:
#5735

[AEgit]

Update 2: As far as I can tell, this performance issue appears, if you open a JabRef database that was created with version 3.8.2, without resaving it in the new format used by version 5. If you save it in the new format, and open it afterwards, the CPU load will still shoot up, but only for a few seconds (which I am already used to from previous developer versions).

If you cannot reproduce the behaviour I report here, I can send you the respective JabRef databases (one saved in the old format, the other one in the new one). For various reasons, I do not want to make this database public, so if you need it, please let me know, how I can provide you with the database without making it available to everyone.

[tobiasdiez]

Do you know if this issue appeared recently, i.e. could you previously open the 3.8.2 file without any big performance issues?

[AEgit]

Fortunately I still had an older .msi installer left on my disk. I tried the following version:

JabRef 5.0-alpha--2019-12-04--5dcb565
Windows 10 10.0 amd64
Java 13.0.1

With this version, CPU load stays high only for 30 sec with the 3.8.2. file (I think JabRef mainly spends this time calculating the number of elements each group has; note, that my database also contains thousands of static groups). So the issue must have been introduced somewhere between version JabRef 5.0-alpha--2019-12-04--5dcb565 and version JabRef 5.0-alpha--2019-12-11--c3b7ee7

[Siedlerchr]

Another general performance issue seems to be related to sorting by author. While testing #5730 I started profiling and noticed that the main table names formatting is called and seems to be taking a long time, nearly 1sec. Tested with a huge bib file and 6,400 entries

JavaFX Application Thread  Runnable CPU usage on sample: 937ms
  java.util.prefs.WindowsPreferences.WindowsRegOpenKey(long, byte[], int) WindowsPreferences.java (native)
  java.util.prefs.WindowsPreferences.WindowsRegOpenKey1(long, byte[], int) WindowsPreferences.java:182
  java.util.prefs.WindowsPreferences.openKey(byte[], int, int) WindowsPreferences.java:534
  java.util.prefs.WindowsPreferences.openKey(int, int) WindowsPreferences.java:515
  java.util.prefs.WindowsPreferences.openKey(int) WindowsPreferences.java:501
  java.util.prefs.WindowsPreferences.getSpi(String) WindowsPreferences.java:680
  java.util.prefs.AbstractPreferences.get(String, String) AbstractPreferences.java:303
  java.util.prefs.AbstractPreferences.getBoolean(String, boolean) AbstractPreferences.java:531
  org.jabref.preferences.JabRefPreferences.getBoolean(String) JabRefPreferences.java:1011
  org.jabref.gui.maintable.MainTableNameFormatter.formatName(String) MainTableNameFormatter.java:23
  org.jabref.gui.maintable.FieldColumn.computeText(BibEntryTableViewModel) FieldColumn.java:70
  org.jabref.gui.maintable.FieldColumn.lambda$3(BibEntryTableViewModel) FieldColumn.java:52
  org.jabref.gui.maintable.FieldColumn$$Lambda$1360.call()
  javafx.beans.binding.Bindings$7.computeValue() Bindings.java:398
  javafx.beans.binding.StringBinding.get() StringBinding.java:150
  javafx.beans.binding.StringBinding.get() StringBinding.java:60
  javafx.beans.binding.StringExpression.getValue() StringExpression.java:51
  javafx.beans.binding.StringExpression.getValue() StringExpression.java:47
  javafx.scene.control.TableColumnBase.getCellData(Object) TableColumnBase.java:686
  com.sun.javafx.scene.control.TableColumnComparatorBase.compare(Object, Object) TableColumnComparatorBase.java:59
  javafx.collections.transformation.SortedList$ElementComparator.compare(SortedList$Element, SortedList$Element) SortedList.java:299
  javafx.collections.transformation.SortedList$ElementComparator.compare(Object, Object) SortedList.java:288
  com.sun.javafx.collections.SortHelper.mergeSort(Object[], Object[], int, int, int, Comparator) SortHelper.java:271  <8 recursive calls>
  com.sun.javafx.collections.SortHelper.sort(Object[], int, int, Comparator) SortHelper.java:96
  javafx.collections.transformation.SortedList.doSortWithPermutationChange() SortedList.java:185
  javafx.collections.transformation.SortedList$1.invalidated() SortedList.java:132
  javafx.beans.property.ObjectPropertyBase.markInvalid() ObjectPropertyBase.java:112
  javafx.beans.property.ObjectPropertyBase$Listener.invalidated(Observable) ObjectPropertyBase.java:234
  com.sun.javafx.binding.ExpressionHelper$SingleInvalidation.fireValueChangedEvent() ExpressionHelper.java:136
  com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper) ExpressionHelper.java:80
  javafx.beans.property.ReadOnlyObjectPropertyBase.fireValueChangedEvent() ReadOnlyObjectPropertyBase.java:74
  javafx.beans.property.ReadOnlyObjectWrapper.fireValueChangedEvent() ReadOnlyObjectWrapper.java:102
  javafx.beans.property.ObjectPropertyBase.markInvalid() ObjectPropertyBase.java:113
  javafx.beans.property.ObjectPropertyBase.set(Object) ObjectPropertyBase.java:147
  javafx.scene.control.TableView.setComparator(Comparator) TableView.java:1208
  javafx.scene.control.TableView.sort() TableView.java:1551
  javafx.scene.control.TableView.doSort(TableUtil$SortEventType, Object[]) TableView.java:1651
  javafx.scene.control.TableView.lambda$new$0(ListChangeListener$Change) TableView.java:545
  javafx.scene.control.TableView$$Lambda$1317.onChanged(ListChangeListener$Change)
  com.sun.javafx.collections.ListListenerHelper$Generic.fireValueChangedEvent(ListChangeListener$Change) ListListenerHelper.java:329
  com.sun.javafx.collections.ListListenerHelper.fireValueChangedEvent(ListListenerHelper, ListChangeListener$Change) ListListenerHelper.java:73
  javafx.collections.ObservableListBase.fireChange(ListChangeListener$Change) ObservableListBase.java:233
  javafx.collections.ListChangeBuilder.commit() ListChangeBuilder.java:482
  javafx.collections.ListChangeBuilder.endChange() ListChangeBuilder.java:541
  javafx.collections.ObservableListBase.endChange() ObservableListBase.java:205
  javafx.collections.ModifiableObservableListBase.setAll(Collection) ModifiableObservableListBase.java:90
  javafx.collections.ObservableListBase.setAll(Object[]) ObservableListBase.java:251
  javafx.scene.control.skin.TableColumnHeader.sortColumn(boolean) TableColumnHeader.java:868
  javafx.scene.control.skin.TableColumnHeader.lambda$static$14(MouseEvent) TableColumnHeader.java:272
  javafx.scene.control.skin.TableColumnHeader$$Lambda$899.handle(Event)
  com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(Event) CompositeEventHandler.java:86
  com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventType, Event) EventHandlerManager.java:238
  com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(Event) EventHandlerManager.java:191
  com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent(Event) CompositeEventDispatcher.java:59
  com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(Event, EventDispatchChain) BasicEventDispatcher.java:58
  com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(Event) EventDispatchChainImpl.java:114
  com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(Event, EventDispatchChain) BasicEventDispatcher.java:56
  com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(Event) EventDispatchChainImpl.java:114
  com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(Event, EventDispatchChain) BasicEventDispatcher.java:56
  com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(Event) EventDispatchChainImpl.java:114
  com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(Event, EventDispatchChain) BasicEventDispatcher.java:56
  com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(Event) EventDispatchChainImpl.java:114
  com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(Event, EventDispatchChain) BasicEventDispatcher.java:56
  com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(Event) EventDispatchChainImpl.java:114
  com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(Event, EventDispatchChain) BasicEventDispatcher.java:56
  com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(Event) EventDispatchChainImpl.java:114
  com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(Event, EventDispatchChain) BasicEventDispatcher.java:56
  com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(Event) EventDispatchChainImpl.java:114
  com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(Event, EventDispatchChain) BasicEventDispatcher.java:56
  com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(Event) EventDispatchChainImpl.java:114
  com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(Event, EventDispatchChain) BasicEventDispatcher.java:56
  com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(Event) EventDispatchChainImpl.java:114
  com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(Event, EventDispatchChain) BasicEventDispatcher.java:56
  com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(Event) EventDispatchChainImpl.java:114
  com.sun.javafx.event.EventUtil.fireEventImpl(EventDispatchChain, EventTarget, Event) EventUtil.java:74
  com.sun.javafx.event.EventUtil.fireEvent(EventTarget, Event) EventUtil.java:54
  javafx.event.Event.fireEvent(EventTarget, Event) Event.java:198
  javafx.scene.Scene$MouseHandler.process(MouseEvent, boolean) Scene.java:3862
  javafx.scene.Scene.processMouseEvent(MouseEvent) Scene.java:1849
  javafx.scene.Scene$ScenePeerListener.mouseEvent(EventType, double, double, double, double, MouseButton, boolean, boolean, boolean, boolean, boolean, boolean, boolean, boolean, boolean, boolean, boolean) Scene.java:2590
  com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run() GlassViewEventHandler.java:409
  com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run() GlassViewEventHandler.java:299
  java.security.AccessController.executePrivileged(PrivilegedAction, AccessControlContext, Class) AccessController.java:753
  java.security.AccessController.doPrivileged(PrivilegedAction, AccessControlContext) AccessController.java:391
  com.sun.javafx.tk.quantum.GlassViewEventHandler.lambda$handleMouseEvent$2() GlassViewEventHandler.java:447
  com.sun.javafx.tk.quantum.GlassViewEventHandler$$Lambda$1233.get()
  com.sun.javafx.tk.quantum.QuantumToolkit.runWithoutRenderLock(Supplier) QuantumToolkit.java:412
  com.sun.javafx.tk.quantum.GlassViewEventHandler.handleMouseEvent(View, long, int, int, int, int, int, int, int, boolean, boolean) GlassViewEventHandler.java:446
  com.sun.glass.ui.View.handleMouseEvent(long, int, int, int, int, int, int, int, boolean, boolean) View.java:556
  com.sun.glass.ui.View.notifyMouse(int, int, int, int, int, int, int, boolean, boolean) View.java:942
  com.sun.glass.ui.win.WinApplication._runLoop(Runnable) WinApplication.java (native)
  com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(int, Runnable) WinApplication.java:174
  com.sun.glass.ui.win.WinApplication$$Lambda$225.run()
  java.lang.Thread.run() Thread.java:830

[AEgit]

JabRef 5.0-beta.352--2020-01-18--49e8ee2
Windows 10 10.0 amd64
Java 13.0.2

Ok, in newer versions of JabRef 5.0 the performance problems has become much worse. Now no matter whether you use a database in old format (3.8.2) or in 5.0 format, the CPU load hits 100% all the time and does not seem drop at all. Furthermore, JabRef uses way more RAM (for a database with >19,000 entries it has a memory footprint of 8.5 GB).

EDIT: Actually, the problem is different and appears again to be related to the way the database is stored. If you open a database in the old 3.8.2 format OR a database saved with an older version of the JabRef 5.0 developer version the database will be shown (indicated by the asterisk) as having experienced some changes (despite no user interaction except for opening the database). If you save this database with the current dev version of JabRef 5.0, then close JabRef, and the re-open the newly saved database in JabRef, the performance problem I mentioned above does not appear anymore.
Instead, the behaviour returns to what has been described here: #5734 (comment)
and here:
#5734 (comment)
The CPU load stays high for a couple of seconds (I think it is like 30 secs or so), after which it drops and stays low. The RAM consumption is also down to 2 GB.

So, as a summary. For people migrating from older databases to the current JabRef 5.0 developer version (even if it is an older JabRef 5.0 database): The first time you open this database you should save it with the current JabRef version, otherwise you will experience massive performance problems. If you perform the save action, the performance is still not at the 3.8.2 level, but definitely much better than it was in the past for older JabRef 5.0 dev versions.

See also:
#5736

[tobiasdiez]

This should be fixed in the latest development version. At least opening a very big bib file in the 3.8.2 file version, leads to 30 sec of high CPU and 2 gb ram consumption. After this period, CPU goes down to almost 0 and ram also to about 1gb, which seems reasonable.
Could you please check the build from http://builds.jabref.org/master/. Thanks! Please remember to make a backup of your library before trying-out this version.

[AEgit]

JabRef 5.0-beta.354--2020-01-18--2612e22
Windows 10 10.0 amd64
Java 13.0.2

Cannot confirm when using a bib file stored in 3.8.2 format. The issues mentioned in #5734 (comment) persist. But as mentioned there as well, if you save this database in the 5.0 format and then re-open it, the performance is as described in #5734 (comment)

So, the behaviour of the older version JabRef 5.0-alpha--2019-12-04--5dcb565 (described here), which had a much better performance with files stored in 3.8.2 format, has NOT been restored.
But if the workaround is accepted, that the database has to be saved in 5.0 format, then this particular problem is, indeed, somewhat reasonably solved (you still have to wait ca. 30 secs to be actually able to use the database, but I reckon that might be fine for people).
It might be necessary, though, for people coming from older versions to know about this behaviour. Otherwise, if you just load your old database into JabRef 5.0 you will face an abyssmal performance as long as you do not save your database in the new format.