Performance Issues with Large Family Trees (8000+ members)

[pedroguedes]

I'm working on a project where we're using Topola-Viewer to visualize a large family tree that includes around 8300 family members. While the tool works well for smaller trees, we've been encountering significant performance issues when dealing with this large dataset.

Specifically, we've noticed the following problems:

• Slow Performance in "Hourglass" and "All Relatives" Views: These views become extremely sluggish, especially when navigating or zooming in/out.
• Unresponsive UI: The interface can become unresponsive or start glitching when viewing family members with a large number of relatives.

Has anyone else encountered performance issues with large family trees in Topola-Viewer?
Any insights or advice would be greatly appreciated. Thanks in advance for your help!

[syedatif]

I am interested in the response for this as I am building a family tree with about 29000 members.

[PeWu]

Hi,

Improving performance is something that is always a good direction.

My database has around 6k people and for me the performance is acceptable. I have already made performance improvements earlier. There are a few places where you can look for performance bottlenecks.

1. Loading and parsing the GEDCOM file. This is a one time operation and should not affect the responsiveness of the UI after data is loaded. This can be improved but parsing the GEDOM file is a long process by itself.
2. Looking up data and rebuilding the tree when refocusing. I don't think this is an issue but I don't have concrete evidence.
3. Large number of objects being animated during a transition and during zooming and panning. I think this is the main bottleneck for a large tree.

To reduce the number of visible objects on the chart, we could add a parameter that limits the number of generations being shown. Loading data from WikiTree already has limits on the number of generations shown mostly for limiting the number of requests to the API.

I'm not sure how to improve performance while not limiting the number of visible objects.

To be more "scientific" about performance improvements, it would be good to have a benchmark GEDCOM file or set of files of different sizes to test with. Do you know of a source where such file could be taken from? We could also generate such a file programatically. Or maybe take a large enough subset of WikiTree data?
Moreover, we could create a set of benchmarks measuring the time of various actions, like loading data, displaying the chart, changing the focused person, etc.

I am quite limited with the time I can dedicate to this effort but I can definitely help if you are interested in looking into this issue.

[KentGrigo]

There's a public dataset on the royal families in Europe here: https://webtreeprint.com/tp_famous_gedcoms.php
Performance-wise, the biggest issue on this dataset is zooming. Loading and panning isn't a problem for me.

For this dataset, there's also the issue of married cousins, which causes a big part of the tree to be duplicated many times.
So, only rendering a person once is also a way of optimizing some datasets. (Of course, this raises the question of: how should this be designed?)
See #30

[Bertg]

Here is one general strategy that would work very well, but I'm not sure how easily that would fit in the current code base:

The strategy renders an acceptable amount of data initially, and then the user can "expand" nodes as desired.

1. Render the person that is "focused"
2. Render "adjacent" relations (eg, parents, children, partners), keep track of the path that is rendered for each node.
3. For all newly rendered nodes render their "adjacent" relations, again keeping track of the full path for each.
4. Repeat those steps until either:
   • A node has already been rendered (eg, don't render the same subtree twice)
   • A pre-defined path (length) is achieved
   • No more nodes can be discovered
5. When a sub-tree is not rendered, render a "expand" button instead referring to the "path" that was taken

To be clear, these above steps should not actually "render" but instead create an in memory representation of what should be rendered. Also, depending on the visualisation type the definition of "adjacent" types will be different.

Note that in step 4 we can have different rules for different directions. Eg in the relatives chart we could choose to keep going "up" indecently, but don't allow any "down" directly after the grandparents. This would visualise your 1st degree cousins, uncles and aunts; and all your great grand parents, but not your more distant cousins.

Once rendered, when the user clicks on one of these "expand" nodes, we can take the in-memory representation, and repeat the algorithm, for that node. The reason we refer to these "expand" buttons by path is to prevent accidental explosion of the visualisation when the person appears multiple times in the tree.

If we want to be really fancy, when switching to an other person in focus, we could try to rebuild all the paths the user has expanded, and try to re-render those as well. This could prevent jarring "jumps" in the visualisation.