Handling big genomes

[celine-mercier]

Hello,

I am looking at using JBrowse2 to work with our assembly of the Pinus radiata genome which is around 25 Gbp. Eventually we might want to develop plugins for new tracks etc depending on our specific needs. I have never developed in JavaScript though (I’m mostly used to C and Python), so for now I’m just trying to get a feel of what’s possible.

I installed a development server as per the instructions, and loaded the bgzipped indexed fasta data with our current scaffolds, but get an “Out of Memory” if I try to load the entire assembly.

What would be the best way to deal with a big genome like this one? Could I modify the code to handle the memory differently? Or would that be impossible or too much of a task, especially for someone who’s never touched JavaScript? Then I guess we would just stick to displaying the genome in smaller pieces.

Cheers,
Celine

[garrettjstevens]

Hi @celine-mercier, thanks for trying out JBrowse 2! I was wondering if we could get a bit more information to best know how to help. First of all, how many scaffolds are in the FASTA you are using?

When you say you get an out of memory error, at what point does the error show up? Does the app not load at all, or does it load but then crash when you try to open a view?

Thanks,
Garrett

[celine-mercier]

Hi!

Thanks for your answer.
We have 318,699 scaffolds.

Here's what happens:

• 1. I choose "Linear Genome view"
• 2. I usually get a couple of "Page Unresponsive" warnings from Chrome that I ignore while the list of scaffolds is loaded I guess, but the 'scaffold menu' loads fine after a minute
• 3. Then I can choose a specific scaffold by typing its id in the 'Enter a sequence or locstring' menu and it loads fine
• 4. But if from the menu I click the "Show all regions in assembly" button, I get an "Out of Memory" error in the browser after about one minute.
• I also tried loading another genome of similar size just to check (https://www.ncbi.nlm.nih.gov/assembly/GCA_000404065.3/), but I get the "Out of Memory" error at step 2 if I choose this assembly in the drop down menu, meaning I can't even load the scaffold menu (too many scaffolds?).

Does that help? Thanks,
Celine

[garrettjstevens]

Thanks for that info, and for providing the link to another large genome, it was very helpful. I also loaded up the Pinus taeda assembly and got the same error at step 2. The number of scaffolds/contigs/chromosomes in the FASTA will likely have more of an effect on the performance of JBrowse than the actual sizes of those scaffolds/contigs/chromosomes, and since that Pinus taeda assembly has ~1.8 million scaffolds, JBrowse definitely struggles.

The "typical" usage of JBrowse, though, is to have a single scaffold being displayed at a time and to switch between the scaffolds using the menu at the top of the Linear Genome View. The "Show all regions in assembly" option is really for more of a special case where you need to see the whole assembly on the screen at once, but with hundreds of thousands of scaffolds, I don't think that type of display would be very useful for you. So since you are able to select a scaffold in the "Enter sequence or locstring" box, you should still be able to use JBrowse in the way it's most commonly used.

That being said, having the page become unresponsive and taking a long time to load isn't the best experience. I've created an issue here to help us remember to track handling this better as a feature request: #1847

We welcome contributions to our code, but since this is somewhat involved internal part of the architecture, I don't anticipate it would be a quick fix.

Let us know if you have any more questions.

[celine-mercier]

Thank you, that's a helpful answer. Do you have any idea how long a sequence has to be before it starts being an issue?

[cmdcolin]

@celine-mercier we found a fix for this issue recently in #1847 and will probably get this into the next release...it can sometimes still be slow just due to bandwidth of downloading the large .fai file upfront but it shouldn't freeze the program now to use an assembly like this.

[celine-mercier]

Sounds good, thanks!

[cmdcolin]

Should be fixed in next release after fix from #2060