exceed 2^31-1 bytes in methylDB object during conversion to non-DB variant

[alexg9010]

People are running into an issue when trying to convert a big methlDB object into the in-memory variant.
This Code caused the explicit error:

> objDB=as(methDB,"methylBase")

Error in paste(tabixRes[[1]], collapse = "\n") : 
  result would exceed 2^31-1 bytes

I was able to pinpoint the problematic line (https://github.com/al2na/methylKit/blob/master/R/tabix.functions.R#L317) which is called when we are using a select or subsetting ([) call.

select/`[` --> headTabix --> getTabixByChunk --> tabix2dt --> fread(paste(tabixRes[[1]],collapse="\n"),"\n" )

, where tabixRes is a list, with one element per region. Each element of the list is a character vector representing records in the region.
The error occurs because size of the pasted string exceeds the length limit of strings (
https://stackoverflow.com/questions/53120436/error-in-pastev-collapse-n-result-would-exceed-231-1-bytes ) .

[al2na]

any idea how to fix it?

…

On Mon, Jan 14, 2019 at 12:29 PM Alexander Gosdschan < ***@***.***> wrote: People are running into an issue when trying to convert a big methlDB object into the in-memory variant. This Code caused the explicit error: > objDB=as(methDB,"methylBase") Error in paste(tabixRes[[1]], collapse = "\n") : result would exceed 2^31-1 bytes I was able to pinpoint the problematic line ( https://github.com/al2na/methylKit/blob/master/R/tabix.functions.R#L317) which is called when we are using a select or subsetting ([) call. select/`[` --> headTabix --> getTabixByChunk --> tabix2dt --> fread(paste(tabixRes[[1]],collapse="\n"),"\n" ) , where tabixRes is a list, with one element per region. Each element of the list is a character vector representing records in the region. The error occurs because size of the pasted string exceeds the length limit of strings ( https://stackoverflow.com/questions/53120436/error-in-pastev-collapse-n-result-would-exceed-231-1-bytes ) . — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#141>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAm9EQkzYS1lzL8Ke0_ZsNBXHR89csMWks5vDGn_gaJpZM4Z-EXC> .

[alexg9010]

My idea is to replace headTabix/ getTabixByChunk with applyTabixByChunk in all functions that fetch the whole methylDB into memory, such that this will be done by chunks (of 1e6 lines). This way we should not reach the string length anymore.

[al2na]

maybe we can look at what these guys are doing: https://www.rdocumentation.org/packages/seqminer/versions/6.7/topics/tabix.read.table

…

On Tue, Jan 15, 2019 at 10:18 AM Alexander Gosdschan < ***@***.***> wrote: My idea is to replace headTabix/ getTabixByChunk with applyTabixByChunk in all functions that fetch the whole methylDB into memory, such that this will be done by chunks (of 1e6 lines). This way we should not reach the string length anymore. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#141 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAm9EQsDai_mTrIcC_rBK_cKPeanHYW9ks5vDZz-gaJpZM4Z-EXC> .

[al2na]

you can also look at the example in rsamtools, I don't know how fast it would be, they use map instead of reading everything at once. res <- scanTabix(tbx) dff <- Map(function(elt) { read.csv(textConnection(elt), sep="\t", header=FALSE) }, res) this would be the easiest to change tabix2dt, tabix2df and tabix2gr functions with map. This way we will not get into this problem. I don't know if it would be faster than depending on seqminer which is C/C++ based for reading the tabix to data frame. But it is the easiest option

…

On Tue, Jan 15, 2019 at 3:06 PM Altuna Akalin ***@***.***> wrote: maybe we can look at what these guys are doing: https://www.rdocumentation.org/packages/seqminer/versions/6.7/topics/tabix.read.table On Tue, Jan 15, 2019 at 10:18 AM Alexander Gosdschan < ***@***.***> wrote: > My idea is to replace headTabix/ getTabixByChunk with applyTabixByChunk > in all functions that fetch the whole methylDB into memory, such that this > will be done by chunks (of 1e6 lines). This way we should not reach the > string length anymore. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#141 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAm9EQsDai_mTrIcC_rBK_cKPeanHYW9ks5vDZz-gaJpZM4Z-EXC> > . >

[al2na]

@alexg9010! @katwre uses data.table::fread to read the whole tabix file into memory, maybe it it is something to replace scanBam when the user wants to read the whole file.

[katwre]

yes, I sometimes use data.table::fread to read even really big tabix files (1.7G) and it's quite fast, e.g.:

fread('zcat ~/methylBase_meth.deT.nodups.txt.bgz')

[alexg9010]

yes, I would like to stay with fread too, and we are already using it extensively.
tabix2dt and alikes where supposed to only read chunks and not the whole files, so I think we need to limit their use to those cases.

[alexg9010]

should be fixed in aa6a0d1