singlem threads and 16

[wwood]

Hey,

On a large-ish metagenome (~20Gbp) singlem_pipe_reads timed out in the queue. Since singlem pipe can make use of >1 thread (though tbf, improvements could be made), how come it is restricted to 1 thread?

Also, how come the binning methods are set to a maximum of 16 threads with e.g.

    threads:
        min(config["max_threads"], 16)

? If -t is >16, why not use more?

[rhysnewell]

I believe this was implemented to try and get the binners and assemblers to run in parallel, but I agree 16 threads is not enough. Probably just best to leave it out how it originally was and let them use max threads instead.

I'll look into the singlem pipe reads threading issue

[wwood]

Pipe seems to be running with more threads when now. Maybe when queue submitted it is different? Sorry don't have time to investigate atm -\------------- Ben Woodcroft Group leader, Centre for Microbiome Research, QUT

…

________________________________ From: Rhys Newell ***@***.***> Sent: Monday, December 4, 2023 11:44:56 AM To: rhysnewell/aviary ***@***.***> Cc: Ben J Woodcroft ***@***.***>; Author ***@***.***> Subject: Re: [rhysnewell/aviary] singlem threads and 16 (Issue #178) I believe this was implemented to try and get the binners and assemblers to run in parallel, but I agree 16 threads is not enough. Probably just best to leave it out how it originally was and let them use max threads instead. I'll look into the singlem pipe reads threading issue ― Reply to this email directly, view it on GitHub<#178 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAADX5GUXLRWTMF7J7VRWHLYHUTJRAVCNFSM6AAAAABADO3NCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZXG4YDSNZVGU>. You are receiving this because you authored the thread.Message ID: ***@***.***>