Support Parallel computing #71
Conversation
|
thanks @geertvandeweyer cfor the PR. I am aware of it just found no time yet to look deeper into it. Maybe you can give some more explanations what was the intention with the rule specific threding limits. |
| @@ -14,6 +14,7 @@ where: | |||
| -q print basic information about snakemake run | |||
| -p print full information about the snakemake run | |||
| -d do not remove temporary directory for debug puroposes | |||
| -t specify location for temporary directory [default: /tmp/] | |||
There was a problem hiding this comment.
I thought about that too. But in tehory it can be controlled via the variable $TMPDIR
Using you code will change the default behaviour creating always a tmp folder under /Ttmp/ and not in $TMPDIR if set (or not specified via -d)
There was a problem hiding this comment.
ok, that makes sense. you might default to $TMPDIR as well, as I actually wasn't aware of $TMPDIR :-)
| # wildcard_constraints: | ||
| # file=".*(?<!\\.vcf)$" | ||
| #wildcard_constraints: | ||
| # basefile="[^/]+" |
There was a problem hiding this comment.
that's some leftover i missed. It's commented in both version, so no impact. but it can be reverted
Sorry for the late reply. The rule specific threading was setup rather empirically. Some threads used high memory, so there thatwas limit (hence the resource scaling); for others (I think vep), it was mainly disk I/O. So the rules I had max'd out my SSD setup. Any higher for those would bring I/O into massive wait states... I'm not saying it's a good for all set, but it might be worth adding options to tune the threading per step somehow ? |
This PR includes:
For benchmarking, see : https://hub.docker.com/repository/docker/cmgantwerpen/cadd/general