Sharing data for independent testing

[abhi18av]

Hi @TimHHH ,

I was wondering whether it'd be possible for you to share with me the data such as resources and reference files?

This way I can test the pipeline from end to end independently, preferably on 5 genome samples, and make sure that the results are as expected.

If possible, could you can share the data via OneDrive

[TimHHH]

Hi @abhi18av ,
I believe all the resources can now be found in the resources branch. I am assuming that you also require the fastqs, I will get these to you via OneDrive. Note that I will supply a couple more samples to ensure that VQSR behaves reasonably well.

[TimHHH]

You can find the fastqs here:
http://transfer.sh/qC7cUo/M_16_F0425_EC550A-330_Myco-F5_Mycobacterium_S32_L005_R1_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_F0425_EC550A-330_Myco-F5_Mycobacterium_S32_L005_R2_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_F0458_FS015-12_Myco-F5_Mycobacterium_S65_L005_R1_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_F0458_FS015-12_Myco-F5_Mycobacterium_S65_L005_R2_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_F0465_FS021A-289_Myco-F5_Mycobacterium_S72_L005_R1_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_F0465_FS021A-289_Myco-F5_Mycobacterium_S72_L005_R2_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_F0467_FS023-73_Myco-F5_Mycobacterium_S74_L005_R1_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_F0467_FS023-73_Myco-F5_Mycobacterium_S74_L005_R2_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_F0586_EC012B-270_Myco_F-7_Mycobacterium_ACCACTGT_L008_R1_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_F0586_EC012B-270_Myco_F-7_Mycobacterium_ACCACTGT_L008_R2_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_F0644_EC083-202_Myco_F-7_Mycobacterium_TCTTCACA_L008_R1_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_F0644_EC083-202_Myco_F-7_Mycobacterium_TCTTCACA_L008_R2_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_F0649_EC088-207_Myco_F-7_Mycobacterium_TTCACGCA_L008_R1_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_F0649_EC088-207_Myco_F-7_Mycobacterium_TTCACGCA_L008_R2_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_F0650_EC090-209_Myco_F-7_Mycobacterium_AACTCACC_L008_R1_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_F0650_EC090-209_Myco_F-7_Mycobacterium_AACTCACC_L008_R2_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_J0666_FS007_2_85_NFHT58_L006_R1_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_J0666_FS007_2_85_NFHT58_L006_R2_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_J0673_fs031_2_89_NFHT86_L006_R1_001.fastq.gz
http://transfer.sh/qC7cUo/M_16_J0673_fs031_2_89_NFHT86_L006_R2_001.fastq.gz
http://transfer.sh/qC7cUo/xbs-nf.test.csv
Note the new and improved sample sheet, this one can be included in the development branch.
Next time I'll combine the lot in one zip 😉 first time using transfer.sh.

[abhi18av]

This sounds great @TimHHH , appreciate it!

[abhi18av]

I've downloaded the files from there now.

[abhi18av]

Hi @TimHHH @LennertVerboven ,

Could you please share these FASTQ files with me again? I have lost access to the server I was using to test these, I might be able to access it again by tomorrow - but just in case, I'd rather have the data again as a backup.

[TimHHH]

Hi @abhi18av
Do you still need these files? Or have you managed to find a copy?

[abhi18av]

Yup, found a copy on my end - thanks!

[abhi18av]

I think that there are a couple resource remaining 🤔

1. resistance_db

https://github.com/abhi18av/xbs-nf/blob/07c8b73e037f89a2f821ff6d4e935e91b05d3ea8/conf/dev.config#L52

This is from the original, XBS_main.py (Line-79)

	parser.add_argument('--resistance_db', metavar='resistance_db', type=str, default='resistance_db_v1', help='The database to use for resistance calling')

2. Is this file https://github.com/abhi18av/xbs-nf/blob/master/resources/regions/UVP_List_of_Excluded_loci.list the same as DRgenes.list which is mentioned in XBS_call.sh (Line-140) ?

3. Similarly, there are 3 vcf files which I can't find.

    ref_gvcf="${projectDir}/data/test_data/reference_gvcfs_v02.g.vcf.gz"

    ref_gvcf_study=""

    dbsnp_vcf = "${projectDir}/data/test_data/tbvar.v3.vcf"

Or perhaps I've overlooked these and they were already included as part of #23 albeit named differently?

[abhi18av]

The GATK_BASE_RECALIBRATOR process, doesn't seem to have been used anywhere else.

[TimHHH]

Hi @abhi18av
Actually I like this piece of code and it would be nice to keep. You can simply use Benavente2015 as the dbsnp.
This piece of code should not be default and only activated with some argument stating that the dataset is uncontaminated, but I think this has already been implemented?

[abhi18av]

Yes, this is already implemented and is only activated when the dataset_is_not_contaminated is marked as true.

However, since there is no dbsnp_vcf it fails currently. Based on your comment, I'll add Benavente2015.UVPapproved.rRNAexcluded.vcf.gz as the default one - or do you wish to leave it to the user to use within their config?

[TimHHH]

Benavente2015.UVPapproved.rRNAexcluded.vcf.gz can be default.

[abhi18av]

Oki doki, I've made that one as the default.

dfd932b

[LennertVerboven]

For the first point; these databases have to be loaded in TBProfiler, this needs to happen during install using some reference files and a list of variants and their association. I can look up which files are required and send them to you. Lennert Verboven [logo_uantwerp] <https://www.uantwerp.be> [logo_torch] <https://torch-consortium.com/> TORCH - Tuberculosis Omics ResearCH consortium Global Health Institute Gouverneur Kinsbergencentrum - Room 1.16 Doornstraat 331 - 2610 Wilrijk - Belgium ***@***.*** ***@***.***>

…

________________________________ From: Tim H. Heupink ***@***.***> Sent: 04 November 2021 10:00:21 To: abhi18av/xbs-nf Cc: Lennert Verboven; Mention Subject: Re: [abhi18av/xbs-nf] Sharing data for independent testing (Discussion #20) CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. 1. resistance_db We have used two versions of this db, the original (?) TBprofiler version and Lennert's WHO version. @LennertVerboven<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FLennertVerboven&data=04%7C01%7Clennert.verboven%40uantwerpen.be%7C5594ae43d4e84656c06b08d99f7188c6%7C792e08fb2d544a8eaf72202548136ef6%7C0%7C0%7C637716132244709245%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9vvexEJY8lxZpr7JnrHcs0K0cMT8yYo4FGfBC%2BAvC38%3D&reserved=0> can you suggest which version is best to use? 2. Is this file https://github.com/abhi18av/xbs-nf/blob/master/resources/regions/UVP_List_of_Excluded_loci.list<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fabhi18av%2Fxbs-nf%2Fblob%2Fmaster%2Fresources%2Fregions%2FUVP_List_of_Excluded_loci.list&data=04%7C01%7Clennert.verboven%40uantwerpen.be%7C5594ae43d4e84656c06b08d99f7188c6%7C792e08fb2d544a8eaf72202548136ef6%7C0%7C0%7C637716132244719244%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5FM7p7aqh9T%2BmvP0U%2B5Xt2mCqkkwbsWatnpTCIeRbms%3D&reserved=0> the same as DRgenes.list which is mentioned in XBS_call.sh (Line-140) ? These are different, I'll locate DRgenes.list. 3. Similarly, there are 3 vcf files which I can't find. The are old redundant files. Code can be scrapped. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fabhi18av%2Fxbs-nf%2Fdiscussions%2F20%23discussioncomment-1587132&data=04%7C01%7Clennert.verboven%40uantwerpen.be%7C5594ae43d4e84656c06b08d99f7188c6%7C792e08fb2d544a8eaf72202548136ef6%7C0%7C0%7C637716132244719244%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ys9%2FTs647WEvl1L%2FT%2Fa3onJPRXBzK7KO0jtgim9soUQ%3D&reserved=0>, or unsubscribe<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACWGSFNVWZF5MZ6YQEAQQ43UKJDSLANCNFSM5GNJGIYA&data=04%7C01%7Clennert.verboven%40uantwerpen.be%7C5594ae43d4e84656c06b08d99f7188c6%7C792e08fb2d544a8eaf72202548136ef6%7C0%7C0%7C637716132244729243%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FauvBZBCqdp2zgoqF40TCkMBJVBQtYGCrk7pQrTTi4k%3D&reserved=0>. Triage notifications on the go with GitHub Mobile for iOS<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Clennert.verboven%40uantwerpen.be%7C5594ae43d4e84656c06b08d99f7188c6%7C792e08fb2d544a8eaf72202548136ef6%7C0%7C0%7C637716132244739232%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UcbCyPcDigl8NtGMII%2FC6aOibqZ6C30eTpq0kf1qDuI%3D&reserved=0> or Android<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Clennert.verboven%40uantwerpen.be%7C5594ae43d4e84656c06b08d99f7188c6%7C792e08fb2d544a8eaf72202548136ef6%7C0%7C0%7C637716132244739232%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=XS%2B%2B1O%2B19byscLyGC%2F7s%2B7KQtrMo%2BC1pTsSbyECs4Bs%3D&reserved=0>.

[TimHHH]

is the WHO database now in a usable format or do you prefer to stick to the standard TBprofilerdb? did you also not supplement the TB standard TBprofiler db?

[LennertVerboven]

WHO is not yes usable, the TBprofiler db is installed by default, Imade some additions, I will supply the required files and the the commands to load the db. You can always omit the db paramter and it will use the default TBProfiler one for testin and development. Lennert Verboven [logo_uantwerp] <https://www.uantwerp.be> [logo_torch] <https://torch-consortium.com/> TORCH - Tuberculosis Omics ResearCH consortium Global Health Institute Gouverneur Kinsbergencentrum - Room 1.16 Doornstraat 331 - 2610 Wilrijk - Belgium ***@***.*** ***@***.***>

…

________________________________ From: Tim H. Heupink ***@***.***> Sent: 04 November 2021 11:33:46 To: abhi18av/xbs-nf Cc: Lennert Verboven; Mention Subject: Re: [abhi18av/xbs-nf] Sharing data for independent testing (Discussion #20) CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. is the WHO database now in a usable format or do you prefer to stick to the standard TBprofilerdb? did you also not supplement the TB standard TBprofiler db? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fabhi18av%2Fxbs-nf%2Fdiscussions%2F20%23discussioncomment-1587651&data=04%7C01%7Clennert.verboven%40uantwerpen.be%7C8e1e12d0bd4641c2bb2608d99f7e95e8%7C792e08fb2d544a8eaf72202548136ef6%7C0%7C0%7C637716188293757561%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=X3d1vdLl8Qy6V6uxR6jOvoJW%2Fkn7aTfc4VPBLRdILxA%3D&reserved=0>, or unsubscribe<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACWGSFP6NO4RB2F344QHWMLUKJOQVANCNFSM5GNJGIYA&data=04%7C01%7Clennert.verboven%40uantwerpen.be%7C8e1e12d0bd4641c2bb2608d99f7e95e8%7C792e08fb2d544a8eaf72202548136ef6%7C0%7C0%7C637716188293767518%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=1gik6XWcc2nWU%2BaGblki1Gbobb8y0Xl5xizrEveClaU%3D&reserved=0>. Triage notifications on the go with GitHub Mobile for iOS<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Clennert.verboven%40uantwerpen.be%7C8e1e12d0bd4641c2bb2608d99f7e95e8%7C792e08fb2d544a8eaf72202548136ef6%7C0%7C0%7C637716188293767518%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=u2woM6jCZE%2F%2Bqk05c7eguXS%2FU%2BqWAfk2Q3jg5Nz0Fk4%3D&reserved=0> or Android<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Clennert.verboven%40uantwerpen.be%7C8e1e12d0bd4641c2bb2608d99f7e95e8%7C792e08fb2d544a8eaf72202548136ef6%7C0%7C0%7C637716188293777474%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=pswA8QoMT7HMtPx2Ora3HwVCFiYzMlyk2hsAkrrTDa0%3D&reserved=0>.

[abhi18av]

You can always omit the db paramter and it will use the default TBProfiler one for testin and development.

Ah, this is the crucial bit - it means this needs to be exposed as an optional database (file).

Thanks for letting me know @LennertVerboven 👍

[TimHHH]

Hi.
@abhi18av the original code speaks of a resistance_db which is normally loaded from the tbprofiler venv (a set of files all starting with the same name). Would be great to have the possibility to easily specify different db's as we anticipate updated versions continuously.
When @LennertVerboven supplies the resistance_db this will include a .bed file that can be used as the DRgenes.list file, this also ensures that the DR genes match those from the database being specified.

[abhi18av]

Hi @TimHHH ,

Would be great to have the possibility to easily specify different db's as we anticipate updated versions continuously.

Sure, this makes sense and should be possible using the existing design of passing this as params for example, in the pbs.config you can specify this

https://github.com/abhi18av/xbs-nf/blob/07c8b73e037f89a2f821ff6d4e935e91b05d3ea8/conf/dev.config#L52

When you wish to use a different version of the resistance_db_v100 then you'd simply need to update the pbs.config file for the same

 resistance_db="${projectDir}/data/mock_data/resistance_db_v100" 

Does this suffice the use-case?

Though, it must be kept in mind that if the resistance_db is updated on an analysis which you want to resume (after adding more data) - this would not be possible, since the md5sum of the inputs of the process would be different and Nextflow would re-evaluate all samples within that process since one of the input files i.e. resistance_db changed.

... supplies the resistance_db this will include a .bed file that can be used as the DRgenes.list file, this also ensures that the DR genes match those from the database being specified.

Sure, that would be great to have to finish the testing on my end 👍

[TimHHH]

okay, that sounds good I think.