How are all the fields in the cohort CSV derived?

[abhi18av]

Hi everyone,

I'd like to clearly document (and understand) the derivation of all fields in the cohort CSV file before moving forward.

My guess is that, instead of placing the RELATIVE_ABUNDANCE_THRESHOLD_MET and NTM_FRACTION_THRESHOLD_MET fields in the middle as I have done here - these should be simply appended in the end - which is the reason why some fields don't get populated in CSV below

|SAMPLE                           |AVG_INSERT_SIZE|MAPPED_%|RAW_TOTAL_SEQS|AVERAGE_QUALITY|QUANTTB_RELATIVE_ABUNDANCE|RELATIVE_ABUNDANCE_THRESHOLD_MET|QUANTTB_DEPTH|MEAN_COVERAGE|SD_COVERAGE|NTM_FRACTION_THRESHOLD_MET|MEDIAN_COVERAGE|MAD_COVERAGE|PCT_EXC_ADAPTER|PCT_EXC_MAPQ|PCT_EXC_DUPE|PCT_EXC_UNPAIRED|PCT_EXC_BASEQ|PCT_EXC_OVERLAP|PCT_EXC_CAPPED|PCT_EXC_TOTAL|PCT_1X  |PCT_5X  |PCT_10X |PCT_30X |PCT_50X |PCT_100X|NTM_FRACTION|
|---------------------------------|---------------|--------|--------------|---------------|--------------------------|--------------------------------|-------------|-------------|-----------|--------------------------|---------------|------------|---------------|------------|------------|----------------|-------------|---------------|--------------|-------------|--------|--------|--------|--------|--------|--------|------------|
|EXIT-RIF.EC012B-270.L1.A196.1.1.1|523.0          |97.45   |3506452       |31.7           |1.0                       |                                |89.589       |83.342031    |21.686654  |85                        |11             |0.000006    |0.013383       |0.088419    |0           |0.141238        |0.002489     |0              |0.245536      |0.987641     |0.984446|0.982526|0.973326|0.951572|0.183733|0       |            |
|EXIT-RIF.EC083-202.L1.A254.1.1.1 |535.5          |97.19   |3546147       |31.4           |1.0                       |                                |91.559       |83.760148    |21.873153  |85                        |11             |0.000004    |0.01252        |0.085023    |0           |0.146826        |0.001784     |0              |0.246157      |0.987062     |0.983496|0.98146 |0.972495|0.951153|0.191673|0       |            |
|EXIT-RIF.EC088-207.L1.A259.1.1.1 |518.3          |97.49   |3122416       |31.9           |1.0                       |                                |78.093       |75.183627    |18.449009  |76                        |10             |0.000007    |0.009515       |0.088893    |0           |0.140051        |0.001744     |0              |0.240209      |0.990618     |0.9882  |0.985845|0.975198|0.93931 |0.057507|0       |            |
|EXIT-RIF.EC090-209.L1.A260.1.1.1 |517.3          |94.63   |3594520       |31.9           |1.0                       |                                |81.714       |84.374798    |23.13964   |86                        |11             |0.000003    |0.010123       |0.085864    |0           |0.139407        |0.001788     |0              |0.237185      |0.985736     |0.981516|0.979527|0.970905|0.950222|0.206509|0       |            |
|EXIT-RIF.EC550A-330.L1.A35.1.1.1 |441.8          |87.88   |3753074       |32.6           |1.0                       |                                |87.358       |84.276615    |23.571236  |87                        |12             |0.000016    |0.012219       |0.020734    |0           |0.119355        |0.038917     |0              |0.191241      |0.986566     |0.9785  |0.974647|0.962712|0.940086|0.229799|0       |            |
|EXIT-RIF.FS007-2.L1.A457.1.1.1   |511.1          |76.31   |3993905       |30.6           |0.205;0.795               |                                |13.3;51.686  |73.678728    |18.422225  |75                        |10             |0.000003    |0.011833       |0.028722    |0           |0.163045        |0.033694     |0              |0.237297      |0.990259     |0.985019|0.98108 |0.968489|0.932203|0.044592|0       |            |
|EXIT-RIF.FS015-12.L1.A68.1.1.1   |444.2          |95.96   |3217693       |32.9           |1.0                       |                                |83.58        |84.185857    |22.083922  |87                        |10             |0.000004    |0.013934       |0.014679    |0           |0.113414        |0.010076     |0              |0.152106      |0.983192     |0.976263|0.972926|0.962951|0.945395|0.200666|0       |            |
|EXIT-RIF.FS021A-289.L1.A75.1.1.1 |342.8          |80.04   |4137586       |34.5           |1.0                       |                                |89.929       |79.628111    |20.894355  |82                        |11             |0.00004     |0.014943       |0.029575    |0           |0.082773        |0.110566     |0              |0.237897      |0.987384     |0.982065|0.978342|0.966292|0.941363|0.119821|0       |            |
|EXIT-RIF.FS023-73.L1.A77.1.1.1   |448.8          |86.41   |3585303       |32.7           |1.0                       |                                |78.146       |80.470621    |20.951666  |83                        |11             |0.000023    |0.009988       |0.019432    |0           |0.11788         |0.034915     |0              |0.182239      |0.990006     |0.982765|0.978555|0.966547|0.93907 |0.141801|0       |            |
|EXIT-RIF.FS031-2.L1.A464.1.1.1   |512.4          |68.99   |4353287       |30.5           |1.0                       |                                |77.886       |74.300743    |19.913616  |76                        |10             |0.000006    |0.008985       |0.029352    |0           |0.166612        |0.020136     |0              |0.225092      |0.98712      |0.978556|0.974574|0.962886|0.927151|0.056141|0       |            |

My current understanding (derived from ) is documented in the

// SAMPLE    AVG_INSERT_SIZE      MAPPED_%    RAW_TOTAL_SEQS    AVERAGE_QUALITY    QUANTTB_RELATIVE_ABUNDANCE     RELATIVE_ABUNDANCE_THRESHOLD_MET    QUANTTB_DEPTH    MEAN_COVERAGE    SD_COVERAGE       NTM_FRACTION_THRESHOLD_MET MEDIAN_COVERAGE    MAD_COVERAGE    PCT_EXC_ADAPTER    PCT_EXC_MAPQ    PCT_EXC_DUPE    PCT_EXC_UNPAIRED    PCT_EXC_BASEQ    PCT_EXC_OVERLAP    PCT_EXC_CAPPED    PCT_EXC_TOTAL    PCT_1X    PCT_5X    PCT_10X    PCT_30X    PCT_50X    PCT_100X    NTM_FRACTION
// ${i}      ${TOTAL_SEQS}      ${MAPPED_P}    ${INS_SIZE}    ${AVG_QUAL}          ${REL_ABUNDANCE}                 $(rel_abundance_threshold_met)       ${DEPTH}       ${WGS_METR}      ${NTM_FRACTION}   $(ntm_fraction_threshold_met)

Trying to translate the following snippet from XBS_merge.sh (L-65)

read -ra SAMPLE_LIST <<< "$SAMPLES"
for i in "${SAMPLE_LIST[@]}"
do
    COVERAGE=$(cat ${OUT_DIR}/stats/${i}.recal_reads.WgsMetrics | grep "^4411532" | cut -f 4)
    BREADTH_OF_COVERAGE=$(cat ${OUT_DIR}/stats/${i}.recal_reads.WgsMetrics | grep "^4411532" | cut -f 14)
    REL_ABUNDANCE=$(tail -n +2 ${OUT_DIR}/quanttb/${i}.qtb.txt | cut -d , -f 4 | paste -s -d ';')
    DEPTH=$(tail -n +2 ${OUT_DIR}/quanttb/${i}.qtb.txt | cut -d , -f 5 | paste -s -d ';')
    TOTAL_SEQS=$(cat ${OUT_DIR}/stats/${i}.recal_reads.SamtoolStats | grep "insert size average" | cut -f 3)
    MAPPED_P=$(cat ${OUT_DIR}/stats/${i}.recal_reads.FlagStat | grep "mapped (" | cut -f 2 -d "(" | cut -f 1 -d "%")
    INS_SIZE=$(cat ${OUT_DIR}/stats/${i}.recal_reads.SamtoolStats | grep "raw total sequences" | cut -f 3)
    AVG_QUAL=$(cat ${OUT_DIR}/stats/${i}.recal_reads.SamtoolStats | grep "average quality" | cut -f 3)
    WGS_METR=$(cat ${OUT_DIR}/stats/${i}.recal_reads.WgsMetrics | grep "^4411532" | cut -f 2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,20,22,27)
    NTM_FRACTION=$(cat ${OUT_DIR}/stats/${i}.potential_NTM_fraction.txt)
    echo -e "${i}\t${TOTAL_SEQS}\t${MAPPED_P}\t${INS_SIZE}\t${AVG_QUAL}\t${REL_ABUNDANCE}\t${DEPTH}\t${WGS_METR}\t${NTM_FRACTION}" >> ${OUT_DIR}/stats/$JOINT_NAME.summary.tsv
    if [ "$COVERAGE" -ge "$MEDIAN_COVERAGE_CUTOFF" ] && [ 1 -eq "$(echo "$BREADTH_OF_COVERAGE >= $BREADTH_OF_COVERAGE_CUTOFF" | bc)" ] && rel_abundance_threshold_met $REL_ABUNDANCE_CUTOFF $REL_ABUNDANCE && ntm_fraction_threshold_met $NTM_FRACTION_CUTOFF $NTM_FRACTION;
    then
        echo "Adding ${i}"
        #echo "Adding ${i}, has a median coverage of ${COVERAGE} (${MEDIAN_COVERAGE_CUTOFF} required), breadth of coverage of ${BREADTH_OF_COVERAGE} (${BREADTH_OF_COVERAGE_CUTOFF} required), relative abundance of ${REL_ABUNDANCE} (${REL_ABUNDANCE_CUTOFF} required)"
        GVCFs+=" -V ${OUT_DIR}/gvcf/${i}.g.vcf.gz"
    else
        echo "Not adding ${i}, has a median coverage of ${COVERAGE} (${MEDIAN_COVERAGE_CUTOFF} minimum), breadth of coverage of ${BREADTH_OF_COVERAGE} (${BREADTH_OF_COVERAGE_CUTOFF} minimum), relative abundance of ${REL_ABUNDANCE} (${REL_ABUNDANCE_CUTOFF} minimum), NTM fraction of ${NTM_FRACTION} (${NTM_FRACTION_CUTOFF} maximum)"
    fi;
done

... I don't quite understand how are the following fields derived?

MEDIAN_COVERAGE    MAD_COVERAGE    PCT_EXC_ADAPTER    PCT_EXC_MAPQ    PCT_EXC_DUPE    PCT_EXC_UNPAIRED    PCT_EXC_BASEQ    PCT_EXC_OVERLAP    PCT_EXC_CAPPED    PCT_EXC_TOTAL    PCT_1X    PCT_5X    PCT_10X    PCT_30X    PCT_50X    PCT_100X    NTM_FRACTION

This is crucial for me to understand since we need to filter out the samples based on the following parameter

if [ "$COVERAGE" -ge "$MEDIAN_COVERAGE_CUTOFF" ] &&
       [ 1 -eq "$(echo "$BREADTH_OF_COVERAGE >= $BREADTH_OF_COVERAGE_CUTOFF" | bc)" ] &&
       rel_abundance_threshold_met $REL_ABUNDANCE_CUTOFF $REL_ABUNDANCE &&
       ntm_fraction_threshold_met $NTM_FRACTION_CUTOFF $NTM_FRACTION;

... however, with what I have at the moment some of the fields are not really captured in the CSV , as highlighted here #16 (comment)

[abhi18av]

My guess is that, instead of placing the RELATIVE_ABUNDANCE_THRESHOLD_MET and NTM_FRACTION_THRESHOLD_MET fields in the middle as I have done here - these should be simply appended in the end - which is the reason why some fields don't get populated in CSV below

Assuming that this was actually the reason, I have made the relevant changes (+ fixed a bug) and now this is the resultant CSV file - could anyone please confirm these results?

|SAMPLE                           |AVG_INSERT_SIZE|MAPPED_%|RAW_TOTAL_SEQS|AVERAGE_QUALITY|QUANTTB_RELATIVE_ABUNDANCE|QUANTTB_DEPTH|MEAN_COVERAGE|SD_COVERAGE|MEDIAN_COVERAGE|MAD_COVERAGE|PCT_EXC_ADAPTER|PCT_EXC_MAPQ|PCT_EXC_DUPE|PCT_EXC_UNPAIRED|PCT_EXC_BASEQ|PCT_EXC_OVERLAP|PCT_EXC_CAPPED|PCT_EXC_TOTAL|PCT_1X  |PCT_5X  |PCT_10X |PCT_30X |PCT_50X |PCT_100X|NTM_FRACTION|NTM_FRACTION_THRESHOLD_MET|RELATIVE_ABUNDANCE_THRESHOLD_MET|
|---------------------------------|---------------|--------|--------------|---------------|--------------------------|-------------|-------------|-----------|---------------|------------|---------------|------------|------------|----------------|-------------|---------------|--------------|-------------|--------|--------|--------|--------|--------|--------|------------|--------------------------|--------------------------------|
|EXIT-RIF.EC012B-270.L1.A196.1.1.1|523.0          |97.45   |3506452       |31.7           |1.0                       |89.589       |83.342031    |21.686654  |85             |11          |0.000006       |0.013383    |0.088419    |0               |0.141238     |0.002489       |0             |0.245536     |0.987641|0.984446|0.982526|0.973326|0.951572|0.183733|0           |0                         |0                               |
|EXIT-RIF.EC083-202.L1.A254.1.1.1 |535.5          |97.19   |3546147       |31.4           |1.0                       |91.559       |83.760148    |21.873153  |85             |11          |0.000004       |0.01252     |0.085023    |0               |0.146826     |0.001784       |0             |0.246157     |0.987062|0.983496|0.98146 |0.972495|0.951153|0.191673|0           |0                         |0                               |
|EXIT-RIF.EC088-207.L1.A259.1.1.1 |518.3          |97.49   |3122416       |31.9           |1.0                       |78.093       |75.183627    |18.449009  |76             |10          |0.000007       |0.009515    |0.088893    |0               |0.140051     |0.001744       |0             |0.240209     |0.990618|0.9882  |0.985845|0.975198|0.93931 |0.057507|0           |0                         |0                               |
|EXIT-RIF.EC090-209.L1.A260.1.1.1 |517.3          |94.63   |3594520       |31.9           |1.0                       |81.714       |84.374798    |23.13964   |86             |11          |0.000003       |0.010123    |0.085864    |0               |0.139407     |0.001788       |0             |0.237185     |0.985736|0.981516|0.979527|0.970905|0.950222|0.206509|0           |0                         |0                               |
|EXIT-RIF.EC550A-330.L1.A35.1.1.1 |441.8          |87.88   |3753074       |32.6           |1.0                       |87.358       |84.276615    |23.571236  |87             |12          |0.000016       |0.012219    |0.020734    |0               |0.119355     |0.038917       |0             |0.191241     |0.986566|0.9785  |0.974647|0.962712|0.940086|0.229799|0           |0                         |0                               |
|EXIT-RIF.FS007-2.L1.A457.1.1.1   |511.1          |76.31   |3993905       |30.6           |0.205;0.795               |13.3;51.686  |73.678728    |18.422225  |75             |10          |0.000003       |0.011833    |0.028722    |0               |0.163045     |0.033694       |0             |0.237297     |0.990259|0.985019|0.98108 |0.968489|0.932203|0.044592|0           |0                         |1                               |
|EXIT-RIF.FS015-12.L1.A68.1.1.1   |444.2          |95.96   |3217693       |32.9           |1.0                       |83.58        |84.185857    |22.083922  |87             |10          |0.000004       |0.013934    |0.014679    |0               |0.113414     |0.010076       |0             |0.152106     |0.983192|0.976263|0.972926|0.962951|0.945395|0.200666|0           |0                         |0                               |
|EXIT-RIF.FS021A-289.L1.A75.1.1.1 |342.8          |80.04   |4137586       |34.5           |1.0                       |89.929       |79.628111    |20.894355  |82             |11          |0.00004        |0.014943    |0.029575    |0               |0.082773     |0.110566       |0             |0.237897     |0.987384|0.982065|0.978342|0.966292|0.941363|0.119821|0           |0                         |0                               |
|EXIT-RIF.FS023-73.L1.A77.1.1.1   |448.8          |86.41   |3585303       |32.7           |1.0                       |78.146       |80.470621    |20.951666  |83             |11          |0.000023       |0.009988    |0.019432    |0               |0.11788      |0.034915       |0             |0.182239     |0.990006|0.982765|0.978555|0.966547|0.93907 |0.141801|0           |0                         |0                               |
|EXIT-RIF.FS031-2.L1.A464.1.1.1   |512.4          |68.99   |4353287       |30.5           |1.0                       |77.886       |74.300743    |19.913616  |76             |10          |0.000006       |0.008985    |0.029352    |0               |0.166612     |0.020136       |0             |0.225092     |0.98712 |0.978556|0.974574|0.962886|0.927151|0.056141|0           |0                         |0                               |

[abhi18av]

Though the problem now seems to be that none of the samples end up with NTM_FRACTION_THRESHOLD_MET as 1 (for NTM cutoff 0.20 or 0.10, 0.05 ) therefore only 1 sample passes the intended test condition ntm_fraction_threshold_met $NTM_FRACTION_CUTOFF $NTM_FRACTION 🤔

The next step

• is ideally to confirm whether this result is the same as XBS-v1.0 for these functions (as these columns are not captured in the CSV in XBS-v1.0)
• to confirm whether this is how the eventual proportion of passing/failing samples is supposed to look like?

Another thing which would be helpful, as mentioned in #16 (comment) (repeated below).

Could you please share the CSV file that you have for these samples, generated from the XBS-v1.0 ? I do want to confirm whether the shell script in UTILS_SAMPLE_STATS is tweaked correctly or not.

[TimHHH]

I have now supplied the csv in the other thread. This out put looks correct now, just the issue you mention with the sample passing.

• If the NTM fraction is below 0.20 then it is approved.
• If one of the Mtb strains identified by quantb reaches a fraction above or equal to 0.80 (QUANTTB_RELATIVE_ABUNDANCE) then it is approved.
  ergo, the current threshold-met-values are inverse of what they should be.