Test failing PR from mvdbeek

[lldelisle]

No description provided.

[lldelisle]

Test Results (powered by Planemo)

Test Summary

Test State	Count
Total	4
Passed	2
Error	0
Failure	2
Skipped	0

Failed Tests

• ❌ parallel-accession-download.ga_0

  Problems:

  • Output with path /tmp/tmpqz16_dgs/SRR044777__538ad9ef-faac-4dcc-97b3-25096e6eb109 different than expected, difference (using contains):
    ( /home/runner/work/iwc/iwc/workflows/data-fetching/parallel-accession-download/test-data/SRR044777_head.fastq v. /tmp/tmpk3wunwobSRR044777_head.fastq )
    Failed to find 'b'@F47USSH02H1LGA/4'' in history data. (lines_diff=0).
    

  Workflow invocation details

  • Invocation Messages

  • Steps

    • Step 1: Run accessions:

      • step_state: scheduled

    • Step 2: Split accessions to collection:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • mkdir ./out && python '/tmp/shed_dir/toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/6cbe2f30c2d7/split_file_to_collection/split_file_to_collection.py' --out ./out --in '/tmp/tmppcmmhl1i/files/9/d/7/dataset_9d775c9d-db00-4dae-b105-8bcbede97a16.dat' --ftype 'txt' --chunksize 1 --file_names 'split_file' --file_ext 'txt'

          Exit Code:

          • 0

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"txt"
            __workflow_invocation_uuid__	"eac1e115847a11ee8802d319867c0200"
            chromInfo	"/tmp/tmppcmmhl1i/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            split_parms	{"__current_case__": 5, "input": {"values": [{"id": 1, "src": "hda"}]}, "newfilenames": "split_file", "select_allocate": {"__current_case__": 2, "allocate": "byrow"}, "select_ftype": "txt", "select_mode": {"__current_case__": 0, "chunksize": "1", "mode": "chunk"}}

    • Step 3: to parameter:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • cd ../; python _evaluate_expression_.py

          Exit Code:

          • 0

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"txt"
            __workflow_invocation_uuid__	"eac1e115847a11ee8802d319867c0200"
            chromInfo	"/tmp/tmppcmmhl1i/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            param_type	"text"
            remove_newlines	true

    • Step 4: fasterq-dump:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • set -o | grep -q pipefail && set -o pipefail;  mkdir -p ~/.ncbi && cp '/tmp/tmppcmmhl1i/job_working_directory/000/4/configs/tmppjd5to6y' ~/.ncbi/user-settings.mkfg && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s "/repository/user/ad/public/root=$PWD" && vdb-config -s "/repository/user/default-path=$PWD" && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s /http/timeout/read=10000 &&   export SRA_PREFETCH_RETRIES=3 && export SRA_PREFETCH_ATTEMPT=1 &&   echo 'SRR044777' | sed -r 's/(\,|\;|__cn__)/\n/g' > accessions && for acc in $(cat ./accessions); do ( echo "Downloading accession: $acc..." &&  while [ $SRA_PREFETCH_ATTEMPT -le $SRA_PREFETCH_RETRIES ] ; do fasterq-dump "$acc" -e ${GALAXY_SLOTS:-1} --seq-defline '@$sn/$ri' --qual-defline '+' --split-3 --skip-technical 2>&1 | tee -a '/tmp/tmppcmmhl1i/job_working_directory/000/4/outputs/dataset_02460ed0-fd57-4928-8786-af355027965e.dat'; if [ $? == 0 ] && [ $(ls *.fastq | wc -l) -ge 1 ]; then break ; else echo "Prefetch attempt $SRA_PREFETCH_ATTEMPT of $SRA_PREFETCH_RETRIES exited with code $?" ; SRA_PREFETCH_ATTEMPT=`expr $SRA_PREFETCH_ATTEMPT + 1` ; sleep 1 ; fi ; done && mkdir -p output && mkdir -p outputOther && count="$(ls *.fastq | wc -l)" && echo "There are $count fastq files" && data=($(ls *.fastq)) && if [ "$count" -eq 1 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"__single.fastqsanger.gz && rm "${data[0]}"; elif [ "--split-3" = "--split-3" ]; then if [ -e "${acc}".fastq ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${acc}".fastq > outputOther/"${acc}"__single.fastqsanger.gz; fi && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_1.fastq > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_2.fastq > output/"${acc}"_reverse.fastqsanger.gz && rm "${acc}"*.fastq; elif [ "$count" -eq 2 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${data[1]}" > output/"${acc}"_reverse.fastqsanger.gz && rm "${data[0]}" && rm "${data[1]}"; else for file in ${data[*]}; do pigz -cqp ${GALAXY_SLOTS:-1} "$file" > outputOther/"$file"sanger.gz && rm "$file"; done; fi;  ); done; echo "Done with all accessions."

          Exit Code:

          • 0

          Standard Output:

          • Downloading accession: SRR044777...
            spots read      : 7,882
            reads read      : 31,528
            reads written   : 7,882
            technical reads : 23,646
            There are 1 fastq files
            Done with all accessions.
            

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"input"
            __job_resource	{"__current_case__": 0, "__job_resource__select": "no"}
            __workflow_invocation_uuid__	"eac1e115847a11ee8802d319867c0200"
            adv	{"minlen": null, "seq_defline": "@$sn/$ri", "skip_technical": true, "split": "--split-3"}
            chromInfo	"/tmp/tmppcmmhl1i/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            input	{"__current_case__": 0, "accession": "SRR044777", "input_select": "accession_number"}

    • Step 5: flatten paired output:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"eac1e115847a11ee8802d319867c0200"
            input	{"values": [{"id": 3, "src": "hdca"}]}
            rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}, {"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [2], "connectable": true, "is_workflow": false, "type": "paired_identifier"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier2", "warn": null}]}

    • Step 6: flatten single end output:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"eac1e115847a11ee8802d319867c0200"
            input	{"values": [{"id": 4, "src": "hdca"}]}
            rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}]}

  • Other invocation details

    • history_id

      • 4bfc8047508a87c5

    • history_state

      • ok

    • invocation_id

      • 4bfc8047508a87c5

    • invocation_state

      • scheduled

    • workflow_id

      • ca037634a55aabd7

❌ parallel-accession-download.ga_0

Problems:

• Output with path /tmp/tmpqz16_dgs/SRR044777__538ad9ef-faac-4dcc-97b3-25096e6eb109 different than expected, difference (using contains):
  ( /home/runner/work/iwc/iwc/workflows/data-fetching/parallel-accession-download/test-data/SRR044777_head.fastq v. /tmp/tmpk3wunwobSRR044777_head.fastq )
  Failed to find 'b'@F47USSH02H1LGA/4'' in history data. (lines_diff=0).
  

Workflow invocation details

• Invocation Messages

• Steps

  • Step 1: Run accessions:

    • step_state: scheduled

  • Step 2: Split accessions to collection:

    • step_state: scheduled

    • Jobs

      • Job 1:

        • Job state is ok

        Command Line:

        • mkdir ./out && python '/tmp/shed_dir/toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/6cbe2f30c2d7/split_file_to_collection/split_file_to_collection.py' --out ./out --in '/tmp/tmppcmmhl1i/files/9/d/7/dataset_9d775c9d-db00-4dae-b105-8bcbede97a16.dat' --ftype 'txt' --chunksize 1 --file_names 'split_file' --file_ext 'txt'

        Exit Code:

        • 0

        Job Parameters:

        • Job parameter	Parameter value
          __input_ext	"txt"
          __workflow_invocation_uuid__	"eac1e115847a11ee8802d319867c0200"
          chromInfo	"/tmp/tmppcmmhl1i/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
          dbkey	"?"
          split_parms	{"__current_case__": 5, "input": {"values": [{"id": 1, "src": "hda"}]}, "newfilenames": "split_file", "select_allocate": {"__current_case__": 2, "allocate": "byrow"}, "select_ftype": "txt", "select_mode": {"__current_case__": 0, "chunksize": "1", "mode": "chunk"}}

  • Step 3: to parameter:

    • step_state: scheduled

    • Jobs

      • Job 1:

        • Job state is ok

        Command Line:

        • cd ../; python _evaluate_expression_.py

        Exit Code:

        • 0

        Job Parameters:

        • Job parameter	Parameter value
          __input_ext	"txt"
          __workflow_invocation_uuid__	"eac1e115847a11ee8802d319867c0200"
          chromInfo	"/tmp/tmppcmmhl1i/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
          dbkey	"?"
          param_type	"text"
          remove_newlines	true

  • Step 4: fasterq-dump:

    • step_state: scheduled

    • Jobs

      • Job 1:

        • Job state is ok

        Command Line:

        • set -o | grep -q pipefail && set -o pipefail;  mkdir -p ~/.ncbi && cp '/tmp/tmppcmmhl1i/job_working_directory/000/4/configs/tmppjd5to6y' ~/.ncbi/user-settings.mkfg && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s "/repository/user/ad/public/root=$PWD" && vdb-config -s "/repository/user/default-path=$PWD" && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s /http/timeout/read=10000 &&   export SRA_PREFETCH_RETRIES=3 && export SRA_PREFETCH_ATTEMPT=1 &&   echo 'SRR044777' | sed -r 's/(\,|\;|__cn__)/\n/g' > accessions && for acc in $(cat ./accessions); do ( echo "Downloading accession: $acc..." &&  while [ $SRA_PREFETCH_ATTEMPT -le $SRA_PREFETCH_RETRIES ] ; do fasterq-dump "$acc" -e ${GALAXY_SLOTS:-1} --seq-defline '@$sn/$ri' --qual-defline '+' --split-3 --skip-technical 2>&1 | tee -a '/tmp/tmppcmmhl1i/job_working_directory/000/4/outputs/dataset_02460ed0-fd57-4928-8786-af355027965e.dat'; if [ $? == 0 ] && [ $(ls *.fastq | wc -l) -ge 1 ]; then break ; else echo "Prefetch attempt $SRA_PREFETCH_ATTEMPT of $SRA_PREFETCH_RETRIES exited with code $?" ; SRA_PREFETCH_ATTEMPT=`expr $SRA_PREFETCH_ATTEMPT + 1` ; sleep 1 ; fi ; done && mkdir -p output && mkdir -p outputOther && count="$(ls *.fastq | wc -l)" && echo "There are $count fastq files" && data=($(ls *.fastq)) && if [ "$count" -eq 1 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"__single.fastqsanger.gz && rm "${data[0]}"; elif [ "--split-3" = "--split-3" ]; then if [ -e "${acc}".fastq ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${acc}".fastq > outputOther/"${acc}"__single.fastqsanger.gz; fi && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_1.fastq > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_2.fastq > output/"${acc}"_reverse.fastqsanger.gz && rm "${acc}"*.fastq; elif [ "$count" -eq 2 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${data[1]}" > output/"${acc}"_reverse.fastqsanger.gz && rm "${data[0]}" && rm "${data[1]}"; else for file in ${data[*]}; do pigz -cqp ${GALAXY_SLOTS:-1} "$file" > outputOther/"$file"sanger.gz && rm "$file"; done; fi;  ); done; echo "Done with all accessions."

        Exit Code:

        • 0

        Standard Output:

        • Downloading accession: SRR044777...
          spots read      : 7,882
          reads read      : 31,528
          reads written   : 7,882
          technical reads : 23,646
          There are 1 fastq files
          Done with all accessions.
          

        Job Parameters:

        • Job parameter	Parameter value
          __input_ext	"input"
          __job_resource	{"__current_case__": 0, "__job_resource__select": "no"}
          __workflow_invocation_uuid__	"eac1e115847a11ee8802d319867c0200"
          adv	{"minlen": null, "seq_defline": "@$sn/$ri", "skip_technical": true, "split": "--split-3"}
          chromInfo	"/tmp/tmppcmmhl1i/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
          dbkey	"?"
          input	{"__current_case__": 0, "accession": "SRR044777", "input_select": "accession_number"}

  • Step 5: flatten paired output:

    • step_state: scheduled

    • Jobs

      • Job 1:

        • Job state is ok

        Job Parameters:

        • Job parameter	Parameter value
          __workflow_invocation_uuid__	"eac1e115847a11ee8802d319867c0200"
          input	{"values": [{"id": 3, "src": "hdca"}]}
          rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}, {"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [2], "connectable": true, "is_workflow": false, "type": "paired_identifier"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier2", "warn": null}]}

  • Step 6: flatten single end output:

    • step_state: scheduled

    • Jobs

      • Job 1:

        • Job state is ok

        Job Parameters:

        • Job parameter	Parameter value
          __workflow_invocation_uuid__	"eac1e115847a11ee8802d319867c0200"
          input	{"values": [{"id": 4, "src": "hdca"}]}
          rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}]}

• Other invocation details

  • history_id

    • 4bfc8047508a87c5

  • history_state

    • ok

  • invocation_id

    • 4bfc8047508a87c5

  • invocation_state

    • scheduled

  • workflow_id

    • ca037634a55aabd7

Passed Tests

• ✅ parallel-accession-download.ga_1

  Workflow invocation details

  • Invocation Messages

  • Steps

    • Step 1: Run accessions:

      • step_state: scheduled

    • Step 2: Split accessions to collection:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • mkdir ./out && python '/tmp/shed_dir/toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/6cbe2f30c2d7/split_file_to_collection/split_file_to_collection.py' --out ./out --in '/tmp/tmppcmmhl1i/files/9/c/6/dataset_9c64945c-8dad-4439-9caa-95f3cee1a30f.dat' --ftype 'txt' --chunksize 1 --file_names 'split_file' --file_ext 'txt'

          Exit Code:

          • 0

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"txt"
            __workflow_invocation_uuid__	"0b5b81dd847b11ee8802d319867c0200"
            chromInfo	"/tmp/tmppcmmhl1i/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            split_parms	{"__current_case__": 5, "input": {"values": [{"id": 7, "src": "hda"}]}, "newfilenames": "split_file", "select_allocate": {"__current_case__": 2, "allocate": "byrow"}, "select_ftype": "txt", "select_mode": {"__current_case__": 0, "chunksize": "1", "mode": "chunk"}}

    • Step 3: to parameter:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • cd ../; python _evaluate_expression_.py

          Exit Code:

          • 0

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"txt"
            __workflow_invocation_uuid__	"0b5b81dd847b11ee8802d319867c0200"
            chromInfo	"/tmp/tmppcmmhl1i/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            param_type	"text"
            remove_newlines	true

    • Step 4: fasterq-dump:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • set -o | grep -q pipefail && set -o pipefail;  mkdir -p ~/.ncbi && cp '/tmp/tmppcmmhl1i/job_working_directory/000/10/configs/tmplq6_ljvl' ~/.ncbi/user-settings.mkfg && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s "/repository/user/ad/public/root=$PWD" && vdb-config -s "/repository/user/default-path=$PWD" && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s /http/timeout/read=10000 &&   export SRA_PREFETCH_RETRIES=3 && export SRA_PREFETCH_ATTEMPT=1 &&   echo 'SRR11953971' | sed -r 's/(\,|\;|__cn__)/\n/g' > accessions && for acc in $(cat ./accessions); do ( echo "Downloading accession: $acc..." &&  while [ $SRA_PREFETCH_ATTEMPT -le $SRA_PREFETCH_RETRIES ] ; do fasterq-dump "$acc" -e ${GALAXY_SLOTS:-1} --seq-defline '@$sn/$ri' --qual-defline '+' --split-3 --skip-technical 2>&1 | tee -a '/tmp/tmppcmmhl1i/job_working_directory/000/10/outputs/dataset_6118aee3-6ee7-4ecc-ab5b-82f0473dbb49.dat'; if [ $? == 0 ] && [ $(ls *.fastq | wc -l) -ge 1 ]; then break ; else echo "Prefetch attempt $SRA_PREFETCH_ATTEMPT of $SRA_PREFETCH_RETRIES exited with code $?" ; SRA_PREFETCH_ATTEMPT=`expr $SRA_PREFETCH_ATTEMPT + 1` ; sleep 1 ; fi ; done && mkdir -p output && mkdir -p outputOther && count="$(ls *.fastq | wc -l)" && echo "There are $count fastq files" && data=($(ls *.fastq)) && if [ "$count" -eq 1 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"__single.fastqsanger.gz && rm "${data[0]}"; elif [ "--split-3" = "--split-3" ]; then if [ -e "${acc}".fastq ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${acc}".fastq > outputOther/"${acc}"__single.fastqsanger.gz; fi && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_1.fastq > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_2.fastq > output/"${acc}"_reverse.fastqsanger.gz && rm "${acc}"*.fastq; elif [ "$count" -eq 2 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${data[1]}" > output/"${acc}"_reverse.fastqsanger.gz && rm "${data[0]}" && rm "${data[1]}"; else for file in ${data[*]}; do pigz -cqp ${GALAXY_SLOTS:-1} "$file" > outputOther/"$file"sanger.gz && rm "$file"; done; fi;  ); done; echo "Done with all accessions."

          Exit Code:

          • 0

          Standard Output:

          • Downloading accession: SRR11953971...
            spots read      : 2,057
            reads read      : 4,114
            reads written   : 4,114
            There are 2 fastq files
            Done with all accessions.
            

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"input"
            __job_resource	{"__current_case__": 0, "__job_resource__select": "no"}
            __workflow_invocation_uuid__	"0b5b81dd847b11ee8802d319867c0200"
            adv	{"minlen": null, "seq_defline": "@$sn/$ri", "skip_technical": true, "split": "--split-3"}
            chromInfo	"/tmp/tmppcmmhl1i/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            input	{"__current_case__": 0, "accession": "SRR11953971", "input_select": "accession_number"}

    • Step 5: flatten paired output:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"0b5b81dd847b11ee8802d319867c0200"
            input	{"values": [{"id": 11, "src": "hdca"}]}
            rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}, {"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [2], "connectable": true, "is_workflow": false, "type": "paired_identifier"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier2", "warn": null}]}

    • Step 6: flatten single end output:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"0b5b81dd847b11ee8802d319867c0200"
            input	{"values": [{"id": 12, "src": "hdca"}]}
            rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}]}

  • Other invocation details

    • history_id

      • ca037634a55aabd7

    • history_state

      • ok

    • invocation_id

      • ca037634a55aabd7

    • invocation_state

      • scheduled

    • workflow_id

      • ca037634a55aabd7

✅ parallel-accession-download.ga_1

Workflow invocation details

• Invocation Messages

• Steps

  • Step 1: Run accessions:

    • step_state: scheduled

  • Step 2: Split accessions to collection:

    • step_state: scheduled

    • Jobs

      • Job 1:

        • Job state is ok

        Command Line:

        • mkdir ./out && python '/tmp/shed_dir/toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/6cbe2f30c2d7/split_file_to_collection/split_file_to_collection.py' --out ./out --in '/tmp/tmppcmmhl1i/files/9/c/6/dataset_9c64945c-8dad-4439-9caa-95f3cee1a30f.dat' --ftype 'txt' --chunksize 1 --file_names 'split_file' --file_ext 'txt'

        Exit Code:

        • 0

        Job Parameters:

        • Job parameter	Parameter value
          __input_ext	"txt"
          __workflow_invocation_uuid__	"0b5b81dd847b11ee8802d319867c0200"
          chromInfo	"/tmp/tmppcmmhl1i/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
          dbkey	"?"
          split_parms	{"__current_case__": 5, "input": {"values": [{"id": 7, "src": "hda"}]}, "newfilenames": "split_file", "select_allocate": {"__current_case__": 2, "allocate": "byrow"}, "select_ftype": "txt", "select_mode": {"__current_case__": 0, "chunksize": "1", "mode": "chunk"}}

  • Step 3: to parameter:

    • step_state: scheduled

    • Jobs

      • Job 1:

        • Job state is ok

        Command Line:

        • cd ../; python _evaluate_expression_.py

        Exit Code:

        • 0

        Job Parameters:

        • Job parameter	Parameter value
          __input_ext	"txt"
          __workflow_invocation_uuid__	"0b5b81dd847b11ee8802d319867c0200"
          chromInfo	"/tmp/tmppcmmhl1i/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
          dbkey	"?"
          param_type	"text"
          remove_newlines	true

  • Step 4: fasterq-dump:

    • step_state: scheduled

    • Jobs

      • Job 1:

        • Job state is ok

        Command Line:

        • set -o | grep -q pipefail && set -o pipefail;  mkdir -p ~/.ncbi && cp '/tmp/tmppcmmhl1i/job_working_directory/000/10/configs/tmplq6_ljvl' ~/.ncbi/user-settings.mkfg && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s "/repository/user/ad/public/root=$PWD" && vdb-config -s "/repository/user/default-path=$PWD" && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s /http/timeout/read=10000 &&   export SRA_PREFETCH_RETRIES=3 && export SRA_PREFETCH_ATTEMPT=1 &&   echo 'SRR11953971' | sed -r 's/(\,|\;|__cn__)/\n/g' > accessions && for acc in $(cat ./accessions); do ( echo "Downloading accession: $acc..." &&  while [ $SRA_PREFETCH_ATTEMPT -le $SRA_PREFETCH_RETRIES ] ; do fasterq-dump "$acc" -e ${GALAXY_SLOTS:-1} --seq-defline '@$sn/$ri' --qual-defline '+' --split-3 --skip-technical 2>&1 | tee -a '/tmp/tmppcmmhl1i/job_working_directory/000/10/outputs/dataset_6118aee3-6ee7-4ecc-ab5b-82f0473dbb49.dat'; if [ $? == 0 ] && [ $(ls *.fastq | wc -l) -ge 1 ]; then break ; else echo "Prefetch attempt $SRA_PREFETCH_ATTEMPT of $SRA_PREFETCH_RETRIES exited with code $?" ; SRA_PREFETCH_ATTEMPT=`expr $SRA_PREFETCH_ATTEMPT + 1` ; sleep 1 ; fi ; done && mkdir -p output && mkdir -p outputOther && count="$(ls *.fastq | wc -l)" && echo "There are $count fastq files" && data=($(ls *.fastq)) && if [ "$count" -eq 1 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"__single.fastqsanger.gz && rm "${data[0]}"; elif [ "--split-3" = "--split-3" ]; then if [ -e "${acc}".fastq ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${acc}".fastq > outputOther/"${acc}"__single.fastqsanger.gz; fi && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_1.fastq > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_2.fastq > output/"${acc}"_reverse.fastqsanger.gz && rm "${acc}"*.fastq; elif [ "$count" -eq 2 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${data[1]}" > output/"${acc}"_reverse.fastqsanger.gz && rm "${data[0]}" && rm "${data[1]}"; else for file in ${data[*]}; do pigz -cqp ${GALAXY_SLOTS:-1} "$file" > outputOther/"$file"sanger.gz && rm "$file"; done; fi;  ); done; echo "Done with all accessions."

        Exit Code:

        • 0

        Standard Output:

        • Downloading accession: SRR11953971...
          spots read      : 2,057
          reads read      : 4,114
          reads written   : 4,114
          There are 2 fastq files
          Done with all accessions.
          

        Job Parameters:

        • Job parameter	Parameter value
          __input_ext	"input"
          __job_resource	{"__current_case__": 0, "__job_resource__select": "no"}
          __workflow_invocation_uuid__	"0b5b81dd847b11ee8802d319867c0200"
          adv	{"minlen": null, "seq_defline": "@$sn/$ri", "skip_technical": true, "split": "--split-3"}
          chromInfo	"/tmp/tmppcmmhl1i/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
          dbkey	"?"
          input	{"__current_case__": 0, "accession": "SRR11953971", "input_select": "accession_number"}

  • Step 5: flatten paired output:

    • step_state: scheduled

    • Jobs

      • Job 1:

        • Job state is ok

        Job Parameters:

        • Job parameter	Parameter value
          __workflow_invocation_uuid__	"0b5b81dd847b11ee8802d319867c0200"
          input	{"values": [{"id": 11, "src": "hdca"}]}
          rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}, {"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [2], "connectable": true, "is_workflow": false, "type": "paired_identifier"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier2", "warn": null}]}

  • Step 6: flatten single end output:

    • step_state: scheduled

    • Jobs

      • Job 1:

        • Job state is ok

        Job Parameters:

        • Job parameter	Parameter value
          __workflow_invocation_uuid__	"0b5b81dd847b11ee8802d319867c0200"
          input	{"values": [{"id": 12, "src": "hdca"}]}
          rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}]}

• Other invocation details

  • history_id

    • ca037634a55aabd7

  • history_state

    • ok

  • invocation_id

    • ca037634a55aabd7

  • invocation_state

    • scheduled

  • workflow_id

    • ca037634a55aabd7

[lldelisle]

Test Results (powered by Planemo)

Test Summary

Test State	Count
Total	2
Passed	1
Error	0
Failure	1
Skipped	0

Failed Tests

• ❌ parallel-accession-download.ga_0

  Problems:

  • Output with path /tmp/tmpslbe5836/SRR044777__46079c79-aba4-4fe3-949f-bcf4d48df663 different than expected, difference (using contains):
    ( /home/runner/work/iwc/iwc/workflows/data-fetching/parallel-accession-download/test-data/SRR044777_head.fastq v. /tmp/tmp91136hnuSRR044777_head.fastq )
    Failed to find 'b'@F47USSH02H1LGA/4'' in history data. (lines_diff=0).
    

  Workflow invocation details

  • Invocation Messages

  • Steps

    • Step 1: Run accessions:

      • step_state: scheduled

    • Step 2: Split accessions to collection:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • mkdir ./out && python '/tmp/shed_dir/toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/6cbe2f30c2d7/split_file_to_collection/split_file_to_collection.py' --out ./out --in '/tmp/tmphx9vfffu/files/4/c/8/dataset_4c85cd48-a5ed-4241-9ce1-cdd9d55a7b7a.dat' --ftype 'txt' --chunksize 1 --file_names 'split_file' --file_ext 'txt'

          Exit Code:

          • 0

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"txt"
            __workflow_invocation_uuid__	"bc7f567f892511ee8e44b99b86b3f598"
            chromInfo	"/tmp/tmphx9vfffu/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            split_parms	{"__current_case__": 5, "input": {"values": [{"id": 1, "src": "hda"}]}, "newfilenames": "split_file", "select_allocate": {"__current_case__": 2, "allocate": "byrow"}, "select_ftype": "txt", "select_mode": {"__current_case__": 0, "chunksize": "1", "mode": "chunk"}}

    • Step 3: to parameter:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • cd ../; python _evaluate_expression_.py

          Exit Code:

          • 0

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"txt"
            __workflow_invocation_uuid__	"bc7f567f892511ee8e44b99b86b3f598"
            chromInfo	"/tmp/tmphx9vfffu/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            param_type	"text"
            remove_newlines	true

    • Step 4: fasterq-dump:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • set -o | grep -q pipefail && set -o pipefail;  mkdir -p ~/.ncbi && cp '/tmp/tmphx9vfffu/job_working_directory/000/4/configs/tmpp2r53kxu' ~/.ncbi/user-settings.mkfg && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s "/repository/user/ad/public/root=$PWD" && vdb-config -s "/repository/user/default-path=$PWD" && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s /http/timeout/read=10000 &&   export SRA_PREFETCH_RETRIES=3 && export SRA_PREFETCH_ATTEMPT=1 &&   echo 'SRR044777' | sed -r 's/(\,|\;|__cn__)/\n/g' > accessions && for acc in $(cat ./accessions); do ( echo "Downloading accession: $acc..." &&  while [ $SRA_PREFETCH_ATTEMPT -le $SRA_PREFETCH_RETRIES ] ; do fasterq-dump "$acc" -e ${GALAXY_SLOTS:-1} --seq-defline '@$sn/$ri' --qual-defline '+' --split-3 --skip-technical 2>&1 | tee -a '/tmp/tmphx9vfffu/job_working_directory/000/4/outputs/dataset_b6be4f30-a932-451f-b616-036d894a6254.dat'; if [ $? == 0 ] && [ $(ls *.fastq | wc -l) -ge 1 ]; then break ; else echo "Prefetch attempt $SRA_PREFETCH_ATTEMPT of $SRA_PREFETCH_RETRIES exited with code $?" ; SRA_PREFETCH_ATTEMPT=`expr $SRA_PREFETCH_ATTEMPT + 1` ; sleep 1 ; fi ; done && mkdir -p output && mkdir -p outputOther && count="$(ls *.fastq | wc -l)" && echo "There are $count fastq files" && data=($(ls *.fastq)) && if [ "$count" -eq 1 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"__single.fastqsanger.gz && rm "${data[0]}"; elif [ "--split-3" = "--split-3" ]; then if [ -e "${acc}".fastq ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${acc}".fastq > outputOther/"${acc}"__single.fastqsanger.gz; fi && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_1.fastq > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_2.fastq > output/"${acc}"_reverse.fastqsanger.gz && rm "${acc}"*.fastq; elif [ "$count" -eq 2 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${data[1]}" > output/"${acc}"_reverse.fastqsanger.gz && rm "${data[0]}" && rm "${data[1]}"; else for file in ${data[*]}; do pigz -cqp ${GALAXY_SLOTS:-1} "$file" > outputOther/"$file"sanger.gz && rm "$file"; done; fi;  ); done; echo "Done with all accessions."

          Exit Code:

          • 0

          Standard Output:

          • Downloading accession: SRR044777...
            spots read      : 7,882
            reads read      : 31,528
            reads written   : 7,882
            technical reads : 23,646
            There are 1 fastq files
            Done with all accessions.
            

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"input"
            __job_resource	{"__current_case__": 0, "__job_resource__select": "no"}
            __workflow_invocation_uuid__	"bc7f567f892511ee8e44b99b86b3f598"
            adv	{"minlen": null, "seq_defline": "@$sn/$ri", "skip_technical": true, "split": "--split-3"}
            chromInfo	"/tmp/tmphx9vfffu/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            input	{"__current_case__": 0, "accession": "SRR044777", "input_select": "accession_number"}

    • Step 5: flatten paired output:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"bc7f567f892511ee8e44b99b86b3f598"
            input	{"values": [{"id": 3, "src": "hdca"}]}
            rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}, {"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [2], "connectable": true, "is_workflow": false, "type": "paired_identifier"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier2", "warn": null}]}

    • Step 6: flatten single end output:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"bc7f567f892511ee8e44b99b86b3f598"
            input	{"values": [{"id": 4, "src": "hdca"}]}
            rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}]}

  • Other invocation details

    • history_id

      • cf81082452a404cf

    • history_state

      • ok

    • invocation_id

      • cf81082452a404cf

    • invocation_state

      • scheduled

    • workflow_id

      • 57e045a23ee9a2d2

Passed Tests

• ✅ parallel-accession-download.ga_1

  Workflow invocation details

  • Invocation Messages

  • Steps

    • Step 1: Run accessions:

      • step_state: scheduled

    • Step 2: Split accessions to collection:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • mkdir ./out && python '/tmp/shed_dir/toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/6cbe2f30c2d7/split_file_to_collection/split_file_to_collection.py' --out ./out --in '/tmp/tmphx9vfffu/files/b/b/e/dataset_bbed98bb-955a-4a26-86d2-b8e8c02c24f9.dat' --ftype 'txt' --chunksize 1 --file_names 'split_file' --file_ext 'txt'

          Exit Code:

          • 0

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"txt"
            __workflow_invocation_uuid__	"f9e0c5cb892511ee8e44b99b86b3f598"
            chromInfo	"/tmp/tmphx9vfffu/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            split_parms	{"__current_case__": 5, "input": {"values": [{"id": 7, "src": "hda"}]}, "newfilenames": "split_file", "select_allocate": {"__current_case__": 2, "allocate": "byrow"}, "select_ftype": "txt", "select_mode": {"__current_case__": 0, "chunksize": "1", "mode": "chunk"}}

    • Step 3: to parameter:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • cd ../; python _evaluate_expression_.py

          Exit Code:

          • 0

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"txt"
            __workflow_invocation_uuid__	"f9e0c5cb892511ee8e44b99b86b3f598"
            chromInfo	"/tmp/tmphx9vfffu/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            param_type	"text"
            remove_newlines	true

    • Step 4: fasterq-dump:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • set -o | grep -q pipefail && set -o pipefail;  mkdir -p ~/.ncbi && cp '/tmp/tmphx9vfffu/job_working_directory/000/10/configs/tmpcwl_6h16' ~/.ncbi/user-settings.mkfg && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s "/repository/user/ad/public/root=$PWD" && vdb-config -s "/repository/user/default-path=$PWD" && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s /http/timeout/read=10000 &&   export SRA_PREFETCH_RETRIES=3 && export SRA_PREFETCH_ATTEMPT=1 &&   echo 'SRR11953971' | sed -r 's/(\,|\;|__cn__)/\n/g' > accessions && for acc in $(cat ./accessions); do ( echo "Downloading accession: $acc..." &&  while [ $SRA_PREFETCH_ATTEMPT -le $SRA_PREFETCH_RETRIES ] ; do fasterq-dump "$acc" -e ${GALAXY_SLOTS:-1} --seq-defline '@$sn/$ri' --qual-defline '+' --split-3 --skip-technical 2>&1 | tee -a '/tmp/tmphx9vfffu/job_working_directory/000/10/outputs/dataset_27dedfa3-8df0-48fe-a6a4-31f421b47eb1.dat'; if [ $? == 0 ] && [ $(ls *.fastq | wc -l) -ge 1 ]; then break ; else echo "Prefetch attempt $SRA_PREFETCH_ATTEMPT of $SRA_PREFETCH_RETRIES exited with code $?" ; SRA_PREFETCH_ATTEMPT=`expr $SRA_PREFETCH_ATTEMPT + 1` ; sleep 1 ; fi ; done && mkdir -p output && mkdir -p outputOther && count="$(ls *.fastq | wc -l)" && echo "There are $count fastq files" && data=($(ls *.fastq)) && if [ "$count" -eq 1 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"__single.fastqsanger.gz && rm "${data[0]}"; elif [ "--split-3" = "--split-3" ]; then if [ -e "${acc}".fastq ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${acc}".fastq > outputOther/"${acc}"__single.fastqsanger.gz; fi && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_1.fastq > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_2.fastq > output/"${acc}"_reverse.fastqsanger.gz && rm "${acc}"*.fastq; elif [ "$count" -eq 2 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${data[1]}" > output/"${acc}"_reverse.fastqsanger.gz && rm "${data[0]}" && rm "${data[1]}"; else for file in ${data[*]}; do pigz -cqp ${GALAXY_SLOTS:-1} "$file" > outputOther/"$file"sanger.gz && rm "$file"; done; fi;  ); done; echo "Done with all accessions."

          Exit Code:

          • 0

          Standard Output:

          • Downloading accession: SRR11953971...
            spots read      : 2,057
            reads read      : 4,114
            reads written   : 4,114
            There are 2 fastq files
            Done with all accessions.
            

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"input"
            __job_resource	{"__current_case__": 0, "__job_resource__select": "no"}
            __workflow_invocation_uuid__	"f9e0c5cb892511ee8e44b99b86b3f598"
            adv	{"minlen": null, "seq_defline": "@$sn/$ri", "skip_technical": true, "split": "--split-3"}
            chromInfo	"/tmp/tmphx9vfffu/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            input	{"__current_case__": 0, "accession": "SRR11953971", "input_select": "accession_number"}

    • Step 5: flatten paired output:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"f9e0c5cb892511ee8e44b99b86b3f598"
            input	{"values": [{"id": 11, "src": "hdca"}]}
            rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}, {"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [2], "connectable": true, "is_workflow": false, "type": "paired_identifier"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier2", "warn": null}]}

    • Step 6: flatten single end output:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"f9e0c5cb892511ee8e44b99b86b3f598"
            input	{"values": [{"id": 12, "src": "hdca"}]}
            rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}]}

  • Other invocation details

    • history_id

      • 57e045a23ee9a2d2

    • history_state

      • ok

    • invocation_id

      • 57e045a23ee9a2d2

    • invocation_state

      • scheduled

    • workflow_id

      • 57e045a23ee9a2d2

[lldelisle]

Test Results (powered by Planemo)

Test Summary

Test State	Count
Total	2
Passed	1
Error	0
Failure	1
Skipped	0

Failed Tests

• ❌ parallel-accession-download.ga_0

  Problems:

  • Output with path /tmp/tmp811u32yk/SRR044777__0f26ecca-4561-4f8c-a158-11022c1b81ff different than expected, difference (using contains):
    ( /home/runner/work/iwc/iwc/workflows/data-fetching/parallel-accession-download/test-data/SRR044777_head.fastq v. /tmp/tmpcm1_t2qaSRR044777_head.fastq )
    Failed to find 'b'@F47USSH02H1LGA/4'' in history data. (lines_diff=0).
    

  Workflow invocation details

  • Invocation Messages

  • Steps

    • Step 1: Run accessions:

      • step_state: scheduled

    • Step 2: Split accessions to collection:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • mkdir ./out && python '/tmp/shed_dir/toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/6cbe2f30c2d7/split_file_to_collection/split_file_to_collection.py' --out ./out --in '/tmp/tmpymn_lcfl/files/a/2/e/dataset_a2e71cfd-4253-47f3-a15d-b88cfdc444ad.dat' --ftype 'txt' --chunksize 1 --file_names 'split_file' --file_ext 'txt'

          Exit Code:

          • 0

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"txt"
            __workflow_invocation_uuid__	"d9707a2d894011ee9e3ff5a83c47b9ae"
            chromInfo	"/tmp/tmpymn_lcfl/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            split_parms	{"__current_case__": 5, "input": {"values": [{"id": 1, "src": "hda"}]}, "newfilenames": "split_file", "select_allocate": {"__current_case__": 2, "allocate": "byrow"}, "select_ftype": "txt", "select_mode": {"__current_case__": 0, "chunksize": "1", "mode": "chunk"}}

    • Step 3: to parameter:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • cd ../; python _evaluate_expression_.py

          Exit Code:

          • 0

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"txt"
            __workflow_invocation_uuid__	"d9707a2d894011ee9e3ff5a83c47b9ae"
            chromInfo	"/tmp/tmpymn_lcfl/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            param_type	"text"
            remove_newlines	true

    • Step 4: fasterq-dump:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • set -o | grep -q pipefail && set -o pipefail;  mkdir -p ~/.ncbi && cp '/tmp/tmpymn_lcfl/job_working_directory/000/4/configs/tmpohid4fd5' ~/.ncbi/user-settings.mkfg && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s "/repository/user/ad/public/root=$PWD" && vdb-config -s "/repository/user/default-path=$PWD" && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s /http/timeout/read=10000 &&   export SRA_PREFETCH_RETRIES=3 && export SRA_PREFETCH_ATTEMPT=1 &&   echo 'SRR044777' | sed -r 's/(\,|\;|__cn__)/\n/g' > accessions && for acc in $(cat ./accessions); do ( echo "Downloading accession: $acc..." &&  while [ $SRA_PREFETCH_ATTEMPT -le $SRA_PREFETCH_RETRIES ] ; do fasterq-dump "$acc" -e ${GALAXY_SLOTS:-1} --seq-defline '@$sn/$ri' --qual-defline '+' --split-3 --skip-technical 2>&1 | tee -a '/tmp/tmpymn_lcfl/job_working_directory/000/4/outputs/dataset_f3f15cac-6c96-47b6-ad99-0dec48bd915e.dat'; if [ $? == 0 ] && [ $(ls *.fastq | wc -l) -ge 1 ]; then break ; else echo "Prefetch attempt $SRA_PREFETCH_ATTEMPT of $SRA_PREFETCH_RETRIES exited with code $?" ; SRA_PREFETCH_ATTEMPT=`expr $SRA_PREFETCH_ATTEMPT + 1` ; sleep 1 ; fi ; done && mkdir -p output && mkdir -p outputOther && count="$(ls *.fastq | wc -l)" && echo "There are $count fastq files" && data=($(ls *.fastq)) && if [ "$count" -eq 1 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"__single.fastqsanger.gz && rm "${data[0]}"; elif [ "--split-3" = "--split-3" ]; then if [ -e "${acc}".fastq ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${acc}".fastq > outputOther/"${acc}"__single.fastqsanger.gz; fi && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_1.fastq > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_2.fastq > output/"${acc}"_reverse.fastqsanger.gz && rm "${acc}"*.fastq; elif [ "$count" -eq 2 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${data[1]}" > output/"${acc}"_reverse.fastqsanger.gz && rm "${data[0]}" && rm "${data[1]}"; else for file in ${data[*]}; do pigz -cqp ${GALAXY_SLOTS:-1} "$file" > outputOther/"$file"sanger.gz && rm "$file"; done; fi;  ); done; echo "Done with all accessions."

          Exit Code:

          • 0

          Standard Output:

          • Downloading accession: SRR044777...
            spots read      : 7,882
            reads read      : 31,528
            reads written   : 7,882
            technical reads : 23,646
            There are 1 fastq files
            Done with all accessions.
            

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"input"
            __job_resource	{"__current_case__": 0, "__job_resource__select": "no"}
            __workflow_invocation_uuid__	"d9707a2d894011ee9e3ff5a83c47b9ae"
            adv	{"minlen": null, "seq_defline": "@$sn/$ri", "skip_technical": true, "split": "--split-3"}
            chromInfo	"/tmp/tmpymn_lcfl/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            input	{"__current_case__": 0, "accession": "SRR044777", "input_select": "accession_number"}

    • Step 5: flatten paired output:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"d9707a2d894011ee9e3ff5a83c47b9ae"
            input	{"values": [{"id": 3, "src": "hdca"}]}
            rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}, {"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [2], "connectable": true, "is_workflow": false, "type": "paired_identifier"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier2", "warn": null}]}

    • Step 6: flatten single end output:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"d9707a2d894011ee9e3ff5a83c47b9ae"
            input	{"values": [{"id": 4, "src": "hdca"}]}
            rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}]}

  • Other invocation details

    • history_id

      • c288193257a840dc

    • history_state

      • ok

    • invocation_id

      • c288193257a840dc

    • invocation_state

      • scheduled

    • workflow_id

      • 22cbb50759f9776d

Passed Tests

• ✅ parallel-accession-download.ga_1

  Workflow invocation details

  • Invocation Messages

  • Steps

    • Step 1: Run accessions:

      • step_state: scheduled

    • Step 2: Split accessions to collection:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • mkdir ./out && python '/tmp/shed_dir/toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/6cbe2f30c2d7/split_file_to_collection/split_file_to_collection.py' --out ./out --in '/tmp/tmpymn_lcfl/files/e/3/3/dataset_e3361491-a8b9-43e5-be1d-bab32b5ad152.dat' --ftype 'txt' --chunksize 1 --file_names 'split_file' --file_ext 'txt'

          Exit Code:

          • 0

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"txt"
            __workflow_invocation_uuid__	"029b1665894111ee9e3ff5a83c47b9ae"
            chromInfo	"/tmp/tmpymn_lcfl/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            split_parms	{"__current_case__": 5, "input": {"values": [{"id": 7, "src": "hda"}]}, "newfilenames": "split_file", "select_allocate": {"__current_case__": 2, "allocate": "byrow"}, "select_ftype": "txt", "select_mode": {"__current_case__": 0, "chunksize": "1", "mode": "chunk"}}

    • Step 3: to parameter:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • cd ../; python _evaluate_expression_.py

          Exit Code:

          • 0

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"txt"
            __workflow_invocation_uuid__	"029b1665894111ee9e3ff5a83c47b9ae"
            chromInfo	"/tmp/tmpymn_lcfl/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            param_type	"text"
            remove_newlines	true

    • Step 4: fasterq-dump:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • set -o | grep -q pipefail && set -o pipefail;  mkdir -p ~/.ncbi && cp '/tmp/tmpymn_lcfl/job_working_directory/000/10/configs/tmp3buqv7b_' ~/.ncbi/user-settings.mkfg && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s "/repository/user/ad/public/root=$PWD" && vdb-config -s "/repository/user/default-path=$PWD" && vdb-config -s "/repository/user/main/public/root=$PWD" && vdb-config -s /http/timeout/read=10000 &&   export SRA_PREFETCH_RETRIES=3 && export SRA_PREFETCH_ATTEMPT=1 &&   echo 'SRR11953971' | sed -r 's/(\,|\;|__cn__)/\n/g' > accessions && for acc in $(cat ./accessions); do ( echo "Downloading accession: $acc..." &&  while [ $SRA_PREFETCH_ATTEMPT -le $SRA_PREFETCH_RETRIES ] ; do fasterq-dump "$acc" -e ${GALAXY_SLOTS:-1} --seq-defline '@$sn/$ri' --qual-defline '+' --split-3 --skip-technical 2>&1 | tee -a '/tmp/tmpymn_lcfl/job_working_directory/000/10/outputs/dataset_63937aff-a407-4b3f-8069-afe7d1ac9832.dat'; if [ $? == 0 ] && [ $(ls *.fastq | wc -l) -ge 1 ]; then break ; else echo "Prefetch attempt $SRA_PREFETCH_ATTEMPT of $SRA_PREFETCH_RETRIES exited with code $?" ; SRA_PREFETCH_ATTEMPT=`expr $SRA_PREFETCH_ATTEMPT + 1` ; sleep 1 ; fi ; done && mkdir -p output && mkdir -p outputOther && count="$(ls *.fastq | wc -l)" && echo "There are $count fastq files" && data=($(ls *.fastq)) && if [ "$count" -eq 1 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"__single.fastqsanger.gz && rm "${data[0]}"; elif [ "--split-3" = "--split-3" ]; then if [ -e "${acc}".fastq ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${acc}".fastq > outputOther/"${acc}"__single.fastqsanger.gz; fi && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_1.fastq > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${acc}"_2.fastq > output/"${acc}"_reverse.fastqsanger.gz && rm "${acc}"*.fastq; elif [ "$count" -eq 2 ]; then pigz -cqp ${GALAXY_SLOTS:-1} "${data[0]}" > output/"${acc}"_forward.fastqsanger.gz && pigz -cqp ${GALAXY_SLOTS:-1} "${data[1]}" > output/"${acc}"_reverse.fastqsanger.gz && rm "${data[0]}" && rm "${data[1]}"; else for file in ${data[*]}; do pigz -cqp ${GALAXY_SLOTS:-1} "$file" > outputOther/"$file"sanger.gz && rm "$file"; done; fi;  ); done; echo "Done with all accessions."

          Exit Code:

          • 0

          Standard Output:

          • Downloading accession: SRR11953971...
            spots read      : 2,057
            reads read      : 4,114
            reads written   : 4,114
            There are 2 fastq files
            Done with all accessions.
            

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"input"
            __job_resource	{"__current_case__": 0, "__job_resource__select": "no"}
            __workflow_invocation_uuid__	"029b1665894111ee9e3ff5a83c47b9ae"
            adv	{"minlen": null, "seq_defline": "@$sn/$ri", "skip_technical": true, "split": "--split-3"}
            chromInfo	"/tmp/tmpymn_lcfl/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            input	{"__current_case__": 0, "accession": "SRR11953971", "input_select": "accession_number"}

    • Step 5: flatten paired output:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"029b1665894111ee9e3ff5a83c47b9ae"
            input	{"values": [{"id": 11, "src": "hdca"}]}
            rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}, {"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [2], "connectable": true, "is_workflow": false, "type": "paired_identifier"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier2", "warn": null}]}

    • Step 6: flatten single end output:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"029b1665894111ee9e3ff5a83c47b9ae"
            input	{"values": [{"id": 12, "src": "hdca"}]}
            rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}]}

  • Other invocation details

    • history_id

      • 22cbb50759f9776d

    • history_state

      • ok

    • invocation_id

      • 22cbb50759f9776d

    • invocation_state

      • scheduled

    • workflow_id

      • 22cbb50759f9776d