From 7043c10e2a3d1272fd2a57eca3f08b5337a315e4 Mon Sep 17 00:00:00 2001 From: Henrique Ribeiro Date: Tue, 9 May 2023 10:08:37 +0100 Subject: [PATCH] Version 85 (#30) * 85 release (#28) * Update cromwell version from 83 to 84 * BW-1255 Implement POST /runs endpoint (#6779) * Adding route * Fixing HTTP method error * All formFields made optional * A compliling state * Saving * Saving * All three endpoints functioning as expected; updated RESTAPI.md * Updated response for submission from 200 to 201 to pass tests * Test submission response * Moved updated submission response to askSubmit * test * updating RESTAPI.md * saving * Adding utility file for submitRequest * cleanup * Update awssdkv from 2.17.152 to 2.17.194 (#6814) * BW-1305 Swagger Update (#6818) * Properly documenting metadataArchiveStatus in WorkflowQueryResult model * Update docs * BT-710 Add configs for BlobPathBuilderFactory (#6817) BT-710 Add configs for BlobPathBuilderFactory * BW-1305 Make "name" optional in workflow query response (#6821) * BT-724 Fix BlobPathBuilder failing on retrieving existing filesystem (#6816) Modify blobPathBuilder to fallback to creating a filesystem if one is not found * Logging updates: (#6813) * [BT-698] first pass on BlobTokenGenerator with E2E test (#6824) * first pass on BlobTokenGenerator with E2E test * update BlobPathBuilder constructor args in test * account -> container level client * [BT-687] specify correct types (#6829) * specify correct types * fix test with new type * remove type declarations in function call * remove unnecessary sas-token config * BW-1206 - Combine all Wes Endpoints & add Tests (#6833) * Add tests, getting frid of WesRunRoutes.scala * wesWorkflowId fix, ec implicits errors gone * Refactoring path for GET /runs * Indentation fix * Commit to rollback * Revert "Indentation fix" This reverts commit 63fc4842c9d4eff68ec9cb7c3ef19e110696598b. * PR trigger * Optimize imports * Missed import * BW-1354 - Porting CBAS preliminary step (#6837) * Getting rid of shared utility file; Adding/Updating WES version of submit. * Edit spec file * Adding Wes-like error * BW-1378 Addl CromIAM user enablement checks (#6826) * Update cromwell version from 84 to 85 * BW-1393 Release doc updates (#6839) * BT-732 Checksum validation for blobs read by engine (#6838) * Draft support for optional FileHash * Draft getMd5 for BlobPath * Resolve non-parallel IO to fix tests * Checksum validation for BlobPath * Nicer error message * Test for missing Blob hash * Break attr acquisition into separate method * Cleanup, comments * In-progress tests of blob hash command * Remove test * Remove unused import * BT-711 Refresh SAS token for filesystem on expiry (#6831) * BT-711 Refresh SAS token for filesystem on expiry * Rough cut of token refresh using exceptions * Ignore tests, and minor cleanup * Remove stray line * Draft of manager class for handling expiring file systems * Style fixes * Refactor of blobfilesystemManager and tests covering its functionality * Refined tests to validate close filesystem as separate unit * Ignore connected tests * Clean up of some things * Refactor BlobFileSystemManager to separate file, and some other cleanup * Some additional scala-ifying * Small cleanup * Correcting imports * trigger tests * trigger tests * Batch 1 of scala steward updates (#6903) * Batch 1 of scala steward updates * Rollback snakeYAML * Attempt 3, with only the passing dependancies * Revert google API and Big Query udpates * Winding back other google deps * rollback remaining google updates * trigger tests * trigger tests * [BW-1398] Migrate PKs to BIGINT (#6907) * BT-745 Batch 2 of scala steward updates (#6906) * Update SBT to 2.0.0 * Fix sbt-git import * Update mouse to 1.0.11 * Update rhino 1.7.14 * SUP-692 Retry with more memory after RC 137 (#6912) * Reorder execution result checks so 137 can retry with more memory * Test for memory retry after 137 RC * Fix test expectations * Make memory retry checks consistent * Revert changes to existing test * Rename retryWithMoreMemory to outOfMemoryDetected * Scala steward updates batch 3 (#6913) * Scala steward updates batch 3 * WX-745 Batch 4 scala steward updates (#6916) * WX-746 Localize all DRS inputs in a single Action (#6914) Co-authored-by: Janet Gainer-Dewar * WX-755 Build all images instead of just Cromwell (#6919) * WX-755 Add `isRelease` option for Docker builds (#6923) * WX-755 Cromwell/CromIAM automatically board train (#6924) * WX-755 Fix environment variable syntax (#6926) * WX-743 Enable TES task creation with BlobPaths (#6921) * Give blob SAS tokens write permission * Case class wrapper for subscription id * Resolve duplicate container name in absolute BlobPath * Ignored test demonstrating correct absolute path generation * Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala Co-authored-by: Brian Reilly * PR feedback Co-authored-by: Brian Reilly * [WX-765] Update snakeyaml to 1.33 (#6927) * update snakeyaml to 1.33 * Don't use deprecated no-arg Constructor constructor Co-authored-by: Janet Gainer-Dewar * WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (#6925) * Upgrade Postgres to 42.4.1 (#6932) * WX-735 Fix incorrect and/or nondeterministic filesystem ordering (#6930) * WX-772 Update Scala to 2.13.9 (#6928) * Update Scala to 2.13.9 * Try updating sbt-scoverage * Does this version exist anywhere we can see? * This version actually exists * Update library version to remove conflict * Codegen version * Fix fun new 2.13.9 compiler errors * Resolve warnings * Newest Scala? * I guess not * Does this please Travis? * force ci * Back out changes to generated code Co-authored-by: Adam Nichols * WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (#6935) Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1. - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) --- updated-dependencies: - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-type: direct:production ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * WX-808 Host allowlist for HTTP imports (#6938) * `hostAllowlist` that allows everything * Refactor * Stick allow list in HttpResolver * Better default config * Allow list tests * Make it build Co-authored-by: Janet Gainer-Dewar * Update commons text to 1.10.0 (#6937) * WX-751 Token refresh signal for monitoring (#6939) * Log messages * `DEBUG` -> `INFO` * WX-744 Optionally rewrite blob paths to appear as local paths (#6941) * Modify blob paths for TES * Make blob transformation configurable * Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala Co-authored-by: Adam Nichols * Apply PR feedback in second place Co-authored-by: Adam Nichols * Update changelog for wdl http allow list (#6944) * WM-1491 Fixing Cromwell-client (#6943) * More updated client for use in cbas * Removing excess code * Fix client build script (#6945) * WX-837: Remove CWL references from documentation (#6949) * wx-837 removed cwl references in markdown doc files * wx-837 removed cwlParsingOverview.md, updated mkdocs.yml * wx-837 updated cromwell.yaml, generated new RESTAPI file * WX-728 Add configurable WSM client to Cromwell (#6948) * Dependencies * Compiles but no tests * Formatting * Moar exclusions * Update to latest WSM * Add additional dependency * We need some UUID here to make the request * Formatting * Clarify what is fake * Formatting * Use our own version of Jersey and Jackson stuff * Port-in Khalid's changes (thank you!) Co-authored-by: Khalid Shakir * Test longevity Don't break the test if someone decides to add a cert to `ws.org` * Cleanup * Cleanup * Cleanup * Adjust TES config file for CI Co-authored-by: Janet Gainer-Dewar Co-authored-by: Khalid Shakir * CROM-6554: Removed PAPIv1 references from doc (#6950) * crom-6554 removed references to PAPI v1 from doc * crom-6554 pr feedback, reworded doc to use example conf as a starting point * WX-833 Real Azure DRS Credentials (#6952) * Remove B2C reference from name * Get token for current user rather than getting from KeyVault * Remove KeyVault config for engine * Remove KeyVault config for DRSLocalizer * Remove KeyVault dependency * Remove KeyVault support from localizer repo template * Cleaned up and working Azure token acquisition for engine * Collapse localizer's AccessTokenStrategy into DrsCredentials * Cleanup * WX-853 Remove most CWL (#6955) * WX-696 Enable getting SAS token from WSM (#6954) * WX-696 Enable getting SAS token from WSM * Wire container resource id from config * Move resource-container-id config path * First pass at config for WSM * Remove unused singleton config * Tests for new config * Fix config parsing * Modified b2c token to be provided each time * Remove singletonConfig arg from factory * Restore types to factory configs * Clean up comments and empty token default * Default to config b2c before searching environment * Fix token default on api client * Fix test * Refactor error handling for when there is no token * Remove token constructor arg for clientProvider * Move configs to global singleton config * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala * default -> override * Add override token to test * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala Co-authored-by: Adam Nichols * Parentheses * Reduce token timeout * Move AzureCredentials to separate file * Make AzureCredentials an object * WSM token cleanup * Config refactor (#6960) Co-authored-by: Janet Gainer-Dewar * Initial blob token documentation * Refine language in BlobSasTokenGenerator * Update comment and formatting Co-authored-by: Janet Gainer-Dewar Co-authored-by: Adam Nichols * WX-853 Remove CWL language factory, Centaur runner (#6961) * WX-842 Add Pact Dependency for Cromwell (#6962) * WX-842 Add Pact Dependency for Cromwell * Remove incomplete test spec * Initial Pact Test * Fix pact so it compiles * Add breadcrumb comment and clean up * ID-125 Add support for drshub, rename all the things (#6959) * Add support for drshub, rename all the things * fallback to martha if resolver is not in config * WX-867 Translate crc32c hashes to b64 for getm (#6970) * Translate crc32c hashes to b64 for getm * Update tests * Remove obsolete b64 handling for md5, centralize hex validation * Restore old test, fix other test * WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (#6965) * WX-859 Accept workflow execution identity in config (#6967) * WX-892 Trim down `ValueStore` logging to prevent OOMs (#6981) * Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (#6975) * WX-863 Turn off Azure NIO logging (#6982) * Turn off Azure NIO logging * Poke Travis * WM-1616: Allow repeating attempts at initialization (take 2) (#6985) * WX-878 Single shared BlobFileSystemManager (#6986) * Make BlobFileSystemManager shared across all BlobPathBuilders * Update TES conf file to reflect new singleton config * Shell escape reference image files [VS-796] [WX-910] (#6989) * WX-769 `disks` compatibility for TES backend (#6991) * Update FiveMinuteIntro.md (#6994) * WX-906 Sbt Unit Tests as Github Actions (#6992) * WX-926 Support falling back to OCI Manifest Format (#7003) * WX-926 Support falling back to OCI Manifest Forma * Only mount reference disks if requested [WX-925] (#7001) * [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (#7004) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (#6980) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status * Address feedback * Address feedback (#6997) * Address additional feedback (#7000) * Fix copy/paste error (#7005) * Address additional feedback * Fix copy/paste error * Trigger CI --------- Co-authored-by: Blair Murri Co-authored-by: Janet Gainer-Dewar * Centaur reference image test should validate symlinks [VS-796] (#6996) * WX-903 Pre-GHA test suite disablement * WX-877 Update CHANGELOG for release 85 (#7011) --------- Signed-off-by: dependabot[bot] Co-authored-by: Janet Gainer-Dewar Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com> Co-authored-by: Chris Llanwarne Co-authored-by: Christian Freitas Co-authored-by: Saloni Shah Co-authored-by: kshakir Co-authored-by: mspector Co-authored-by: Adam Nichols Co-authored-by: Brian Reilly Co-authored-by: Adam Nichols Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Justin Variath Thomas Co-authored-by: Christian Freitas Co-authored-by: Trevyn Langsford Co-authored-by: Miguel Covarrubias Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com> Co-authored-by: Tom Wiseman Co-authored-by: Blair Murri * Develop aws (#29) * stuck on globbing * efs works, no callcaching * update readme * extended EFS support * fix for globbing in nested scatters * updated config for globbing, to prevent issues with empty folders --------- Signed-off-by: dependabot[bot] Co-authored-by: Janet Gainer-Dewar Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com> Co-authored-by: Chris Llanwarne Co-authored-by: Christian Freitas Co-authored-by: Saloni Shah Co-authored-by: kshakir Co-authored-by: mspector Co-authored-by: Adam Nichols Co-authored-by: Brian Reilly Co-authored-by: Adam Nichols Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Justin Variath Thomas Co-authored-by: Christian Freitas Co-authored-by: Trevyn Langsford Co-authored-by: Miguel Covarrubias Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com> Co-authored-by: Tom Wiseman Co-authored-by: Blair Murri Co-authored-by: geertvandeweyer --- .github/set_up_cromwell_action/action.yml | 47 + .github/workflows/chart_update_on_merge.yml | 26 +- .github/workflows/cromwell_unit_tests.yml | 32 + .gitignore | 3 + .travis.yml | 23 +- CHANGELOG.md | 326 +- CromIAM/src/main/resources/application.conf | 2 +- CromIAM/src/main/resources/logback.xml | 14 +- .../CromIamInstrumentation.scala | 1 + .../main/scala/cromiam/sam/SamClient.scala | 33 + .../webservice/CromIamApiService.scala | 16 +- .../cromiam/webservice/QuerySupport.scala | 2 +- .../cromiam/webservice/RequestSupport.scala | 26 +- .../webservice/SubmissionSupport.scala | 2 +- .../webservice/WomtoolRouteSupport.scala | 10 +- .../webservice/CromIamApiServiceSpec.scala | 38 +- .../cromiam/webservice/MockClients.scala | 9 + .../webservice/SwaggerServiceSpec.scala | 7 +- .../webservice/WomtoolRouteSupportSpec.scala | 54 +- CromwellRefdiskManifestCreator/pom.xml | 2 +- .../StandardAsyncExecutionActor.scala | 26 +- .../standard/StandardCachingActorHelper.scala | 2 +- .../RootWorkflowFileHashCacheActor.scala | 2 +- .../StandardCacheHitCopyingActor.scala | 5 + build.sbt | 30 +- .../centaur/reporting/BigQueryReporter.scala | 2 +- .../bcbio_joint_gvcf.test | 23 - .../integrationTestCases/bcbio_prealign.test | 20 - .../integrationTestCases/bcbio_rnaseq.test | 19 - .../integrationTestCases/bcbio_somatic.test | 22 - .../integrationTestCases/bcbio_svcall.test | 22 - .../bcbio_wes-chr21-test-workflow.test | 22 - .../cwl/bcbio/bcbio.options | 5 - .../main-gvcf-joint-samples.json | 615 - .../gvcf-joint-workflow/main-gvcf-joint.cwl | 794 - .../steps/alignment_to_rec.cwl | 198 - .../steps/batch_for_jointvc.cwl | 329 - .../steps/batch_for_variantcall.cwl | 401 - .../steps/combine_sample_regions.cwl | 99 - .../steps/compare_to_rm.cwl | 337 - .../steps/concat_batch_variantcalls.cwl | 196 - .../concat_batch_variantcalls_jointvc.cwl | 209 - .../steps/finalize_jointvc.cwl | 331 - .../steps/get_parallel_regions.cwl | 171 - .../steps/get_parallel_regions_jointvc.cwl | 184 - .../steps/merge_split_alignments.cwl | 168 - .../steps/multiqc_summary.cwl | 95 - .../steps/pipeline_summary.cwl | 219 - .../steps/postprocess_alignment.cwl | 223 - .../steps/postprocess_alignment_to_rec.cwl | 233 - .../steps/postprocess_variants.cwl | 195 - .../steps/prep_align_inputs.cwl | 142 - .../steps/prep_samples.cwl | 95 - .../steps/prep_samples_to_rec.cwl | 85 - .../steps/process_alignment.cwl | 198 - .../gvcf-joint-workflow/steps/qc_to_rec.cwl | 295 - .../gvcf-joint-workflow/steps/run_jointvc.cwl | 203 - .../steps/summarize_vc.cwl | 225 - .../steps/variantcall_batch_region.cwl | 266 - .../gvcf-joint-workflow/wf-alignment.cwl | 143 - .../gvcf-joint-workflow/wf-jointcall.cwl | 360 - .../gvcf-joint-workflow/wf-variantcall.cwl | 329 - .../main-prealign-samples.json | 526 - .../bcbio/prealign-workflow/main-prealign.cwl | 672 - .../steps/batch_for_variantcall.cwl | 407 - .../steps/combine_sample_regions.cwl | 101 - .../prealign-workflow/steps/compare_to_rm.cwl | 315 - .../steps/concat_batch_variantcalls.cwl | 199 - .../steps/get_parallel_regions.cwl | 174 - .../steps/multiqc_summary.cwl | 95 - .../steps/organize_noalign.cwl | 59 - .../steps/pipeline_summary.cwl | 221 - .../steps/postprocess_alignment.cwl | 221 - .../steps/postprocess_alignment_to_rec.cwl | 229 - .../steps/postprocess_variants.cwl | 183 - .../prealign-workflow/steps/prep_samples.cwl | 93 - .../steps/prep_samples_to_rec.cwl | 81 - .../prealign-workflow/steps/qc_to_rec.cwl | 299 - .../prealign-workflow/steps/summarize_vc.cwl | 198 - .../steps/variantcall_batch_region.cwl | 269 - .../prealign-workflow/wf-variantcall.cwl | 316 - .../rnaseq-workflow/main-rnaseq-samples.json | 111 - .../cwl/bcbio/rnaseq-workflow/main-rnaseq.cwl | 304 - .../rnaseq-workflow/steps/detect_fusions.cwl | 97 - .../rnaseq-workflow/steps/multiqc_summary.cwl | 96 - .../steps/pipeline_summary.cwl | 155 - .../rnaseq-workflow/steps/prepare_sample.cwl | 146 - .../steps/process_alignment.cwl | 108 - .../bcbio/rnaseq-workflow/steps/qc_to_rec.cwl | 135 - .../steps/rnaseq_quantitate.cwl | 123 - .../rnaseq-workflow/steps/trim_sample.cwl | 140 - .../main-somatic-samples.json | 680 - .../bcbio/somatic-workflow/main-somatic.cwl | 973 - .../steps/alignment_to_rec.cwl | 200 - .../somatic-workflow/steps/batch_for_sv.cwl | 375 - .../steps/batch_for_variantcall.cwl | 407 - .../steps/calculate_sv_bins.cwl | 225 - .../steps/calculate_sv_coverage.cwl | 218 - .../steps/combine_sample_regions.cwl | 99 - .../somatic-workflow/steps/compare_to_rm.cwl | 311 - .../steps/concat_batch_variantcalls.cwl | 198 - .../somatic-workflow/steps/detect_sv.cwl | 414 - .../steps/get_parallel_regions.cwl | 173 - .../steps/merge_split_alignments.cwl | 169 - .../steps/multiqc_summary.cwl | 96 - .../steps/normalize_sv_coverage.cwl | 236 - .../steps/pipeline_summary.cwl | 223 - .../steps/postprocess_alignment.cwl | 224 - .../steps/postprocess_alignment_to_rec.cwl | 235 - .../steps/postprocess_variants.cwl | 182 - .../steps/prep_align_inputs.cwl | 143 - .../somatic-workflow/steps/prep_samples.cwl | 95 - .../steps/prep_samples_to_rec.cwl | 85 - .../steps/process_alignment.cwl | 199 - .../somatic-workflow/steps/qc_to_rec.cwl | 310 - .../somatic-workflow/steps/summarize_sv.cwl | 242 - .../somatic-workflow/steps/summarize_vc.cwl | 195 - .../steps/variantcall_batch_region.cwl | 268 - .../bcbio/somatic-workflow/wf-alignment.cwl | 144 - .../cwl/bcbio/somatic-workflow/wf-svcall.cwl | 328 - .../bcbio/somatic-workflow/wf-variantcall.cwl | 312 - .../svcall-workflow/main-svcall-samples.json | 707 - .../cwl/bcbio/svcall-workflow/main-svcall.cwl | 975 - .../steps/alignment_to_rec.cwl | 206 - .../svcall-workflow/steps/batch_for_sv.cwl | 359 - .../steps/batch_for_variantcall.cwl | 401 - .../steps/calculate_sv_bins.cwl | 221 - .../steps/calculate_sv_coverage.cwl | 207 - .../steps/combine_sample_regions.cwl | 99 - .../svcall-workflow/steps/compare_to_rm.cwl | 307 - .../steps/concat_batch_variantcalls.cwl | 196 - .../bcbio/svcall-workflow/steps/detect_sv.cwl | 396 - .../steps/get_parallel_regions.cwl | 171 - .../steps/merge_split_alignments.cwl | 171 - .../svcall-workflow/steps/multiqc_summary.cwl | 95 - .../steps/normalize_sv_coverage.cwl | 224 - .../steps/pipeline_summary.cwl | 221 - .../steps/postprocess_alignment.cwl | 223 - .../steps/postprocess_alignment_to_rec.cwl | 233 - .../steps/postprocess_variants.cwl | 180 - .../steps/prep_align_inputs.cwl | 145 - .../svcall-workflow/steps/prep_samples.cwl | 93 - .../steps/prep_samples_to_rec.cwl | 79 - .../steps/process_alignment.cwl | 201 - .../bcbio/svcall-workflow/steps/qc_to_rec.cwl | 308 - .../svcall-workflow/steps/summarize_sv.cwl | 228 - .../svcall-workflow/steps/summarize_vc.cwl | 193 - .../steps/variantcall_batch_region.cwl | 266 - .../bcbio/svcall-workflow/wf-alignment.cwl | 146 - .../cwl/bcbio/svcall-workflow/wf-svcall.cwl | 310 - .../bcbio/svcall-workflow/wf-variantcall.cwl | 308 - .../main-wes_chr21_test-samples.json | 1059 - .../main-wes_chr21_test.cwl | 997 - .../steps/alignment_to_rec.cwl | 194 - .../steps/batch_for_ensemble.cwl | 273 - .../steps/batch_for_sv.cwl | 363 - .../steps/batch_for_variantcall.cwl | 395 - .../steps/calculate_sv_bins.cwl | 219 - .../steps/calculate_sv_coverage.cwl | 206 - .../steps/combine_calls.cwl | 180 - .../steps/combine_sample_regions.cwl | 99 - .../steps/compare_to_rm.cwl | 299 - .../steps/concat_batch_variantcalls.cwl | 190 - .../steps/detect_sv.cwl | 402 - .../steps/get_parallel_regions.cwl | 165 - .../steps/merge_split_alignments.cwl | 165 - .../steps/multiqc_summary.cwl | 94 - .../steps/normalize_sv_coverage.cwl | 224 - .../steps/pipeline_summary.cwl | 219 - .../steps/postprocess_alignment.cwl | 221 - .../steps/postprocess_alignment_to_rec.cwl | 237 - .../steps/postprocess_variants.cwl | 174 - .../steps/prep_align_inputs.cwl | 139 - .../steps/prep_samples.cwl | 95 - .../steps/prep_samples_to_rec.cwl | 85 - .../steps/process_alignment.cwl | 195 - .../steps/qc_to_rec.cwl | 307 - .../steps/summarize_sv.cwl | 236 - .../steps/summarize_vc.cwl | 212 - .../steps/variantcall_batch_region.cwl | 260 - .../wf-alignment.cwl | 140 - .../wes_chr21_test-workflow-gcp/wf-svcall.cwl | 316 - .../wf-variantcall.cwl | 300 - ...InitialWorkDirRequirement.inline_file.test | 16 - ...uirement.inline_file_custom_entryname.test | 16 - ...nitialWorkDirRequirement.input_string.test | 16 - ...kDirRequirement.input_string_function.test | 16 - .../file_array_listing.cwl | 26 - .../InitialWorkDirRequirement/inline_file.cwl | 43 - .../inline_file_custom_entryname.cwl | 47 - .../input_string.cwl | 50 - .../input_string_function.cwl | 49 - .../ad_hoc_file_test.aws.test | 14 - .../ad_hoc_file_test/cwl-test.cwl | 21 - .../ad_hoc_file_test/workflow.cwl | 10 - .../blob_md5/blob_md5.inputs.json | 3 + .../standardTestCases/blob_md5/blob_md5.wdl | 20 + .../cwl_cache_between_workflows.test | 19 - .../cwl_cache_between_workflows.cwl | 79 - .../cwl_cache_between_workflows.json | 1 - .../cwl_cache_within_workflow.test | 26 - .../cwl_cache_within_workflow.cwl | 129 - .../cwl_cache_within_workflow.json | 4 - .../cwl_disk_resources_papiv2.test | 17 - .../standardTestCases/cwl_docker_size.test | 21 - .../cwl_docker_size/cwl_docker_size.cwl | 40 - .../cwl_docker_size/cwl_docker_size.options | 3 - .../cwl_docker_size/cwl_docker_size.yaml | 3 - .../cwl_docker_size/cwl_docker_size_input.txt | 1 - .../cwl_dynamic_initial_workdir.test | 20 - .../cwl_dynamic_initial_workdir.cwl | 29 - .../cwl_dynamic_initial_workdir.inputs | 3 - .../cwl_dynamic_initial_workdir/testdir/a | 0 .../cwl_dynamic_initial_workdir/testdir/b | 0 .../cwl_dynamic_initial_workdir/testdir/c/d | 0 .../cwl_format/cwl_format.cwl | 25 - .../cwl_format/cwl_format.options | 3 - .../cwl_format/cwl_format_invalid.yaml | 5 - .../cwl_format/cwl_format_missing.yaml | 5 - .../cwl_format/cwl_format_url.cwl | 27 - .../cwl_format/cwl_format_valid.yaml | 5 - .../standardTestCases/cwl_format/foo.fasta | 2 - .../standardTestCases/cwl_format/foo.fastq | 4 - .../standardTestCases/cwl_format_invalid.test | 22 - .../standardTestCases/cwl_format_missing.test | 22 - .../standardTestCases/cwl_format_valid.test | 22 - .../cwl_format_valid_with_workflow_url.test | 22 - .../standardTestCases/cwl_glob_sort.test | 16 - .../cwl_glob_sort/cwl_glob_sort.cwl | 19 - .../cwl_glob_sort_with_workflow_url.test | 16 - .../standardTestCases/cwl_import_type.test | 14 - .../cwl_import_type/capture_kit.yml | 5 - .../cwl_import_type/test_wf.cwl | 30 - .../cwl_import_type/test_wf_inputs.json | 6 - .../cwl_import_type/touch.cwl | 23 - .../cwl_import_type_packed.test | 11 - .../cwl_import_type_packed/test_pack.cwl | 91 - .../test_wf_inputs.json | 6 - .../cwl_input_binding_expression.test | 17 - .../cwl_input_binding_expression.cwl | 25 - .../cwl_input_binding_expression.json | 3 - .../standardTestCases/cwl_input_json.test | 18 - .../cwl_input_json/cwl_input_json.yaml | 70 - .../cwl_input_typearray.test | 18 - .../cwl_input_typearray/input.txt | 0 .../cwl_input_typearray/input_typearray.cwl | 42 - .../cwl_input_typearray/input_typearray.yml | 4 - ..._inputdir_zero_doesnt_localize_papiv2.test | 17 - .../cwl_interpolated_strings.test | 17 - .../cwl_interpolated_strings.cwl | 25 - .../cwl_interpolated_strings.json | 1 - .../standardTestCases/cwl_optionals.test | 13 - .../cwl_optionals/cwl_optionals.cwl | 12 - .../cwl_optionals/cwl_optionals.json | 1 - .../cwl_output_json/cwl_output_json.cwl | 34 - .../cwl_output_json/cwl_output_json.inputs | 22 - .../cwl_output_json_jes.test | 25 - .../cwl_prefix_for_array.test | 17 - .../cwl_prefix_for_array/prefix_for_array.cwl | 23 - .../cwl_prefix_for_array/prefix_for_array.yml | 1 - .../cwl_recursive_link_directories.test | 18 - .../cwl_recursive_link_directories.cwl | 19 - .../cwl_relative_imports/cwl_glob_sort.cwl | 16 - .../cwl_relative_imports/workflow.cwl | 16 - .../cwl_relative_imports_url.test | 20 - .../cwl_relative_imports_zip.test | 22 - .../cwl_resources/cwl_disk_resources.cwl | 38 - .../cwl_inputdir_zero_doesnt_localize.cwl | 129 - .../cwl_resources/cwl_resources.cwl | 27 - .../cwl_resources_papiv2.test | 17 - .../cwl_restart/cwl_restart.cwl | 117 - .../cwl_restart_local_with_recover.test | 21 - .../standardTestCases/cwl_run/1st-tool.cwl | 9 - .../standardTestCases/cwl_run/echo-job.yml | 1 - .../cwl_secondary_files.test | 21 - .../cwl_secondary_files/bam.txt | 0 .../cwl_secondary_files/bam.txt.also | 0 .../cwl_secondary_files/bar.txt | 0 .../cwl_secondary_files/bar.txt.also | 0 .../cwl_secondary_files/baz.txt | 0 .../cwl_secondary_files/baz.txt.also | 0 .../cwl_secondary_files/bim.txt | 0 .../cwl_secondary_files/bim.txt.also | 0 .../cwl_secondary_files.cwl | 44 - .../cwl_secondary_files.options | 3 - .../cwl_secondary_files.yaml | 17 - .../cwl_secondary_files/foo.txt | 0 .../cwl_secondary_files/foo.txt.also | 0 .../cwl_secondary_files_workflow.test | 21 - .../cwl_secondary_files_workflow.cwl | 55 - .../cwl_secondary_files_workflow.options | 3 - .../cwl_secondary_files_workflow.yaml | 10 - .../cwl_stdout_expression.cwl | 17 - .../cwl_stdout_expression.json | 4 - .../cwl_stdout_expression_local.test | 21 - .../cwl_stdout_expression_papi.test | 18 - .../standardTestCases/cwls/scatter-job1.json | 4 - .../standardTestCases/cwls/scatter-wf1.cwl | 44 - .../drs_tests/drs_usa_jdr.inputs | 2 +- .../standardTestCases/drs_usa_jdr.test | 1 + .../drs_usa_jdr_preresolve.test | 1 + .../standardTestCases/expressionLib.test | 15 - .../expressionLib/expressionLib.cwl | 14 - .../standardTestCases/hello_cwl.test | 12 - .../standardTestCases/hello_cwl/hello.cwl | 21 - .../standardTestCases/hello_cwl/hello.inputs | 3 - .../http_inputs/http_inputs.cwl | 49 - .../http_inputs/http_inputs_cwl.inputs | 3 - .../standardTestCases/http_inputs_cwl.test | 15 - ...ference_disk_mounted_only_if_requested.wdl | 49 + .../reference_disk/reference_disk_test.inputs | 4 +- .../reference_disk/reference_disk_test.wdl | 55 +- .../reference_disk_false_options.test | 4 +- ..._disk_mounted_only_if_requested_false.test | 14 + ...e_disk_mounted_only_if_requested_true.test | 14 + ...mounted_only_if_requested_unspecified.test | 14 + .../reference_disk_true_options.test | 4 +- .../reference_disk_unspecified_options.test | 4 +- .../retry_with_more_memory_after_137.wdl | 22 + .../retry_with_more_memory_after_137.test | 21 + .../standardTestCases/scatter-wf1.test | 11 - .../standardTestCases/three_step.test | 21 - .../standardTestCases/three_step/inputs.json | 3 - .../three_step/three_step.cwl | 114 - .../three_step/three_step_caller_wf.cwl | 24 - .../three_step_subworkflow.test | 16 - .../workflow_type_and_version_cwl.test | 16 - .../src/bin/centaur-cwl-runner.bash | 26 - .../src/main/resources/application.conf | 23 - .../src/main/resources/logback.xml | 48 - .../src/main/resources/reference.conf | 29 - .../src/main/resources/skipped_tests.csv | 2 - .../scala/centaur/cwl/CentaurCwlRunner.scala | 255 - .../centaur/cwl/CentaurCwlRunnerConfig.scala | 7 - .../centaur/cwl/CentaurCwlRunnerRunMode.scala | 94 - .../scala/centaur/cwl/CloudPreprocessor.scala | 102 - .../scala/centaur/cwl/OutputManipulator.scala | 237 - .../src/main/scala/centaur/cwl/Outputs.scala | 93 - .../src/test/resources/application.conf | 1 - .../test/scala/CloudPreprocessorSpec.scala | 342 - .../impl/drs/DrsCloudNioFileProvider.scala | 30 +- .../drs/DrsCloudNioFileSystemProvider.scala | 3 +- .../DrsCloudNioRegularFileAttributes.scala | 4 +- .../scala/cloud/nio/impl/drs/DrsConfig.scala | 53 +- .../cloud/nio/impl/drs/DrsCredentials.scala | 90 +- .../cloud/nio/impl/drs/DrsPathResolver.scala | 115 +- ...DrsResolverHttpRequestRetryStrategy.scala} | 2 +- .../drs/DrsCloudNioFileProviderSpec.scala | 42 +- .../nio/impl/drs/DrsPathResolverSpec.scala | 66 +- ...esolverHttpRequestRetryStrategySpec.scala} | 14 +- .../MockDrsCloudNioFileSystemProvider.scala | 4 +- .../cloud/nio/impl/drs/MockDrsPaths.scala | 6 +- .../impl/drs/MockEngineDrsPathResolver.scala | 36 +- codegen_java/build.sbt | 2 +- .../main/scala/common/util/TerminalUtil.scala | 4 +- .../collections/EnhancedCollectionsSpec.scala | 6 +- core/src/main/resources/reference.conf | 31 +- .../scala/cromwell/core/io/IoCommand.scala | 6 + .../core/io/IoPromiseProxyActor.scala | 2 + .../core/logging/EnhancedDateConverter.scala | 70 + .../core/logging/EnhancedSlf4jLogger.scala | 16 + .../logging/EnhancedThreadConverter.scala | 21 + .../core/logging/JavaLoggingBridge.scala | 39 + .../core/path/DefaultPathBuilderFactory.scala | 3 + .../core/path/PathBuilderFactory.scala | 21 +- .../cromwell/core/io/IoClientHelperSpec.scala | 4 +- .../cromwell/core/retry/BackoffSpec.scala | 2 +- .../drs/localizer/CommandLineParser.scala | 54 +- .../DrsLocalizerDrsPathResolver.scala | 7 +- .../drs/localizer/DrsLocalizerMain.scala | 53 +- .../accesstokens/AccessTokenStrategy.scala | 7 - .../AzureB2CAccessTokenStrategy.scala | 40 - .../GoogleAccessTokenStrategy.scala | 28 - .../downloaders/GcsUriDownloader.scala | 8 +- .../localizer/downloaders/GetmChecksum.scala | 36 +- .../drs/localizer/CommandLineParserSpec.scala | 89 +- .../drs/localizer/DrsLocalizerMainSpec.scala | 26 +- .../downloaders/GcsUriDownloaderSpec.scala | 8 +- .../downloaders/GetmChecksumSpec.scala | 22 +- .../cromwell.examples.conf | 11 - cwl/src/main/resources/reference.conf | 3 - .../scala/cwl/ArgumentToCommandPart.scala | 26 - .../scala/cwl/BaseCommandToCommandParts.scala | 11 - cwl/src/main/scala/cwl/CommandLineTool.scala | 439 - .../main/scala/cwl/CommandOutputBinding.scala | 295 - cwl/src/main/scala/cwl/CwlCodecs.scala | 153 - cwl/src/main/scala/cwl/CwlDecoder.scala | 84 - .../scala/cwl/CwlExecutableValidation.scala | 38 - .../scala/cwl/CwlExpressionCommandPart.scala | 146 - .../CwlJsonToDelayedCoercionFunction.scala | 90 - cwl/src/main/scala/cwl/CwlToRun.scala | 11 - cwl/src/main/scala/cwl/CwlType.scala | 284 - cwl/src/main/scala/cwl/CwlVersion.scala | 7 - cwl/src/main/scala/cwl/CwlWomExpression.scala | 260 - cwl/src/main/scala/cwl/CwltoolRunner.scala | 71 - cwl/src/main/scala/cwl/EnumSchema.scala | 35 - .../main/scala/cwl/EvaluateExpression.scala | 20 - .../main/scala/cwl/ExpressionEvaluator.scala | 101 - .../scala/cwl/ExpressionInterpolator.scala | 173 - cwl/src/main/scala/cwl/ExpressionTool.scala | 136 - cwl/src/main/scala/cwl/FileParameter.scala | 150 - .../main/scala/cwl/FullyQualifiedName.scala | 61 - cwl/src/main/scala/cwl/GlobEvaluator.scala | 67 - .../scala/cwl/InitialWorkDirRequirement.scala | 63 - cwl/src/main/scala/cwl/InputParameter.scala | 190 - cwl/src/main/scala/cwl/LinkMergeMethod.scala | 8 - .../cwl/MyriadInputTypeToSecondaryFiles.scala | 25 - .../MyriadInputTypeToSortedCommandParts.scala | 254 - .../scala/cwl/MyriadInputTypeToWomType.scala | 95 - .../cwl/MyriadOutputTypeToWomFiles.scala | 66 - .../scala/cwl/MyriadOutputTypeToWomType.scala | 105 - .../cwl/MyriadOutputTypeToWomValue.scala | 100 - cwl/src/main/scala/cwl/OutputParameter.scala | 62 - .../scala/cwl/OutputParameterExpression.scala | 115 - cwl/src/main/scala/cwl/ParameterContext.scala | 17 - .../main/scala/cwl/RequirementsAndHints.scala | 9 - .../main/scala/cwl/RunOutputsToTypeMap.scala | 45 - cwl/src/main/scala/cwl/RunToEmbeddedCwl.scala | 36 - .../main/scala/cwl/RunToInputTypeMap.scala | 37 - cwl/src/main/scala/cwl/ScatterLogic.scala | 263 - cwl/src/main/scala/cwl/ScatterMethod.scala | 9 - .../cwl/StringOrStringArrayToStringList.scala | 12 - cwl/src/main/scala/cwl/Tool.scala | 189 - cwl/src/main/scala/cwl/TypeAliases.scala | 115 - cwl/src/main/scala/cwl/Workflow.scala | 246 - cwl/src/main/scala/cwl/WorkflowStep.scala | 537 - .../main/scala/cwl/WorkflowStepInput.scala | 155 - .../cwl/WorkflowStepInputExpression.scala | 51 - .../WorkflowStepInputMergeExpression.scala | 84 - .../main/scala/cwl/command/ParentName.scala | 10 - .../scala/cwl/command/StringCommandPart.scala | 18 - .../CommandPartSortingAlgorithm.scala | 71 - .../cwl/internal/CwlEcmaScriptDecoder.scala | 124 - .../cwl/internal/EcmaScriptEncoder.scala | 123 - .../scala/cwl/internal/EcmaScriptUtil.scala | 128 - .../cwl/internal/EnhancedRhinoSandbox.scala | 167 - .../scala/cwl/internal/GigabytesToBytes.scala | 26 - cwl/src/main/scala/cwl/model.scala | 310 - .../cwl/ontology/CacheConfiguration.scala | 15 - .../cwl/ontology/OntologyConfiguration.scala | 27 - cwl/src/main/scala/cwl/ontology/Schema.scala | 180 - cwl/src/main/scala/cwl/package.scala | 154 - .../cwl/preprocessor/CwlCanonicalizer.scala | 249 - .../cwl/preprocessor/CwlPreProcessor.scala | 236 - .../scala/cwl/preprocessor/CwlReference.scala | 82 - .../RequirementToAttributeMap.scala | 79 - .../ResourceRequirementToWomExpression.scala | 15 - cwl/src/test/resources/1st-tool.cwl | 9 - cwl/src/test/resources/1st-workflow.cwl | 23 - cwl/src/test/resources/application.conf | 14 - cwl/src/test/resources/arguments.cwl | 22 - cwl/src/test/resources/bad.cwl | 1 - cwl/src/test/resources/bad2.cwl | 1 - cwl/src/test/resources/brokenlinks.cwl | 23 - cwl/src/test/resources/cwl/lodash.js | 17112 ------ cwl/src/test/resources/cwl/ontology/EDAM.owl | 51294 ---------------- .../test/resources/cwl/ontology/gx_edam.ttl | 7 - .../complex_workflow/expected_result.json | 204 - .../complex_workflow/root_workflow.cwl | 41 - .../complex_workflow/sub/sub_workflow.cwl | 34 - .../deep_nesting/expected_result.json | 132 - .../deep_nesting/parseInt-tool.cwl | 14 - .../deep_nesting/root_workflow.cwl | 21 - .../cwl/preprocessor/deep_nesting/wc-tool.cwl | 15 - .../resources/cwl/preprocessor/echo_tool.cwl | 13 - .../root_workflow.cwl | 34 - .../self_reference/expected_result.json | 58 - .../self_reference/root_workflow.cwl | 31 - .../simple_workflow/expected_result.json | 100 - .../simple_workflow/root_workflow.cwl | 23 - .../root_workflow.cwl | 16 - .../sub_workflow_1.cwl | 16 - .../sub_workflow_2.cwl | 16 - cwl/src/test/resources/dir_example.cwl | 17 - cwl/src/test/resources/env.cwl | 10 - cwl/src/test/resources/exampledir/example.txt | 1 - cwl/src/test/resources/file_example.cwl | 17 - cwl/src/test/resources/file_size_example.cwl | 18 - cwl/src/test/resources/links_dont_parse.cwl | 23 - cwl/src/test/resources/logback-test.xml | 17 - cwl/src/test/resources/nestedworkflows.cwl | 52 - cwl/src/test/resources/tar-param.cwl | 17 - cwl/src/test/resources/three_step.cwl | 112 - .../test/scala/cwl/CommandLineToolSpec.scala | 279 - .../cwl/CommandOutputExpressionSpec.scala | 57 - cwl/src/test/scala/cwl/CwlDecoderSpec.scala | 48 - .../scala/cwl/CwlEcmaScriptDecoderSpec.scala | 77 - .../scala/cwl/CwlEcmaScriptEncoderSpec.scala | 35 - .../cwl/CwlExpressionCommandPartSpec.scala | 34 - .../scala/cwl/CwlInputValidationSpec.scala | 203 - .../test/scala/cwl/CwlWorkflowWomSpec.scala | 216 - cwl/src/test/scala/cwl/DirectorySpec.scala | 41 - .../test/scala/cwl/ExportCwlSamplesSpec.scala | 146 - .../scala/cwl/ExpressionEvaluatorSpec.scala | 24 - .../cwl/ExpressionInterpolatorSpec.scala | 121 - cwl/src/test/scala/cwl/FileSpec.scala | 54 - .../scala/cwl/FullyQualifiedNameSpec.scala | 27 - .../test/scala/cwl/LocalIoFunctionSet.scala | 48 - .../test/scala/cwl/ParseBigThreeSpec.scala | 34 - cwl/src/test/scala/cwl/ParsingSpec.scala | 63 - cwl/src/test/scala/cwl/ScatterLogicSpec.scala | 140 - cwl/src/test/scala/cwl/TestSetup.scala | 9 - .../scala/cwl/WomTypeConversionSpec.scala | 134 - .../scala/cwl/WomValueJsonFormatter.scala | 27 - .../cwl/WorkflowStepInputExpressionSpec.scala | 59 - .../scala/cwl/WorkflowStepInputSpec.scala | 109 - .../cwl/internal/EcmaScriptUtilSpec.scala | 96 - .../internal/EnhancedRhinoSandboxSpec.scala | 141 - .../test/scala/cwl/ontology/SchemaSpec.scala | 114 - .../preprocessor/CwlPreProcessorSpec.scala | 125 - .../cwl/preprocessor/CwlReferenceSpec.scala | 79 - .../src/main/resources/changelog.xml | 3 + .../enlarge_docker_hash_store_entry_id.xml | 41 + .../enlarge_sub_workflow_store_entry_id.xml | 41 + .../enlarge_workflow_store_entry_id.xml | 82 + .../migration/liquibase/LiquibaseUtils.scala | 13 +- .../DockerHashStoreEntryComponent.scala | 2 +- .../SubWorkflowStoreEntryComponent.scala | 6 +- .../tables/WorkflowStoreEntryComponent.scala | 2 +- .../sql/tables/DockerHashStoreEntry.scala | 2 +- .../sql/tables/SubWorkflowStoreEntry.scala | 4 +- .../sql/tables/WorkflowStoreEntry.scala | 2 +- .../registryv2/DockerRegistryV2Abstract.scala | 44 +- .../DockerRegistryV2AbstractSpec.scala | 6 +- docs/LanguageSupport.md | 7 - docs/RuntimeAttributes.md | 60 +- docs/WOMtool.md | 4 +- docs/api/RESTAPI.md | 105 +- docs/backends/TES.md | 27 +- docs/cromwell_features/HogFactors.md | 2 +- docs/developers/CwlParsing.md | 66 - .../workflowParsing/cwlParsingOverview.md | 1 - .../workflowParsing/wdlParsingOverview.md | 4 +- docs/filesystems/DataRepositoryService.md | 4 +- docs/filesystems/Filesystems.md | 9 +- docs/tutorials/Containers.md | 22 +- docs/tutorials/FiveMinuteIntro.md | 14 +- docs/tutorials/PipelinesApi101.md | 23 +- docs/wf_options/Overview.md | 4 +- .../src/main/resources/swagger/cromwell.yaml | 174 +- .../cromwell/engine/EngineFilesystems.scala | 14 +- .../cromwell/engine/io/nio/NioFlow.scala | 47 +- .../engine/workflow/WorkflowActor.scala | 139 +- .../callcaching/FetchCachedResultsActor.scala | 2 +- .../job/EngineJobExecutionActor.scala | 2 +- .../execution/stores/ValueStore.scala | 5 +- .../cromwell/server/CromwellServer.scala | 4 +- .../routes/CromwellApiService.scala | 130 +- .../webservice/routes/wes/WesResponse.scala | 16 +- .../routes/wes/WesRouteSupport.scala | 210 +- .../webservice/routes/wes/WesRunLog.scala | 13 - .../webservice/routes/wes/WesRunRoutes.scala | 74 - .../webservice/routes/wes/WesSubmission.scala | 44 + .../cromwell/engine/io/nio/NioFlowSpec.scala | 39 +- .../ValidatingCachingConfigSpec.scala | 2 +- .../webservice/SwaggerServiceSpec.scala | 6 +- .../routes/CromwellApiServiceSpec.scala | 21 +- .../routes/wes/ServiceInfoSpec.scala | 2 +- .../routes/wes/WesRouteSupportSpec.scala | 69 +- .../filesystems/blob/AzureCredentials.scala | 52 + .../blob/BlobFileSystemConfig.scala | 75 + .../blob/BlobFileSystemManager.scala | 267 + .../filesystems/blob/BlobPathBuilder.scala | 115 +- .../blob/BlobPathBuilderFactory.scala | 25 +- .../WorkspaceManagerApiClientProvider.scala | 30 + .../blob/BlobFileSystemConfigSpec.scala | 75 + .../blob/BlobFileSystemContractSpec.scala | 50 + .../blob/BlobPathBuilderFactorySpec.scala | 166 + .../blob/BlobPathBuilderSpec.scala | 84 +- .../cromwell/filesystems/drs/DrsPath.scala | 2 +- .../filesystems/drs/DrsPathBuilder.scala | 2 +- .../drs/DrsPathBuilderFactory.scala | 26 +- .../cromwell/filesystems/drs/DrsReader.scala | 18 +- .../filesystems/drs/DrsResolver.scala | 42 +- .../drs/DrsPathBuilderFactorySpec.scala | 4 +- .../filesystems/drs/DrsPathBuilderSpec.scala | 12 +- .../filesystems/drs/DrsReaderSpec.scala | 18 +- .../filesystems/drs/DrsResolverSpec.scala | 12 +- .../s3/batch/S3BatchCommandBuilder.scala | 8 +- .../cwl/CwlV1_0LanguageFactory.scala | 75 - .../languages/util/ImportResolver.scala | 53 +- .../languages/util/ImportResolverSpec.scala | 43 +- .../NamespaceCacheSpec.scala | 2 +- mkdocs.yml | 2 - processes/release_processes/README.MD | 43 +- project/ContinuousIntegration.scala | 4 +- project/Dependencies.scala | 236 +- project/GenerateRestApiDocs.scala | 9 +- project/Publishing.scala | 20 +- project/Settings.scala | 3 +- project/Version.scala | 18 +- project/plugins.sbt | 7 +- project/swagger2markup.sbt | 2 +- publish/publish_inputs.json | 10 +- .../Repo template_ CromIAM.run.xml | 2 +- ...o template_ Cromwell DRS Localizer.run.xml | 4 +- ...Repo template_ Cromwell server TES.run.xml | 21 + scripts/gen_java_client.sh | 3 +- server/src/main/resources/application.conf | 2 +- server/src/main/resources/logback.xml | 17 +- .../scala/cromwell/CommandLineArguments.scala | 44 +- .../scala/cromwell/CromwellEntryPoint.scala | 8 + server/src/test/resources/cwl_glob_sort.cwl | 16 - .../cromwell/CromwellCommandLineSpec.scala | 26 - .../engine/workflow/WorkflowActorSpec.scala | 108 +- ...ectiveTsvInstrumentationServiceActor.scala | 2 +- .../impl/MetadataDatabaseAccessSpec.scala | 2 +- src/ci/bin/testSingleWorkflowRunner.sh | 23 - .../resources/papi_application.inc.conf.ctmpl | 2 +- .../papi_v2_reference_image_manifest.conf | 4989 +- src/ci/resources/tes_application.conf | 52 + ...wsBatchAsyncBackendJobExecutionActor.scala | 159 +- .../backend/impl/aws/AwsBatchAttributes.scala | 33 +- .../AwsBatchBackendInitializationData.scala | 1 + ...AwsBatchBackendLifecycleActorFactory.scala | 9 +- .../impl/aws/AwsBatchConfiguration.scala | 2 + .../aws/AwsBatchExpressionFunctions.scala | 8 +- .../aws/AwsBatchInitializationActor.scala | 1 + .../backend/impl/aws/AwsBatchJob.scala | 58 +- .../aws/AwsBatchJobCachingActorHelper.scala | 2 +- .../backend/impl/aws/AwsBatchJobPaths.scala | 20 +- .../impl/aws/AwsBatchWorkflowPaths.scala | 14 +- .../scala/cromwell/backend/impl/aws/README.md | 112 +- .../AwsBatchBackendCacheHitCopyingActor.scala | 12 +- .../AwsBatchBackendFileHashingActor.scala | 26 + .../impl/aws/io/AwsBatchGlobFunctions.scala | 96 + ...inesApiAsyncBackendJobExecutionActor.scala | 8 +- .../PipelinesApiJobCachingActorHelper.scala | 2 +- .../common/PipelinesApiJobPaths.scala | 1 + .../pipelines/v2alpha1/GenomicsFactory.scala | 2 +- ...inesApiAsyncBackendJobExecutionActor.scala | 7 +- .../PipelinesParameterConversions.scala | 4 +- .../v2alpha1/PipelinesConversionsSpec.scala | 10 +- .../v2alpha1/api/DeserializationSpec.scala | 6 +- .../v2beta/LifeSciencesFactory.scala | 2 +- ...inesApiAsyncBackendJobExecutionActor.scala | 35 +- .../PipelinesParameterConversions.scala | 23 +- .../pipelines/v2beta/api/Localization.scala | 53 +- ...ApiAsyncBackendJobExecutionActorSpec.scala | 49 + .../v2beta/PipelinesConversionsSpec.scala | 96 - .../v2beta/api/DeserializationSpec.scala | 6 +- .../v2beta/api/LocalizationSpec.scala | 67 + .../TesAsyncBackendJobExecutionActor.scala | 55 +- .../backend/impl/tes/TesConfiguration.scala | 4 +- .../impl/tes/TesRuntimeAttributes.scala | 46 +- .../cromwell/backend/impl/tes/TesTask.scala | 67 +- .../impl/tes/TesRuntimeAttributesSpec.scala | 48 + .../backend/impl/tes/TesTaskSpec.scala | 62 +- .../scala/wdl/draft2/model/AstTools.scala | 4 +- wom/src/main/scala/wom/util/YamlUtils.scala | 6 +- .../scala/wom/types/WomFileTypeSpec.scala | 2 +- .../test/scala/wom/types/WomTypeSpec.scala | 21 +- .../main/scala/womtool/graph/WomGraph.scala | 22 +- .../scala/womtool/input/WomGraphMaker.scala | 4 +- 654 files changed, 5118 insertions(+), 128310 deletions(-) create mode 100644 .github/set_up_cromwell_action/action.yml create mode 100644 .github/workflows/cromwell_unit_tests.yml delete mode 100644 centaur/src/main/resources/integrationTestCases/bcbio_joint_gvcf.test delete mode 100644 centaur/src/main/resources/integrationTestCases/bcbio_prealign.test delete mode 100644 centaur/src/main/resources/integrationTestCases/bcbio_rnaseq.test delete mode 100644 centaur/src/main/resources/integrationTestCases/bcbio_somatic.test delete mode 100644 centaur/src/main/resources/integrationTestCases/bcbio_svcall.test delete mode 100644 centaur/src/main/resources/integrationTestCases/bcbio_wes-chr21-test-workflow.test delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/bcbio.options delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/main-gvcf-joint-samples.json delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/main-gvcf-joint.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/alignment_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/batch_for_jointvc.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/batch_for_variantcall.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/combine_sample_regions.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/compare_to_rm.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/concat_batch_variantcalls.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/concat_batch_variantcalls_jointvc.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/finalize_jointvc.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/get_parallel_regions.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/get_parallel_regions_jointvc.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/merge_split_alignments.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/multiqc_summary.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/pipeline_summary.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/postprocess_alignment.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/postprocess_alignment_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/postprocess_variants.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/prep_align_inputs.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/prep_samples.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/prep_samples_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/process_alignment.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/qc_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/run_jointvc.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/summarize_vc.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/variantcall_batch_region.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/wf-alignment.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/wf-jointcall.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/wf-variantcall.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/main-prealign-samples.json delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/main-prealign.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/batch_for_variantcall.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/combine_sample_regions.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/compare_to_rm.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/concat_batch_variantcalls.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/get_parallel_regions.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/multiqc_summary.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/organize_noalign.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/pipeline_summary.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/postprocess_alignment.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/postprocess_alignment_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/postprocess_variants.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/prep_samples.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/prep_samples_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/qc_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/summarize_vc.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/variantcall_batch_region.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/wf-variantcall.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/main-rnaseq-samples.json delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/main-rnaseq.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/detect_fusions.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/multiqc_summary.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/pipeline_summary.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/prepare_sample.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/process_alignment.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/qc_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/rnaseq_quantitate.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/trim_sample.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/main-somatic-samples.json delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/main-somatic.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/alignment_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/batch_for_sv.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/batch_for_variantcall.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/calculate_sv_bins.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/calculate_sv_coverage.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/combine_sample_regions.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/compare_to_rm.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/concat_batch_variantcalls.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/detect_sv.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/get_parallel_regions.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/merge_split_alignments.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/multiqc_summary.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/normalize_sv_coverage.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/pipeline_summary.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/postprocess_alignment.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/postprocess_alignment_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/postprocess_variants.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/prep_align_inputs.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/prep_samples.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/prep_samples_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/process_alignment.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/qc_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/summarize_sv.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/summarize_vc.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/variantcall_batch_region.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/wf-alignment.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/wf-svcall.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/wf-variantcall.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/main-svcall-samples.json delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/main-svcall.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/alignment_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/batch_for_sv.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/batch_for_variantcall.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/calculate_sv_bins.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/calculate_sv_coverage.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/combine_sample_regions.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/compare_to_rm.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/concat_batch_variantcalls.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/detect_sv.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/get_parallel_regions.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/merge_split_alignments.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/multiqc_summary.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/normalize_sv_coverage.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/pipeline_summary.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/postprocess_alignment.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/postprocess_alignment_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/postprocess_variants.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/prep_align_inputs.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/prep_samples.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/prep_samples_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/process_alignment.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/qc_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/summarize_sv.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/summarize_vc.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/variantcall_batch_region.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/wf-alignment.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/wf-svcall.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/wf-variantcall.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/main-wes_chr21_test-samples.json delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/main-wes_chr21_test.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/alignment_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/batch_for_ensemble.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/batch_for_sv.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/batch_for_variantcall.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/calculate_sv_bins.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/calculate_sv_coverage.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/combine_calls.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/combine_sample_regions.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/compare_to_rm.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/concat_batch_variantcalls.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/detect_sv.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/get_parallel_regions.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/merge_split_alignments.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/multiqc_summary.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/normalize_sv_coverage.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/pipeline_summary.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/postprocess_alignment.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/postprocess_alignment_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/postprocess_variants.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/prep_align_inputs.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/prep_samples.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/prep_samples_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/process_alignment.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/qc_to_rec.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/summarize_sv.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/summarize_vc.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/variantcall_batch_region.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/wf-alignment.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/wf-svcall.cwl delete mode 100644 centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/wf-variantcall.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.inline_file.test delete mode 100644 centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.inline_file_custom_entryname.test delete mode 100644 centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.input_string.test delete mode 100644 centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.input_string_function.test delete mode 100644 centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/file_array_listing.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/inline_file.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/inline_file_custom_entryname.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/input_string.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/input_string_function.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/ad_hoc_file_test.aws.test delete mode 100644 centaur/src/main/resources/standardTestCases/ad_hoc_file_test/cwl-test.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/ad_hoc_file_test/workflow.cwl create mode 100644 centaur/src/main/resources/standardTestCases/blob_md5/blob_md5.inputs.json create mode 100644 centaur/src/main/resources/standardTestCases/blob_md5/blob_md5.wdl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_cache_between_workflows.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_cache_between_workflows/cwl_cache_between_workflows.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_cache_between_workflows/cwl_cache_between_workflows.json delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_cache_within_workflow.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_cache_within_workflow/cwl_cache_within_workflow.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_cache_within_workflow/cwl_cache_within_workflow.json delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_disk_resources_papiv2.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_docker_size.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size.options delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size.yaml delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size_input.txt delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/cwl_dynamic_initial_workdir.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/cwl_dynamic_initial_workdir.inputs delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/testdir/a delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/testdir/b delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/testdir/c/d delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_format/cwl_format.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_format/cwl_format.options delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_invalid.yaml delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_missing.yaml delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_url.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_valid.yaml delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_format/foo.fasta delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_format/foo.fastq delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_format_invalid.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_format_missing.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_format_valid.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_format_valid_with_workflow_url.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_glob_sort.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_glob_sort/cwl_glob_sort.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_glob_sort_with_workflow_url.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_import_type.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_import_type/capture_kit.yml delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_import_type/test_wf.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_import_type/test_wf_inputs.json delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_import_type/touch.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_import_type_packed.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_import_type_packed/test_pack.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_import_type_packed/test_wf_inputs.json delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_input_binding_expression.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_input_binding_expression/cwl_input_binding_expression.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_input_binding_expression/cwl_input_binding_expression.json delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_input_json.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_input_json/cwl_input_json.yaml delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_input_typearray.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_input_typearray/input.txt delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_input_typearray/input_typearray.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_input_typearray/input_typearray.yml delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_inputdir_zero_doesnt_localize_papiv2.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_interpolated_strings.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_interpolated_strings/cwl_interpolated_strings.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_interpolated_strings/cwl_interpolated_strings.json delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_optionals.test delete mode 100755 centaur/src/main/resources/standardTestCases/cwl_optionals/cwl_optionals.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_optionals/cwl_optionals.json delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_output_json/cwl_output_json.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_output_json/cwl_output_json.inputs delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_output_json_jes.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_prefix_for_array.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_prefix_for_array/prefix_for_array.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_prefix_for_array/prefix_for_array.yml delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_recursive_link_directories.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_recursive_link_directories/cwl_recursive_link_directories.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_relative_imports/cwl_glob_sort.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_relative_imports/workflow.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_relative_imports_url.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_relative_imports_zip.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_resources/cwl_disk_resources.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_resources/cwl_inputdir_zero_doesnt_localize.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_resources/cwl_resources.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_resources_papiv2.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_restart/cwl_restart.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_restart_local_with_recover.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_run/1st-tool.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_run/echo-job.yml delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files/bam.txt delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files/bam.txt.also delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files/bar.txt delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files/bar.txt.also delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files/baz.txt delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files/baz.txt.also delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files/bim.txt delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files/bim.txt.also delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files/cwl_secondary_files.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files/cwl_secondary_files.options delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files/cwl_secondary_files.yaml delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files/foo.txt delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files/foo.txt.also delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow/cwl_secondary_files_workflow.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow/cwl_secondary_files_workflow.options delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow/cwl_secondary_files_workflow.yaml delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_stdout_expression/cwl_stdout_expression.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_stdout_expression/cwl_stdout_expression.json delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_stdout_expression_local.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwl_stdout_expression_papi.test delete mode 100644 centaur/src/main/resources/standardTestCases/cwls/scatter-job1.json delete mode 100644 centaur/src/main/resources/standardTestCases/cwls/scatter-wf1.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/expressionLib.test delete mode 100644 centaur/src/main/resources/standardTestCases/expressionLib/expressionLib.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/hello_cwl.test delete mode 100644 centaur/src/main/resources/standardTestCases/hello_cwl/hello.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/hello_cwl/hello.inputs delete mode 100644 centaur/src/main/resources/standardTestCases/http_inputs/http_inputs.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/http_inputs/http_inputs_cwl.inputs delete mode 100644 centaur/src/main/resources/standardTestCases/http_inputs_cwl.test create mode 100644 centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_mounted_only_if_requested.wdl create mode 100644 centaur/src/main/resources/standardTestCases/reference_disk_mounted_only_if_requested_false.test create mode 100644 centaur/src/main/resources/standardTestCases/reference_disk_mounted_only_if_requested_true.test create mode 100644 centaur/src/main/resources/standardTestCases/reference_disk_mounted_only_if_requested_unspecified.test create mode 100644 centaur/src/main/resources/standardTestCases/retry_with_more_memory/retry_with_more_memory_after_137.wdl create mode 100644 centaur/src/main/resources/standardTestCases/retry_with_more_memory_after_137.test delete mode 100644 centaur/src/main/resources/standardTestCases/scatter-wf1.test delete mode 100644 centaur/src/main/resources/standardTestCases/three_step.test delete mode 100644 centaur/src/main/resources/standardTestCases/three_step/inputs.json delete mode 100644 centaur/src/main/resources/standardTestCases/three_step/three_step.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/three_step/three_step_caller_wf.cwl delete mode 100644 centaur/src/main/resources/standardTestCases/three_step_subworkflow.test delete mode 100644 centaur/src/main/resources/standardTestCases/workflow_type_and_version_cwl.test delete mode 100755 centaurCwlRunner/src/bin/centaur-cwl-runner.bash delete mode 100644 centaurCwlRunner/src/main/resources/application.conf delete mode 100644 centaurCwlRunner/src/main/resources/logback.xml delete mode 100644 centaurCwlRunner/src/main/resources/reference.conf delete mode 100644 centaurCwlRunner/src/main/resources/skipped_tests.csv delete mode 100644 centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunner.scala delete mode 100644 centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunnerConfig.scala delete mode 100644 centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunnerRunMode.scala delete mode 100644 centaurCwlRunner/src/main/scala/centaur/cwl/CloudPreprocessor.scala delete mode 100644 centaurCwlRunner/src/main/scala/centaur/cwl/OutputManipulator.scala delete mode 100644 centaurCwlRunner/src/main/scala/centaur/cwl/Outputs.scala delete mode 100644 centaurCwlRunner/src/test/resources/application.conf delete mode 100644 centaurCwlRunner/src/test/scala/CloudPreprocessorSpec.scala rename cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/{MarthaHttpRequestRetryStrategy.scala => DrsResolverHttpRequestRetryStrategy.scala} (96%) rename cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/{MarthaHttpRequestRetryStrategySpec.scala => DrsResolverHttpRequestRetryStrategySpec.scala} (89%) create mode 100644 core/src/main/scala/cromwell/core/logging/EnhancedDateConverter.scala create mode 100644 core/src/main/scala/cromwell/core/logging/EnhancedSlf4jLogger.scala create mode 100644 core/src/main/scala/cromwell/core/logging/EnhancedThreadConverter.scala create mode 100644 core/src/main/scala/cromwell/core/logging/JavaLoggingBridge.scala delete mode 100644 cromwell-drs-localizer/src/main/scala/drs/localizer/accesstokens/AccessTokenStrategy.scala delete mode 100644 cromwell-drs-localizer/src/main/scala/drs/localizer/accesstokens/AzureB2CAccessTokenStrategy.scala delete mode 100644 cromwell-drs-localizer/src/main/scala/drs/localizer/accesstokens/GoogleAccessTokenStrategy.scala delete mode 100644 cwl/src/main/resources/reference.conf delete mode 100644 cwl/src/main/scala/cwl/ArgumentToCommandPart.scala delete mode 100644 cwl/src/main/scala/cwl/BaseCommandToCommandParts.scala delete mode 100644 cwl/src/main/scala/cwl/CommandLineTool.scala delete mode 100644 cwl/src/main/scala/cwl/CommandOutputBinding.scala delete mode 100644 cwl/src/main/scala/cwl/CwlCodecs.scala delete mode 100644 cwl/src/main/scala/cwl/CwlDecoder.scala delete mode 100644 cwl/src/main/scala/cwl/CwlExecutableValidation.scala delete mode 100644 cwl/src/main/scala/cwl/CwlExpressionCommandPart.scala delete mode 100644 cwl/src/main/scala/cwl/CwlJsonToDelayedCoercionFunction.scala delete mode 100644 cwl/src/main/scala/cwl/CwlToRun.scala delete mode 100644 cwl/src/main/scala/cwl/CwlType.scala delete mode 100644 cwl/src/main/scala/cwl/CwlVersion.scala delete mode 100644 cwl/src/main/scala/cwl/CwlWomExpression.scala delete mode 100644 cwl/src/main/scala/cwl/CwltoolRunner.scala delete mode 100644 cwl/src/main/scala/cwl/EnumSchema.scala delete mode 100644 cwl/src/main/scala/cwl/EvaluateExpression.scala delete mode 100644 cwl/src/main/scala/cwl/ExpressionEvaluator.scala delete mode 100644 cwl/src/main/scala/cwl/ExpressionInterpolator.scala delete mode 100644 cwl/src/main/scala/cwl/ExpressionTool.scala delete mode 100644 cwl/src/main/scala/cwl/FileParameter.scala delete mode 100644 cwl/src/main/scala/cwl/FullyQualifiedName.scala delete mode 100644 cwl/src/main/scala/cwl/GlobEvaluator.scala delete mode 100644 cwl/src/main/scala/cwl/InitialWorkDirRequirement.scala delete mode 100644 cwl/src/main/scala/cwl/InputParameter.scala delete mode 100644 cwl/src/main/scala/cwl/LinkMergeMethod.scala delete mode 100644 cwl/src/main/scala/cwl/MyriadInputTypeToSecondaryFiles.scala delete mode 100644 cwl/src/main/scala/cwl/MyriadInputTypeToSortedCommandParts.scala delete mode 100644 cwl/src/main/scala/cwl/MyriadInputTypeToWomType.scala delete mode 100644 cwl/src/main/scala/cwl/MyriadOutputTypeToWomFiles.scala delete mode 100644 cwl/src/main/scala/cwl/MyriadOutputTypeToWomType.scala delete mode 100644 cwl/src/main/scala/cwl/MyriadOutputTypeToWomValue.scala delete mode 100644 cwl/src/main/scala/cwl/OutputParameter.scala delete mode 100644 cwl/src/main/scala/cwl/OutputParameterExpression.scala delete mode 100644 cwl/src/main/scala/cwl/ParameterContext.scala delete mode 100644 cwl/src/main/scala/cwl/RequirementsAndHints.scala delete mode 100644 cwl/src/main/scala/cwl/RunOutputsToTypeMap.scala delete mode 100644 cwl/src/main/scala/cwl/RunToEmbeddedCwl.scala delete mode 100644 cwl/src/main/scala/cwl/RunToInputTypeMap.scala delete mode 100644 cwl/src/main/scala/cwl/ScatterLogic.scala delete mode 100644 cwl/src/main/scala/cwl/ScatterMethod.scala delete mode 100644 cwl/src/main/scala/cwl/StringOrStringArrayToStringList.scala delete mode 100644 cwl/src/main/scala/cwl/Tool.scala delete mode 100644 cwl/src/main/scala/cwl/TypeAliases.scala delete mode 100644 cwl/src/main/scala/cwl/Workflow.scala delete mode 100644 cwl/src/main/scala/cwl/WorkflowStep.scala delete mode 100644 cwl/src/main/scala/cwl/WorkflowStepInput.scala delete mode 100644 cwl/src/main/scala/cwl/WorkflowStepInputExpression.scala delete mode 100644 cwl/src/main/scala/cwl/WorkflowStepInputMergeExpression.scala delete mode 100644 cwl/src/main/scala/cwl/command/ParentName.scala delete mode 100644 cwl/src/main/scala/cwl/command/StringCommandPart.scala delete mode 100644 cwl/src/main/scala/cwl/internal/CommandPartSortingAlgorithm.scala delete mode 100644 cwl/src/main/scala/cwl/internal/CwlEcmaScriptDecoder.scala delete mode 100644 cwl/src/main/scala/cwl/internal/EcmaScriptEncoder.scala delete mode 100644 cwl/src/main/scala/cwl/internal/EcmaScriptUtil.scala delete mode 100644 cwl/src/main/scala/cwl/internal/EnhancedRhinoSandbox.scala delete mode 100644 cwl/src/main/scala/cwl/internal/GigabytesToBytes.scala delete mode 100644 cwl/src/main/scala/cwl/model.scala delete mode 100644 cwl/src/main/scala/cwl/ontology/CacheConfiguration.scala delete mode 100644 cwl/src/main/scala/cwl/ontology/OntologyConfiguration.scala delete mode 100644 cwl/src/main/scala/cwl/ontology/Schema.scala delete mode 100644 cwl/src/main/scala/cwl/package.scala delete mode 100644 cwl/src/main/scala/cwl/preprocessor/CwlCanonicalizer.scala delete mode 100644 cwl/src/main/scala/cwl/preprocessor/CwlPreProcessor.scala delete mode 100644 cwl/src/main/scala/cwl/preprocessor/CwlReference.scala delete mode 100644 cwl/src/main/scala/cwl/requirement/RequirementToAttributeMap.scala delete mode 100644 cwl/src/main/scala/cwl/requirement/ResourceRequirementToWomExpression.scala delete mode 100755 cwl/src/test/resources/1st-tool.cwl delete mode 100644 cwl/src/test/resources/1st-workflow.cwl delete mode 100644 cwl/src/test/resources/application.conf delete mode 100644 cwl/src/test/resources/arguments.cwl delete mode 100644 cwl/src/test/resources/bad.cwl delete mode 100644 cwl/src/test/resources/bad2.cwl delete mode 100644 cwl/src/test/resources/brokenlinks.cwl delete mode 100644 cwl/src/test/resources/cwl/lodash.js delete mode 100644 cwl/src/test/resources/cwl/ontology/EDAM.owl delete mode 100644 cwl/src/test/resources/cwl/ontology/gx_edam.ttl delete mode 100644 cwl/src/test/resources/cwl/preprocessor/complex_workflow/expected_result.json delete mode 100644 cwl/src/test/resources/cwl/preprocessor/complex_workflow/root_workflow.cwl delete mode 100644 cwl/src/test/resources/cwl/preprocessor/complex_workflow/sub/sub_workflow.cwl delete mode 100644 cwl/src/test/resources/cwl/preprocessor/deep_nesting/expected_result.json delete mode 100644 cwl/src/test/resources/cwl/preprocessor/deep_nesting/parseInt-tool.cwl delete mode 100644 cwl/src/test/resources/cwl/preprocessor/deep_nesting/root_workflow.cwl delete mode 100644 cwl/src/test/resources/cwl/preprocessor/deep_nesting/wc-tool.cwl delete mode 100644 cwl/src/test/resources/cwl/preprocessor/echo_tool.cwl delete mode 100644 cwl/src/test/resources/cwl/preprocessor/same_file_cyclic_dependency/root_workflow.cwl delete mode 100644 cwl/src/test/resources/cwl/preprocessor/self_reference/expected_result.json delete mode 100644 cwl/src/test/resources/cwl/preprocessor/self_reference/root_workflow.cwl delete mode 100644 cwl/src/test/resources/cwl/preprocessor/simple_workflow/expected_result.json delete mode 100644 cwl/src/test/resources/cwl/preprocessor/simple_workflow/root_workflow.cwl delete mode 100644 cwl/src/test/resources/cwl/preprocessor/transitive_cyclic_dependency/root_workflow.cwl delete mode 100644 cwl/src/test/resources/cwl/preprocessor/transitive_cyclic_dependency/sub_workflow_1.cwl delete mode 100644 cwl/src/test/resources/cwl/preprocessor/transitive_cyclic_dependency/sub_workflow_2.cwl delete mode 100644 cwl/src/test/resources/dir_example.cwl delete mode 100644 cwl/src/test/resources/env.cwl delete mode 100644 cwl/src/test/resources/exampledir/example.txt delete mode 100644 cwl/src/test/resources/file_example.cwl delete mode 100644 cwl/src/test/resources/file_size_example.cwl delete mode 100644 cwl/src/test/resources/links_dont_parse.cwl delete mode 100644 cwl/src/test/resources/logback-test.xml delete mode 100644 cwl/src/test/resources/nestedworkflows.cwl delete mode 100644 cwl/src/test/resources/tar-param.cwl delete mode 100644 cwl/src/test/resources/three_step.cwl delete mode 100644 cwl/src/test/scala/cwl/CommandLineToolSpec.scala delete mode 100644 cwl/src/test/scala/cwl/CommandOutputExpressionSpec.scala delete mode 100644 cwl/src/test/scala/cwl/CwlDecoderSpec.scala delete mode 100644 cwl/src/test/scala/cwl/CwlEcmaScriptDecoderSpec.scala delete mode 100644 cwl/src/test/scala/cwl/CwlEcmaScriptEncoderSpec.scala delete mode 100644 cwl/src/test/scala/cwl/CwlExpressionCommandPartSpec.scala delete mode 100644 cwl/src/test/scala/cwl/CwlInputValidationSpec.scala delete mode 100644 cwl/src/test/scala/cwl/CwlWorkflowWomSpec.scala delete mode 100644 cwl/src/test/scala/cwl/DirectorySpec.scala delete mode 100644 cwl/src/test/scala/cwl/ExportCwlSamplesSpec.scala delete mode 100644 cwl/src/test/scala/cwl/ExpressionEvaluatorSpec.scala delete mode 100644 cwl/src/test/scala/cwl/ExpressionInterpolatorSpec.scala delete mode 100644 cwl/src/test/scala/cwl/FileSpec.scala delete mode 100644 cwl/src/test/scala/cwl/FullyQualifiedNameSpec.scala delete mode 100644 cwl/src/test/scala/cwl/LocalIoFunctionSet.scala delete mode 100644 cwl/src/test/scala/cwl/ParseBigThreeSpec.scala delete mode 100644 cwl/src/test/scala/cwl/ParsingSpec.scala delete mode 100644 cwl/src/test/scala/cwl/ScatterLogicSpec.scala delete mode 100644 cwl/src/test/scala/cwl/TestSetup.scala delete mode 100644 cwl/src/test/scala/cwl/WomTypeConversionSpec.scala delete mode 100644 cwl/src/test/scala/cwl/WomValueJsonFormatter.scala delete mode 100644 cwl/src/test/scala/cwl/WorkflowStepInputExpressionSpec.scala delete mode 100644 cwl/src/test/scala/cwl/WorkflowStepInputSpec.scala delete mode 100644 cwl/src/test/scala/cwl/internal/EcmaScriptUtilSpec.scala delete mode 100644 cwl/src/test/scala/cwl/internal/EnhancedRhinoSandboxSpec.scala delete mode 100644 cwl/src/test/scala/cwl/ontology/SchemaSpec.scala delete mode 100644 cwl/src/test/scala/cwl/preprocessor/CwlPreProcessorSpec.scala delete mode 100644 cwl/src/test/scala/cwl/preprocessor/CwlReferenceSpec.scala create mode 100644 database/migration/src/main/resources/changesets/enlarge_docker_hash_store_entry_id.xml create mode 100644 database/migration/src/main/resources/changesets/enlarge_sub_workflow_store_entry_id.xml create mode 100644 database/migration/src/main/resources/changesets/enlarge_workflow_store_entry_id.xml delete mode 100644 docs/developers/CwlParsing.md delete mode 100644 docs/developers/bitesize/workflowParsing/cwlParsingOverview.md delete mode 100644 engine/src/main/scala/cromwell/webservice/routes/wes/WesRunRoutes.scala create mode 100644 engine/src/main/scala/cromwell/webservice/routes/wes/WesSubmission.scala create mode 100644 filesystems/blob/src/main/scala/cromwell/filesystems/blob/AzureCredentials.scala create mode 100644 filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemConfig.scala create mode 100644 filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala create mode 100644 filesystems/blob/src/main/scala/cromwell/filesystems/blob/WorkspaceManagerApiClientProvider.scala create mode 100644 filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobFileSystemConfigSpec.scala create mode 100644 filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobFileSystemContractSpec.scala create mode 100644 filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderFactorySpec.scala delete mode 100644 languageFactories/cwl-v1-0/src/main/scala/languages/cwl/CwlV1_0LanguageFactory.scala create mode 100644 runConfigurations/Repo template_ Cromwell server TES.run.xml delete mode 100644 server/src/test/resources/cwl_glob_sort.cwl create mode 100644 supportedBackends/aws/src/main/scala/cromwell/backend/impl/aws/io/AwsBatchGlobFunctions.scala delete mode 100644 supportedBackends/google/pipelines/v2beta/src/test/scala/cromwell/backend/google/pipelines/v2beta/PipelinesConversionsSpec.scala create mode 100644 supportedBackends/google/pipelines/v2beta/src/test/scala/cromwell/backend/google/pipelines/v2beta/api/LocalizationSpec.scala diff --git a/.github/set_up_cromwell_action/action.yml b/.github/set_up_cromwell_action/action.yml new file mode 100644 index 00000000000..13cb11601db --- /dev/null +++ b/.github/set_up_cromwell_action/action.yml @@ -0,0 +1,47 @@ +#This is a series of steps that will setup Java/sbt/Cromwell on the local runner. +#These steps are meant to be re-used and invoked by other Github Action Workflows +name: 'Set Up Cromwell Steps' +description: Specific steps that will set up git secrets, java, sbt, and Cromwell on the local machine. +inputs: + cromwell_repo_token: #As an input to this action, you are required to pass in a token that can be used to authenticate while checking out Cromwell. + required: true + +runs: + using: "composite" # <-- this allows these steps to be used by other workflows. + steps: + #Allows this github action to use a cache to store stuff like Java and sbt files between runs. + - uses: actions/checkout@v3 + name: Checkout Coursier Cache + - uses: coursier/cache-action@v6 + name: Enable Coursier Cache + + #Cromwell requires git-secrets be setup. Here, we set up secrets and verify success with a script. + - name: Git secrets setup + run: | + git clone https://github.com/awslabs/git-secrets.git ~/git-secrets + cd ~/git-secrets + git checkout ad82d68ee924906a0401dfd48de5057731a9bc84 + sudo make install + shell: bash + + - name: Secrets check + run: | + sudo ln -s "$(which echo)" /usr/local/bin/say + ./minnie-kenny.sh --force + git secrets --scan-history + shell: bash + + #Clone the cromwell repo to this VM. + - name: Clone Cromwell + uses: actions/checkout@v3 + with: + repository: broadinstitute/cromwell + token: ${{ inputs.cromwell_repo_token }} + + #Install Java to this VM. This Java version and distribution is compatible with Cromwell. + - name: Setup JDK + uses: actions/setup-java@v3 + with: + distribution: temurin + java-version: 11 + diff --git a/.github/workflows/chart_update_on_merge.yml b/.github/workflows/chart_update_on_merge.yml index 9e847bc17eb..aa7a2c9c9ad 100644 --- a/.github/workflows/chart_update_on_merge.yml +++ b/.github/workflows/chart_update_on_merge.yml @@ -41,19 +41,33 @@ jobs: echo "CROMWELL_NUMBER=$((previous_version + 1))" >> $GITHUB_ENV - name: Save complete image ID run: | - echo "CROMWELL_SNAP_VERSION=`echo "$CROMWELL_NUMBER-$CROMWELL_SHORT_SHA-SNAP"`" >> $GITHUB_ENV + echo "CROMWELL_VERSION=`echo "$CROMWELL_NUMBER-$CROMWELL_SHORT_SHA"`" >> $GITHUB_ENV # `DSDEJENKINS_PASSWORD` auto syncs from vault with https://github.com/broadinstitute/terraform-ap-deployments/pull/614 - name: Login to Docker Hub uses: docker/login-action@v1 with: username: dsdejenkins password: ${{ secrets.DSDEJENKINS_PASSWORD }} + # Build & push `cromwell`, `womtool`, `cromiam`, and `cromwell-drs-localizer` - name: Build Cromwell Docker run: | set -e cd cromwell - sbt server/docker - docker push broadinstitute/cromwell:$CROMWELL_SNAP_VERSION + sbt -Dproject.isSnapshot=false -Dproject.isRelease=false dockerBuildAndPush + - name: Deploy to dev and board release train (Cromwell) + uses: broadinstitute/repository-dispatch@master + with: + token: ${{ secrets.BROADBOT_GITHUB_TOKEN }} + repository: broadinstitute/terra-helmfile + event-type: update-service + client-payload: '{"service": "cromwell", "version": "${{ env.CROMWELL_VERSION }}", "dev_only": false}' + - name: Deploy to dev and board release train (CromIAM) + uses: broadinstitute/repository-dispatch@master + with: + token: ${{ secrets.BROADBOT_GITHUB_TOKEN }} + repository: broadinstitute/terra-helmfile + event-type: update-service + client-payload: '{"service": "cromiam", "version": "${{ env.CROMWELL_VERSION }}", "dev_only": false}' - name: Edit & push chart env: BROADBOT_GITHUB_TOKEN: ${{ secrets.BROADBOT_GITHUB_TOKEN }} @@ -62,10 +76,10 @@ jobs: cd cromwhelm git checkout main ls -la - sed -i "s/appVersion.*/appVersion: \"$CROMWELL_SNAP_VERSION\"/" cromwell-helm/Chart.yaml - sed -i "s/image: broadinstitute\/cromwell.*/image: broadinstitute\/cromwell:$CROMWELL_SNAP_VERSION/" cromwell-helm/templates/cromwell.yaml + sed -i "s/appVersion.*/appVersion: \"$CROMWELL_VERSION\"/" cromwell-helm/Chart.yaml + sed -i "s/image: broadinstitute\/cromwell.*/image: broadinstitute\/cromwell:$CROMWELL_VERSION/" cromwell-helm/templates/cromwell.yaml git diff git config --global user.name "broadbot" git config --global user.email "broadbot@broadinstitute.org" - git commit -am "Auto update to Cromwell $CROMWELL_SNAP_VERSION" + git commit -am "Auto update to Cromwell $CROMWELL_VERSION" git push https://broadbot:$BROADBOT_GITHUB_TOKEN@github.com/broadinstitute/cromwhelm.git main diff --git a/.github/workflows/cromwell_unit_tests.yml b/.github/workflows/cromwell_unit_tests.yml new file mode 100644 index 00000000000..d0927f8b954 --- /dev/null +++ b/.github/workflows/cromwell_unit_tests.yml @@ -0,0 +1,32 @@ +name: 'Cromwell unit tests' + +#This github action runs all of Cromwell's unit tests. + +#This is what shows up in the github workflows page as the title. +run-name: ${{ github.actor }} running Cromwell sbt unit tests. + +#What will trigger the workflow to run. +on: + workflow_dispatch: #Manual trigger from GitHub UI + push: + +permissions: + contents: read + +jobs: + build-and-test: + #This action is using a Github free runner, rather than a Broad self-hosted one. + #This is because the Broad ones don't have sbt installed by default. + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v3 # checkout the cromwell repo + - uses: ./.github/set_up_cromwell_action #Exectute this reusable github action. It will set up java/sbt/git-secrets/cromwell. + with: + cromwell_repo_token: ${{ secrets.BROADBOT_GITHUB_TOKEN }} + + #Invoke SBT to run all unit tests for Cromwell. + - name: Run tests + run: | + set -e + sbt "test" diff --git a/.gitignore b/.gitignore index 571a12c5873..c7d6e562544 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ # common scala config *~ +.metals/ .DS_Store .artifactory .bsp @@ -50,3 +51,5 @@ tesk_application.conf **/venv/ exome_germline_single_sample_v1.3/ **/*.pyc +.scalafmt.conf +.vscode/ diff --git a/.travis.yml b/.travis.yml index d733b16432c..9ffb48b1147 100644 --- a/.travis.yml +++ b/.travis.yml @@ -28,24 +28,13 @@ env: - >- BUILD_TYPE=centaurEngineUpgradeLocal BUILD_MYSQL=5.7 + # Temporarily keeping until `centaurEngineUpgradePapiV2beta` or similar exists - >- BUILD_TYPE=centaurEngineUpgradePapiV2alpha1 BUILD_MYSQL=5.7 - - >- - BUILD_TYPE=centaurHoricromtalPapiV2alpha1 - BUILD_MYSQL=5.7 - >- BUILD_TYPE=centaurHoricromtalPapiV2beta BUILD_MYSQL=5.7 - - >- - BUILD_TYPE=centaurHoricromtalEngineUpgradePapiV2alpha1 - BUILD_MYSQL=5.7 - - >- - BUILD_TYPE=centaurPapiUpgradePapiV2alpha1 - BUILD_MYSQL=5.7 - - >- - BUILD_TYPE=centaurPapiUpgradeNewWorkflowsPapiV2alpha1 - BUILD_MYSQL=5.7 - >- BUILD_TYPE=centaurLocal BUILD_MARIADB=10.3 @@ -55,9 +44,6 @@ env: - >- BUILD_TYPE=centaurLocal BUILD_POSTGRESQL=11.3 - - >- - BUILD_TYPE=centaurPapiV2alpha1 - BUILD_MYSQL=5.7 - >- BUILD_TYPE=centaurPapiV2beta BUILD_MYSQL=5.7 @@ -67,15 +53,10 @@ env: - >- BUILD_TYPE=centaurTes BUILD_MYSQL=5.7 - - >- - BUILD_TYPE=centaurWdlUpgradeLocal - BUILD_MYSQL=5.7 - >- BUILD_TYPE=checkPublish - >- BUILD_TYPE=horicromtalDeadlock - - >- - BUILD_TYPE=dockerScripts - >- BUILD_TYPE=sbt BUILD_SBT_INCLUDE=engine @@ -92,8 +73,6 @@ env: BUILD_TYPE=dbms - >- BUILD_TYPE=singleWorkflowRunner - - >- - BUILD_TYPE=metadataComparisonPython - >- BUILD_TYPE=referenceDiskManifestBuilderApp script: diff --git a/CHANGELOG.md b/CHANGELOG.md index 0c40f47b651..0084c05171b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,97 @@ # Cromwell Change Log +## 85 Release Notes + +### Migration of PKs to BIGINT + +The PK of below tables will be migrated from INT to BIGINT. Also, since `ROOT_WORKFLOW_ID` in `SUB_WORKFLOW_STORE_ENTRY` is a FK to `WORKFLOW_STORE_ENTRY_ID` in `WORKFLOW_STORE_ENTRY` +it is also being migrated from INT to BIGINT. +* DOCKER_HASH_STORE_ENTRY +* WORKFLOW_STORE_ENTRY +* SUB_WORKFLOW_STORE_ENTRY + +### Improvement to "retry with more memory" behavior + +Cromwell will now retry a task with more memory after it fails with return code 137, provided all +the other requirements for retrying with more memory are met. + +### DRS Improvements + +#### Support for invoking `CromwellDRSLocalizer` with manifest file + +`CromwellDRSLocalizer` can now handle multiple file localizations in a single invocation. Users can provide a +manifest file containing multiple (DRS id, local container path) pairs in CSV format, and they will be localized in +sequence, with the program exiting if any fail. +``` +java -jar /path/to/localizer.jar [options] -m /local/path/to/manifest/file.txt +``` + +The previous method of passing in a single DRS file and container destination using positional arguments is still +supported. + +#### Improvement to DRS localization in GCP papiv2beta backend + +All DRS inputs to a task are now localized in a single PAPI action, which should improve speed and resolve +failures observed when attempting to localize a large number of DRS files. + + +### Allow list for HTTP WDL resolution + +Administrators can now configure Cromwell with an allow list that limits the domains from which WDLs can be resolved and imported. +Default behavior is unchanged (Cromwell attempts to resolve WDL files from any URI). Example configuration: +``` +languages { + WDL { + http-allow-list { + enabled: true + allowed-http-hosts: [ + "my.wdl.repo.org", + "raw.githubusercontent.com" + ] + } + } +} +``` + +### CWL implementation removed + +This release removes the `cwl` top-level artifact. Some nonfunctional references may remain, and will be addressed over time. + +For more information, see the [Cromwell 79 release notes](https://github.com/broadinstitute/cromwell/releases/tag/79). + +### TES Improvments + +* Tes system errors are are now reported in Cromwell execution logs when the TES backend returns a task error. + +* Cromwell now attempts to translate `disks` attributes [written for GCP](https://cromwell.readthedocs.io/en/stable/RuntimeAttributes/#disks) into valid `disk` attributes for TES. For information on supported conversions, refer to the [TES documentation](https://cromwell.readthedocs.io/en/stable/backends/TES/). + +### Bug Fixes + +* Reference disks are only mounted if configured in the workflow options. + +* Recent docker images of Ubuntu use a new manifest format, ensure that these newer image versions can be pulled from Docker Registry without issue. + +* When converting ValueStore objects to strings for logging, we truncate long values to limit memory usage. + + +### Security Patching + +Updates to dependencies to fix security vulnerabilities. + +## 84 Release Notes + +### CromIAM enabled user checks + +For Cromwell instances utilizing the optional CromIAM identity and access management component, the following endpoints now verify that the calling user is enabled before forwarding the request. +* `/api/workflows/v1/backends` +* `/api/womtool/v1/describe` + +This change makes the above endpoints consistent with the existing behavior of all the other endpoints in the `/api/` path of CromIAM. + +## 83 Release Notes + +* Changes the type of several primary key columns in call caching tables from int to bigint. The database migration may be lengthy if your database contains a large amount of call caching data. + ## 82 Release Notes * Restored missing example configuration file @@ -27,7 +119,7 @@ The BCS backend and OSS filesystem (both of which support Alibaba Cloud) have be Cromwell 80 no longer supports the wes2cromwell project within the Cromwell repository. In the previous release, 3 Wes2Cromwell endpoints in the Cromwell project were implemented and documented in the Swagger API. Three new endpoints, -located within the wes2cromwell project, will also be moved, implemented, and documented within Cromwell. As a result of this, we can safely remove +located within the wes2cromwell project, will also be moved, implemented, and documented within Cromwell. As a result of this, we can safely remove and deprecate the wes2cromwell project from the repo. Previous endpoints: @@ -111,11 +203,11 @@ Previously: ### New 'requestedWorkflowId' API Option -Allows users to choose their own workflow IDs at workflow submission time. +Allows users to choose their own workflow IDs at workflow submission time. If supplied for single workflows, this value must be a JSON string containing a valid, and not already used, UUID. For batch submissions, this value must be a JSON array of valid UUIDs. -If not supplied, the behavior is as today: Cromwell will generate a random workflow ID for every workflow submitted. +If not supplied, the behavior is as today: Cromwell will generate a random workflow ID for every workflow submitted. ### Bug Fixes @@ -228,8 +320,8 @@ AccessURLs](https://ga4gh.github.io/data-repository-service-schemas/preview/rele ### No labels update for Archived workflows -If **- and ONLY if -** you have metadata archiving turned on, then for a workflow whose metadata has been archived by Cromwell -according to the lifecycle policy, Cromwell will no longer add new labels or update existing labels for this workflow +If **- and ONLY if -** you have metadata archiving turned on, then for a workflow whose metadata has been archived by Cromwell +according to the lifecycle policy, Cromwell will no longer add new labels or update existing labels for this workflow coming through PATCH `/labels` endpoint. ## 60 Release Notes @@ -241,12 +333,12 @@ containerized using [AdoptOpenJDK 11 HotSpot](https://adoptopenjdk.net/). ### Hybrid metadata storage ("carboniting") removed -Carboniting functionality has been removed from Cromwell. +Carboniting functionality has been removed from Cromwell. There will be no effect for customers who store metadata permanently in the relational database (most common), and there will also be no effect for customers who use the in-memory database. Breaking change only for customers who explicitly enabled `carbonite-metadata-service` in their configuration to split -metadata storage between a relational database and Google Cloud Storage. If you had previously enabled carboniting and +metadata storage between a relational database and Google Cloud Storage. If you had previously enabled carboniting and deletion, any workflows marked as `ArchivedAndPurged` in your database will no longer be accessible via the Cromwell metadata API. ## 59 Release Notes @@ -273,9 +365,9 @@ rather than through a level of indirection to a manifest file stored in GCS. Mor ### Retry with More Memory as workflow option The experimental memory retry feature gains per-workflow customization and includes breaking changes: -* The per-backend configuration key `.config.memory-retry.error-keys` has been removed and replaced +* The per-backend configuration key `.config.memory-retry.error-keys` has been removed and replaced with global key `system.memory-retry-error-keys` -* The per-backend configuration key `.config.memory-retry.multiplier` has been replaced with **workflow option** +* The per-backend configuration key `.config.memory-retry.multiplier` has been replaced with **workflow option** `memory_retry_multiplier` More details can be found [here](https://cromwell.readthedocs.io/en/develop/wf_options/Overview.md#retry-with-more-memory-multiplier). @@ -284,7 +376,7 @@ More details can be found [here](https://cromwell.readthedocs.io/en/develop/wf_o * Fixed a bug that caused Cromwell to mark workflows as failed after a single `500`, `503`, or `504` error from Google Cloud Storage. * Cromwell will now retry these errors as designed. - * The default retry count is `5` and may be customized with `system.io.number-of-attempts`. + * The default retry count is `5` and may be customized with `system.io.number-of-attempts`. ## 55 Release Notes @@ -294,7 +386,7 @@ Users with access to the new Mac hardware should review [important information p ### Bug Fixes -* Fixed a bug that prevented `read_json()` from working with arrays and primitives. The function now works as expected for all valid JSON data inputs. +* Fixed a bug that prevented `read_json()` from working with arrays and primitives. The function now works as expected for all valid JSON data inputs. More information on JSON Type to WDL Type conversion can be found [here](https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md#mixed-read_jsonstringfile). * Now retries HTTP 408 responses as well as HTTP 429 responses during DOS/DRS resolution requests. @@ -322,19 +414,19 @@ to the worker VM on the next attempt if the task is interrupted. More details [h ### Bug Fixes * Fixed a bug that prevented `write_json()` from working with arrays and primitives. The function now works as expected for `Boolean`, `String`, `Integer`, `Float`, - `Pair[_, _]`, `Object`, `Map[_, _]` and `Array[_]` (including array of objects) type inputs. More information on WDL Type to JSON Type + `Pair[_, _]`, `Object`, `Map[_, _]` and `Array[_]` (including array of objects) type inputs. More information on WDL Type to JSON Type conversion can be found [here](https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md#mixed-read_jsonstringfile). ### Spark backend support removal -Spark backend was not widely used and it was decided to remove it from the codebase in order to narrow the scope of Cromwell code. +Spark backend was not widely used and it was decided to remove it from the codebase in order to narrow the scope of Cromwell code. ### Improved DRS Localizer logging Error logging while localizing a DRS URI should now be more clear especially when there is a Requester Pays bucket involved. ### Per-backend hog factors -Cromwell now allows overriding system-level log factors on back-end level. First, Cromwell will try to use hog-factor +Cromwell now allows overriding system-level log factors on back-end level. First, Cromwell will try to use hog-factor defined in the backend config, and if it is not defined, it will default to using system-wide hog factor. ```conf backend { @@ -402,16 +494,16 @@ https://cromwell.readthedocs.io/en/stable/tutorials/Containers/#singularity). ### Google library upgrade [(#5565)](https://github.com/broadinstitute/cromwell/pull/5565) -All previous versions of Cromwell shipped with Google Cloud Storage (GCS) libraries that are now deprecated and will [stop working in August 2020](https://developers.googleblog.com/2018/03/discontinuing-support-for-json-rpc-and.html). This release adopts updated libraries to ensure uninterrupted operation. The only user action required is upgrading Cromwell. +All previous versions of Cromwell shipped with Google Cloud Storage (GCS) libraries that are now deprecated and will [stop working in August 2020](https://developers.googleblog.com/2018/03/discontinuing-support-for-json-rpc-and.html). This release adopts updated libraries to ensure uninterrupted operation. The only user action required is upgrading Cromwell. ### Bug fixes * Fixed a bug that required Cromwell to be restarted in order to pick up DNS changes. * By default, the JVM caches DNS records with a TTL of infinity. - * Cromwell now configures its JVM with a 3-minute TTL. This value can be customized by setting `system.dns-cache-ttl`. + * Cromwell now configures its JVM with a 3-minute TTL. This value can be customized by setting `system.dns-cache-ttl`. * Clarified an error message that Cromwell emits when the compute backend terminates a job of its own volition (as opposed to termination in response to an abort request from Cromwell) * Previously, the error read `The job was aborted from outside Cromwell` - * The new error reads `The compute backend terminated the job. If this termination is unexpected, examine likely causes such as preemption, running out of disk or memory on the compute instance, or exceeding the backend's maximum job duration.` + * The new error reads `The compute backend terminated the job. If this termination is unexpected, examine likely causes such as preemption, running out of disk or memory on the compute instance, or exceeding the backend's maximum job duration.` ## 51 Release Notes @@ -422,8 +514,8 @@ https://cromwell.readthedocs.io/en/stable/Configuring/#call-caching) for details ### Bug fixes -* Fixed a bug where the `size(...)` function did not work correctly on files - from a shared filesystem if `size(...)` was called in the input section on a +* Fixed a bug where the `size(...)` function did not work correctly on files + from a shared filesystem if `size(...)` was called in the input section on a relative path. + Fixed a bug where the `use_relative_output_paths` option would not preserve intermediate folders. @@ -431,17 +523,17 @@ https://cromwell.readthedocs.io/en/stable/Configuring/#call-caching) for details #### Call caching blacklisting improvements -Cromwell previously supported blacklisting GCS buckets containing cache hits which could not be copied for permissions +Cromwell previously supported blacklisting GCS buckets containing cache hits which could not be copied for permissions reasons. Cromwell now adds support for blacklisting individual cache hits which could not be copied for any reason, as well as grouping blacklist caches according to a workflow option key. More information available in the [ -call caching documentation]( https://cromwell.readthedocs.io/en/stable/Configuring/#call-caching). +call caching documentation]( https://cromwell.readthedocs.io/en/stable/Configuring/#call-caching). #### new xxh64 and fingerprint strategies for call caching -Existing call cache strategies `path` and `path+modtime` don't work when using docker on shared filesystems +Existing call cache strategies `path` and `path+modtime` don't work when using docker on shared filesystems (SFS backend, i.e. not in cloud storage). The `file` (md5sum) strategy works, but uses a lot of resources. -Two faster strategies have been added for this use case: `xxh64` and -`fingerprint`. `xxh64` is a lightweight hashing algorithm, `fingerprint` is a strategy designed to be very +Two faster strategies have been added for this use case: `xxh64` and +`fingerprint`. `xxh64` is a lightweight hashing algorithm, `fingerprint` is a strategy designed to be very lightweight. Read more about it in the [call caching documentation]( https://cromwell.readthedocs.io/en/stable/Configuring/#call-caching). @@ -475,7 +567,7 @@ SELECT table_rows FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = 'cromwell' #### Execution Directory Layout (cache copies) -When an attempt to copy a cache result is made, you'll now see a `cacheCopy` directory in the call root directory. +When an attempt to copy a cache result is made, you'll now see a `cacheCopy` directory in the call root directory. This prevents them clashing with the files staged to the same directory for attempt 1 if the cache copy fails (see also: Bug Fixes). The directory layout used to be: @@ -512,7 +604,7 @@ but is now: #### Disable call-caching for tasks -It is now possible to indicate in a workflow that a task should not be call-cached. See details +It is now possible to indicate in a workflow that a task should not be call-cached. See details [here](https://cromwell.readthedocs.io/en/stable/optimizations/VolatileTasks). #### Delete Intermediate Outputs on PapiV2 @@ -525,13 +617,13 @@ for more information. #### Metadata Archival Support Cromwell 49 now offers the option to archive metadata to GCS and remove the equivalent metadata from relational -database storage. Please see -[the documentation](https://cromwell.readthedocs.io/en/stable/Configuring#hybrid-metadata-storage-classic-carbonite) for more details. +database storage. Please see +[the documentation](https://cromwell.readthedocs.io/en/stable/Configuring#hybrid-metadata-storage-classic-carbonite) for more details. #### Adding support for Google Cloud Life Sciences v2beta -Cromwell now supports running workflows using Google Cloud Life Sciences v2beta API in addition to Google Cloud Genomics v2alpha1. -More information about migration to the new API from v2alpha1 -[here](https://cromwell.readthedocs.io/en/stable/backends/Google#migration-from-google-cloud-genomics-v2alpha1-to-google-cloud-life-sciences-v2beta). +Cromwell now supports running workflows using Google Cloud Life Sciences v2beta API in addition to Google Cloud Genomics v2alpha1. +More information about migration to the new API from v2alpha1 +[here](https://cromwell.readthedocs.io/en/stable/backends/Google#migration-from-google-cloud-genomics-v2alpha1-to-google-cloud-life-sciences-v2beta). * **Note** Google Cloud Life Sciences is the new name for newer versions of Google Cloud Genomics. * **Note** Support for Google Cloud Genomics v2alpha1 will be removed in a future version of Cromwell. Advance notice will be provided. @@ -539,23 +631,23 @@ More information about migration to the new API from v2alpha1 #### Installation methods -Links to the conda package and docker container are now available in +Links to the conda package and docker container are now available in [the install documentation](https://cromwell.readthedocs.io/en/stable/Getting/). ### Bug Fixes -+ Fix a bug where zip files with directories could not be imported. - For example a zip with `a.wdl` and `b.wdl` could be imported but one with `sub_workflows/a.wdl` ++ Fix a bug where zip files with directories could not be imported. + For example a zip with `a.wdl` and `b.wdl` could be imported but one with `sub_workflows/a.wdl` and `imports/b.wdl` could not. + Fix a bug which sometimes allowed execution scripts copied by a failed cache-copy to be run instead - of the attempt-1 script for a live job execution. - + of the attempt-1 script for a live job execution. + ## 48 Release Notes ### Womtool Graph for WDL 1.0 -The `womtool graph` command now supports WDL 1.0 workflows. +The `womtool graph` command now supports WDL 1.0 workflows. * **Note:** Generated graphs - including in WDL draft 2 - may look slightly different than they did in version 47. ### Documentation @@ -567,7 +659,7 @@ The `womtool graph` command now supports WDL 1.0 workflows. ### Retry with more memory on Papiv2 [(#5180)](https://github.com/broadinstitute/cromwell/pull/5180) -Cromwell now allows user defined retries. With `memory-retry` config you can specify an array of strings which when encountered in the `stderr` +Cromwell now allows user defined retries. With `memory-retry` config you can specify an array of strings which when encountered in the `stderr` file by Cromwell, allows the task to be retried with multiplier factor mentioned in the config. More information [here](https://cromwell.readthedocs.io/en/stable/backends/Google/). ### GCS Parallel Composite Upload Support @@ -601,7 +693,7 @@ PAPI error code 14. Task was preempted for the 2nd time. ``` Cromwell 44 introduced special handling that detects both preemption indicators and re-runs the job consistent with the `preemptible` setting. -Cromwell 46 enhances this handling in response to user reports of possible continued issues. +Cromwell 46 enhances this handling in response to user reports of possible continued issues. ## 45 Release Notes @@ -619,7 +711,7 @@ More info [here](https://cromwell.readthedocs.io/en/stable/WOMtool/) ### BCS backend new Features support #### New docker registry -Alibaba Cloud Container Registry is now supported for the `docker` runtime attribute, and the previous `dockerTag` +Alibaba Cloud Container Registry is now supported for the `docker` runtime attribute, and the previous `dockerTag` runtime attribute continues to be available for Alibaba Cloud OSS Registry. #### Call caching Cromwell now supports Call caching when using the BCS backend. @@ -647,7 +739,7 @@ Cromwell's PAPI v2 backend will now handle this type of preemption. ### Virtual Private Cloud with Subnetworks -Cromwell now allows PAPIV2 jobs to run on a specific subnetwork inside a private network by adding the subnetwork key +Cromwell now allows PAPIV2 jobs to run on a specific subnetwork inside a private network by adding the subnetwork key `subnetwork-label-key` inside `virtual-private-cloud` in backend configuration. More info [here](https://cromwell.readthedocs.io/en/stable/backends/Google/). ### Call caching database refactoring @@ -655,7 +747,7 @@ Cromwell now allows PAPIV2 jobs to run on a specific subnetwork inside a private Cromwell's `CALL_CACHING_HASH_ENTRY` primary key has been refactored to use a `BIGINT` datatype in place of the previous `INT` datatype. Cromwell will not be usable during the time the Liquibase migration for this refactor is running. In the Google Cloud SQL with SSD environment this migration runs at a rate of approximately 100,000 `CALL_CACHING_HASH_ENTRY` -rows per second. In deployments with millions or billions of `CALL_CACHING_HASH_ENTRY` rows the migration may require +rows per second. In deployments with millions or billions of `CALL_CACHING_HASH_ENTRY` rows the migration may require a significant amount of downtime so please plan accordingly. The following SQL could be used to estimate the number of rows in this table: @@ -665,7 +757,7 @@ select max(CALL_CACHING_HASH_ENTRY_ID) from CALL_CACHING_HASH_ENTRY ### Stackdriver Instrumentation -Cromwell now supports sending metrics to [Google's Stackdriver API](https://cloud.google.com/monitoring/api/v3/). +Cromwell now supports sending metrics to [Google's Stackdriver API](https://cloud.google.com/monitoring/api/v3/). Learn more on how to configure [here](https://cromwell.readthedocs.io/en/stable/developers/Instrumentation/). ### BigQuery in PAPI @@ -682,7 +774,7 @@ which now has been updated to `services.Instrumentation.config`. More info on it #### cached-copy -A new experimental feature, the `cached-copy` localization strategy is available for the shared filesystem. +A new experimental feature, the `cached-copy` localization strategy is available for the shared filesystem. More information can be found in the [documentation on localization](https://cromwell.readthedocs.io/en/stable/backends/HPC). #### Yaml node limits @@ -716,7 +808,7 @@ Specifically, the new `validWorkflow` key indicates whether the workflow file is ### Configuration Changes * Virtual private networks can now be configured. See the section below for details. - + #### Batch Request Timeouts The timeout on Cromwell's requests to PAPIv2 can now be configured. See the sample PAPIv2.conf for more documentation: @@ -725,7 +817,7 @@ The timeout on Cromwell's requests to PAPIv2 can now be configured. See the samp backend { providers { PAPIv2 { - config { + config { batch-requests { timeouts { read = 10 seconds @@ -802,14 +894,14 @@ This field is now accepted within WDL files as well as within the configuration #### Logging long running jobs -All backends can now emit slow job warnings after a configurable time running. +All backends can now emit slow job warnings after a configurable time running. NB This example shows how to configure this setting for the PAPIv2 backend: ```conf # Emit a warning if jobs last longer than this amount of time. This might indicate that something got stuck. backend { providers { PAPIv2 { - config { + config { slow-job-warning-time: 24 hours } } @@ -824,7 +916,7 @@ backend { * The `gpuType` attribute is no longer validated against a whitelist at workflow submission time. Instead, validation now happens at runtime. This allows any valid accelerator to be used. * The `nvidiaDriverVersion` attribute is now available in WDL `runtime` sections. The default continues to be `390.46` which applies if and only if GPUs are being used. * A default `gpuType` ("nvidia-tesla-k80") will now be applied if `gpuCount` is specified but `gpuType` is not. -* Similarly, a default `gpuCount` (1) will be applied if `gpuType` is specified but `cpuCount` is not. +* Similarly, a default `gpuCount` (1) will be applied if `gpuType` is specified but `cpuCount` is not. ### Bug fixes @@ -865,7 +957,7 @@ services { class = "cromwell.services.healthmonitor.impl.standard.StandardHealthMonitorServiceActor" } } -``` +``` With this one: ``` services { @@ -876,7 +968,7 @@ services { } } } -``` +``` ###### From `WorkbenchHealthMonitorServiceActor`: Replace this stanza: ``` @@ -893,7 +985,7 @@ services { } } } -``` +``` With this one: ``` services { @@ -909,12 +1001,12 @@ services { } } } -``` +``` ### Workflow options changes -A new workflow option is added. If the `final_workflow_outputs_dir` is set -`use_relative_output_paths` can be used. When set to `true` this will copy -all the outputs relative to their execution directory. +A new workflow option is added. If the `final_workflow_outputs_dir` is set +`use_relative_output_paths` can be used. When set to `true` this will copy +all the outputs relative to their execution directory. my_final_workflow_outputs_dir/~~MyWorkflow/af76876d8-6e8768fa/call-MyTask/execution/~~output_of_interest. More information can be found in [the workflow options documentation](https://cromwell.readthedocs.io/en/stable/wf_options/Overview/#output-copying). @@ -944,10 +1036,10 @@ shutdown. ### Bug fixes -#### Format fix for `write_map()` +#### Format fix for `write_map()` Fixed an issue that caused the `write_map()` function in Cromwell's WDL 1.0 implementation to produce output in the wrong format. Specifically, the output's rows and columns were swapped. WDL draft-2 was not affected. - + Incorrect `write_map()` output in Cromwell 38 and earlier: ``` key1 key2 key3 @@ -987,7 +1079,7 @@ available as `StandardHealthMonitorServiceActor`. ### Docker - Adds support for retrieving docker digests of asia.gcr.io images -- Adds configuration settings for docker digest lookups. See the `docker` section of the `reference.conf` for more information +- Adds configuration settings for docker digest lookups. See the `docker` section of the `reference.conf` for more information - Attempt to automatically adjust the boot disk size on the Google Cloud Backend (version 2) if the size of the image is greater than the default disk size or the required disk size in the runtime attributes. Only works for registries that support the version 2 of the manifest schema (https://docs.docker.com/registry/spec/manifest-v2-2/) At this date (12/09/18) this includes GCR and Dockerhub. @@ -1032,28 +1124,28 @@ Support `InputResourceRequirement` hint In cases where its not obvious why jobs are queued in Cromwell, you can enable logging for the Job Execution Token Dispenser, using the `system.hog-safety.token-log-interval-seconds` configuration value. -The default, `0`, means that no logging will occur. +The default, `0`, means that no logging will occur. #### HTTP Filesystem -- The HTTP filesystem is now enabled for engine use by default. To continue without an HTTP filesystem, you can add the +- The HTTP filesystem is now enabled for engine use by default. To continue without an HTTP filesystem, you can add the following content into the appropriate stanza of your configuration file: ``` engine { filesystems { - http { - enabled: false + http { + enabled: false } } } -``` +``` - When the value `exit-code-timeout-seconds` is set, `check-alive` command is now only called once every timeout interval instead of each poll. ### Beta preview of new Womtool `/describe` endpoint This new endpoint brings the functionality of Womtool to the world of web services. Submit workflows for validation and receive a JSON description in response. -The endpoint is still undergoing heavy development and should not be used in production. The final version will ship in a future release of Cromwell; watch this space. +The endpoint is still undergoing heavy development and should not be used in production. The final version will ship in a future release of Cromwell; watch this space. ### Bug fixes @@ -1093,7 +1185,7 @@ Details [here](https://cromwell.readthedocs.io/en/develop/backends/HPC/#exit-cod Coalesce metadata requests to eliminate expensive and redundant queries and metadata construction. -#### Eliminate redundant SFS logging and metadata +#### Eliminate redundant SFS logging and metadata Eliminate superfluous logging and metadata publishing in the shared filesystem backend on poll intervals where there was not a state change. @@ -1106,7 +1198,7 @@ Previously US-EAST-1 was hardcoded in places. ### Submit workflow using URL Cromwell now allows for a user to submit the URL pointing to workflow file to run a workflow. -More details on how to use it in: +More details on how to use it in: - `Server` mode can be found [here](https://cromwell.readthedocs.io/en/develop/api/RESTAPI/). - `Run` mode can be found [here](https://cromwell.readthedocs.io/en/develop/CommandLine/#run). @@ -1132,7 +1224,7 @@ Cromwell now offers the ability to cache file hashes on a root workflow level ba ### Extra configuration options -The value `dockerRoot` can now be set in a backend configuration. +The value `dockerRoot` can now be set in a backend configuration. This will set the execution folder in the container (default: `/cromwell-executions`). ### Bug Fixes @@ -1238,7 +1330,7 @@ Cromwell now supports retrying failed tasks up to a specified count by declaring * Cromwell now publishes the labels as soon as the workflow is submitted (whether started or on hold). If the labels are invalid, the workflow will not be submitted and request will fail. ### Scala 2.11 Removed -From version 32 onwards we will no longer be publishing build artifacts compatible with Scala 2.11. +From version 32 onwards we will no longer be publishing build artifacts compatible with Scala 2.11. * If you don't import the classes into your own scala project then this should have no impact on you. * If you **are** importing the classes into your own scala project, make sure you are using Scala 2.12. @@ -1330,12 +1422,12 @@ Workflow metadata for jobs run on a Google Pipelines API backend will report the ## 31 Release Notes -* **Cromwell server** -The Cromwell server source code is now located under `server/src`. `sbt assembly` will build the runnable Cromwell JAR in +* **Cromwell server** +The Cromwell server source code is now located under `server/src`. `sbt assembly` will build the runnable Cromwell JAR in `server/target/scala-2.12/` with a name like `cromwell-.jar`. * **Robustness** - + The rate at which jobs are being started can now be controlled using the `system.job-rate-control` configuration stanza. + + The rate at which jobs are being started can now be controlled using the `system.job-rate-control` configuration stanza. + A load controller service has been added to allow Cromwell to self-monitor and adjust its load accordingly. The load controller is currently a simple on/off switch controlling the job start rate. It gathers metrics from different parts of the system to inform its decision to stop the creation of jobs. @@ -1343,7 +1435,7 @@ You can find relevant configuration in the `services.LoadController` section of as well as in the `load-control` section in `reference.conf`. The load level of the monitored sub-systems are instrumented and can be found under the `cromwell.load` statsD path. + The statsD metrics have been re-shuffled a bit. If you had a dashboard you might find that you need to update it. -Changes include: +Changes include: + Removed artificially inserted "count" and "timing" the path + Added a `load` section + Metrics were prefixed twice with `cromwell` (`cromwell.cromwell.my_metric`), now they're only prefixed once @@ -1352,8 +1444,8 @@ Changes include: * Added a configuration option under `docker.hash-lookup.enabled` to disable docker hash lookup. Disabling it will also disable call caching for jobs with floating docker tags. - -* **API** + +* **API** + Updated the `/query` response to include the total number of query results returned. See [here](http://cromwell.readthedocs.io/en/develop/api/RESTAPI/#workflowqueryresponse) for more information. ## 30.1 Release Notes @@ -1368,14 +1460,14 @@ Changes include: ### Other changes -* **New Cromwell documentation** +* **New Cromwell documentation** Our documentation has moved from our [README](https://github.com/broadinstitute/cromwell/blob/29_hotfix/README.md) to a new website: [Cromwell Documentation](http://cromwell.readthedocs.io/en/develop/). There are new [Tutorials](http://cromwell.readthedocs.io/en/develop/tutorials/FiveMinuteIntro/) and much of the documentation has been re-written. The source files are in the [/docs](https://github.com/broadinstitute/cromwell/tree/develop/docs) directory. -* **API** +* **API** + Cromwell now supports input files in the yaml format (JSON format is still supported). + Added a [GET version for the `labels` endpoint](http://cromwell.readthedocs.io/en/develop/api/RESTAPI/#retrieves-the-current-labels-for-a-workflow) which will return current labels for a workflow. -* **Database** +* **Database** You have the option of storing the metadata in a separate SQL database than the database containing the internal engine data. When switching connection information for an existing database containing historical data, the tables should be manually replicated from one database instance to another using the tools appropriate for your specific @@ -1384,48 +1476,48 @@ and likely to change in the future. See the [Database Documentation](https://cro [cromwell.examples.conf](https://www.github.com/broadinstitute/cromwell/tree/develop/cromwell.example.backends/cromwell.examples.conf) for more information. -* **StatsD** +* **StatsD** Added initial support for StatsD instrumentation. See the [Instrumentation Documentation](https://cromwell.readthedocs.io/en/develop/Instrumentation) for details on how to use it. -* **User Service Account auth mode for Google** +* **User Service Account auth mode for Google** Added a new authentication mode for [Google Cloud Platform](https://cromwell.readthedocs.io/en/develop/backends/Google) which will allow a user to supply the JSON key file in their workflow options to allow for per-workflow authentication via service account. This is analogous to the previously existing refresh token authentication scheme. As with the refresh token scheme it is encouraged that the **user_service_account_json** workflow option field is added to the **encrypted-fields** list in the configuration. -* **Bugfixes** +* **Bugfixes** Abort of Dockerized tasks on the Local backend should now work as expected. Cromwell uses `docker kill` to kill the Docker container. ## 29 Release Notes ### Breaking Changes -* **Command line** +* **Command line** In preparation for supporting CWL scripts (yes, you read that right!), we have extensively revised the Command Line in Cromwell 29. For more details about the usage changes please see the [README](https://github.com/broadinstitute/cromwell#command-line-usage). And stay tuned to the [WDL/Cromwell blog](https://software.broadinstitute.org/wdl/blog) over the next couple of months for more news about CWL. -* **Request timeouts** +* **Request timeouts** Cromwell now returns more specific `503 Service Unavailable` error codes on request timeouts, rather than the more generic `500 Internal Server Error`. The response for a request timeout will now be plain text, rather than a JSON format. -* **Metadata endpoint** +* **Metadata endpoint** The response from the metadata endpoint can be quite large depending on your workflow. You can now opt-in to have Cromwell gzip your metadata file, in order to reduce file size, by sending the `Accept-Encoding: gzip` header. The default behavior now does not gzip encode responses. -* **Engine endpoints** +* **Engine endpoints** Previously the engine endpoints were available under `/api/engine` but now the endpoints are under `/engine` so they don't require authentication. Workflow endpoints are still available under `/api/workflows`. We also deprecated the setting `api.routeUnwrapped` as a part of this internal consistency effort. -* **Call caching diff** +* **Call caching diff** We updated the response format of the [callcaching/diff](https://github.com/broadinstitute/cromwell#get-apiworkflowsversioncallcachingdiff) endpoint. ### Other changes -* **Cromwell server** +* **Cromwell server** When running in server mode, Cromwell now attempts to gracefully shutdown after receiving a `SIGINT` (`Ctrl-C`) or `SIGTERM` (`kill`) signal. This means that Cromwell waits for all pending database writes before exiting, as long as you include `application.conf` at the top of your config file. You can find detailed information about how to configure this feature in the [Cromwell Wiki](https://github.com/broadinstitute/cromwell/wiki/DevZone#graceful-server-shutdown). -* **Concurrent jobs** +* **Concurrent jobs** You can now limit the number of concurrent jobs for any backend. Previously this was only possible in some backend implementations. Please see the [README](https://github.com/broadinstitute/cromwell#backend-job-limits) for details. ### WDL -* **Optional WDL variables** +* **Optional WDL variables** Empty optional WDL values are now rendered as the `null` JSON value instead of the JSON string `"null"` in the metadata and output endpoints. You do not need to migrate previous workflows. Workflows run on Cromwell 28 and prior will still render empty values as `"null"`. -* **Empty WDL variables** +* **Empty WDL variables** Cromwell now accepts `null` JSON values in the input file and coerces them as an empty WDL value. WDL variables must be declared optional in order to be supplied with a `null` JSON value. input.json @@ -1474,8 +1566,8 @@ task writer { #### `ContinueWhilePossible` A workflow utilizing the WorkflowFailureMode Workflow Option `ContinueWhilePossible` will now successfully reach a terminal state once all runnable jobs have completed. -#### `FailOnStderr` -When `FailOnStderr` is set to false, Cromwell no longer checks for the existence of a stderr file for that task. +#### `FailOnStderr` +When `FailOnStderr` is set to false, Cromwell no longer checks for the existence of a stderr file for that task. ### WDL Functions @@ -1491,7 +1583,7 @@ task foo { command { ... } runtime { docker: "..." - memory: ceil(size(in_file)) * 4 + memory: ceil(size(in_file)) * 4 } } ``` @@ -1506,7 +1598,7 @@ If the hashes fail to be calculated, the reason is indicated in a `hashFailures` See the [README](https://github.com/broadinstitute/cromwell#get-apiworkflowsversionidmetadata) for an example metadata response. -* New endpoint returning the hash differential for 2 calls. +* New endpoint returning the hash differential for 2 calls. `GET /api/workflows/:version/callcaching/diff` @@ -1535,7 +1627,7 @@ A second value is allowed, `reference`, that will instead point to the original filesystems { gcs { auth = "application-default" - + caching { duplication-strategy = "reference" } @@ -1563,7 +1655,7 @@ than the previous default. The migration will only be executed on MySQL. Other databases will lose their previous cached jobs. In order to run properly on MySQL, **the following flag needs to be adjusted**: https://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html#sysvar_group_concat_max_len The following query will give you a minimum to set the group_concat_max_len value to: - + ```sql SELECT MAX(aggregated) as group_concat_max_len FROM ( @@ -1574,15 +1666,15 @@ SELECT MAX(aggregated) as group_concat_max_len FROM ``` Here is the SQL command to run to set the group_concat_max_len flag to the proper value: - + ```sql SET GLOBAL group_concat_max_len = value ``` - + Where `value` is replaced with the value you want to set it to. - + Note that the migration will fail if the flag is not set properly. - + ### Breaking Changes * The update to Slick 3.2 requires a database stanza to @@ -1630,7 +1722,7 @@ run the call: ### Docker -* The Docker section of the configuration has been slightly reworked +* The Docker section of the configuration has been slightly reworked An option to specify how a Docker hash should be looked up has been added. Two methods are available. "local" will try to look for the image on the machine where cromwell is running. If it can't be found, Cromwell will try to `pull` the image and use the hash from the retrieved image. "remote" will try to look up the image hash directly on the remote repository where the image is located (Docker Hub and GCR are supported) @@ -1694,14 +1786,14 @@ failures: [{ * Added a configuration option under `system.io` to throttle the number of I/O queries that Cromwell makes, as well as configure retry parameters. This is mostly useful for the JES backend and should be updated to match the GCS quota available for the project. - + ``` system.io { # Global Throttling - This is mostly useful for GCS and can be adjusted to match # the quota availble on the GCS API number-of-requests = 100000 per = 100 seconds - + # Number of times an I/O operation should be attempted before giving up and failing it. number-of-attempts = 5 } @@ -1725,7 +1817,7 @@ system.io { ### Config Changes -* Added a field `insert-batch-size` to the `database` stanza which defines how many values from a batch insert will be processed at a time. This value defaults to 2000. +* Added a field `insert-batch-size` to the `database` stanza which defines how many values from a batch insert will be processed at a time. This value defaults to 2000. * Moved the config value `services.MetadataService.metadata-summary-refresh-interval` to `services.MetadataService.config.metadata-summary-refresh-interval` * Added ability to override the default zone(s) used by JES via the config structure by setting `genomics.default-zones` in the JES configuration * The cromwell server TCP binding timeout is now configurable via the config key `webservice.binding-timeout`, defaulted @@ -1748,15 +1840,15 @@ system.io { * In some cases the SFS backend, used for Local, SGE, etc., coerced `WdlFile` to `WdlString` by using `.toUri`. This resulted in strings prepended with `file:///path/to/file`. Now absolute file paths will not contain the uri scheme. * Launch jobs on servers that support the GA4GH Task Execution Schema using the TES backend. -* **Call caching: Cromwell will no longer try to use the cache for WDL tasks that contain a floating docker tag.** +* **Call caching: Cromwell will no longer try to use the cache for WDL tasks that contain a floating docker tag.** Call caching will still behave the same for tasks having a docker image with a specific hash. - See https://github.com/broadinstitute/cromwell#call-caching-docker-tags for more details. + See https://github.com/broadinstitute/cromwell#call-caching-docker-tags for more details. * Added docker hash lookup. Cromwell will try to lookup the hash for a docker image with a floating tag, and use that hash when executing the job. This will be reflected in the metadata where the docker runtime attribute will contains the hash that was used. If Cromwell is unable to lookup the docker hash, the job will be run with the original user defined floating tag. Cromwell is currently able to lookup public and private docker hashes for images on Docker Hub and Google Container Engine for job running on the JES backend. For other backends, cromwell is able to lookup public docker hashes for Docker Hub and Google Container Engine. - See https://github.com/broadinstitute/cromwell#call-caching-docker-tags for more details. + See https://github.com/broadinstitute/cromwell#call-caching-docker-tags for more details. ### Database schema changes * Added CUSTOM_LABELS as a field of WORKFLOW_STORE_ENTRY, to store workflow store entries. @@ -1775,7 +1867,7 @@ resulted in strings prepended with `file:///path/to/file`. Now absolute file pat * Timing diagrams and metadata now receive more fine grained workflow states between submission and Running. * Support for the Pair WDL type (e.g. `Pair[Int, File] floo = (3, "gs://blar/blaz/qlux.txt")`) * Added support for new WDL functions: - * `zip: (Array[X], Array[Y]) => Array[Pair[X, Y]]` - align items in the two arrays by index and return them as WDL pairs + * `zip: (Array[X], Array[Y]) => Array[Pair[X, Y]]` - align items in the two arrays by index and return them as WDL pairs * `cross: (Array[X], Array[Y]) => Array[Pair[X, Y]]` - create every possible pair from the two input arrays and return them all as WDL pairs * `transpose: (Array[Array[X]]) => Array[Array[X]]` compute the matrix transpose for a 2D array. Assumes each inner array has the same length. * By default, `system.abort-jobs-on-terminate` is false when running `java -jar cromwell.jar server`, and true when running `java -jar cromwell.jar run `. @@ -1791,9 +1883,9 @@ resulted in strings prepended with `file:///path/to/file`. Now absolute file pat * Fix bugs related to the behavior of Cromwell in Single Workflow Runner Mode. Cromwell will now exit once a workflow completes in Single Workflow Runner Mode. Additionally, when restarting Cromwell in Single Workflow Runner Mode, Cromwell will no longer restart incomplete workflows from a previous session. ### Annex A - Workflow outputs - + The WDL specification has changed regarding [workflow outputs](https://github.com/openwdl/wdl/blob/master/versions/draft-2/SPEC.md#outputs) to accommodate sub workflows. -This change is backward compatible in terms of runnable WDLs (WDL files using the deprecated workflow outputs syntax will still run the same). +This change is backward compatible in terms of runnable WDLs (WDL files using the deprecated workflow outputs syntax will still run the same). The only visible change lies in the metadata (as well as the console output in single workflow mode, when workflow outputs are printed out at the end of a successful workflow). TL;DR Unless you are parsing or manipulating the "key" by which workflow outputs are referenced in the metadata (and/or the console output for single workflow mode), you can skip the following explanation. @@ -1846,7 +1938,7 @@ task t { The new syntax allows for type checking of the outputs as well as expressions. It also allows for explicitly naming to the outputs. The old syntax doesn't give the ability to name workflow outputs. For consistency reasons, Cromwell will generate a "new syntax" workflow output for each task output, and name them. -Their name will be generated using their FQN, which would give +Their name will be generated using their FQN, which would give ``` output { @@ -1854,8 +1946,8 @@ output { String w.t.out2 = t.out2 } ``` - -However as the FQN separator is `.`, the name itself cannot contain any `.`. + +However as the FQN separator is `.`, the name itself cannot contain any `.`. For that reason, `.` are replaced with `_` : *Old syntax expanded to new syntax* @@ -1866,16 +1958,16 @@ output { } ``` -The consequence is that the workflow outputs section of the metadata for `old_syntax` would previously look like - +The consequence is that the workflow outputs section of the metadata for `old_syntax` would previously look like + ``` outputs { "w.t.out1": "hello", "w.t.out2": "hello" } ``` - -but it will now look like + +but it will now look like ``` outputs { diff --git a/CromIAM/src/main/resources/application.conf b/CromIAM/src/main/resources/application.conf index c809d0daafb..ff1006ab91b 100644 --- a/CromIAM/src/main/resources/application.conf +++ b/CromIAM/src/main/resources/application.conf @@ -24,7 +24,7 @@ swagger_oauth { akka { log-dead-letters = "off" - loggers = ["akka.event.slf4j.Slf4jLogger"] + loggers = ["cromwell.core.logging.EnhancedSlf4jLogger"] http { server { diff --git a/CromIAM/src/main/resources/logback.xml b/CromIAM/src/main/resources/logback.xml index 030868d3259..e9e2b753466 100644 --- a/CromIAM/src/main/resources/logback.xml +++ b/CromIAM/src/main/resources/logback.xml @@ -1,5 +1,15 @@ + + + + @@ -18,7 +28,7 @@ - %date %X{sourceThread} %-5level - %msg%n + %ed{yyyy-MM-dd HH:mm:ss,SSS} %et %-5level - %msg%n @@ -68,7 +78,7 @@ - %d{yyyy-MM-dd HH:mm:ss,SSS} [%thread] %-5level %logger{35} - %msg%n + %ed{yyyy-MM-dd HH:mm:ss,SSS} [%et] %-5level %logger{35} - %msg%n diff --git a/CromIAM/src/main/scala/cromiam/instrumentation/CromIamInstrumentation.scala b/CromIAM/src/main/scala/cromiam/instrumentation/CromIamInstrumentation.scala index 63f48073146..65b164f00f6 100644 --- a/CromIAM/src/main/scala/cromiam/instrumentation/CromIamInstrumentation.scala +++ b/CromIAM/src/main/scala/cromiam/instrumentation/CromIamInstrumentation.scala @@ -20,6 +20,7 @@ trait CromIamInstrumentation extends CromwellInstrumentation { val samPrefix: NonEmptyList[String] = NonEmptyList.one("sam") val getWhitelistPrefix = NonEmptyList.one("get-whitelist") + val getUserEnabledPrefix = NonEmptyList.one("get-user-enabled") val userCollectionPrefix = NonEmptyList.one("user-collection") val authCollectionPrefix = NonEmptyList.one("auth-collection") val registerCollectionPrefix = NonEmptyList.one("register-collection") diff --git a/CromIAM/src/main/scala/cromiam/sam/SamClient.scala b/CromIAM/src/main/scala/cromiam/sam/SamClient.scala index f289251a2fb..d6a315f8241 100644 --- a/CromIAM/src/main/scala/cromiam/sam/SamClient.scala +++ b/CromIAM/src/main/scala/cromiam/sam/SamClient.scala @@ -18,6 +18,7 @@ import cromiam.sam.SamResourceJsonSupport._ import cromiam.server.status.StatusCheckedSubsystem import cromwell.api.model._ import mouse.boolean._ +import spray.json.RootJsonFormat import scala.concurrent.ExecutionContextExecutor @@ -73,6 +74,33 @@ class SamClient(scheme: String, } yield whitelisted } + def isUserEnabledSam(user: User, cromIamRequest: HttpRequest): FailureResponseOrT[Boolean] = { + val request = HttpRequest( + method = HttpMethods.GET, + uri = samUserStatusUri, + headers = List[HttpHeader](user.authorization) + ) + + for { + response <- instrumentRequest( + () => Http().singleRequest(request).asFailureResponseOrT, + cromIamRequest, + instrumentationPrefixForSam(getUserEnabledPrefix) + ) + userEnabled <- response.status match { + case StatusCodes.OK => + val unmarshal: IO[UserStatusInfo] = IO.fromFuture(IO(Unmarshal(response.entity).to[UserStatusInfo])) + FailureResponseOrT.right[HttpResponse](unmarshal).map { userInfo => + if (!userInfo.enabled) log.info("Access denied for user {}", user.userId) + userInfo.enabled + } + case _ => + log.error("Could not verify access with Sam for user {}, error was {} {}", user.userId, response.status, response.toString().take(100)) + FailureResponseOrT.pure[IO, HttpResponse](false) + } + } yield userEnabled + } + def collectionsForUser(user: User, cromIamRequest: HttpRequest): FailureResponseOrT[List[Collection]] = { val request = HttpRequest(method = HttpMethods.GET, uri = samBaseCollectionUri, headers = List[HttpHeader](user.authorization)) @@ -170,6 +198,7 @@ class SamClient(scheme: String, private lazy val samBaseResourceUri = s"$samBaseUri/api/resource" private lazy val samBaseCollectionUri = s"$samBaseResourceUri/workflow-collection" private lazy val samSubmitWhitelistUri = s"$samBaseResourceUri/caas/submit/action/get_whitelist" + private lazy val samUserStatusUri = s"$samBaseUri/register/user/v2/self/info" } @@ -188,4 +217,8 @@ object SamClient { def SamRegisterCollectionExceptionResp(statusCode: StatusCode) = HttpResponse(status = statusCode, entity = SamRegisterCollectionException(statusCode).getMessage) + case class UserStatusInfo(adminEnabled: Boolean, enabled: Boolean, userEmail: String, userSubjectId: String) + + implicit val UserStatusInfoFormat: RootJsonFormat[UserStatusInfo] = jsonFormat4(UserStatusInfo) + } diff --git a/CromIAM/src/main/scala/cromiam/webservice/CromIamApiService.scala b/CromIAM/src/main/scala/cromiam/webservice/CromIamApiService.scala index 63694599476..7a16e5ea797 100644 --- a/CromIAM/src/main/scala/cromiam/webservice/CromIamApiService.scala +++ b/CromIAM/src/main/scala/cromiam/webservice/CromIamApiService.scala @@ -81,7 +81,7 @@ trait CromIamApiService extends RequestSupport def abortRoute: Route = path("api" / "workflows" / Segment / Segment / Abort) { (_, workflowId) => post { - extractUserAndRequest { (user, req) => + extractUserAndStrictRequest { (user, req) => logUserWorkflowAction(user, workflowId, Abort) complete { authorizeAbortThenForwardToCromwell(user, workflowId, req).asHttpResponse @@ -93,7 +93,7 @@ trait CromIamApiService extends RequestSupport //noinspection MutatorLikeMethodIsParameterless def releaseHoldRoute: Route = path("api" / "workflows" / Segment / Segment / ReleaseHold) { (_, workflowId) => post { - extractUserAndRequest { (user, req) => + extractUserAndStrictRequest { (user, req) => logUserWorkflowAction(user, workflowId, ReleaseHold) complete { authorizeUpdateThenForwardToCromwell(user, workflowId, req).asHttpResponse @@ -112,7 +112,7 @@ trait CromIamApiService extends RequestSupport def labelPatchRoute: Route = { path("api" / "workflows" / Segment / Segment / Labels) { (_, workflowId) => patch { - extractUserAndRequest { (user, req) => + extractUserAndStrictRequest { (user, req) => entity(as[String]) { labels => logUserWorkflowAction(user, workflowId, Labels) validateLabels(Option(labels)) { _ => // Not using the labels, just using this to verify they didn't specify labels we don't want them to @@ -130,7 +130,7 @@ trait CromIamApiService extends RequestSupport def callCacheDiffRoute: Route = path("api" / "workflows" / Segment / "callcaching" / "diff") { _ => get { - extractUserAndRequest { (user, req) => + extractUserAndStrictRequest { (user, req) => logUserAction(user, "call caching diff") parameterSeq { parameters => val paramMap = parameters.toMap @@ -150,11 +150,9 @@ trait CromIamApiService extends RequestSupport */ private def workflowGetRoute(urlSuffix: String): Route = path("api" / "workflows" / Segment / urlSuffix) { _ => get { - extractUserAndRequest { (user, req) => + extractUserAndStrictRequest { (user, req) => logUserAction(user, urlSuffix) - complete { - cromwellClient.forwardToCromwell(req).asHttpResponse - } + forwardIfUserEnabled(user, req, cromwellClient, samClient) } } } @@ -166,7 +164,7 @@ trait CromIamApiService extends RequestSupport private def workflowRoute(urlSuffix: String, method: Directive0): Route = path("api" / "workflows" / Segment / Segment / urlSuffix) { (_, workflowId) => method { - extractUserAndRequest { (user, req) => + extractUserAndStrictRequest { (user, req) => logUserWorkflowAction(user, workflowId, urlSuffix) complete { authorizeReadThenForwardToCromwell(user, List(workflowId), req).asHttpResponse diff --git a/CromIAM/src/main/scala/cromiam/webservice/QuerySupport.scala b/CromIAM/src/main/scala/cromiam/webservice/QuerySupport.scala index cddabe74a57..e9397605c6a 100644 --- a/CromIAM/src/main/scala/cromiam/webservice/QuerySupport.scala +++ b/CromIAM/src/main/scala/cromiam/webservice/QuerySupport.scala @@ -55,7 +55,7 @@ trait QuerySupport extends RequestSupport { * directive */ private def preprocessQuery: Directive[(User, List[Collection], HttpRequest)] = { - extractUserAndRequest tflatMap { case (user, cromIamRequest) => + extractUserAndStrictRequest tflatMap { case (user, cromIamRequest) => log.info("Received query " + cromIamRequest.method.value + " request for user " + user.userId) onComplete(samClient.collectionsForUser(user, cromIamRequest).value.unsafeToFuture()) flatMap { diff --git a/CromIAM/src/main/scala/cromiam/webservice/RequestSupport.scala b/CromIAM/src/main/scala/cromiam/webservice/RequestSupport.scala index 12b231485f7..c9b6a196368 100644 --- a/CromIAM/src/main/scala/cromiam/webservice/RequestSupport.scala +++ b/CromIAM/src/main/scala/cromiam/webservice/RequestSupport.scala @@ -6,6 +6,13 @@ import akka.http.scaladsl.server.Directives._ import akka.http.scaladsl.server._ import cromiam.auth.User import org.broadinstitute.dsde.workbench.model.WorkbenchUserId +import akka.http.scaladsl.model.HttpResponse +import akka.http.scaladsl.server.Directives.{authorize, complete, onComplete} +import akka.http.scaladsl.server.Route +import cromiam.cromwell.CromwellClient +import cromiam.sam.SamClient + +import scala.util.{Failure, Success} trait RequestSupport { def extractStrictRequest: Directive1[HttpRequest] = { @@ -26,10 +33,27 @@ trait RequestSupport { } } - def extractUserAndRequest: Directive[(User, HttpRequest)] = { + def extractUserAndStrictRequest: Directive[(User, HttpRequest)] = { for { user <- extractUser request <- extractStrictRequest } yield (user, request) } + + def forwardIfUserEnabled(user: User, req: HttpRequest, cromwellClient: CromwellClient, samClient: SamClient): Route = { + import cromwell.api.model.EnhancedFailureResponseOrHttpResponseT + + onComplete(samClient.isUserEnabledSam(user, req).value.unsafeToFuture()) { + case Success(Left(httpResponse: HttpResponse)) => complete(httpResponse) + case Success(Right(isEnabled: Boolean)) => + authorize(isEnabled) { + complete { + cromwellClient.forwardToCromwell(req).asHttpResponse + } + } + case Failure(e) => + val message = s"Unable to look up enablement status for user ${user.userId}: ${e.getMessage}. Please try again later." + throw new RuntimeException(message, e) + } + } } diff --git a/CromIAM/src/main/scala/cromiam/webservice/SubmissionSupport.scala b/CromIAM/src/main/scala/cromiam/webservice/SubmissionSupport.scala index c1f5477475b..52a05d1cdc7 100644 --- a/CromIAM/src/main/scala/cromiam/webservice/SubmissionSupport.scala +++ b/CromIAM/src/main/scala/cromiam/webservice/SubmissionSupport.scala @@ -31,7 +31,7 @@ trait SubmissionSupport extends RequestSupport { // FIXME - getting pathPrefix to shrink this keeps hosing up, there's gotta be some way to do this def submitRoute: Route = (path("api" / "workflows" / Segment) | path("api" / "workflows" / Segment / "batch")) { _ => post { - extractUserAndRequest { (user, request) => + extractUserAndStrictRequest { (user, request) => log.info("Received submission request from user " + user.userId) onComplete(samClient.isSubmitWhitelisted(user, request).value.unsafeToFuture()) { case Success(Left(httpResponse)) => complete(httpResponse) diff --git a/CromIAM/src/main/scala/cromiam/webservice/WomtoolRouteSupport.scala b/CromIAM/src/main/scala/cromiam/webservice/WomtoolRouteSupport.scala index 671d4a76543..a6098b1fae0 100644 --- a/CromIAM/src/main/scala/cromiam/webservice/WomtoolRouteSupport.scala +++ b/CromIAM/src/main/scala/cromiam/webservice/WomtoolRouteSupport.scala @@ -2,20 +2,18 @@ package cromiam.webservice import akka.http.scaladsl.server.Directives._ import cromiam.cromwell.CromwellClient -import cromwell.api.model._ +import cromiam.sam.SamClient trait WomtoolRouteSupport extends RequestSupport { // When this trait is mixed into `CromIamApiService` the value of `cromwellClient` is the reader (non-abort) address val cromwellClient: CromwellClient + val samClient: SamClient val womtoolRoutes = path("api" / "womtool" / Segment / "describe") { _ => post { - extractStrictRequest { req => - complete { - // This endpoint requires authn which it gets for free from the proxy, does not care about authz - cromwellClient.forwardToCromwell(req).asHttpResponse - } + extractUserAndStrictRequest { (user, req) => + forwardIfUserEnabled(user, req, cromwellClient, samClient) } } } diff --git a/CromIAM/src/test/scala/cromiam/webservice/CromIamApiServiceSpec.scala b/CromIAM/src/test/scala/cromiam/webservice/CromIamApiServiceSpec.scala index 5ad6a976204..89945c5bfcb 100644 --- a/CromIAM/src/test/scala/cromiam/webservice/CromIamApiServiceSpec.scala +++ b/CromIAM/src/test/scala/cromiam/webservice/CromIamApiServiceSpec.scala @@ -4,7 +4,8 @@ import akka.event.NoLogging import akka.http.scaladsl.model.StatusCodes._ import akka.http.scaladsl.model.headers.{Authorization, OAuth2BearerToken, RawHeader} import akka.http.scaladsl.model.{ContentTypes, HttpEntity, HttpHeader} -import akka.http.scaladsl.server.MissingHeaderRejection +import akka.http.scaladsl.server.Route.seal +import akka.http.scaladsl.server.{AuthorizationFailedRejection, MissingHeaderRejection} import akka.http.scaladsl.testkit.ScalatestRouteTest import com.typesafe.config.Config import common.assertion.CromwellTimeoutSpec @@ -303,12 +304,45 @@ class CromIamApiServiceSpec extends AnyFlatSpec with CromwellTimeoutSpec with Ma } } - it should "reject request if it doesn't contain OIDC_CLAIM_user_id in header" in { + it should "reject request if it doesn't contain OIDC_CLAIM_user_id or token" in { Get(s"/api/workflows/$version/backends") ~> allRoutes ~> check { rejection shouldEqual MissingHeaderRejection("OIDC_CLAIM_user_id") } } + it should "return 403 when we request with a disabled user" in { + Get( + s"/api/workflows/$version/backends" + ).withHeaders( + List(Authorization(OAuth2BearerToken("my-token")), RawHeader("OIDC_CLAIM_user_id", "disabled@example.com")) + ) ~> allRoutes ~> check { + rejection shouldEqual AuthorizationFailedRejection + } + } + + it should "reject request if it contains a token and no OIDC_CLAIM_user_id in header" in { + Get( + s"/api/workflows/$version/backends" + ).withHeaders( + List(Authorization(OAuth2BearerToken("my-token"))) + ) ~> allRoutes ~> check { + rejection shouldEqual MissingHeaderRejection("OIDC_CLAIM_user_id") + } + } + + it should "return 404 when no auth token provided" in { + Get( + s"/api/workflows/$version/backends" + ).withHeaders( + List(RawHeader("OIDC_CLAIM_user_id", "enabled@example.com")) + // "[An] explicit call on the Route.seal method is needed in test code, but in your application code it is not necessary." + // https://doc.akka.io/docs/akka-http/current/routing-dsl/testkit.html#testing-sealed-routes + // https://doc.akka.io/docs/akka-http/current/routing-dsl/routes.html#sealing-a-route + ) ~> seal(allRoutes) ~> check { + responseAs[String] shouldEqual "The requested resource could not be found." + status shouldBe NotFound + } + } behavior of "ReleaseHold endpoint" it should "return 200 for authorized user who has collection associated with root workflow" in { diff --git a/CromIAM/src/test/scala/cromiam/webservice/MockClients.scala b/CromIAM/src/test/scala/cromiam/webservice/MockClients.scala index 3fb9689e5cc..59e75347159 100644 --- a/CromIAM/src/test/scala/cromiam/webservice/MockClients.scala +++ b/CromIAM/src/test/scala/cromiam/webservice/MockClients.scala @@ -141,6 +141,15 @@ class MockSamClient(checkSubmitWhitelist: Boolean = true) FailureResponseOrT.pure(!user.userId.value.equalsIgnoreCase(NotWhitelistedUser)) } + override def isUserEnabledSam(user: User, cromIamRequest: HttpRequest): FailureResponseOrT[Boolean] = { + if (user.userId.value == "enabled@example.com" || user.userId.value == MockSamClient.AuthorizedUserCollectionStr) + FailureResponseOrT.pure(true) + else if (user.userId.value == "disabled@example.com") + FailureResponseOrT.pure(false) + else + throw new Exception("Misconfigured test") + } + override def requestAuth(authorizationRequest: CollectionAuthorizationRequest, cromIamRequest: HttpRequest): FailureResponseOrT[Unit] = { authorizationRequest.user.userId.value match { diff --git a/CromIAM/src/test/scala/cromiam/webservice/SwaggerServiceSpec.scala b/CromIAM/src/test/scala/cromiam/webservice/SwaggerServiceSpec.scala index c0277fd92c0..838a523f4eb 100644 --- a/CromIAM/src/test/scala/cromiam/webservice/SwaggerServiceSpec.scala +++ b/CromIAM/src/test/scala/cromiam/webservice/SwaggerServiceSpec.scala @@ -12,7 +12,7 @@ import org.scalatest.prop.TableDrivenPropertyChecks import org.yaml.snakeyaml.constructor.Constructor import org.yaml.snakeyaml.error.YAMLException import org.yaml.snakeyaml.nodes.MappingNode -import org.yaml.snakeyaml.{Yaml => SnakeYaml} +import org.yaml.snakeyaml.{LoaderOptions, Yaml => SnakeYaml} import scala.jdk.CollectionConverters._ @@ -21,6 +21,7 @@ class SwaggerServiceSpec extends AnyFlatSpec with CromwellTimeoutSpec with Swagg with TableDrivenPropertyChecks { def actorRefFactory = system override def oauthConfig: SwaggerOauthConfig = SwaggerOauthConfig("clientId", "realm", "appName") + val yamlLoaderOptions = new LoaderOptions behavior of "SwaggerService" @@ -32,7 +33,7 @@ class SwaggerServiceSpec extends AnyFlatSpec with CromwellTimeoutSpec with Swagg contentType should be(ContentTypes.`application/octet-stream`) val body = responseAs[String] - val yaml = new SnakeYaml(new UniqueKeyConstructor()).loadAs(body, classOf[java.util.Map[String, AnyRef]]) + val yaml = new SnakeYaml(new UniqueKeyConstructor(new LoaderOptions)).loadAs(body, classOf[java.util.Map[String, AnyRef]]) yaml.get("swagger") should be("2.0") } @@ -109,7 +110,7 @@ class SwaggerServiceSpec extends AnyFlatSpec with CromwellTimeoutSpec with Swagg * Adapted from: * https://bitbucket.org/asomov/snakeyaml/src/e9cd9f5e8d76c61eb983e29b3dc039c1fac9c393/src/test/java/org/yaml/snakeyaml/issues/issue139/UniqueKeyTest.java?fileviewer=file-view-default#UniqueKeyTest.java-43:62 */ -class UniqueKeyConstructor extends Constructor { +class UniqueKeyConstructor(val loaderOptions: LoaderOptions) extends Constructor(loaderOptions) { import java.util.{Map => JMap} diff --git a/CromIAM/src/test/scala/cromiam/webservice/WomtoolRouteSupportSpec.scala b/CromIAM/src/test/scala/cromiam/webservice/WomtoolRouteSupportSpec.scala index 7ba1c495f23..785c887c374 100644 --- a/CromIAM/src/test/scala/cromiam/webservice/WomtoolRouteSupportSpec.scala +++ b/CromIAM/src/test/scala/cromiam/webservice/WomtoolRouteSupportSpec.scala @@ -2,6 +2,9 @@ package cromiam.webservice import akka.http.scaladsl.model.ContentTypes import akka.http.scaladsl.model.StatusCodes._ +import akka.http.scaladsl.model.headers.{Authorization, OAuth2BearerToken, RawHeader} +import akka.http.scaladsl.server.Route.seal +import akka.http.scaladsl.server.{AuthorizationFailedRejection, MissingHeaderRejection} import akka.http.scaladsl.testkit.ScalatestRouteTest import common.assertion.CromwellTimeoutSpec import org.scalatest.flatspec.AnyFlatSpec @@ -11,15 +14,64 @@ import org.scalatest.matchers.should.Matchers class WomtoolRouteSupportSpec extends AnyFlatSpec with CromwellTimeoutSpec with Matchers with WomtoolRouteSupport with ScalatestRouteTest { override lazy val cromwellClient = new MockCromwellClient() + override lazy val samClient = new MockSamClient() behavior of "Womtool endpoint routes" it should "return 200 when we request to the right path" in { - Post(s"/api/womtool/v1/describe") ~> womtoolRoutes ~> check { + Post( + s"/api/womtool/v1/describe" + ).withHeaders( + List(Authorization(OAuth2BearerToken("my-token")), RawHeader("OIDC_CLAIM_user_id", "enabled@example.com")) + ) ~> womtoolRoutes ~> check { status shouldBe OK responseAs[String] shouldBe "Hey there, workflow describer" contentType should be(ContentTypes.`text/plain(UTF-8)`) } } + it should "return 403 when we request with a disabled user" in { + Post( + s"/api/womtool/v1/describe" + ).withHeaders( + List(Authorization(OAuth2BearerToken("my-token")), RawHeader("OIDC_CLAIM_user_id", "disabled@example.com")) + ) ~> womtoolRoutes ~> check { + rejection shouldEqual AuthorizationFailedRejection + } + } + + it should "bail out with no user ID" in { + Post( + s"/api/womtool/v1/describe" + ).withHeaders( + List(Authorization(OAuth2BearerToken("my-token"))) + ) ~> womtoolRoutes ~> check { + rejection shouldEqual MissingHeaderRejection("OIDC_CLAIM_user_id") + } + } + + it should "return 404 when no auth token provided" in { + Post( + s"/api/womtool/v1/describe" + ).withHeaders( + List(RawHeader("OIDC_CLAIM_user_id", "enabled@example.com")) + // "[An] explicit call on the Route.seal method is needed in test code, but in your application code it is not necessary." + // https://doc.akka.io/docs/akka-http/current/routing-dsl/testkit.html#testing-sealed-routes + // https://doc.akka.io/docs/akka-http/current/routing-dsl/routes.html#sealing-a-route + ) ~> seal(womtoolRoutes) ~> check { + responseAs[String] shouldEqual "The requested resource could not be found." + status shouldBe NotFound + } + } + + it should "bail out with no headers" in { + Post( + s"/api/womtool/v1/describe" + ).withHeaders( + List.empty + ) ~> womtoolRoutes ~> check { + rejection shouldEqual MissingHeaderRejection("OIDC_CLAIM_user_id") + } + } + } diff --git a/CromwellRefdiskManifestCreator/pom.xml b/CromwellRefdiskManifestCreator/pom.xml index 987aad421be..829f64b5fd3 100644 --- a/CromwellRefdiskManifestCreator/pom.xml +++ b/CromwellRefdiskManifestCreator/pom.xml @@ -48,7 +48,7 @@ com.fasterxml.jackson.core jackson-databind - 2.13.2.2 + 2.13.4.1 org.apache.logging.log4j diff --git a/backend/src/main/scala/cromwell/backend/standard/StandardAsyncExecutionActor.scala b/backend/src/main/scala/cromwell/backend/standard/StandardAsyncExecutionActor.scala index 5c415cc3c50..02c2a2577a2 100644 --- a/backend/src/main/scala/cromwell/backend/standard/StandardAsyncExecutionActor.scala +++ b/backend/src/main/scala/cromwell/backend/standard/StandardAsyncExecutionActor.scala @@ -244,6 +244,8 @@ trait StandardAsyncExecutionActor } } + lazy val memoryRetryRequested: Boolean = memoryRetryFactor.nonEmpty + /** * Returns the shell scripting for finding all files listed within a directory. * @@ -1265,8 +1267,8 @@ trait StandardAsyncExecutionActor */ def handleExecutionResult(status: StandardAsyncRunState, oldHandle: StandardAsyncPendingExecutionHandle): Future[ExecutionHandle] = { - - // get the memory retry code. + + // Returns true if the task has written an RC file that indicates OOM, false otherwise def memoryRetryRC: Future[Boolean] = { // convert int to boolean def returnCodeAsBoolean(codeAsOption: Option[String]): Boolean = { @@ -1329,11 +1331,11 @@ trait StandardAsyncExecutionActor // Only check stderr size if we need to, otherwise this results in a lot of unnecessary I/O that // may fail due to race conditions on quickly-executing jobs. stderrSize <- if (failOnStdErr) asyncIo.sizeAsync(stderr) else Future.successful(0L) - retryWithMoreMemory <- memoryRetryRC - } yield (stderrSize, returnCodeAsString, retryWithMoreMemory) + outOfMemoryDetected <- memoryRetryRC + } yield (stderrSize, returnCodeAsString, outOfMemoryDetected) stderrSizeAndReturnCodeAndMemoryRetry flatMap { - case (stderrSize, returnCodeAsString, retryWithMoreMemory) => + case (stderrSize, returnCodeAsString, outOfMemoryDetected) => val tryReturnCodeAsInt = Try(returnCodeAsString.trim.toInt) if (isDone(status)) { @@ -1341,13 +1343,15 @@ trait StandardAsyncExecutionActor case Success(returnCodeAsInt) if failOnStdErr && stderrSize.intValue > 0 => val executionHandle = Future.successful(FailedNonRetryableExecutionHandle(StderrNonEmpty(jobDescriptor.key.tag, stderrSize, stderrAsOption), Option(returnCodeAsInt), None)) retryElseFail(executionHandle) - case Success(returnCodeAsInt) if isAbort(returnCodeAsInt) => - Future.successful(AbortedExecutionHandle) case Success(returnCodeAsInt) if continueOnReturnCode.continueFor(returnCodeAsInt) => handleExecutionSuccess(status, oldHandle, returnCodeAsInt) - case Success(returnCodeAsInt) if retryWithMoreMemory => + // It's important that we check retryWithMoreMemory case before isAbort. RC could be 137 in either case; + // if it was caused by OOM killer, want to handle as OOM and not job abort. + case Success(returnCodeAsInt) if outOfMemoryDetected && memoryRetryRequested => val executionHandle = Future.successful(FailedNonRetryableExecutionHandle(RetryWithMoreMemory(jobDescriptor.key.tag, stderrAsOption, memoryRetryErrorKeys, log), Option(returnCodeAsInt), None)) - retryElseFail(executionHandle, retryWithMoreMemory) + retryElseFail(executionHandle, outOfMemoryDetected) + case Success(returnCodeAsInt) if isAbort(returnCodeAsInt) => + Future.successful(AbortedExecutionHandle) case Success(returnCodeAsInt) => val executionHandle = Future.successful(FailedNonRetryableExecutionHandle(WrongReturnCode(jobDescriptor.key.tag, returnCodeAsInt, stderrAsOption), Option(returnCodeAsInt), None)) retryElseFail(executionHandle) @@ -1356,9 +1360,9 @@ trait StandardAsyncExecutionActor } } else { tryReturnCodeAsInt match { - case Success(returnCodeAsInt) if retryWithMoreMemory && !continueOnReturnCode.continueFor(returnCodeAsInt) => + case Success(returnCodeAsInt) if outOfMemoryDetected && memoryRetryRequested && !continueOnReturnCode.continueFor(returnCodeAsInt) => val executionHandle = Future.successful(FailedNonRetryableExecutionHandle(RetryWithMoreMemory(jobDescriptor.key.tag, stderrAsOption, memoryRetryErrorKeys, log), Option(returnCodeAsInt), None)) - retryElseFail(executionHandle, retryWithMoreMemory) + retryElseFail(executionHandle, outOfMemoryDetected) case _ => val failureStatus = handleExecutionFailure(status, tryReturnCodeAsInt.toOption) retryElseFail(failureStatus) diff --git a/backend/src/main/scala/cromwell/backend/standard/StandardCachingActorHelper.scala b/backend/src/main/scala/cromwell/backend/standard/StandardCachingActorHelper.scala index dd4254c8439..a64c6a5439c 100644 --- a/backend/src/main/scala/cromwell/backend/standard/StandardCachingActorHelper.scala +++ b/backend/src/main/scala/cromwell/backend/standard/StandardCachingActorHelper.scala @@ -78,7 +78,7 @@ trait StandardCachingActorHelper extends JobCachingActorHelper { val fileMetadata = jobPaths.metadataPaths - runtimeAttributesMetadata ++ fileMetadata ++ nonStandardMetadata + nonStandardMetadata ++ runtimeAttributesMetadata ++ fileMetadata } /** diff --git a/backend/src/main/scala/cromwell/backend/standard/callcaching/RootWorkflowFileHashCacheActor.scala b/backend/src/main/scala/cromwell/backend/standard/callcaching/RootWorkflowFileHashCacheActor.scala index e33625741b3..c3d1452660e 100644 --- a/backend/src/main/scala/cromwell/backend/standard/callcaching/RootWorkflowFileHashCacheActor.scala +++ b/backend/src/main/scala/cromwell/backend/standard/callcaching/RootWorkflowFileHashCacheActor.scala @@ -48,7 +48,7 @@ class RootWorkflowFileHashCacheActor private[callcaching](override val ioActor: // hash to become available. cache.put(key, FileHashValueRequested(requesters = requester :: requesters)) case FileHashSuccess(value) => - sender() ! Tuple2(hashCommand.fileHashContext, IoSuccess(requester.ioCommand, value)) + sender() ! Tuple2[Any, Any](hashCommand.fileHashContext, IoSuccess(requester.ioCommand, value)) case FileHashFailure(error) => sender() ! Tuple2(hashCommand.fileHashContext, IoFailure(requester.ioCommand, new IOException(error))) } diff --git a/backend/src/main/scala/cromwell/backend/standard/callcaching/StandardCacheHitCopyingActor.scala b/backend/src/main/scala/cromwell/backend/standard/callcaching/StandardCacheHitCopyingActor.scala index f1662db858b..e23ecd277af 100644 --- a/backend/src/main/scala/cromwell/backend/standard/callcaching/StandardCacheHitCopyingActor.scala +++ b/backend/src/main/scala/cromwell/backend/standard/callcaching/StandardCacheHitCopyingActor.scala @@ -228,6 +228,8 @@ abstract class StandardCacheHitCopyingActor(val standardParams: StandardCacheHit data.commandsToWaitFor.flatten.headOption match { case Some(command: IoCopyCommand) => logCacheHitCopyCommand(command) + case Some(command: IoTouchCommand) => + logCacheHitTouchCommand(command) case huh => log.warning(s"BT-322 {} unexpected commandsToWaitFor: {}", jobTag, huh) } @@ -307,6 +309,9 @@ abstract class StandardCacheHitCopyingActor(val standardParams: StandardCacheHit case _ => } + private def logCacheHitTouchCommand(command: IoTouchCommand): Unit = + log.info(s"BT-322 {} cache hit for file : {}", jobTag, command.toString) + def succeedAndStop(returnCode: Option[Int], copiedJobOutputs: CallOutputs, detritusMap: DetritusMap): State = { import cromwell.services.metadata.MetadataService.implicits.MetadataAutoPutter serviceRegistryActor.putMetadata(jobDescriptor.workflowDescriptor.id, Option(jobDescriptor.key), startMetadataKeyValues) diff --git a/build.sbt b/build.sbt index 3b255d0e0e5..76f6f27e2c9 100644 --- a/build.sbt +++ b/build.sbt @@ -74,12 +74,6 @@ lazy val wdlTransformsBiscayne = (project in wdlTransformsRoot / "biscayne") .dependsOn(common % "test->test") .dependsOn(wom % "test->test") -lazy val cwl = project - .withLibrarySettings("cromwell-cwl", cwlDependencies) - .dependsOn(wom) - .dependsOn(wom % "test->test") - .dependsOn(common % "test->test") - lazy val core = project .withLibrarySettings("cromwell-core", coreDependencies) .dependsOn(wom) @@ -183,7 +177,6 @@ lazy val services = project .dependsOn(wdlDraft2LanguageFactory % "test->test") // because the WaaS tests init language config with all languages .dependsOn(wdlDraft3LanguageFactory % "test->test") .dependsOn(wdlBiscayneLanguageFactory % "test->test") - .dependsOn(cwlV1_0LanguageFactory % "test->test") .dependsOn(core % "test->test") .dependsOn(ftpFileSystem % "test->test") .dependsOn(common % "test->test") @@ -245,6 +238,9 @@ lazy val tesBackend = (project in backendRoot / "tes") .dependsOn(sfsBackend) .dependsOn(ftpFileSystem) .dependsOn(drsFileSystem) + .dependsOn(azureBlobFileSystem) + // TES backend provides a compatibility layer to run WDLs with PAPI runtime attributes [WX-769] + .dependsOn(googlePipelinesCommon) .dependsOn(backend % "test->test") .dependsOn(common % "test->test") @@ -263,7 +259,6 @@ lazy val engine = project .dependsOn(azureBlobFileSystem % "test->test") .dependsOn(`cloud-nio-spi`) .dependsOn(languageFactoryCore) - .dependsOn(cwlV1_0LanguageFactory % "test->test") .dependsOn(wdlDraft2LanguageFactory % "test->test") .dependsOn(wdlDraft3LanguageFactory % "test->test") .dependsOn(wdlBiscayneLanguageFactory % "test->test") @@ -278,20 +273,11 @@ lazy val engine = project // Executables -lazy val centaurCwlRunner = project - .withExecutableSettings("centaur-cwl-runner", centaurCwlRunnerDependencies, buildDocker = false) - .dependsOn(cwl) - .dependsOn(centaur) - .dependsOn(gcsFileSystem) - .dependsOn(ftpFileSystem) - .dependsOn(common % "test->test") - lazy val womtool = project .withExecutableSettings("womtool", womtoolDependencies) .dependsOn(wdlDraft2LanguageFactory) .dependsOn(wdlDraft3LanguageFactory) .dependsOn(wdlBiscayneLanguageFactory) - .dependsOn(cwlV1_0LanguageFactory) .dependsOn(wom % "test->test") .dependsOn(common % "test->test") @@ -337,12 +323,6 @@ lazy val wdlBiscayneLanguageFactory = (project in languageFactoryRoot / "wdl-bis .dependsOn(wdlTransformsBiscayne) .dependsOn(common % "test->test") -lazy val cwlV1_0LanguageFactory = (project in languageFactoryRoot / "cwl-v1-0") - .withLibrarySettings("cwl-v1-0") - .dependsOn(languageFactoryCore) - .dependsOn(cwl) - .dependsOn(common % "test->test") - lazy val `cloud-nio-spi` = (project in cloudNio / "cloud-nio-spi") .withLibrarySettings(libraryName = "cloud-nio-spi", dependencies = spiDependencies) .dependsOn(common % "test->test") @@ -385,7 +365,6 @@ lazy val server = project .dependsOn(wdlDraft2LanguageFactory) .dependsOn(wdlDraft3LanguageFactory) .dependsOn(wdlBiscayneLanguageFactory) - .dependsOn(cwlV1_0LanguageFactory) .dependsOn(engine % "test->test") .dependsOn(common % "test->test") @@ -402,14 +381,11 @@ lazy val root = (project in file(".")) .aggregate(azureBlobFileSystem) .aggregate(backend) .aggregate(centaur) - .aggregate(centaurCwlRunner) .aggregate(cloudSupport) .aggregate(common) .aggregate(core) .aggregate(cromiam) .aggregate(cromwellApiClient) - .aggregate(cwl) - .aggregate(cwlV1_0LanguageFactory) .aggregate(databaseMigration) .aggregate(databaseSql) .aggregate(dockerHashing) diff --git a/centaur/src/it/scala/centaur/reporting/BigQueryReporter.scala b/centaur/src/it/scala/centaur/reporting/BigQueryReporter.scala index 16463742632..9a88a9ac109 100644 --- a/centaur/src/it/scala/centaur/reporting/BigQueryReporter.scala +++ b/centaur/src/it/scala/centaur/reporting/BigQueryReporter.scala @@ -189,7 +189,7 @@ class BigQueryReporter(override val params: ErrorReporterParams) extends ErrorRe } private def toJobKeyValueRow(jobKeyValueEntry: JobKeyValueEntry): RowToInsert = { - RowToInsert of Map( + RowToInsert of Map[String, Any]( "call_fully_qualified_name" -> jobKeyValueEntry.callFullyQualifiedName, "job_attempt" -> jobKeyValueEntry.jobAttempt, "job_index" -> jobKeyValueEntry.jobIndex, diff --git a/centaur/src/main/resources/integrationTestCases/bcbio_joint_gvcf.test b/centaur/src/main/resources/integrationTestCases/bcbio_joint_gvcf.test deleted file mode 100644 index ef32cbc7d7c..00000000000 --- a/centaur/src/main/resources/integrationTestCases/bcbio_joint_gvcf.test +++ /dev/null @@ -1,23 +0,0 @@ -name: bcbio_joint_gvcf -testFormat: workflowsuccess -backends: [Papiv2] -workflowType: CWL -workflowTypeVersion: v1.0 -tags: [bcbio] - - -files { - workflow: cwl/bcbio/gvcf-joint-workflow/main-gvcf-joint.cwl - inputs: cwl/bcbio/gvcf-joint-workflow/main-gvcf-joint-samples.json - imports: [ - cwl/bcbio/gvcf-joint-workflow/steps, - cwl/bcbio/gvcf-joint-workflow/wf-alignment.cwl, - cwl/bcbio/gvcf-joint-workflow/wf-jointcall.cwl, - cwl/bcbio/gvcf-joint-workflow/wf-variantcall.cwl - ] - options: cwl/bcbio/bcbio.options -} - -metadata { - status: Succeeded -} diff --git a/centaur/src/main/resources/integrationTestCases/bcbio_prealign.test b/centaur/src/main/resources/integrationTestCases/bcbio_prealign.test deleted file mode 100644 index 2c4a6db3f45..00000000000 --- a/centaur/src/main/resources/integrationTestCases/bcbio_prealign.test +++ /dev/null @@ -1,20 +0,0 @@ -name: bcbio_prealign -testFormat: workflowsuccess -backends: [Papiv2] -workflowType: CWL -workflowTypeVersion: v1.0 -tags: [bcbio] - -files { - workflow: cwl/bcbio/prealign-workflow/main-prealign.cwl - inputs: cwl/bcbio/prealign-workflow/main-prealign-samples.json - imports: [ - cwl/bcbio/prealign-workflow/steps, - cwl/bcbio/prealign-workflow/wf-variantcall.cwl - ] - options: cwl/bcbio/bcbio.options -} - -metadata { - status: Succeeded -} diff --git a/centaur/src/main/resources/integrationTestCases/bcbio_rnaseq.test b/centaur/src/main/resources/integrationTestCases/bcbio_rnaseq.test deleted file mode 100644 index 64ac876c568..00000000000 --- a/centaur/src/main/resources/integrationTestCases/bcbio_rnaseq.test +++ /dev/null @@ -1,19 +0,0 @@ -name: bcbio_rnaseq -testFormat: workflowsuccess -backends: [Papiv2] -workflowType: CWL -workflowTypeVersion: v1.0 -tags: [bcbio] - -files { - workflow: cwl/bcbio/rnaseq-workflow/main-rnaseq.cwl - inputs: cwl/bcbio/rnaseq-workflow/main-rnaseq-samples.json - imports: [ - cwl/bcbio/rnaseq-workflow/steps - ] - options: cwl/bcbio/bcbio.options -} - -metadata { - status: Succeeded -} diff --git a/centaur/src/main/resources/integrationTestCases/bcbio_somatic.test b/centaur/src/main/resources/integrationTestCases/bcbio_somatic.test deleted file mode 100644 index 440099f280e..00000000000 --- a/centaur/src/main/resources/integrationTestCases/bcbio_somatic.test +++ /dev/null @@ -1,22 +0,0 @@ -name: bcbio_somatic -testFormat: workflowsuccess -backends: [Papiv2] -workflowType: CWL -workflowTypeVersion: v1.0 -tags: [bcbio] - -files { - workflow: cwl/bcbio/somatic-workflow/main-somatic.cwl - inputs: cwl/bcbio/somatic-workflow/main-somatic-samples.json - imports: [ - cwl/bcbio/somatic-workflow/steps, - cwl/bcbio/somatic-workflow/wf-alignment.cwl, - cwl/bcbio/somatic-workflow/wf-svcall.cwl, - cwl/bcbio/somatic-workflow/wf-variantcall.cwl - ] - options: cwl/bcbio/bcbio.options -} - -metadata { - status: Succeeded -} diff --git a/centaur/src/main/resources/integrationTestCases/bcbio_svcall.test b/centaur/src/main/resources/integrationTestCases/bcbio_svcall.test deleted file mode 100644 index 9116814fe6d..00000000000 --- a/centaur/src/main/resources/integrationTestCases/bcbio_svcall.test +++ /dev/null @@ -1,22 +0,0 @@ -name: bcbio_svcall -testFormat: workflowsuccess -backends: [Papiv2] -workflowType: CWL -workflowTypeVersion: v1.0 -tags: [bcbio] - -files { - workflow: cwl/bcbio/svcall-workflow/main-svcall.cwl - inputs: cwl/bcbio/svcall-workflow/main-svcall-samples.json - imports: [ - cwl/bcbio/svcall-workflow/steps, - cwl/bcbio/svcall-workflow/wf-alignment.cwl, - cwl/bcbio/svcall-workflow/wf-svcall.cwl, - cwl/bcbio/svcall-workflow/wf-variantcall.cwl - ] - options: cwl/bcbio/bcbio.options -} - -metadata { - status: Succeeded -} diff --git a/centaur/src/main/resources/integrationTestCases/bcbio_wes-chr21-test-workflow.test b/centaur/src/main/resources/integrationTestCases/bcbio_wes-chr21-test-workflow.test deleted file mode 100644 index 1969d6bef66..00000000000 --- a/centaur/src/main/resources/integrationTestCases/bcbio_wes-chr21-test-workflow.test +++ /dev/null @@ -1,22 +0,0 @@ -name: wes_chr21_test-workflow -testFormat: workflowsuccess -backends: [Papiv2] -workflowType: CWL -workflowTypeVersion: v1.0 -tags: [bcbio] - -files { - workflow: cwl/bcbio/wes_chr21_test-workflow-gcp/main-wes_chr21_test.cwl - inputs: cwl/bcbio/wes_chr21_test-workflow-gcp/main-wes_chr21_test-samples.json - imports: [ - cwl/bcbio/wes_chr21_test-workflow-gcp/steps, - cwl/bcbio/wes_chr21_test-workflow-gcp/wf-alignment.cwl, - cwl/bcbio/wes_chr21_test-workflow-gcp/wf-svcall.cwl, - cwl/bcbio/wes_chr21_test-workflow-gcp/wf-variantcall.cwl - ] - options: cwl/bcbio/bcbio.options -} - -metadata { - status: Succeeded -} diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/bcbio.options b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/bcbio.options deleted file mode 100644 index 1340bda7f1c..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/bcbio.options +++ /dev/null @@ -1,5 +0,0 @@ -{ - "default_runtime_attributes": { - "bootDiskSizeGb": 20 - } -} diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/main-gvcf-joint-samples.json b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/main-gvcf-joint-samples.json deleted file mode 100644 index b7080e81448..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/main-gvcf-joint-samples.json +++ /dev/null @@ -1,615 +0,0 @@ -{ - "analysis": [ - "variant2", - "variant2" - ], - "config__algorithm__adapters": [ - [ - "polyx" - ], - [ - "polyx" - ] - ], - "config__algorithm__align_split_size": [ - null, - null - ], - "config__algorithm__aligner": [ - "bwa", - "bwa" - ], - "config__algorithm__bam_clean": [ - "False", - "False" - ], - "config__algorithm__coverage": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/automated/coverage_transcripts-bam.bed" - }, - null - ], - "config__algorithm__coverage_interval": [ - null, - null - ], - "config__algorithm__effects": [ - "snpeff", - "snpeff" - ], - "config__algorithm__ensemble": [ - null, - null - ], - "config__algorithm__exclude_regions": [ - [], - [] - ], - "config__algorithm__mark_duplicates": [ - "True", - "True" - ], - "config__algorithm__min_allele_fraction": [ - 10.0, - 10.0 - ], - "config__algorithm__nomap_split_size": [ - 250, - 250 - ], - "config__algorithm__nomap_split_targets": [ - 20, - 20 - ], - "config__algorithm__qc": [ - [ - "contamination", - "coverage", - "fastqc", - "peddy", - "picard", - "qsignature", - "samtools", - "variants" - ], - [ - "contamination", - "coverage", - "fastqc", - "peddy", - "picard", - "qsignature", - "samtools", - "variants" - ] - ], - "config__algorithm__recalibrate": [ - "True", - "True" - ], - "config__algorithm__tools_off": [ - [], - [] - ], - "config__algorithm__tools_on": [ - [ - "gvcf" - ], - [ - "gvcf" - ] - ], - "config__algorithm__trim_reads": [ - "read_through", - "False" - ], - "config__algorithm__validate": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/reference_material/7_100326_FC6107FAAXX-grade.vcf" - }, - null - ], - "config__algorithm__validate_regions": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed" - }, - null - ], - "config__algorithm__variant_regions": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed" - } - ], - "config__algorithm__variantcaller": [ - [ - "gatk-haplotype", - "strelka2" - ], - [ - "gatk-haplotype", - "strelka2" - ] - ], - "config__algorithm__vcfanno": [ - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/config/vcfanno/gemini.conf" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/config/vcfanno/gemini.lua" - } - ], - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/config/vcfanno/gemini.conf" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/config/vcfanno/gemini.lua" - } - ] - ], - "description": [ - "Test1", - "Test2" - ], - "files": [ - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam.bai" - } - ] - } - ], - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam" - } - ] - ], - "genome_build": [ - "hg19", - "hg19" - ], - "genome_resources__aliases__ensembl": [ - "homo_sapiens_vep_83_GRCh37", - "homo_sapiens_vep_83_GRCh37" - ], - "genome_resources__aliases__human": [ - "True", - "True" - ], - "genome_resources__aliases__snpeff": [ - "hg19", - "hg19" - ], - "genome_resources__rnaseq__gene_bed": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rnaseq/ref-transcripts.bed" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rnaseq/ref-transcripts.bed" - } - ], - "genome_resources__variation__1000g": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__clinvar": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__cosmic": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__dbsnp": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__encode_blacklist": [ - null, - null - ], - "genome_resources__variation__esp": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__exac": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__gnomad_exome": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__lcr": [ - null, - null - ], - "genome_resources__variation__polyx": [ - null, - null - ], - "genome_resources__variation__train_hapmap": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__train_indels": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz.tbi" - } - ] - } - ], - "metadata__batch": [ - "b1", - "b1" - ], - "metadata__phenotype": [ - "", - "" - ], - "reference__bwa__indexes": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.ann" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.bwt" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.pac" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.sa" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.ann" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.bwt" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.pac" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.sa" - } - ] - } - ], - "reference__fasta__base": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa.fai" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.dict" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa.fai" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.dict" - } - ] - } - ], - "reference__genome_context": [ - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz.tbi" - } - ] - } - ], - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz.tbi" - } - ] - } - ] - ], - "reference__rtg": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rtg--hg19.sdf-wf.tar.gz" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rtg--hg19.sdf-wf.tar.gz" - } - ], - "reference__snpeff__hg19": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/snpeff--hg19-wf.tar.gz" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/snpeff--hg19-wf.tar.gz" - } - ], - "reference__versions": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/versions.csv" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/versions.csv" - } - ], - "resources": [ - "{}", - "{}" - ], - "rgnames__lane": [ - "Test1", - "Test2" - ], - "rgnames__lb": [ - null, - null - ], - "rgnames__pl": [ - "illumina", - "illumina" - ], - "rgnames__pu": [ - "Test1", - "Test2" - ], - "rgnames__rg": [ - "Test1", - "Test2" - ], - "rgnames__sample": [ - "Test1", - "Test2" - ], - "vrn_file": [ - null, - null - ] -} diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/main-gvcf-joint.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/main-gvcf-joint.cwl deleted file mode 100644 index a798100dc12..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/main-gvcf-joint.cwl +++ /dev/null @@ -1,794 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: config__algorithm__align_split_size - type: - items: - - 'null' - - string - type: array -- id: files - type: - items: - items: File - type: array - type: array -- id: config__algorithm__trim_reads - type: - items: - - string - - 'null' - - boolean - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: config__algorithm__vcfanno - type: - items: - items: File - type: array - type: array -- id: resources - type: - items: string - type: array -- id: config__algorithm__variantcaller - type: - items: - items: string - type: array - type: array -- id: config__algorithm__adapters - type: - items: - items: string - type: array - type: array -- id: genome_resources__variation__1000g - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__coverage_interval - type: - items: - - 'null' - - string - type: array -- id: genome_resources__rnaseq__gene_bed - type: - items: File - type: array -- id: rgnames__lb - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__clinvar - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__esp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: rgnames__rg - type: - items: string - type: array -- id: metadata__batch - type: - items: string - type: array -- id: rgnames__lane - type: - items: string - type: array -- id: genome_resources__variation__lcr - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__min_allele_fraction - type: - items: double - type: array -- id: config__algorithm__nomap_split_targets - type: - items: long - type: array -- id: reference__versions - type: - items: File - type: array -- id: reference__bwa__indexes - secondaryFiles: - - ^.ann - - ^.pac - - ^.sa - - ^.bwt - type: - items: File - type: array -- id: vrn_file - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__train_hapmap - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: reference__genome_context - secondaryFiles: - - .tbi - type: - items: - items: File - type: array - type: array -- id: config__algorithm__bam_clean - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__nomap_split_size - type: - items: long - type: array -- id: config__algorithm__validate - type: - items: - - File - - 'null' - type: array -- id: reference__snpeff__hg19 - type: - items: File - type: array -- id: description - type: - items: string - type: array -- id: config__algorithm__validate_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__aligner - type: - items: string - type: array -- id: rgnames__pl - type: - items: string - type: array -- id: genome_build - type: - items: string - type: array -- id: rgnames__pu - type: - items: string - type: array -- id: genome_resources__variation__exac - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__gnomad_exome - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__recalibrate - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__coverage - type: - items: - - File - - 'null' - type: array -- id: genome_resources__aliases__human - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: genome_resources__variation__dbsnp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__mark_duplicates - type: - items: - - string - - 'null' - - boolean - type: array -- id: genome_resources__variation__polyx - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__encode_blacklist - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__cosmic - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__ensemble - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__qc - type: - items: - items: string - type: array - type: array -- id: analysis - type: - items: string - type: array -- id: rgnames__sample - type: - items: string - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: config__algorithm__effects - type: - items: string - type: array -- id: config__algorithm__variant_regions - type: - items: File - type: array -- id: genome_resources__aliases__ensembl - type: - items: string - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: reference__rtg - type: - items: File - type: array -- id: genome_resources__variation__train_indels - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__aliases__snpeff - type: - items: string - type: array -outputs: -- id: rgnames__sample_out - outputSource: prep_samples/rgnames__sample - type: - items: string - type: array -- id: align_bam - outputSource: postprocess_alignment/align_bam - type: - items: - - File - - 'null' - type: array -- id: regions__sample_callable - outputSource: postprocess_alignment/regions__sample_callable - type: - items: - - File - - 'null' - type: array -- id: validate__grading_summary - outputSource: summarize_vc/validate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: variants__calls - outputSource: summarize_vc/variants__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: variants__gvcf - outputSource: summarize_vc/variants__gvcf - type: - items: - - 'null' - - items: - - File - - 'null' - type: array - type: array -- id: summary__multiqc - outputSource: multiqc_summary/summary__multiqc - type: - items: - - File - - 'null' - type: array -- id: versions__tools - outputSource: multiqc_summary/versions__tools - type: - items: - - File - - 'null' - type: array -- id: versions__data - outputSource: multiqc_summary/versions__data - type: - items: - - File - - 'null' - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: alignment_to_rec - in: - - id: files - source: files - - id: analysis - source: analysis - - id: config__algorithm__align_split_size - source: config__algorithm__align_split_size - - id: reference__fasta__base - source: reference__fasta__base - - id: rgnames__pl - source: rgnames__pl - - id: rgnames__sample - source: rgnames__sample - - id: rgnames__pu - source: rgnames__pu - - id: rgnames__lane - source: rgnames__lane - - id: rgnames__rg - source: rgnames__rg - - id: rgnames__lb - source: rgnames__lb - - id: reference__bwa__indexes - source: reference__bwa__indexes - - id: config__algorithm__aligner - source: config__algorithm__aligner - - id: config__algorithm__trim_reads - source: config__algorithm__trim_reads - - id: config__algorithm__adapters - source: config__algorithm__adapters - - id: config__algorithm__bam_clean - source: config__algorithm__bam_clean - - id: config__algorithm__variant_regions - source: config__algorithm__variant_regions - - id: config__algorithm__mark_duplicates - source: config__algorithm__mark_duplicates - - id: resources - source: resources - - id: description - source: description - out: - - id: alignment_rec - run: steps/alignment_to_rec.cwl -- id: alignment - in: - - id: alignment_rec - source: alignment_to_rec/alignment_rec - out: - - id: align_bam - - id: work_bam_plus__disc - - id: work_bam_plus__sr - - id: hla__fastq - run: wf-alignment.cwl - scatter: - - alignment_rec - scatterMethod: dotproduct -- id: prep_samples_to_rec - in: - - id: config__algorithm__coverage - source: config__algorithm__coverage - - id: rgnames__sample - source: rgnames__sample - - id: config__algorithm__variant_regions - source: config__algorithm__variant_regions - - id: reference__fasta__base - source: reference__fasta__base - - id: resources - source: resources - - id: description - source: description - out: - - id: prep_samples_rec - run: steps/prep_samples_to_rec.cwl -- id: prep_samples - in: - - id: prep_samples_rec - source: prep_samples_to_rec/prep_samples_rec - out: - - id: rgnames__sample - - id: config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - - id: config__algorithm__variant_regions_orig - - id: config__algorithm__coverage - - id: config__algorithm__coverage_merged - - id: config__algorithm__coverage_orig - - id: config__algorithm__seq2c_bed_ready - run: steps/prep_samples.cwl - scatter: - - prep_samples_rec - scatterMethod: dotproduct -- id: postprocess_alignment_to_rec - in: - - id: align_bam - source: alignment/align_bam - - id: config__algorithm__coverage_interval - source: config__algorithm__coverage_interval - - id: config__algorithm__exclude_regions - source: config__algorithm__exclude_regions - - id: config__algorithm__variant_regions - source: prep_samples/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: prep_samples/config__algorithm__variant_regions_merged - - id: config__algorithm__variant_regions_orig - source: prep_samples/config__algorithm__variant_regions_orig - - id: config__algorithm__coverage - source: prep_samples/config__algorithm__coverage - - id: config__algorithm__coverage_merged - source: prep_samples/config__algorithm__coverage_merged - - id: config__algorithm__coverage_orig - source: prep_samples/config__algorithm__coverage_orig - - id: config__algorithm__seq2c_bed_ready - source: prep_samples/config__algorithm__seq2c_bed_ready - - id: config__algorithm__recalibrate - source: config__algorithm__recalibrate - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: genome_resources__rnaseq__gene_bed - source: genome_resources__rnaseq__gene_bed - - id: genome_resources__variation__dbsnp - source: genome_resources__variation__dbsnp - - id: genome_resources__variation__lcr - source: genome_resources__variation__lcr - - id: genome_resources__variation__polyx - source: genome_resources__variation__polyx - - id: genome_resources__variation__encode_blacklist - source: genome_resources__variation__encode_blacklist - - id: reference__fasta__base - source: reference__fasta__base - - id: resources - source: resources - - id: description - source: description - out: - - id: postprocess_alignment_rec - run: steps/postprocess_alignment_to_rec.cwl -- id: postprocess_alignment - in: - - id: postprocess_alignment_rec - source: postprocess_alignment_to_rec/postprocess_alignment_rec - out: - - id: config__algorithm__coverage_interval - - id: config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - - id: config__algorithm__variant_regions_orig - - id: config__algorithm__coverage - - id: config__algorithm__coverage_merged - - id: config__algorithm__coverage_orig - - id: config__algorithm__seq2c_bed_ready - - id: regions__callable - - id: regions__sample_callable - - id: regions__nblock - - id: depth__samtools__stats - - id: depth__samtools__idxstats - - id: depth__variant_regions__regions - - id: depth__variant_regions__dist - - id: depth__sv_regions__regions - - id: depth__sv_regions__dist - - id: depth__coverage__regions - - id: depth__coverage__dist - - id: depth__coverage__thresholds - - id: align_bam - run: steps/postprocess_alignment.cwl - scatter: - - postprocess_alignment_rec - scatterMethod: dotproduct -- id: combine_sample_regions - in: - - id: regions__callable - source: postprocess_alignment/regions__callable - - id: regions__nblock - source: postprocess_alignment/regions__nblock - - id: metadata__batch - source: metadata__batch - - id: config__algorithm__nomap_split_size - source: config__algorithm__nomap_split_size - - id: config__algorithm__nomap_split_targets - source: config__algorithm__nomap_split_targets - - id: reference__fasta__base - source: reference__fasta__base - - id: resources - source: resources - - id: description - source: description - out: - - id: config__algorithm__callable_regions - - id: config__algorithm__non_callable_regions - - id: config__algorithm__callable_count - run: steps/combine_sample_regions.cwl -- id: batch_for_variantcall - in: - - id: analysis - source: analysis - - id: genome_build - source: genome_build - - id: align_bam - source: postprocess_alignment/align_bam - - id: vrn_file - source: vrn_file - - id: metadata__batch - source: metadata__batch - - id: metadata__phenotype - source: metadata__phenotype - - id: config__algorithm__callable_regions - source: combine_sample_regions/config__algorithm__callable_regions - - id: regions__sample_callable - source: postprocess_alignment/regions__sample_callable - - id: config__algorithm__variantcaller - source: config__algorithm__variantcaller - - id: config__algorithm__ensemble - source: config__algorithm__ensemble - - id: config__algorithm__vcfanno - source: config__algorithm__vcfanno - - id: config__algorithm__coverage_interval - source: postprocess_alignment/config__algorithm__coverage_interval - - id: config__algorithm__effects - source: config__algorithm__effects - - id: config__algorithm__min_allele_fraction - source: config__algorithm__min_allele_fraction - - id: config__algorithm__exclude_regions - source: config__algorithm__exclude_regions - - id: config__algorithm__variant_regions - source: postprocess_alignment/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: postprocess_alignment/config__algorithm__variant_regions_merged - - id: config__algorithm__validate - source: config__algorithm__validate - - id: config__algorithm__validate_regions - source: config__algorithm__validate_regions - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: config__algorithm__tools_off - source: config__algorithm__tools_off - - id: reference__fasta__base - source: reference__fasta__base - - id: reference__rtg - source: reference__rtg - - id: reference__genome_context - source: reference__genome_context - - id: genome_resources__variation__clinvar - source: genome_resources__variation__clinvar - - id: genome_resources__variation__cosmic - source: genome_resources__variation__cosmic - - id: genome_resources__variation__dbsnp - source: genome_resources__variation__dbsnp - - id: genome_resources__variation__esp - source: genome_resources__variation__esp - - id: genome_resources__variation__exac - source: genome_resources__variation__exac - - id: genome_resources__variation__gnomad_exome - source: genome_resources__variation__gnomad_exome - - id: genome_resources__variation__1000g - source: genome_resources__variation__1000g - - id: genome_resources__variation__lcr - source: genome_resources__variation__lcr - - id: genome_resources__variation__polyx - source: genome_resources__variation__polyx - - id: genome_resources__variation__encode_blacklist - source: genome_resources__variation__encode_blacklist - - id: genome_resources__aliases__ensembl - source: genome_resources__aliases__ensembl - - id: genome_resources__aliases__human - source: genome_resources__aliases__human - - id: genome_resources__aliases__snpeff - source: genome_resources__aliases__snpeff - - id: reference__snpeff__hg19 - source: reference__snpeff__hg19 - - id: genome_resources__variation__train_hapmap - source: genome_resources__variation__train_hapmap - - id: genome_resources__variation__train_indels - source: genome_resources__variation__train_indels - - id: resources - source: resources - - id: description - source: description - out: - - id: batch_rec - run: steps/batch_for_variantcall.cwl -- id: variantcall - in: - - id: batch_rec - source: batch_for_variantcall/batch_rec - out: - - id: vc_rec - run: wf-variantcall.cwl - scatter: - - batch_rec - scatterMethod: dotproduct -- id: batch_for_jointvc - in: - - id: vc_rec - source: variantcall/vc_rec - out: - - id: jointvc_batch_rec - run: steps/batch_for_jointvc.cwl -- id: jointcall - in: - - id: jointvc_batch_rec - source: batch_for_jointvc/jointvc_batch_rec - out: - - id: jointvc_rec - - id: vrn_file_joint - run: wf-jointcall.cwl - scatter: - - jointvc_batch_rec - scatterMethod: dotproduct -- id: summarize_vc - in: - - id: jointvc_rec - source: jointcall/jointvc_rec - out: - - id: variants__calls - - id: variants__gvcf - - id: variants__samples - - id: validate__grading_summary - - id: validate__grading_plots - run: steps/summarize_vc.cwl -- id: qc_to_rec - in: - - id: align_bam - source: postprocess_alignment/align_bam - - id: analysis - source: analysis - - id: reference__fasta__base - source: reference__fasta__base - - id: reference__versions - source: reference__versions - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: config__algorithm__tools_off - source: config__algorithm__tools_off - - id: genome_build - source: genome_build - - id: config__algorithm__qc - source: config__algorithm__qc - - id: metadata__batch - source: metadata__batch - - id: metadata__phenotype - source: metadata__phenotype - - id: config__algorithm__coverage_interval - source: postprocess_alignment/config__algorithm__coverage_interval - - id: depth__variant_regions__regions - source: postprocess_alignment/depth__variant_regions__regions - - id: depth__variant_regions__dist - source: postprocess_alignment/depth__variant_regions__dist - - id: depth__samtools__stats - source: postprocess_alignment/depth__samtools__stats - - id: depth__samtools__idxstats - source: postprocess_alignment/depth__samtools__idxstats - - id: depth__sv_regions__regions - source: postprocess_alignment/depth__sv_regions__regions - - id: depth__sv_regions__dist - source: postprocess_alignment/depth__sv_regions__dist - - id: depth__coverage__regions - source: postprocess_alignment/depth__coverage__regions - - id: depth__coverage__dist - source: postprocess_alignment/depth__coverage__dist - - id: depth__coverage__thresholds - source: postprocess_alignment/depth__coverage__thresholds - - id: config__algorithm__variant_regions - source: postprocess_alignment/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: postprocess_alignment/config__algorithm__variant_regions_merged - - id: config__algorithm__coverage - source: postprocess_alignment/config__algorithm__coverage - - id: config__algorithm__coverage_merged - source: postprocess_alignment/config__algorithm__coverage_merged - - id: variants__samples - source: summarize_vc/variants__samples - - id: resources - source: resources - - id: description - source: description - out: - - id: qc_rec - run: steps/qc_to_rec.cwl -- id: pipeline_summary - in: - - id: qc_rec - source: qc_to_rec/qc_rec - out: - - id: qcout_rec - run: steps/pipeline_summary.cwl - scatter: - - qc_rec - scatterMethod: dotproduct -- id: multiqc_summary - in: - - id: qcout_rec - source: pipeline_summary/qcout_rec - out: - - id: summary__multiqc - - id: versions__tools - - id: versions__data - run: steps/multiqc_summary.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/alignment_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/alignment_to_rec.cwl deleted file mode 100644 index 1379c575535..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/alignment_to_rec.cwl +++ /dev/null @@ -1,198 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=alignment_rec:resources;description;config__algorithm__align_split_size;files;config__algorithm__trim_reads;reference__fasta__base;config__algorithm__adapters;rgnames__lb;rgnames__rg;rgnames__lane;reference__bwa__indexes;config__algorithm__bam_clean;config__algorithm__aligner;rgnames__pl;rgnames__pu;config__algorithm__mark_duplicates;analysis;rgnames__sample;config__algorithm__variant_regions -- sentinel_inputs=files:var,analysis:var,config__algorithm__align_split_size:var,reference__fasta__base:var,rgnames__pl:var,rgnames__sample:var,rgnames__pu:var,rgnames__lane:var,rgnames__rg:var,rgnames__lb:var,reference__bwa__indexes:var,config__algorithm__aligner:var,config__algorithm__trim_reads:var,config__algorithm__adapters:var,config__algorithm__bam_clean:var,config__algorithm__variant_regions:var,config__algorithm__mark_duplicates:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- alignment_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 2048 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: files - type: - items: - items: File - type: array - type: array -- id: analysis - type: - items: string - type: array -- id: config__algorithm__align_split_size - type: - items: - - 'null' - - string - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: rgnames__pl - type: - items: string - type: array -- id: rgnames__sample - type: - items: string - type: array -- id: rgnames__pu - type: - items: string - type: array -- id: rgnames__lane - type: - items: string - type: array -- id: rgnames__rg - type: - items: string - type: array -- id: rgnames__lb - type: - items: - - 'null' - - string - type: array -- id: reference__bwa__indexes - secondaryFiles: - - ^.ann - - ^.pac - - ^.sa - - ^.bwt - type: - items: File - type: array -- id: config__algorithm__aligner - type: - items: string - type: array -- id: config__algorithm__trim_reads - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__adapters - type: - items: - items: string - type: array - type: array -- id: config__algorithm__bam_clean - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__variant_regions - type: - items: File - type: array -- id: config__algorithm__mark_duplicates - type: - items: - - string - - 'null' - - boolean - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: alignment_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: - - string - - 'null' - - boolean - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: rgnames__pu - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: alignment_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/batch_for_jointvc.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/batch_for_jointvc.cwl deleted file mode 100644 index 523567ed191..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/batch_for_jointvc.cwl +++ /dev/null @@ -1,329 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-batch -- sentinel_outputs=jointvc_batch_rec:resources;description;batch_samples;validate__summary;validate__tp;validate__fp;validate__fn;vrn_file;reference__fasta__base;metadata__phenotype;config__algorithm__vcfanno;config__algorithm__variantcaller;genome_resources__variation__1000g;config__algorithm__coverage_interval;genome_resources__variation__clinvar;genome_resources__variation__esp;metadata__batch;genome_resources__variation__lcr;config__algorithm__min_allele_fraction;genome_resources__variation__train_hapmap;reference__genome_context;config__algorithm__validate;reference__snpeff__hg19;config__algorithm__validate_regions;genome_build;genome_resources__variation__exac;genome_resources__variation__gnomad_exome;genome_resources__aliases__human;config__algorithm__tools_off;genome_resources__variation__dbsnp;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;genome_resources__variation__cosmic;config__algorithm__ensemble;analysis;config__algorithm__tools_on;config__algorithm__effects;config__algorithm__variant_regions;genome_resources__aliases__ensembl;config__algorithm__exclude_regions;reference__rtg;genome_resources__variation__train_indels;genome_resources__aliases__snpeff;config__algorithm__variant_regions_merged;regions__sample_callable;config__algorithm__callable_regions -- sentinel_inputs=vc_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- batch_for_jointvc -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 2048 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: vc_rec - type: - items: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array - type: array -outputs: -- id: jointvc_batch_rec - type: - items: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: jointvc_batch_rec - type: record - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/batch_for_variantcall.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/batch_for_variantcall.cwl deleted file mode 100644 index ca40e5b30cf..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/batch_for_variantcall.cwl +++ /dev/null @@ -1,401 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-batch -- sentinel_outputs=batch_rec:resources;description;reference__fasta__base;metadata__phenotype;config__algorithm__vcfanno;config__algorithm__variantcaller;genome_resources__variation__1000g;config__algorithm__coverage_interval;genome_resources__variation__clinvar;genome_resources__variation__esp;metadata__batch;genome_resources__variation__lcr;config__algorithm__min_allele_fraction;vrn_file;genome_resources__variation__train_hapmap;reference__genome_context;config__algorithm__validate;reference__snpeff__hg19;config__algorithm__validate_regions;genome_build;genome_resources__variation__exac;genome_resources__variation__gnomad_exome;genome_resources__aliases__human;config__algorithm__tools_off;genome_resources__variation__dbsnp;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;genome_resources__variation__cosmic;config__algorithm__ensemble;analysis;config__algorithm__tools_on;config__algorithm__effects;config__algorithm__variant_regions;genome_resources__aliases__ensembl;config__algorithm__exclude_regions;reference__rtg;genome_resources__variation__train_indels;genome_resources__aliases__snpeff;align_bam;config__algorithm__variant_regions_merged;regions__sample_callable;config__algorithm__callable_regions -- sentinel_inputs=analysis:var,genome_build:var,align_bam:var,vrn_file:var,metadata__batch:var,metadata__phenotype:var,config__algorithm__callable_regions:var,regions__sample_callable:var,config__algorithm__variantcaller:var,config__algorithm__ensemble:var,config__algorithm__vcfanno:var,config__algorithm__coverage_interval:var,config__algorithm__effects:var,config__algorithm__min_allele_fraction:var,config__algorithm__exclude_regions:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__validate:var,config__algorithm__validate_regions:var,config__algorithm__tools_on:var,config__algorithm__tools_off:var,reference__fasta__base:var,reference__rtg:var,reference__genome_context:var,genome_resources__variation__clinvar:var,genome_resources__variation__cosmic:var,genome_resources__variation__dbsnp:var,genome_resources__variation__esp:var,genome_resources__variation__exac:var,genome_resources__variation__gnomad_exome:var,genome_resources__variation__1000g:var,genome_resources__variation__lcr:var,genome_resources__variation__polyx:var,genome_resources__variation__encode_blacklist:var,genome_resources__aliases__ensembl:var,genome_resources__aliases__human:var,genome_resources__aliases__snpeff:var,reference__snpeff__hg19:var,genome_resources__variation__train_hapmap:var,genome_resources__variation__train_indels:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- batch_for_variantcall -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1030 - ramMin: 2048 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: analysis - type: - items: string - type: array -- id: genome_build - type: - items: string - type: array -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: vrn_file - type: - items: - - 'null' - - string - type: array -- id: metadata__batch - type: - items: string - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: config__algorithm__callable_regions - type: - items: File - type: array -- id: regions__sample_callable - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variantcaller - type: - items: - items: string - type: array - type: array -- id: config__algorithm__ensemble - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__vcfanno - type: - items: - items: File - type: array - type: array -- id: config__algorithm__coverage_interval - type: - items: - - string - - 'null' - type: array -- id: config__algorithm__effects - type: - items: string - type: array -- id: config__algorithm__min_allele_fraction - type: - items: double - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__validate - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__validate_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: reference__rtg - type: - items: File - type: array -- id: reference__genome_context - secondaryFiles: - - .tbi - type: - items: - items: File - type: array - type: array -- id: genome_resources__variation__clinvar - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__cosmic - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__dbsnp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__esp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__exac - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__gnomad_exome - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__1000g - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__lcr - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__polyx - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__encode_blacklist - type: - items: - - 'null' - - string - type: array -- id: genome_resources__aliases__ensembl - type: - items: string - type: array -- id: genome_resources__aliases__human - type: - items: - - string - - 'null' - - boolean - type: array -- id: genome_resources__aliases__snpeff - type: - items: string - type: array -- id: reference__snpeff__hg19 - type: - items: File - type: array -- id: genome_resources__variation__train_hapmap - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__train_indels - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: batch_rec - type: - items: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/combine_sample_regions.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/combine_sample_regions.cwl deleted file mode 100644 index d876db22bd2..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/combine_sample_regions.cwl +++ /dev/null @@ -1,99 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=config__algorithm__callable_regions,config__algorithm__non_callable_regions,config__algorithm__callable_count -- sentinel_inputs=regions__callable:var,regions__nblock:var,metadata__batch:var,config__algorithm__nomap_split_size:var,config__algorithm__nomap_split_targets:var,reference__fasta__base:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- combine_sample_regions -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1025 - ramMin: 2048 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 -- class: arv:APIRequirement -inputs: -- id: regions__callable - type: - items: - - File - - 'null' - type: array -- id: regions__nblock - type: - items: - - File - - 'null' - type: array -- id: metadata__batch - type: - items: string - type: array -- id: config__algorithm__nomap_split_size - type: - items: long - type: array -- id: config__algorithm__nomap_split_targets - type: - items: long - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: config__algorithm__callable_regions - type: - items: File - type: array -- id: config__algorithm__non_callable_regions - type: - items: File - type: array -- id: config__algorithm__callable_count - type: - items: int - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/compare_to_rm.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/compare_to_rm.cwl deleted file mode 100644 index 06152084c58..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/compare_to_rm.cwl +++ /dev/null @@ -1,337 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-single -- sentinel_outputs=vc_rec:batch_samples;validate__summary;validate__tp;validate__fp;validate__fn;resources;description;vrn_file;reference__fasta__base;metadata__phenotype;config__algorithm__vcfanno;config__algorithm__variantcaller;genome_resources__variation__1000g;config__algorithm__coverage_interval;genome_resources__variation__clinvar;genome_resources__variation__esp;metadata__batch;genome_resources__variation__lcr;config__algorithm__min_allele_fraction;genome_resources__variation__train_hapmap;reference__genome_context;config__algorithm__validate;reference__snpeff__hg19;config__algorithm__validate_regions;genome_build;genome_resources__variation__exac;genome_resources__variation__gnomad_exome;genome_resources__aliases__human;config__algorithm__tools_off;genome_resources__variation__dbsnp;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;genome_resources__variation__cosmic;config__algorithm__ensemble;analysis;config__algorithm__tools_on;config__algorithm__effects;config__algorithm__variant_regions;genome_resources__aliases__ensembl;config__algorithm__exclude_regions;reference__rtg;genome_resources__variation__train_indels;genome_resources__aliases__snpeff;config__algorithm__variant_regions_merged;regions__sample_callable;config__algorithm__callable_regions -- sentinel_inputs=batch_rec:record,vrn_file:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- compare_to_rm -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1028 - ramMin: 4096 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: gvcf-regions - specs: - - https://anaconda.org/bioconda/gvcf-regions - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: rtg-tools - specs: - - https://anaconda.org/bioconda/rtg-tools - - package: vcfanno - specs: - - https://anaconda.org/bioconda/vcfanno -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: vrn_file - secondaryFiles: - - .tbi - type: File -outputs: -- id: vc_rec - type: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/concat_batch_variantcalls.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/concat_batch_variantcalls.cwl deleted file mode 100644 index d486086c79e..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/concat_batch_variantcalls.cwl +++ /dev/null @@ -1,196 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-merge -- sentinel_outputs=vrn_file -- sentinel_inputs=batch_rec:record,region_block:var,vrn_file_region:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- concat_batch_variantcalls -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 2048 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 -- class: arv:APIRequirement -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: region_block - type: - items: - items: string - type: array - type: array -- id: vrn_file_region - secondaryFiles: - - .tbi - type: - items: - - File - - 'null' - type: array -outputs: -- id: vrn_file - secondaryFiles: - - .tbi - type: File -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/concat_batch_variantcalls_jointvc.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/concat_batch_variantcalls_jointvc.cwl deleted file mode 100644 index dbb0f54be94..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/concat_batch_variantcalls_jointvc.cwl +++ /dev/null @@ -1,209 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-merge -- sentinel_outputs=vrn_file_joint -- sentinel_inputs=jointvc_batch_rec:record,region:var,vrn_file_region:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- concat_batch_variantcalls_jointvc -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 2048 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 -- class: arv:APIRequirement -inputs: -- id: jointvc_batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: jointvc_batch_rec - type: record - type: array -- id: region - type: - items: string - type: array -- id: vrn_file_region - secondaryFiles: - - .tbi - type: - items: - - File - - 'null' - type: array -outputs: -- id: vrn_file_joint - secondaryFiles: - - .tbi - type: File -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/finalize_jointvc.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/finalize_jointvc.cwl deleted file mode 100644 index fd1e04762e4..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/finalize_jointvc.cwl +++ /dev/null @@ -1,331 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-single -- sentinel_outputs=jointvc_rec:resources;description;batch_samples;validate__summary;validate__tp;validate__fp;validate__fn;vrn_file;reference__fasta__base;metadata__phenotype;config__algorithm__vcfanno;config__algorithm__variantcaller;genome_resources__variation__1000g;config__algorithm__coverage_interval;genome_resources__variation__clinvar;genome_resources__variation__esp;metadata__batch;genome_resources__variation__lcr;config__algorithm__min_allele_fraction;genome_resources__variation__train_hapmap;reference__genome_context;config__algorithm__validate;reference__snpeff__hg19;config__algorithm__validate_regions;genome_build;genome_resources__variation__exac;genome_resources__variation__gnomad_exome;genome_resources__aliases__human;config__algorithm__tools_off;genome_resources__variation__dbsnp;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;genome_resources__variation__cosmic;config__algorithm__ensemble;analysis;config__algorithm__tools_on;config__algorithm__effects;config__algorithm__variant_regions;genome_resources__aliases__ensembl;config__algorithm__exclude_regions;reference__rtg;genome_resources__variation__train_indels;genome_resources__aliases__snpeff;config__algorithm__variant_regions_merged;regions__sample_callable;config__algorithm__callable_regions;vrn_file_joint -- sentinel_inputs=jointvc_batch_rec:record,vrn_file_joint:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- finalize_jointvc -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 2048 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -inputs: -- id: jointvc_batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: jointvc_batch_rec - type: record - type: array -- id: vrn_file_joint - secondaryFiles: - - .tbi - type: File -outputs: -- id: jointvc_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - - name: vrn_file_joint - type: File - name: jointvc_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/get_parallel_regions.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/get_parallel_regions.cwl deleted file mode 100644 index 18e635c4006..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/get_parallel_regions.cwl +++ /dev/null @@ -1,171 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-split -- sentinel_outputs=region_block -- sentinel_inputs=batch_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- get_parallel_regions -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1030 - ramMin: 2048 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -outputs: -- id: region_block - type: - items: - items: string - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/get_parallel_regions_jointvc.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/get_parallel_regions_jointvc.cwl deleted file mode 100644 index fac484f75ca..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/get_parallel_regions_jointvc.cwl +++ /dev/null @@ -1,184 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-split -- sentinel_outputs=region -- sentinel_inputs=jointvc_batch_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- get_parallel_regions_jointvc -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 2048 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -inputs: -- id: jointvc_batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: jointvc_batch_rec - type: record - type: array -outputs: -- id: region - type: - items: string - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/merge_split_alignments.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/merge_split_alignments.cwl deleted file mode 100644 index 66e17dc7aa9..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/merge_split_alignments.cwl +++ /dev/null @@ -1,168 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=single-merge -- sentinel_outputs=align_bam,work_bam_plus__disc,work_bam_plus__sr,hla__fastq -- sentinel_inputs=alignment_rec:record,work_bam:var,align_bam:var,work_bam_plus__disc:var,work_bam_plus__sr:var,hla__fastq:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- merge_split_alignments -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1035 - ramMin: 4096 - tmpdirMin: 6 -- class: dx:InputResourceRequirement - indirMin: 4 -- class: SoftwareRequirement - packages: - - package: biobambam - specs: - - https://anaconda.org/bioconda/biobambam - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: variantbam - specs: - - https://anaconda.org/bioconda/variantbam -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: - - string - - 'null' - - boolean - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: rgnames__pu - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: alignment_rec - type: record -- id: work_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: align_bam_toolinput - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: work_bam_plus__disc_toolinput - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: work_bam_plus__sr_toolinput - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: hla__fastq_toolinput - type: - items: - - 'null' - - items: File - type: array - type: array -outputs: -- id: align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__disc - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__sr - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: hla__fastq - type: - - 'null' - - items: File - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/multiqc_summary.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/multiqc_summary.cwl deleted file mode 100644 index 45a90274381..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/multiqc_summary.cwl +++ /dev/null @@ -1,95 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=summary__multiqc,versions__tools,versions__data -- sentinel_inputs=qcout_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- multiqc_summary -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1030 - ramMin: 2048 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: multiqc - specs: - - https://anaconda.org/bioconda/multiqc - - package: multiqc-bcbio - specs: - - https://anaconda.org/bioconda/multiqc-bcbio -inputs: -- id: qcout_rec - type: - items: - fields: - - name: summary__qc - type: - - File - - 'null' - - name: summary__metrics - type: - - string - - 'null' - - name: description - type: string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: config__algorithm__tools_on - type: - items: string - type: array - name: qcout_rec - type: record - type: array -outputs: -- id: summary__multiqc - type: - items: - - File - - 'null' - type: array -- id: versions__tools - type: - items: - - File - - 'null' - type: array -- id: versions__data - type: - items: - - File - - 'null' - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/pipeline_summary.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/pipeline_summary.cwl deleted file mode 100644 index 75e0c7a35d5..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/pipeline_summary.cwl +++ /dev/null @@ -1,219 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=qcout_rec:summary__qc;summary__metrics;description;reference__versions;genome_build;config__algorithm__tools_off;config__algorithm__qc;config__algorithm__tools_on -- sentinel_inputs=qc_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- pipeline_summary -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1030 - ramMin: 4096 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: fastqc=0.11.7=5 - specs: - - https://anaconda.org/bioconda/fastqc=0.11.7=5 - - package: goleft - specs: - - https://anaconda.org/bioconda/goleft - - package: hts-nim-tools - specs: - - https://anaconda.org/bioconda/hts-nim-tools - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: picard - specs: - - https://anaconda.org/bioconda/picard - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: qsignature - specs: - - https://anaconda.org/bioconda/qsignature - - package: qualimap - specs: - - https://anaconda.org/bioconda/qualimap - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: preseq - specs: - - https://anaconda.org/bioconda/preseq - - package: peddy - specs: - - https://anaconda.org/bioconda/peddy - - package: verifybamid2 - specs: - - https://anaconda.org/bioconda/verifybamid2 -- class: arv:RuntimeConstraints - keep_cache: 4096 -inputs: -- id: qc_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: depth__samtools__stats - type: - - File - - 'null' - - name: depth__samtools__idxstats - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: depth__variant_regions__dist - type: - - File - - 'null' - - name: depth__sv_regions__regions - type: - - File - - 'null' - - name: depth__sv_regions__dist - type: - - File - - 'null' - - name: depth__coverage__regions - type: - - File - - 'null' - - name: depth__coverage__dist - type: - - File - - 'null' - - name: depth__coverage__thresholds - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - name: qc_rec - type: record -outputs: -- id: qcout_rec - type: - fields: - - name: summary__qc - type: - - File - - 'null' - - name: summary__metrics - type: - - string - - 'null' - - name: description - type: string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: config__algorithm__tools_on - type: - items: string - type: array - name: qcout_rec - type: record -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/postprocess_alignment.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/postprocess_alignment.cwl deleted file mode 100644 index 09728af2814..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/postprocess_alignment.cwl +++ /dev/null @@ -1,223 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=config__algorithm__coverage_interval,config__algorithm__variant_regions,config__algorithm__variant_regions_merged,config__algorithm__variant_regions_orig,config__algorithm__coverage,config__algorithm__coverage_merged,config__algorithm__coverage_orig,config__algorithm__seq2c_bed_ready,regions__callable,regions__sample_callable,regions__nblock,depth__samtools__stats,depth__samtools__idxstats,depth__variant_regions__regions,depth__variant_regions__dist,depth__sv_regions__regions,depth__sv_regions__dist,depth__coverage__regions,depth__coverage__dist,depth__coverage__thresholds,align_bam -- sentinel_inputs=postprocess_alignment_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- postprocess_alignment -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1033 - ramMin: 4096 - tmpdirMin: 5 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: goleft - specs: - - https://anaconda.org/bioconda/goleft - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: sentieon - specs: - - https://anaconda.org/bioconda/sentieon -- class: arv:APIRequirement -inputs: -- id: postprocess_alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - 'null' - - string - - name: genome_resources__rnaseq__gene_bed - type: File - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__recalibrate - type: - - string - - 'null' - - boolean - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__variant_regions_orig - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: config__algorithm__coverage_orig - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - name: postprocess_alignment_rec - type: record -outputs: -- id: config__algorithm__coverage_interval - type: - - string - - 'null' -- id: config__algorithm__variant_regions - type: - - File - - 'null' -- id: config__algorithm__variant_regions_merged - type: - - File - - 'null' -- id: config__algorithm__variant_regions_orig - type: - - File - - 'null' -- id: config__algorithm__coverage - type: - - File - - 'null' -- id: config__algorithm__coverage_merged - type: - - File - - 'null' -- id: config__algorithm__coverage_orig - type: - - File - - 'null' -- id: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' -- id: regions__callable - type: - - File - - 'null' -- id: regions__sample_callable - type: - - File - - 'null' -- id: regions__nblock - type: - - File - - 'null' -- id: depth__samtools__stats - type: - - File - - 'null' -- id: depth__samtools__idxstats - type: - - File - - 'null' -- id: depth__variant_regions__regions - type: - - File - - 'null' -- id: depth__variant_regions__dist - type: - - File - - 'null' -- id: depth__sv_regions__regions - type: - - File - - 'null' -- id: depth__sv_regions__dist - type: - - File - - 'null' -- id: depth__coverage__regions - type: - - File - - 'null' -- id: depth__coverage__dist - type: - - File - - 'null' -- id: depth__coverage__thresholds - type: - - File - - 'null' -- id: align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/postprocess_alignment_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/postprocess_alignment_to_rec.cwl deleted file mode 100644 index 813f3fed737..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/postprocess_alignment_to_rec.cwl +++ /dev/null @@ -1,233 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=postprocess_alignment_rec:resources;description;reference__fasta__base;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;genome_resources__variation__lcr;config__algorithm__recalibrate;config__algorithm__coverage;genome_resources__variation__dbsnp;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__tools_on;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__variant_regions_orig;config__algorithm__coverage_merged;config__algorithm__coverage_orig;config__algorithm__seq2c_bed_ready -- sentinel_inputs=align_bam:var,config__algorithm__coverage_interval:var,config__algorithm__exclude_regions:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__variant_regions_orig:var,config__algorithm__coverage:var,config__algorithm__coverage_merged:var,config__algorithm__coverage_orig:var,config__algorithm__seq2c_bed_ready:var,config__algorithm__recalibrate:var,config__algorithm__tools_on:var,genome_resources__rnaseq__gene_bed:var,genome_resources__variation__dbsnp:var,genome_resources__variation__lcr:var,genome_resources__variation__polyx:var,genome_resources__variation__encode_blacklist:var,reference__fasta__base:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- postprocess_alignment_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 2048 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_interval - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_orig - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_orig - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__seq2c_bed_ready - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__recalibrate - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: genome_resources__rnaseq__gene_bed - type: - items: File - type: array -- id: genome_resources__variation__dbsnp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__lcr - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__polyx - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__encode_blacklist - type: - items: - - 'null' - - string - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: postprocess_alignment_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - 'null' - - string - - name: genome_resources__rnaseq__gene_bed - type: File - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__recalibrate - type: - - string - - 'null' - - boolean - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__variant_regions_orig - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: config__algorithm__coverage_orig - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - name: postprocess_alignment_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/postprocess_variants.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/postprocess_variants.cwl deleted file mode 100644 index a300aa5be18..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/postprocess_variants.cwl +++ /dev/null @@ -1,195 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-single -- sentinel_outputs=vrn_file_joint -- sentinel_inputs=jointvc_batch_rec:record,vrn_file_joint:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- postprocess_variants -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1028 - ramMin: 4096 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: snpeff - specs: - - https://anaconda.org/bioconda/snpeff - version: - - 4.3.1t -inputs: -- id: jointvc_batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: jointvc_batch_rec - type: record - type: array -- id: vrn_file_joint_toolinput - secondaryFiles: - - .tbi - type: File -outputs: -- id: vrn_file_joint - secondaryFiles: - - .tbi - type: File -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/prep_align_inputs.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/prep_align_inputs.cwl deleted file mode 100644 index 8179e185e8b..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/prep_align_inputs.cwl +++ /dev/null @@ -1,142 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=single-split -- sentinel_outputs=process_alignment_rec:files;config__algorithm__quality_format;align_split -- sentinel_inputs=alignment_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- prep_align_inputs -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1028 - ramMin: 4096 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 4 -- class: SoftwareRequirement - packages: - - package: grabix - specs: - - https://anaconda.org/bioconda/grabix - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: biobambam - specs: - - https://anaconda.org/bioconda/biobambam - - package: atropos;env - specs: - - https://anaconda.org/bioconda/atropos;env - version: - - python3 - - package: optitype - specs: - - https://anaconda.org/bioconda/optitype - - package: razers3 - specs: - - https://anaconda.org/bioconda/razers3 - version: - - 3.5.0 - - package: coincbc - specs: - - https://anaconda.org/bioconda/coincbc -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: - - string - - 'null' - - boolean - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: rgnames__pu - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: alignment_rec - type: record -outputs: -- id: process_alignment_rec - type: - items: - fields: - - name: files - type: - - 'null' - - items: File - type: array - - name: config__algorithm__quality_format - type: - - string - - 'null' - - name: align_split - type: - - string - - 'null' - name: process_alignment_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/prep_samples.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/prep_samples.cwl deleted file mode 100644 index 244502016fb..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/prep_samples.cwl +++ /dev/null @@ -1,95 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=rgnames__sample,config__algorithm__variant_regions,config__algorithm__variant_regions_merged,config__algorithm__variant_regions_orig,config__algorithm__coverage,config__algorithm__coverage_merged,config__algorithm__coverage_orig,config__algorithm__seq2c_bed_ready -- sentinel_inputs=prep_samples_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- prep_samples -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1025 - ramMin: 2048 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy -inputs: -- id: prep_samples_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: prep_samples_rec - type: record -outputs: -- id: rgnames__sample - type: string -- id: config__algorithm__variant_regions - type: - - File - - 'null' -- id: config__algorithm__variant_regions_merged - type: - - File - - 'null' -- id: config__algorithm__variant_regions_orig - type: - - File - - 'null' -- id: config__algorithm__coverage - type: - - File - - 'null' -- id: config__algorithm__coverage_merged - type: - - File - - 'null' -- id: config__algorithm__coverage_orig - type: - - File - - 'null' -- id: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/prep_samples_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/prep_samples_to_rec.cwl deleted file mode 100644 index 830e65f0404..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/prep_samples_to_rec.cwl +++ /dev/null @@ -1,85 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=prep_samples_rec:resources;description;reference__fasta__base;config__algorithm__coverage;rgnames__sample;config__algorithm__variant_regions -- sentinel_inputs=config__algorithm__coverage:var,rgnames__sample:var,config__algorithm__variant_regions:var,reference__fasta__base:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- prep_samples_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1025 - ramMin: 2048 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: config__algorithm__coverage - type: - items: - - File - - 'null' - type: array -- id: rgnames__sample - type: - items: string - type: array -- id: config__algorithm__variant_regions - type: - items: File - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: prep_samples_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: prep_samples_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/process_alignment.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/process_alignment.cwl deleted file mode 100644 index b12a9946b5c..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/process_alignment.cwl +++ /dev/null @@ -1,198 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=single-parallel -- sentinel_outputs=work_bam,align_bam,hla__fastq,work_bam_plus__disc,work_bam_plus__sr -- sentinel_inputs=alignment_rec:record,process_alignment_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- process_alignment -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1030 - ramMin: 4096 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 7 -- class: SoftwareRequirement - packages: - - package: bwa - specs: - - https://anaconda.org/bioconda/bwa - - package: bwakit - specs: - - https://anaconda.org/bioconda/bwakit - - package: grabix - specs: - - https://anaconda.org/bioconda/grabix - - package: minimap2 - specs: - - https://anaconda.org/bioconda/minimap2 - - package: novoalign - specs: - - https://anaconda.org/bioconda/novoalign - - package: snap-aligner - specs: - - https://anaconda.org/bioconda/snap-aligner - version: - - 1.0dev.97 - - package: sentieon - specs: - - https://anaconda.org/bioconda/sentieon - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: pysam> - specs: - - https://anaconda.org/bioconda/pysam> - version: - - 0.13.0 - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: fgbio - specs: - - https://anaconda.org/bioconda/fgbio - - package: umis - specs: - - https://anaconda.org/bioconda/umis - - package: biobambam - specs: - - https://anaconda.org/bioconda/biobambam - - package: seqtk - specs: - - https://anaconda.org/bioconda/seqtk - - package: samblaster - specs: - - https://anaconda.org/bioconda/samblaster - - package: variantbam - specs: - - https://anaconda.org/bioconda/variantbam -- class: arv:APIRequirement -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: - - string - - 'null' - - boolean - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: rgnames__pu - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: alignment_rec - type: record -- id: process_alignment_rec - type: - fields: - - name: files - type: - - 'null' - - items: File - type: array - - name: config__algorithm__quality_format - type: - - string - - 'null' - - name: align_split - type: - - string - - 'null' - name: process_alignment_rec - type: record -outputs: -- id: work_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: hla__fastq - type: - - 'null' - - items: File - type: array -- id: work_bam_plus__disc - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__sr - secondaryFiles: - - .bai - type: - - File - - 'null' -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/qc_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/qc_to_rec.cwl deleted file mode 100644 index 53e0a326097..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/qc_to_rec.cwl +++ /dev/null @@ -1,295 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=qc_rec:resources;description;reference__fasta__base;metadata__phenotype;config__algorithm__coverage_interval;metadata__batch;reference__versions;genome_build;config__algorithm__coverage;config__algorithm__tools_off;config__algorithm__qc;analysis;config__algorithm__tools_on;config__algorithm__variant_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__coverage_merged;depth__samtools__stats;depth__samtools__idxstats;depth__variant_regions__regions;depth__variant_regions__dist;depth__sv_regions__regions;depth__sv_regions__dist;depth__coverage__regions;depth__coverage__dist;depth__coverage__thresholds;variants__samples -- sentinel_inputs=align_bam:var,analysis:var,reference__fasta__base:var,reference__versions:var,config__algorithm__tools_on:var,config__algorithm__tools_off:var,genome_build:var,config__algorithm__qc:var,metadata__batch:var,metadata__phenotype:var,config__algorithm__coverage_interval:var,depth__variant_regions__regions:var,depth__variant_regions__dist:var,depth__samtools__stats:var,depth__samtools__idxstats:var,depth__sv_regions__regions:var,depth__sv_regions__dist:var,depth__coverage__regions:var,depth__coverage__dist:var,depth__coverage__thresholds:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__coverage:var,config__algorithm__coverage_merged:var,variants__samples:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- qc_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 2048 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: analysis - type: - items: string - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: reference__versions - type: - items: File - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: genome_build - type: - items: string - type: array -- id: config__algorithm__qc - type: - items: - items: string - type: array - type: array -- id: metadata__batch - type: - items: string - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: config__algorithm__coverage_interval - type: - items: - - string - - 'null' - type: array -- id: depth__variant_regions__regions - type: - items: - - File - - 'null' - type: array -- id: depth__variant_regions__dist - type: - items: - - File - - 'null' - type: array -- id: depth__samtools__stats - type: - items: - - File - - 'null' - type: array -- id: depth__samtools__idxstats - type: - items: - - File - - 'null' - type: array -- id: depth__sv_regions__regions - type: - items: - - File - - 'null' - type: array -- id: depth__sv_regions__dist - type: - items: - - File - - 'null' - type: array -- id: depth__coverage__regions - type: - items: - - File - - 'null' - type: array -- id: depth__coverage__dist - type: - items: - - File - - 'null' - type: array -- id: depth__coverage__thresholds - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_merged - type: - items: - - File - - 'null' - type: array -- id: variants__samples - type: - items: - items: - items: - - File - - 'null' - type: array - type: array - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: qc_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: depth__samtools__stats - type: - - File - - 'null' - - name: depth__samtools__idxstats - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: depth__variant_regions__dist - type: - - File - - 'null' - - name: depth__sv_regions__regions - type: - - File - - 'null' - - name: depth__sv_regions__dist - type: - - File - - 'null' - - name: depth__coverage__regions - type: - - File - - 'null' - - name: depth__coverage__dist - type: - - File - - 'null' - - name: depth__coverage__thresholds - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - name: qc_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/run_jointvc.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/run_jointvc.cwl deleted file mode 100644 index 897c1e02ca8..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/run_jointvc.cwl +++ /dev/null @@ -1,203 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-parallel -- sentinel_outputs=vrn_file_region,region -- sentinel_inputs=jointvc_batch_rec:record,region:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- run_jointvc -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 2048 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 - - package: gvcfgenotyper - specs: - - https://anaconda.org/bioconda/gvcfgenotyper - - package: sentieon - specs: - - https://anaconda.org/bioconda/sentieon -- class: arv:APIRequirement -inputs: -- id: jointvc_batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: jointvc_batch_rec - type: record - type: array -- id: region_toolinput - type: string -outputs: -- id: vrn_file_region - secondaryFiles: - - .tbi - type: - - File - - 'null' -- id: region - type: string -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/summarize_vc.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/summarize_vc.cwl deleted file mode 100644 index 85b8a8f01e0..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/summarize_vc.cwl +++ /dev/null @@ -1,225 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=variants__calls,variants__gvcf,variants__samples,validate__grading_summary,validate__grading_plots -- sentinel_inputs=jointvc_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- summarize_vc -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1025 - ramMin: 2048 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 1 -inputs: -- id: jointvc_rec - type: - items: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - - name: vrn_file_joint - type: File - name: jointvc_rec - type: record - type: array - type: array -outputs: -- id: variants__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: variants__gvcf - type: - items: - - 'null' - - items: - - File - - 'null' - type: array - type: array -- id: variants__samples - type: - items: - items: - items: - - File - - 'null' - type: array - type: array - type: array -- id: validate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: validate__grading_plots - type: - items: - items: - - File - - 'null' - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/variantcall_batch_region.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/variantcall_batch_region.cwl deleted file mode 100644 index eb58aa47177..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/steps/variantcall_batch_region.cwl +++ /dev/null @@ -1,266 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-parallel -- sentinel_outputs=vrn_file_region,region_block -- sentinel_inputs=batch_rec:record,region_block:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- variantcall_batch_region -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1030 - ramMin: 4096 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: freebayes - specs: - - https://anaconda.org/bioconda/freebayes - version: - - 1.1.0.46 - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 - - package: vqsr_cnn - specs: - - https://anaconda.org/bioconda/vqsr_cnn - - package: deepvariant - specs: - - https://anaconda.org/bioconda/deepvariant - - package: sentieon - specs: - - https://anaconda.org/bioconda/sentieon - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: octopus - specs: - - https://anaconda.org/bioconda/octopus - - package: picard - specs: - - https://anaconda.org/bioconda/picard - - package: platypus-variant - specs: - - https://anaconda.org/bioconda/platypus-variant - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: pysam> - specs: - - https://anaconda.org/bioconda/pysam> - version: - - 0.13.0 - - package: strelka - specs: - - https://anaconda.org/bioconda/strelka - - package: vardict - specs: - - https://anaconda.org/bioconda/vardict - - package: vardict-java - specs: - - https://anaconda.org/bioconda/vardict-java - - package: varscan - specs: - - https://anaconda.org/bioconda/varscan - - package: moreutils - specs: - - https://anaconda.org/bioconda/moreutils - - package: vcfanno - specs: - - https://anaconda.org/bioconda/vcfanno - - package: vcflib - specs: - - https://anaconda.org/bioconda/vcflib - - package: vt - specs: - - https://anaconda.org/bioconda/vt - - package: r - specs: - - https://anaconda.org/bioconda/r - version: - - 3.4.1 - - package: r-base=3.4.1=h4fe35fd_8 - specs: - - https://anaconda.org/bioconda/r-base=3.4.1=h4fe35fd_8 - - package: perl - specs: - - https://anaconda.org/bioconda/perl -- class: arv:APIRequirement -- class: arv:RuntimeConstraints - keep_cache: 4096 -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: region_block_toolinput - type: - items: string - type: array -outputs: -- id: vrn_file_region - secondaryFiles: - - .tbi - type: - - File - - 'null' -- id: region_block - type: - items: string - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/wf-alignment.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/wf-alignment.cwl deleted file mode 100644 index 27c76c747c5..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/wf-alignment.cwl +++ /dev/null @@ -1,143 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: - - string - - 'null' - - boolean - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: rgnames__pu - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: alignment_rec - type: record -outputs: -- id: align_bam - outputSource: merge_split_alignments/align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__disc - outputSource: merge_split_alignments/work_bam_plus__disc - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__sr - outputSource: merge_split_alignments/work_bam_plus__sr - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: hla__fastq - outputSource: merge_split_alignments/hla__fastq - type: - - 'null' - - items: File - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: prep_align_inputs - in: - - id: alignment_rec - source: alignment_rec - out: - - id: process_alignment_rec - run: steps/prep_align_inputs.cwl -- id: process_alignment - in: - - id: alignment_rec - source: alignment_rec - - id: process_alignment_rec - source: prep_align_inputs/process_alignment_rec - out: - - id: work_bam - - id: align_bam - - id: hla__fastq - - id: work_bam_plus__disc - - id: work_bam_plus__sr - run: steps/process_alignment.cwl - scatter: - - process_alignment_rec - scatterMethod: dotproduct -- id: merge_split_alignments - in: - - id: alignment_rec - source: alignment_rec - - id: work_bam - source: process_alignment/work_bam - - id: align_bam_toolinput - source: process_alignment/align_bam - - id: work_bam_plus__disc_toolinput - source: process_alignment/work_bam_plus__disc - - id: work_bam_plus__sr_toolinput - source: process_alignment/work_bam_plus__sr - - id: hla__fastq_toolinput - source: process_alignment/hla__fastq - out: - - id: align_bam - - id: work_bam_plus__disc - - id: work_bam_plus__sr - - id: hla__fastq - run: steps/merge_split_alignments.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/wf-jointcall.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/wf-jointcall.cwl deleted file mode 100644 index 3091d1de370..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/wf-jointcall.cwl +++ /dev/null @@ -1,360 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: jointvc_batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: jointvc_batch_rec - type: record - type: array -outputs: -- id: jointvc_rec - outputSource: finalize_jointvc/jointvc_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - - name: vrn_file_joint - type: File - name: jointvc_rec - type: record - type: array -- id: vrn_file_joint - outputSource: postprocess_variants/vrn_file_joint - secondaryFiles: - - .tbi - type: File -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: get_parallel_regions_jointvc - in: - - id: jointvc_batch_rec - source: jointvc_batch_rec - out: - - id: region - run: steps/get_parallel_regions_jointvc.cwl -- id: run_jointvc - in: - - id: jointvc_batch_rec - source: jointvc_batch_rec - - id: region_toolinput - source: get_parallel_regions_jointvc/region - out: - - id: vrn_file_region - - id: region - run: steps/run_jointvc.cwl - scatter: - - region_toolinput - scatterMethod: dotproduct -- id: concat_batch_variantcalls_jointvc - in: - - id: jointvc_batch_rec - source: jointvc_batch_rec - - id: region - source: run_jointvc/region - - id: vrn_file_region - source: run_jointvc/vrn_file_region - out: - - id: vrn_file_joint - run: steps/concat_batch_variantcalls_jointvc.cwl -- id: postprocess_variants - in: - - id: jointvc_batch_rec - source: jointvc_batch_rec - - id: vrn_file_joint_toolinput - source: concat_batch_variantcalls_jointvc/vrn_file_joint - out: - - id: vrn_file_joint - run: steps/postprocess_variants.cwl -- id: finalize_jointvc - in: - - id: jointvc_batch_rec - source: jointvc_batch_rec - - id: vrn_file_joint - source: postprocess_variants/vrn_file_joint - out: - - id: jointvc_rec - run: steps/finalize_jointvc.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/wf-variantcall.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/wf-variantcall.cwl deleted file mode 100644 index ffabd7e9f1e..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/gvcf-joint-workflow/wf-variantcall.cwl +++ /dev/null @@ -1,329 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -outputs: -- id: vc_rec - outputSource: compare_to_rm/vc_rec - type: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - File - - 'null' - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: get_parallel_regions - in: - - id: batch_rec - source: batch_rec - out: - - id: region_block - run: steps/get_parallel_regions.cwl -- id: variantcall_batch_region - in: - - id: batch_rec - source: batch_rec - - id: region_block_toolinput - source: get_parallel_regions/region_block - out: - - id: vrn_file_region - - id: region_block - run: steps/variantcall_batch_region.cwl - scatter: - - region_block_toolinput - scatterMethod: dotproduct -- id: concat_batch_variantcalls - in: - - id: batch_rec - source: batch_rec - - id: region_block - source: variantcall_batch_region/region_block - - id: vrn_file_region - source: variantcall_batch_region/vrn_file_region - out: - - id: vrn_file - run: steps/concat_batch_variantcalls.cwl -- id: compare_to_rm - in: - - id: batch_rec - source: batch_rec - - id: vrn_file - source: concat_batch_variantcalls/vrn_file - out: - - id: vc_rec - run: steps/compare_to_rm.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/main-prealign-samples.json b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/main-prealign-samples.json deleted file mode 100644 index 3e1234addda..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/main-prealign-samples.json +++ /dev/null @@ -1,526 +0,0 @@ -{ - "analysis": [ - "variant2", - "variant2" - ], - "config__algorithm__coverage": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/automated/coverage_transcripts-bam.bed" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/automated/coverage_transcripts-bam.bed" - } - ], - "config__algorithm__coverage_interval": [ - null, - null - ], - "config__algorithm__effects": [ - "snpeff", - "snpeff" - ], - "config__algorithm__ensemble": [ - null, - null - ], - "config__algorithm__exclude_regions": [ - [], - [] - ], - "config__algorithm__min_allele_fraction": [ - 10.0, - 10.0 - ], - "config__algorithm__nomap_split_size": [ - 250, - 250 - ], - "config__algorithm__nomap_split_targets": [ - 20, - 20 - ], - "config__algorithm__qc": [ - [ - "contamination", - "coverage", - "fastqc", - "peddy", - "picard", - "qsignature", - "qualimap", - "samtools", - "variants" - ], - [ - "contamination", - "coverage", - "fastqc", - "peddy", - "picard", - "qsignature", - "qualimap", - "samtools", - "variants" - ] - ], - "config__algorithm__recalibrate": [ - "False", - "False" - ], - "config__algorithm__tools_off": [ - [], - [] - ], - "config__algorithm__tools_on": [ - [ - "qualimap_full" - ], - [ - "qualimap_full" - ] - ], - "config__algorithm__validate": [ - null, - null - ], - "config__algorithm__validate_regions": [ - null, - null - ], - "config__algorithm__variant_regions": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed" - } - ], - "config__algorithm__variantcaller": [ - "False", - "False" - ], - "config__algorithm__vcfanno": [ - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/config/vcfanno/gemini.conf" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/config/vcfanno/gemini.lua" - } - ], - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/config/vcfanno/gemini.conf" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/config/vcfanno/gemini.lua" - } - ] - ], - "description": [ - "Test1", - "Test2" - ], - "files": [ - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam" - } - ], - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX_2.bam" - } - ] - ], - "genome_build": [ - "hg19", - "hg19" - ], - "genome_resources__aliases__ensembl": [ - "homo_sapiens_vep_83_GRCh37", - "homo_sapiens_vep_83_GRCh37" - ], - "genome_resources__aliases__human": [ - "True", - "True" - ], - "genome_resources__aliases__snpeff": [ - "hg19", - "hg19" - ], - "genome_resources__rnaseq__gene_bed": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rnaseq/ref-transcripts.bed" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rnaseq/ref-transcripts.bed" - } - ], - "genome_resources__variation__1000g": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__clinvar": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__cosmic": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__dbsnp": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__encode_blacklist": [ - null, - null - ], - "genome_resources__variation__esp": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__exac": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__gnomad_exome": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__lcr": [ - null, - null - ], - "genome_resources__variation__polyx": [ - null, - null - ], - "genome_resources__variation__train_hapmap": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__train_indels": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz.tbi" - } - ] - } - ], - "metadata__batch": [ - null, - null - ], - "metadata__phenotype": [ - "", - "" - ], - "reference__fasta__base": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa.fai" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.dict" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa.fai" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.dict" - } - ] - } - ], - "reference__genome_context": [ - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz.tbi" - } - ] - } - ], - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz.tbi" - } - ] - } - ] - ], - "reference__rtg": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rtg--hg19.sdf-wf.tar.gz" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rtg--hg19.sdf-wf.tar.gz" - } - ], - "reference__snpeff__hg19": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/snpeff--hg19-wf.tar.gz" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/snpeff--hg19-wf.tar.gz" - } - ], - "reference__versions": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/versions.csv" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/versions.csv" - } - ], - "resources": [ - "{}", - "{}" - ], - "rgnames__sample": [ - "Test1", - "Test2" - ], - "vrn_file": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/Test1-gatk-haplotype.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/Test1-gatk-haplotype.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/Test2-gatk-haplotype.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/Test2-gatk-haplotype.vcf.gz.tbi" - } - ] - } - ] -} diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/main-prealign.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/main-prealign.cwl deleted file mode 100644 index 535082448ed..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/main-prealign.cwl +++ /dev/null @@ -1,672 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: files - type: - items: - items: File - type: array - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: config__algorithm__vcfanno - type: - items: - items: File - type: array - type: array -- id: resources - type: - items: string - type: array -- id: config__algorithm__variantcaller - type: - items: - - string - - 'null' - - boolean - type: array -- id: genome_resources__variation__1000g - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__coverage_interval - type: - items: - - 'null' - - string - type: array -- id: genome_resources__rnaseq__gene_bed - type: - items: File - type: array -- id: genome_resources__variation__clinvar - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__esp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: metadata__batch - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__lcr - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__min_allele_fraction - type: - items: double - type: array -- id: config__algorithm__nomap_split_targets - type: - items: long - type: array -- id: reference__versions - type: - items: File - type: array -- id: vrn_file - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__train_hapmap - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: reference__genome_context - secondaryFiles: - - .tbi - type: - items: - items: File - type: array - type: array -- id: config__algorithm__nomap_split_size - type: - items: long - type: array -- id: config__algorithm__validate - type: - items: - - 'null' - - string - type: array -- id: reference__snpeff__hg19 - type: - items: File - type: array -- id: description - type: - items: string - type: array -- id: config__algorithm__validate_regions - type: - items: - - 'null' - - string - type: array -- id: genome_build - type: - items: string - type: array -- id: genome_resources__variation__exac - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__gnomad_exome - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__recalibrate - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__coverage - type: - items: File - type: array -- id: genome_resources__aliases__human - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: genome_resources__variation__dbsnp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__polyx - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__encode_blacklist - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__cosmic - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__ensemble - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__qc - type: - items: - items: string - type: array - type: array -- id: analysis - type: - items: string - type: array -- id: rgnames__sample - type: - items: string - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: config__algorithm__effects - type: - items: string - type: array -- id: config__algorithm__variant_regions - type: - items: File - type: array -- id: genome_resources__aliases__ensembl - type: - items: string - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: reference__rtg - type: - items: File - type: array -- id: genome_resources__variation__train_indels - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__aliases__snpeff - type: - items: string - type: array -outputs: -- id: rgnames__sample_out - outputSource: prep_samples/rgnames__sample - type: - items: string - type: array -- id: align_bam - outputSource: postprocess_alignment/align_bam - type: - items: - - File - - 'null' - type: array -- id: regions__sample_callable - outputSource: postprocess_alignment/regions__sample_callable - type: - items: - - File - - 'null' - type: array -- id: validate__grading_summary - outputSource: summarize_vc/validate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: variants__calls - outputSource: summarize_vc/variants__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: variants__gvcf - outputSource: summarize_vc/variants__gvcf - type: - items: - - 'null' - - items: - - File - - 'null' - type: array - type: array -- id: summary__multiqc - outputSource: multiqc_summary/summary__multiqc - type: - items: - - File - - 'null' - type: array -- id: versions__tools - outputSource: multiqc_summary/versions__tools - type: - items: - - File - - 'null' - type: array -- id: versions__data - outputSource: multiqc_summary/versions__data - type: - items: - - File - - 'null' - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: organize_noalign - in: - - id: files - source: files - - id: resources - source: resources - - id: description - source: description - out: - - id: align_bam - - id: work_bam_plus__disc - - id: work_bam_plus__sr - - id: hla__fastq - run: steps/organize_noalign.cwl - scatter: - - files - - resources - - description - scatterMethod: dotproduct -- id: prep_samples_to_rec - in: - - id: config__algorithm__coverage - source: config__algorithm__coverage - - id: rgnames__sample - source: rgnames__sample - - id: config__algorithm__variant_regions - source: config__algorithm__variant_regions - - id: reference__fasta__base - source: reference__fasta__base - - id: resources - source: resources - - id: description - source: description - out: - - id: prep_samples_rec - run: steps/prep_samples_to_rec.cwl -- id: prep_samples - in: - - id: prep_samples_rec - source: prep_samples_to_rec/prep_samples_rec - out: - - id: rgnames__sample - - id: config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - - id: config__algorithm__variant_regions_orig - - id: config__algorithm__coverage - - id: config__algorithm__coverage_merged - - id: config__algorithm__coverage_orig - - id: config__algorithm__seq2c_bed_ready - run: steps/prep_samples.cwl - scatter: - - prep_samples_rec - scatterMethod: dotproduct -- id: postprocess_alignment_to_rec - in: - - id: align_bam - source: organize_noalign/align_bam - - id: config__algorithm__coverage_interval - source: config__algorithm__coverage_interval - - id: config__algorithm__exclude_regions - source: config__algorithm__exclude_regions - - id: config__algorithm__variant_regions - source: prep_samples/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: prep_samples/config__algorithm__variant_regions_merged - - id: config__algorithm__variant_regions_orig - source: prep_samples/config__algorithm__variant_regions_orig - - id: config__algorithm__coverage - source: prep_samples/config__algorithm__coverage - - id: config__algorithm__coverage_merged - source: prep_samples/config__algorithm__coverage_merged - - id: config__algorithm__coverage_orig - source: prep_samples/config__algorithm__coverage_orig - - id: config__algorithm__seq2c_bed_ready - source: prep_samples/config__algorithm__seq2c_bed_ready - - id: config__algorithm__recalibrate - source: config__algorithm__recalibrate - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: genome_resources__rnaseq__gene_bed - source: genome_resources__rnaseq__gene_bed - - id: genome_resources__variation__dbsnp - source: genome_resources__variation__dbsnp - - id: genome_resources__variation__lcr - source: genome_resources__variation__lcr - - id: genome_resources__variation__polyx - source: genome_resources__variation__polyx - - id: genome_resources__variation__encode_blacklist - source: genome_resources__variation__encode_blacklist - - id: reference__fasta__base - source: reference__fasta__base - - id: resources - source: resources - - id: description - source: description - out: - - id: postprocess_alignment_rec - run: steps/postprocess_alignment_to_rec.cwl -- id: postprocess_alignment - in: - - id: postprocess_alignment_rec - source: postprocess_alignment_to_rec/postprocess_alignment_rec - out: - - id: config__algorithm__coverage_interval - - id: config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - - id: config__algorithm__variant_regions_orig - - id: config__algorithm__coverage - - id: config__algorithm__coverage_merged - - id: config__algorithm__coverage_orig - - id: config__algorithm__seq2c_bed_ready - - id: regions__callable - - id: regions__sample_callable - - id: regions__nblock - - id: depth__samtools__stats - - id: depth__samtools__idxstats - - id: depth__variant_regions__regions - - id: depth__variant_regions__dist - - id: depth__sv_regions__regions - - id: depth__sv_regions__dist - - id: depth__coverage__regions - - id: depth__coverage__dist - - id: depth__coverage__thresholds - - id: align_bam - run: steps/postprocess_alignment.cwl - scatter: - - postprocess_alignment_rec - scatterMethod: dotproduct -- id: combine_sample_regions - in: - - id: regions__callable - source: postprocess_alignment/regions__callable - - id: regions__nblock - source: postprocess_alignment/regions__nblock - - id: metadata__batch - source: metadata__batch - - id: config__algorithm__nomap_split_size - source: config__algorithm__nomap_split_size - - id: config__algorithm__nomap_split_targets - source: config__algorithm__nomap_split_targets - - id: reference__fasta__base - source: reference__fasta__base - - id: resources - source: resources - - id: description - source: description - out: - - id: config__algorithm__callable_regions - - id: config__algorithm__non_callable_regions - - id: config__algorithm__callable_count - run: steps/combine_sample_regions.cwl -- id: batch_for_variantcall - in: - - id: analysis - source: analysis - - id: genome_build - source: genome_build - - id: align_bam - source: postprocess_alignment/align_bam - - id: vrn_file - source: vrn_file - - id: metadata__batch - source: metadata__batch - - id: metadata__phenotype - source: metadata__phenotype - - id: config__algorithm__callable_regions - source: combine_sample_regions/config__algorithm__callable_regions - - id: regions__sample_callable - source: postprocess_alignment/regions__sample_callable - - id: config__algorithm__variantcaller - source: config__algorithm__variantcaller - - id: config__algorithm__ensemble - source: config__algorithm__ensemble - - id: config__algorithm__vcfanno - source: config__algorithm__vcfanno - - id: config__algorithm__coverage_interval - source: postprocess_alignment/config__algorithm__coverage_interval - - id: config__algorithm__effects - source: config__algorithm__effects - - id: config__algorithm__min_allele_fraction - source: config__algorithm__min_allele_fraction - - id: config__algorithm__exclude_regions - source: config__algorithm__exclude_regions - - id: config__algorithm__variant_regions - source: postprocess_alignment/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: postprocess_alignment/config__algorithm__variant_regions_merged - - id: config__algorithm__validate - source: config__algorithm__validate - - id: config__algorithm__validate_regions - source: config__algorithm__validate_regions - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: config__algorithm__tools_off - source: config__algorithm__tools_off - - id: reference__fasta__base - source: reference__fasta__base - - id: reference__rtg - source: reference__rtg - - id: reference__genome_context - source: reference__genome_context - - id: genome_resources__variation__clinvar - source: genome_resources__variation__clinvar - - id: genome_resources__variation__cosmic - source: genome_resources__variation__cosmic - - id: genome_resources__variation__dbsnp - source: genome_resources__variation__dbsnp - - id: genome_resources__variation__esp - source: genome_resources__variation__esp - - id: genome_resources__variation__exac - source: genome_resources__variation__exac - - id: genome_resources__variation__gnomad_exome - source: genome_resources__variation__gnomad_exome - - id: genome_resources__variation__1000g - source: genome_resources__variation__1000g - - id: genome_resources__variation__lcr - source: genome_resources__variation__lcr - - id: genome_resources__variation__polyx - source: genome_resources__variation__polyx - - id: genome_resources__variation__encode_blacklist - source: genome_resources__variation__encode_blacklist - - id: genome_resources__aliases__ensembl - source: genome_resources__aliases__ensembl - - id: genome_resources__aliases__human - source: genome_resources__aliases__human - - id: genome_resources__aliases__snpeff - source: genome_resources__aliases__snpeff - - id: reference__snpeff__hg19 - source: reference__snpeff__hg19 - - id: genome_resources__variation__train_hapmap - source: genome_resources__variation__train_hapmap - - id: genome_resources__variation__train_indels - source: genome_resources__variation__train_indels - - id: resources - source: resources - - id: description - source: description - out: - - id: batch_rec - run: steps/batch_for_variantcall.cwl -- id: variantcall - in: - - id: batch_rec - source: batch_for_variantcall/batch_rec - out: - - id: vc_rec - run: wf-variantcall.cwl - scatter: - - batch_rec - scatterMethod: dotproduct -- id: summarize_vc - in: - - id: vc_rec - source: variantcall/vc_rec - out: - - id: variants__calls - - id: variants__gvcf - - id: variants__samples - - id: validate__grading_summary - - id: validate__grading_plots - run: steps/summarize_vc.cwl -- id: qc_to_rec - in: - - id: align_bam - source: postprocess_alignment/align_bam - - id: analysis - source: analysis - - id: reference__fasta__base - source: reference__fasta__base - - id: reference__versions - source: reference__versions - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: config__algorithm__tools_off - source: config__algorithm__tools_off - - id: genome_build - source: genome_build - - id: config__algorithm__qc - source: config__algorithm__qc - - id: metadata__batch - source: metadata__batch - - id: metadata__phenotype - source: metadata__phenotype - - id: config__algorithm__coverage_interval - source: postprocess_alignment/config__algorithm__coverage_interval - - id: depth__variant_regions__regions - source: postprocess_alignment/depth__variant_regions__regions - - id: depth__variant_regions__dist - source: postprocess_alignment/depth__variant_regions__dist - - id: depth__samtools__stats - source: postprocess_alignment/depth__samtools__stats - - id: depth__samtools__idxstats - source: postprocess_alignment/depth__samtools__idxstats - - id: depth__sv_regions__regions - source: postprocess_alignment/depth__sv_regions__regions - - id: depth__sv_regions__dist - source: postprocess_alignment/depth__sv_regions__dist - - id: depth__coverage__regions - source: postprocess_alignment/depth__coverage__regions - - id: depth__coverage__dist - source: postprocess_alignment/depth__coverage__dist - - id: depth__coverage__thresholds - source: postprocess_alignment/depth__coverage__thresholds - - id: config__algorithm__variant_regions - source: postprocess_alignment/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: postprocess_alignment/config__algorithm__variant_regions_merged - - id: config__algorithm__coverage - source: postprocess_alignment/config__algorithm__coverage - - id: config__algorithm__coverage_merged - source: postprocess_alignment/config__algorithm__coverage_merged - - id: variants__samples - source: summarize_vc/variants__samples - - id: resources - source: resources - - id: description - source: description - out: - - id: qc_rec - run: steps/qc_to_rec.cwl -- id: pipeline_summary - in: - - id: qc_rec - source: qc_to_rec/qc_rec - out: - - id: qcout_rec - run: steps/pipeline_summary.cwl - scatter: - - qc_rec - scatterMethod: dotproduct -- id: multiqc_summary - in: - - id: qcout_rec - source: pipeline_summary/qcout_rec - out: - - id: summary__multiqc - - id: versions__tools - - id: versions__data - run: steps/multiqc_summary.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/batch_for_variantcall.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/batch_for_variantcall.cwl deleted file mode 100644 index 1ad9465444b..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/batch_for_variantcall.cwl +++ /dev/null @@ -1,407 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-batch -- sentinel_outputs=batch_rec:resources;description;reference__fasta__base;metadata__phenotype;config__algorithm__vcfanno;config__algorithm__variantcaller;genome_resources__variation__1000g;config__algorithm__coverage_interval;genome_resources__variation__clinvar;genome_resources__variation__esp;metadata__batch;genome_resources__variation__lcr;config__algorithm__min_allele_fraction;vrn_file;genome_resources__variation__train_hapmap;reference__genome_context;config__algorithm__validate;reference__snpeff__hg19;config__algorithm__validate_regions;genome_build;genome_resources__variation__exac;genome_resources__variation__gnomad_exome;genome_resources__aliases__human;config__algorithm__tools_off;genome_resources__variation__dbsnp;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;genome_resources__variation__cosmic;config__algorithm__ensemble;analysis;config__algorithm__tools_on;config__algorithm__effects;config__algorithm__variant_regions;genome_resources__aliases__ensembl;config__algorithm__exclude_regions;reference__rtg;genome_resources__variation__train_indels;genome_resources__aliases__snpeff;align_bam;config__algorithm__variant_regions_merged;regions__sample_callable;config__algorithm__callable_regions -- sentinel_inputs=analysis:var,genome_build:var,align_bam:var,vrn_file:var,metadata__batch:var,metadata__phenotype:var,config__algorithm__callable_regions:var,regions__sample_callable:var,config__algorithm__variantcaller:var,config__algorithm__ensemble:var,config__algorithm__vcfanno:var,config__algorithm__coverage_interval:var,config__algorithm__effects:var,config__algorithm__min_allele_fraction:var,config__algorithm__exclude_regions:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__validate:var,config__algorithm__validate_regions:var,config__algorithm__tools_on:var,config__algorithm__tools_off:var,reference__fasta__base:var,reference__rtg:var,reference__genome_context:var,genome_resources__variation__clinvar:var,genome_resources__variation__cosmic:var,genome_resources__variation__dbsnp:var,genome_resources__variation__esp:var,genome_resources__variation__exac:var,genome_resources__variation__gnomad_exome:var,genome_resources__variation__1000g:var,genome_resources__variation__lcr:var,genome_resources__variation__polyx:var,genome_resources__variation__encode_blacklist:var,genome_resources__aliases__ensembl:var,genome_resources__aliases__human:var,genome_resources__aliases__snpeff:var,reference__snpeff__hg19:var,genome_resources__variation__train_hapmap:var,genome_resources__variation__train_indels:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- batch_for_variantcall -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1030 - ramMin: 3072 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: analysis - type: - items: string - type: array -- id: genome_build - type: - items: string - type: array -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: vrn_file - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: metadata__batch - type: - items: - - 'null' - - string - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: config__algorithm__callable_regions - type: - items: File - type: array -- id: regions__sample_callable - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variantcaller - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__ensemble - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__vcfanno - type: - items: - items: File - type: array - type: array -- id: config__algorithm__coverage_interval - type: - items: - - string - - 'null' - type: array -- id: config__algorithm__effects - type: - items: string - type: array -- id: config__algorithm__min_allele_fraction - type: - items: double - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__validate - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__validate_regions - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: reference__rtg - type: - items: File - type: array -- id: reference__genome_context - secondaryFiles: - - .tbi - type: - items: - items: File - type: array - type: array -- id: genome_resources__variation__clinvar - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__cosmic - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__dbsnp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__esp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__exac - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__gnomad_exome - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__1000g - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__lcr - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__polyx - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__encode_blacklist - type: - items: - - 'null' - - string - type: array -- id: genome_resources__aliases__ensembl - type: - items: string - type: array -- id: genome_resources__aliases__human - type: - items: - - string - - 'null' - - boolean - type: array -- id: genome_resources__aliases__snpeff - type: - items: string - type: array -- id: reference__snpeff__hg19 - type: - items: File - type: array -- id: genome_resources__variation__train_hapmap - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__train_indels - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: batch_rec - type: - items: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: - - boolean - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: - - 'null' - - string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: File - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/combine_sample_regions.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/combine_sample_regions.cwl deleted file mode 100644 index 2df313f7a8f..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/combine_sample_regions.cwl +++ /dev/null @@ -1,101 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=config__algorithm__callable_regions,config__algorithm__non_callable_regions,config__algorithm__callable_count -- sentinel_inputs=regions__callable:var,regions__nblock:var,metadata__batch:var,config__algorithm__nomap_split_size:var,config__algorithm__nomap_split_targets:var,reference__fasta__base:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- combine_sample_regions -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1025 - ramMin: 3072 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 -- class: arv:APIRequirement -inputs: -- id: regions__callable - type: - items: - - File - - 'null' - type: array -- id: regions__nblock - type: - items: - - File - - 'null' - type: array -- id: metadata__batch - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__nomap_split_size - type: - items: long - type: array -- id: config__algorithm__nomap_split_targets - type: - items: long - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: config__algorithm__callable_regions - type: - items: File - type: array -- id: config__algorithm__non_callable_regions - type: - items: File - type: array -- id: config__algorithm__callable_count - type: - items: int - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/compare_to_rm.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/compare_to_rm.cwl deleted file mode 100644 index ab4909659ca..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/compare_to_rm.cwl +++ /dev/null @@ -1,315 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-single -- sentinel_outputs=vc_rec:batch_samples;validate__summary;validate__tp;validate__fp;validate__fn;resources;description;vrn_file;reference__fasta__base;metadata__phenotype;config__algorithm__vcfanno;config__algorithm__variantcaller;config__algorithm__coverage_interval;metadata__batch;config__algorithm__min_allele_fraction;reference__genome_context;config__algorithm__validate;reference__snpeff__hg19;config__algorithm__validate_regions;genome_build;genome_resources__aliases__human;config__algorithm__tools_off;config__algorithm__ensemble;analysis;config__algorithm__tools_on;config__algorithm__effects;config__algorithm__variant_regions;genome_resources__aliases__ensembl;config__algorithm__exclude_regions;reference__rtg;genome_resources__aliases__snpeff;config__algorithm__variant_regions_merged;regions__sample_callable;config__algorithm__callable_regions -- sentinel_inputs=batch_rec:record,vrn_file:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- compare_to_rm -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1028 - ramMin: 6144 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: gvcf-regions - specs: - - https://anaconda.org/bioconda/gvcf-regions - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: rtg-tools - specs: - - https://anaconda.org/bioconda/rtg-tools - - package: vcfanno - specs: - - https://anaconda.org/bioconda/vcfanno -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: - - boolean - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: - - 'null' - - string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: File - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: vrn_file - secondaryFiles: - - .tbi - type: File -outputs: -- id: vc_rec - type: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: - - boolean - - 'null' - - string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/concat_batch_variantcalls.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/concat_batch_variantcalls.cwl deleted file mode 100644 index ae12d21651d..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/concat_batch_variantcalls.cwl +++ /dev/null @@ -1,199 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-merge -- sentinel_outputs=vrn_file -- sentinel_inputs=batch_rec:record,region_block:var,vrn_file_region:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- concat_batch_variantcalls -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 3072 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 -- class: arv:APIRequirement -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: - - boolean - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: - - 'null' - - string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: File - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: region_block - type: - items: - items: string - type: array - type: array -- id: vrn_file_region - secondaryFiles: - - .tbi - type: - items: - - File - - 'null' - type: array -outputs: -- id: vrn_file - secondaryFiles: - - .tbi - type: File -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/get_parallel_regions.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/get_parallel_regions.cwl deleted file mode 100644 index c4e3997bca4..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/get_parallel_regions.cwl +++ /dev/null @@ -1,174 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-split -- sentinel_outputs=region_block -- sentinel_inputs=batch_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- get_parallel_regions -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1030 - ramMin: 3072 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: - - boolean - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: - - 'null' - - string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: File - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -outputs: -- id: region_block - type: - items: - items: string - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/multiqc_summary.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/multiqc_summary.cwl deleted file mode 100644 index 82ce9f97a43..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/multiqc_summary.cwl +++ /dev/null @@ -1,95 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=summary__multiqc,versions__tools,versions__data -- sentinel_inputs=qcout_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- multiqc_summary -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1030 - ramMin: 3072 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: multiqc - specs: - - https://anaconda.org/bioconda/multiqc - - package: multiqc-bcbio - specs: - - https://anaconda.org/bioconda/multiqc-bcbio -inputs: -- id: qcout_rec - type: - items: - fields: - - name: summary__qc - type: - - File - - 'null' - - name: summary__metrics - type: - - string - - 'null' - - name: description - type: string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: config__algorithm__tools_on - type: - items: string - type: array - name: qcout_rec - type: record - type: array -outputs: -- id: summary__multiqc - type: - items: - - File - - 'null' - type: array -- id: versions__tools - type: - items: - - File - - 'null' - type: array -- id: versions__data - type: - items: - - File - - 'null' - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/organize_noalign.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/organize_noalign.cwl deleted file mode 100644 index 370f2ccf982..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/organize_noalign.cwl +++ /dev/null @@ -1,59 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=align_bam,work_bam_plus__disc,work_bam_plus__sr,hla__fastq -- sentinel_inputs=files:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- organize_noalign -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1024 - ramMin: 3072 - tmpdirMin: 0 -- class: dx:InputResourceRequirement - indirMin: 4 -inputs: -- id: files - type: - items: File - type: array -- id: resources - type: string -- id: description - type: string -outputs: -- id: align_bam - secondaryFiles: - - .bai - type: File -- id: work_bam_plus__disc - type: - - File - - 'null' -- id: work_bam_plus__sr - type: - - File - - 'null' -- id: hla__fastq - type: - - File - - 'null' -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/pipeline_summary.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/pipeline_summary.cwl deleted file mode 100644 index c0e41e5ba62..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/pipeline_summary.cwl +++ /dev/null @@ -1,221 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=qcout_rec:summary__qc;summary__metrics;description;reference__versions;genome_build;config__algorithm__tools_off;config__algorithm__qc;config__algorithm__tools_on -- sentinel_inputs=qc_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- pipeline_summary -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1030 - ramMin: 6144 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: fastqc=0.11.7=5 - specs: - - https://anaconda.org/bioconda/fastqc=0.11.7=5 - - package: goleft - specs: - - https://anaconda.org/bioconda/goleft - - package: hts-nim-tools - specs: - - https://anaconda.org/bioconda/hts-nim-tools - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: picard - specs: - - https://anaconda.org/bioconda/picard - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: qsignature - specs: - - https://anaconda.org/bioconda/qsignature - - package: qualimap - specs: - - https://anaconda.org/bioconda/qualimap - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: preseq - specs: - - https://anaconda.org/bioconda/preseq - - package: peddy - specs: - - https://anaconda.org/bioconda/peddy - - package: verifybamid2 - specs: - - https://anaconda.org/bioconda/verifybamid2 -- class: arv:RuntimeConstraints - keep_cache: 4096 -inputs: -- id: qc_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: - - 'null' - - string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: depth__samtools__stats - type: - - File - - 'null' - - name: depth__samtools__idxstats - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: depth__variant_regions__dist - type: - - File - - 'null' - - name: depth__sv_regions__regions - type: - - File - - 'null' - - name: depth__sv_regions__dist - type: - - File - - 'null' - - name: depth__coverage__regions - type: - - File - - 'null' - - name: depth__coverage__dist - type: - - File - - 'null' - - name: depth__coverage__thresholds - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - name: qc_rec - type: record -outputs: -- id: qcout_rec - type: - fields: - - name: summary__qc - type: - - File - - 'null' - - name: summary__metrics - type: - - string - - 'null' - - name: description - type: string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: config__algorithm__tools_on - type: - items: string - type: array - name: qcout_rec - type: record -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/postprocess_alignment.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/postprocess_alignment.cwl deleted file mode 100644 index 6a261c205af..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/postprocess_alignment.cwl +++ /dev/null @@ -1,221 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=config__algorithm__coverage_interval,config__algorithm__variant_regions,config__algorithm__variant_regions_merged,config__algorithm__variant_regions_orig,config__algorithm__coverage,config__algorithm__coverage_merged,config__algorithm__coverage_orig,config__algorithm__seq2c_bed_ready,regions__callable,regions__sample_callable,regions__nblock,depth__samtools__stats,depth__samtools__idxstats,depth__variant_regions__regions,depth__variant_regions__dist,depth__sv_regions__regions,depth__sv_regions__dist,depth__coverage__regions,depth__coverage__dist,depth__coverage__thresholds,align_bam -- sentinel_inputs=postprocess_alignment_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- postprocess_alignment -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1033 - ramMin: 6144 - tmpdirMin: 5 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: goleft - specs: - - https://anaconda.org/bioconda/goleft - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: sentieon - specs: - - https://anaconda.org/bioconda/sentieon -- class: arv:APIRequirement -inputs: -- id: postprocess_alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - 'null' - - string - - name: genome_resources__rnaseq__gene_bed - type: File - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__recalibrate - type: - - string - - 'null' - - boolean - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: File - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__variant_regions_orig - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: config__algorithm__coverage_orig - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - name: postprocess_alignment_rec - type: record -outputs: -- id: config__algorithm__coverage_interval - type: - - string - - 'null' -- id: config__algorithm__variant_regions - type: - - File - - 'null' -- id: config__algorithm__variant_regions_merged - type: - - File - - 'null' -- id: config__algorithm__variant_regions_orig - type: - - File - - 'null' -- id: config__algorithm__coverage - type: - - File - - 'null' -- id: config__algorithm__coverage_merged - type: - - File - - 'null' -- id: config__algorithm__coverage_orig - type: - - File - - 'null' -- id: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' -- id: regions__callable - type: - - File - - 'null' -- id: regions__sample_callable - type: - - File - - 'null' -- id: regions__nblock - type: - - File - - 'null' -- id: depth__samtools__stats - type: - - File - - 'null' -- id: depth__samtools__idxstats - type: - - File - - 'null' -- id: depth__variant_regions__regions - type: - - File - - 'null' -- id: depth__variant_regions__dist - type: - - File - - 'null' -- id: depth__sv_regions__regions - type: - - File - - 'null' -- id: depth__sv_regions__dist - type: - - File - - 'null' -- id: depth__coverage__regions - type: - - File - - 'null' -- id: depth__coverage__dist - type: - - File - - 'null' -- id: depth__coverage__thresholds - type: - - File - - 'null' -- id: align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/postprocess_alignment_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/postprocess_alignment_to_rec.cwl deleted file mode 100644 index e61b71f4d28..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/postprocess_alignment_to_rec.cwl +++ /dev/null @@ -1,229 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=postprocess_alignment_rec:resources;description;reference__fasta__base;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;genome_resources__variation__lcr;config__algorithm__recalibrate;config__algorithm__coverage;genome_resources__variation__dbsnp;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__tools_on;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__variant_regions_orig;config__algorithm__coverage_merged;config__algorithm__coverage_orig;config__algorithm__seq2c_bed_ready -- sentinel_inputs=align_bam:var,config__algorithm__coverage_interval:var,config__algorithm__exclude_regions:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__variant_regions_orig:var,config__algorithm__coverage:var,config__algorithm__coverage_merged:var,config__algorithm__coverage_orig:var,config__algorithm__seq2c_bed_ready:var,config__algorithm__recalibrate:var,config__algorithm__tools_on:var,genome_resources__rnaseq__gene_bed:var,genome_resources__variation__dbsnp:var,genome_resources__variation__lcr:var,genome_resources__variation__polyx:var,genome_resources__variation__encode_blacklist:var,reference__fasta__base:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- postprocess_alignment_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 3072 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: align_bam - secondaryFiles: - - .bai - type: - items: File - type: array -- id: config__algorithm__coverage_interval - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_orig - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_orig - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__seq2c_bed_ready - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__recalibrate - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: genome_resources__rnaseq__gene_bed - type: - items: File - type: array -- id: genome_resources__variation__dbsnp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__lcr - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__polyx - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__encode_blacklist - type: - items: - - 'null' - - string - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: postprocess_alignment_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - 'null' - - string - - name: genome_resources__rnaseq__gene_bed - type: File - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__recalibrate - type: - - string - - 'null' - - boolean - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: File - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__variant_regions_orig - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: config__algorithm__coverage_orig - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - name: postprocess_alignment_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/postprocess_variants.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/postprocess_variants.cwl deleted file mode 100644 index dd7e67c7d61..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/postprocess_variants.cwl +++ /dev/null @@ -1,183 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-single -- sentinel_outputs=vrn_file -- sentinel_inputs=batch_rec:record,vrn_file:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- postprocess_variants -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1025 - ramMin: 6144 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: snpeff - specs: - - https://anaconda.org/bioconda/snpeff - version: - - 4.3.1t -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: - - boolean - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: - - 'null' - - string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: File - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: vrn_file_toolinput - secondaryFiles: - - .tbi - type: File -outputs: -- id: vrn_file - secondaryFiles: - - .tbi - type: File -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/prep_samples.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/prep_samples.cwl deleted file mode 100644 index f473b969345..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/prep_samples.cwl +++ /dev/null @@ -1,93 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=rgnames__sample,config__algorithm__variant_regions,config__algorithm__variant_regions_merged,config__algorithm__variant_regions_orig,config__algorithm__coverage,config__algorithm__coverage_merged,config__algorithm__coverage_orig,config__algorithm__seq2c_bed_ready -- sentinel_inputs=prep_samples_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- prep_samples -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1025 - ramMin: 3072 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy -inputs: -- id: prep_samples_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage - type: File - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: prep_samples_rec - type: record -outputs: -- id: rgnames__sample - type: string -- id: config__algorithm__variant_regions - type: - - File - - 'null' -- id: config__algorithm__variant_regions_merged - type: - - File - - 'null' -- id: config__algorithm__variant_regions_orig - type: - - File - - 'null' -- id: config__algorithm__coverage - type: - - File - - 'null' -- id: config__algorithm__coverage_merged - type: - - File - - 'null' -- id: config__algorithm__coverage_orig - type: - - File - - 'null' -- id: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/prep_samples_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/prep_samples_to_rec.cwl deleted file mode 100644 index 54f7781bb21..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/prep_samples_to_rec.cwl +++ /dev/null @@ -1,81 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=prep_samples_rec:resources;description;reference__fasta__base;config__algorithm__coverage;rgnames__sample;config__algorithm__variant_regions -- sentinel_inputs=config__algorithm__coverage:var,rgnames__sample:var,config__algorithm__variant_regions:var,reference__fasta__base:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- prep_samples_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1025 - ramMin: 3072 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: config__algorithm__coverage - type: - items: File - type: array -- id: rgnames__sample - type: - items: string - type: array -- id: config__algorithm__variant_regions - type: - items: File - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: prep_samples_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage - type: File - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: prep_samples_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/qc_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/qc_to_rec.cwl deleted file mode 100644 index 3de03a1a1a8..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/qc_to_rec.cwl +++ /dev/null @@ -1,299 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=qc_rec:resources;description;reference__fasta__base;metadata__phenotype;config__algorithm__coverage_interval;metadata__batch;reference__versions;genome_build;config__algorithm__coverage;config__algorithm__tools_off;config__algorithm__qc;analysis;config__algorithm__tools_on;config__algorithm__variant_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__coverage_merged;depth__samtools__stats;depth__samtools__idxstats;depth__variant_regions__regions;depth__variant_regions__dist;depth__sv_regions__regions;depth__sv_regions__dist;depth__coverage__regions;depth__coverage__dist;depth__coverage__thresholds;variants__samples -- sentinel_inputs=align_bam:var,analysis:var,reference__fasta__base:var,reference__versions:var,config__algorithm__tools_on:var,config__algorithm__tools_off:var,genome_build:var,config__algorithm__qc:var,metadata__batch:var,metadata__phenotype:var,config__algorithm__coverage_interval:var,depth__variant_regions__regions:var,depth__variant_regions__dist:var,depth__samtools__stats:var,depth__samtools__idxstats:var,depth__sv_regions__regions:var,depth__sv_regions__dist:var,depth__coverage__regions:var,depth__coverage__dist:var,depth__coverage__thresholds:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__coverage:var,config__algorithm__coverage_merged:var,variants__samples:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- qc_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 3072 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: analysis - type: - items: string - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: reference__versions - type: - items: File - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: genome_build - type: - items: string - type: array -- id: config__algorithm__qc - type: - items: - items: string - type: array - type: array -- id: metadata__batch - type: - items: - - 'null' - - string - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: config__algorithm__coverage_interval - type: - items: - - string - - 'null' - type: array -- id: depth__variant_regions__regions - type: - items: - - File - - 'null' - type: array -- id: depth__variant_regions__dist - type: - items: - - File - - 'null' - type: array -- id: depth__samtools__stats - type: - items: - - File - - 'null' - type: array -- id: depth__samtools__idxstats - type: - items: - - File - - 'null' - type: array -- id: depth__sv_regions__regions - type: - items: - - File - - 'null' - type: array -- id: depth__sv_regions__dist - type: - items: - - File - - 'null' - type: array -- id: depth__coverage__regions - type: - items: - - File - - 'null' - type: array -- id: depth__coverage__dist - type: - items: - - File - - 'null' - type: array -- id: depth__coverage__thresholds - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_merged - type: - items: - - File - - 'null' - type: array -- id: variants__samples - type: - items: - items: - items: - - File - - 'null' - type: array - type: array - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: qc_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: - - 'null' - - string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: depth__samtools__stats - type: - - File - - 'null' - - name: depth__samtools__idxstats - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: depth__variant_regions__dist - type: - - File - - 'null' - - name: depth__sv_regions__regions - type: - - File - - 'null' - - name: depth__sv_regions__dist - type: - - File - - 'null' - - name: depth__coverage__regions - type: - - File - - 'null' - - name: depth__coverage__dist - type: - - File - - 'null' - - name: depth__coverage__thresholds - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - name: qc_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/summarize_vc.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/summarize_vc.cwl deleted file mode 100644 index e9fde20378c..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/summarize_vc.cwl +++ /dev/null @@ -1,198 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=variants__calls,variants__gvcf,variants__samples,validate__grading_summary,validate__grading_plots -- sentinel_inputs=vc_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- summarize_vc -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1025 - ramMin: 3072 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 1 -inputs: -- id: vc_rec - type: - items: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: - - boolean - - 'null' - - string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array - type: array -outputs: -- id: variants__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: variants__gvcf - type: - items: - - 'null' - - items: - - File - - 'null' - type: array - type: array -- id: variants__samples - type: - items: - items: - items: - - File - - 'null' - type: array - type: array - type: array -- id: validate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: validate__grading_plots - type: - items: - items: - - File - - 'null' - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/variantcall_batch_region.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/variantcall_batch_region.cwl deleted file mode 100644 index e9192828e62..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/steps/variantcall_batch_region.cwl +++ /dev/null @@ -1,269 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-parallel -- sentinel_outputs=vrn_file_region,region_block -- sentinel_inputs=batch_rec:record,region_block:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- variantcall_batch_region -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1030 - ramMin: 6144 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: freebayes - specs: - - https://anaconda.org/bioconda/freebayes - version: - - 1.1.0.46 - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 - - package: vqsr_cnn - specs: - - https://anaconda.org/bioconda/vqsr_cnn - - package: deepvariant - specs: - - https://anaconda.org/bioconda/deepvariant - - package: sentieon - specs: - - https://anaconda.org/bioconda/sentieon - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: octopus - specs: - - https://anaconda.org/bioconda/octopus - - package: picard - specs: - - https://anaconda.org/bioconda/picard - - package: platypus-variant - specs: - - https://anaconda.org/bioconda/platypus-variant - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: pysam> - specs: - - https://anaconda.org/bioconda/pysam> - version: - - 0.13.0 - - package: strelka - specs: - - https://anaconda.org/bioconda/strelka - - package: vardict - specs: - - https://anaconda.org/bioconda/vardict - - package: vardict-java - specs: - - https://anaconda.org/bioconda/vardict-java - - package: varscan - specs: - - https://anaconda.org/bioconda/varscan - - package: moreutils - specs: - - https://anaconda.org/bioconda/moreutils - - package: vcfanno - specs: - - https://anaconda.org/bioconda/vcfanno - - package: vcflib - specs: - - https://anaconda.org/bioconda/vcflib - - package: vt - specs: - - https://anaconda.org/bioconda/vt - - package: r - specs: - - https://anaconda.org/bioconda/r - version: - - 3.4.1 - - package: r-base=3.4.1=h4fe35fd_8 - specs: - - https://anaconda.org/bioconda/r-base=3.4.1=h4fe35fd_8 - - package: perl - specs: - - https://anaconda.org/bioconda/perl -- class: arv:APIRequirement -- class: arv:RuntimeConstraints - keep_cache: 4096 -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: - - boolean - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: - - 'null' - - string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: File - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: region_block_toolinput - type: - items: string - type: array -outputs: -- id: vrn_file_region - secondaryFiles: - - .tbi - type: - - File - - 'null' -- id: region_block - type: - items: string - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/wf-variantcall.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/wf-variantcall.cwl deleted file mode 100644 index b48a8254984..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/prealign-workflow/wf-variantcall.cwl +++ /dev/null @@ -1,316 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: - - boolean - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: - - 'null' - - string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: File - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -outputs: -- id: vc_rec - outputSource: compare_to_rm/vc_rec - type: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: metadata__phenotype - type: string - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: - - boolean - - 'null' - - string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: - - 'null' - - string - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: get_parallel_regions - in: - - id: batch_rec - source: batch_rec - out: - - id: region_block - run: steps/get_parallel_regions.cwl -- id: variantcall_batch_region - in: - - id: batch_rec - source: batch_rec - - id: region_block_toolinput - source: get_parallel_regions/region_block - out: - - id: vrn_file_region - - id: region_block - run: steps/variantcall_batch_region.cwl - scatter: - - region_block_toolinput - scatterMethod: dotproduct -- id: concat_batch_variantcalls - in: - - id: batch_rec - source: batch_rec - - id: region_block - source: variantcall_batch_region/region_block - - id: vrn_file_region - source: variantcall_batch_region/vrn_file_region - out: - - id: vrn_file - run: steps/concat_batch_variantcalls.cwl -- id: postprocess_variants - in: - - id: batch_rec - source: batch_rec - - id: vrn_file_toolinput - source: concat_batch_variantcalls/vrn_file - out: - - id: vrn_file - run: steps/postprocess_variants.cwl -- id: compare_to_rm - in: - - id: batch_rec - source: batch_rec - - id: vrn_file - source: postprocess_variants/vrn_file - out: - - id: vc_rec - run: steps/compare_to_rm.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/main-rnaseq-samples.json b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/main-rnaseq-samples.json deleted file mode 100644 index 3e67d52727e..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/main-rnaseq-samples.json +++ /dev/null @@ -1,111 +0,0 @@ -{ - "analysis": [ - "RNA-seq" - ], - "config__algorithm__aligner": [ - "hisat2" - ], - "config__algorithm__coverage_interval": [ - null - ], - "config__algorithm__expression_caller": [ - [ - "salmon", - "kallisto" - ] - ], - "config__algorithm__fusion_caller": [ - [ - "pizzly" - ] - ], - "config__algorithm__qc": [ - [ - "fastqc", - "qualimap_rnaseq", - "samtools" - ] - ], - "config__algorithm__quality_format": [ - "standard" - ], - "config__algorithm__tools_off": [ - [] - ], - "config__algorithm__tools_on": [ - [] - ], - "description": [ - "Test1" - ], - "files": [ - [ - { - "class": "File", - "path": "gs://bcbio-cromwell-dev-test/test_bcbio_cwl/testdata/test_fusion/fusion-1_1.fq.gz" - }, - { - "class": "File", - "path": "gs://bcbio-cromwell-dev-test/test_bcbio_cwl/testdata/test_fusion/fusion-1_2.fq.gz" - } - ] - ], - "genome_build": [ - "hg19" - ], - "genome_resources__rnaseq__transcripts": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rnaseq/ref-transcripts.gtf", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rnaseq/ref-transcripts.gtf.db" - } - ] - } - ], - "reference__fasta__base": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa.fai" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.dict" - } - ] - } - ], - "reference__hisat2__indexes": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/hisat2-wf.tar.gz" - } - ], - "resources": [ - "{}" - ], - "rgnames__lane": [ - "Test1" - ], - "rgnames__lb": [ - null - ], - "rgnames__pl": [ - "illumina" - ], - "rgnames__pu": [ - "Test1" - ], - "rgnames__rg": [ - "Test1" - ], - "rgnames__sample": [ - "Test1" - ] -} diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/main-rnaseq.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/main-rnaseq.cwl deleted file mode 100644 index 2dd258ac786..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/main-rnaseq.cwl +++ /dev/null @@ -1,304 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: files - type: - items: - items: File - type: array - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: config__algorithm__expression_caller - type: - items: - items: string - type: array - type: array -- id: resources - type: - items: string - type: array -- id: config__algorithm__coverage_interval - type: - items: - - 'null' - - string - type: array -- id: rgnames__lb - type: - items: - - 'null' - - string - type: array -- id: rgnames__rg - type: - items: string - type: array -- id: reference__hisat2__indexes - type: - items: File - type: array -- id: config__algorithm__fusion_caller - type: - items: - items: string - type: array - type: array -- id: description - type: - items: string - type: array -- id: config__algorithm__aligner - type: - items: string - type: array -- id: rgnames__pl - type: - items: string - type: array -- id: genome_build - type: - items: string - type: array -- id: rgnames__pu - type: - items: string - type: array -- id: genome_resources__rnaseq__transcripts - secondaryFiles: - - .db - type: - items: File - type: array -- id: config__algorithm__quality_format - type: - items: string - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__qc - type: - items: - items: string - type: array - type: array -- id: analysis - type: - items: string - type: array -- id: rgnames__sample - type: - items: string - type: array -- id: config__algorithm__tools_on - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: rgnames__lane - type: - items: string - type: array -outputs: -- id: rgnames__sample_out - outputSource: rgnames__sample - type: - items: string - type: array -- id: align_bam - outputSource: process_alignment/align_bam - type: - items: File - type: array -- id: quant__tsv - outputSource: rnaseq_quantitate/quant__tsv - type: - items: File - type: array -- id: summary__multiqc - outputSource: multiqc_summary/summary__multiqc - type: - items: - - File - - 'null' - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: prepare_sample - in: - - id: files - source: files - - id: rgnames__sample - source: rgnames__sample - - id: reference__fasta__base - source: reference__fasta__base - - id: genome_build - source: genome_build - - id: genome_resources__rnaseq__transcripts - source: genome_resources__rnaseq__transcripts - - id: analysis - source: analysis - - id: rgnames__pl - source: rgnames__pl - - id: rgnames__pu - source: rgnames__pu - - id: rgnames__lane - source: rgnames__lane - - id: rgnames__rg - source: rgnames__rg - - id: rgnames__lb - source: rgnames__lb - - id: reference__hisat2__indexes - source: reference__hisat2__indexes - - id: config__algorithm__aligner - source: config__algorithm__aligner - - id: config__algorithm__expression_caller - source: config__algorithm__expression_caller - - id: config__algorithm__fusion_caller - source: config__algorithm__fusion_caller - - id: config__algorithm__quality_format - source: config__algorithm__quality_format - - id: resources - source: resources - - id: description - source: description - out: - - id: prep_rec - run: steps/prepare_sample.cwl - scatter: - - files - - rgnames__sample - - reference__fasta__base - - genome_build - - genome_resources__rnaseq__transcripts - - analysis - - rgnames__pl - - rgnames__pu - - rgnames__lane - - rgnames__rg - - rgnames__lb - - reference__hisat2__indexes - - config__algorithm__aligner - - config__algorithm__expression_caller - - config__algorithm__fusion_caller - - config__algorithm__quality_format - - resources - - description - scatterMethod: dotproduct -- id: trim_sample - in: - - id: prep_rec - source: prepare_sample/prep_rec - out: - - id: trim_rec - run: steps/trim_sample.cwl - scatter: - - prep_rec - scatterMethod: dotproduct -- id: process_alignment - in: - - id: trim_rec - source: trim_sample/trim_rec - out: - - id: align_bam - run: steps/process_alignment.cwl - scatter: - - trim_rec - scatterMethod: dotproduct -- id: rnaseq_quantitate - in: - - id: trim_rec - source: trim_sample/trim_rec - - id: align_bam - source: process_alignment/align_bam - out: - - id: count_file - - id: quant__tsv - - id: quant__hdf5 - - id: quant__fusion - run: steps/rnaseq_quantitate.cwl - scatter: - - trim_rec - - align_bam - scatterMethod: dotproduct -- id: qc_to_rec - in: - - id: align_bam - source: process_alignment/align_bam - - id: analysis - source: analysis - - id: reference__fasta__base - source: reference__fasta__base - - id: genome_resources__rnaseq__transcripts - source: genome_resources__rnaseq__transcripts - - id: genome_build - source: genome_build - - id: config__algorithm__coverage_interval - source: config__algorithm__coverage_interval - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: config__algorithm__tools_off - source: config__algorithm__tools_off - - id: config__algorithm__qc - source: config__algorithm__qc - - id: resources - source: resources - - id: description - source: description - out: - - id: qc_rec - run: steps/qc_to_rec.cwl -- id: pipeline_summary - in: - - id: qc_rec - source: qc_to_rec/qc_rec - out: - - id: qcout_rec - run: steps/pipeline_summary.cwl - scatter: - - qc_rec - scatterMethod: dotproduct -- id: multiqc_summary - in: - - id: qcout_rec - source: pipeline_summary/qcout_rec - out: - - id: summary__multiqc - run: steps/multiqc_summary.cwl -- id: detect_fusions - in: - - id: quant__fusion - source: rnaseq_quantitate/quant__fusion - - id: quant__hdf5 - source: rnaseq_quantitate/quant__hdf5 - - id: trim_rec - source: trim_sample/trim_rec - out: - - id: fusion__fasta - - id: fusion__json - run: steps/detect_fusions.cwl - scatter: - - quant__fusion - - quant__hdf5 - - trim_rec - scatterMethod: dotproduct diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/detect_fusions.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/detect_fusions.cwl deleted file mode 100644 index 8803c214f31..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/detect_fusions.cwl +++ /dev/null @@ -1,97 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=fusion__fasta,fusion__json -- sentinel_inputs=quant__fusion:var,quant__hdf5:var,trim_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- detect_fusions -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1024 - ramMin: 4096 - tmpdirMin: 0 -- class: dx:InputResourceRequirement - indirMin: 4 -- class: SoftwareRequirement - packages: - - package: pizzly - specs: - - https://anaconda.org/bioconda/pizzly -inputs: -- id: quant__fusion - type: File -- id: quant__hdf5 - type: File -- id: trim_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: files - type: - items: File - type: array - - name: reference__fasta__base - type: File - - name: config__algorithm__expression_caller - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: reference__hisat2__indexes - type: File - - name: config__algorithm__fusion_caller - type: - items: string - type: array - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: genome_build - type: string - - name: rgnames__pu - type: string - - name: genome_resources__rnaseq__transcripts - type: File - - name: config__algorithm__quality_format - type: string - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: rgnames__lane - type: string - name: trim_rec - type: record -outputs: -- id: fusion__fasta - type: File -- id: fusion__json - type: File -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/multiqc_summary.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/multiqc_summary.cwl deleted file mode 100644 index 604f9291df7..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/multiqc_summary.cwl +++ /dev/null @@ -1,96 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=summary__multiqc -- sentinel_inputs=qcout_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- multiqc_summary -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1024 - ramMin: 4096 - tmpdirMin: 0 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: multiqc - specs: - - https://anaconda.org/bioconda/multiqc - - package: multiqc-bcbio - specs: - - https://anaconda.org/bioconda/multiqc-bcbio -inputs: -- id: qcout_rec - type: - items: - fields: - - name: summary__qc - type: - - File - - 'null' - - name: summary__metrics - type: - - string - - 'null' - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__rnaseq__transcripts - type: File - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: File - name: qcout_rec - type: record - type: array -outputs: -- id: summary__multiqc - type: - items: - - File - - 'null' - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/pipeline_summary.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/pipeline_summary.cwl deleted file mode 100644 index 230b84afc66..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/pipeline_summary.cwl +++ /dev/null @@ -1,155 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=qcout_rec:summary__qc;summary__metrics;resources;description;reference__fasta__base;config__algorithm__coverage_interval;genome_build;genome_resources__rnaseq__transcripts;config__algorithm__tools_off;config__algorithm__qc;analysis;config__algorithm__tools_on;align_bam -- sentinel_inputs=qc_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- pipeline_summary -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1024 - ramMin: 4096 - tmpdirMin: 0 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: fastqc=0.11.7=4 - specs: - - https://anaconda.org/bioconda/fastqc=0.11.7=4 - - package: goleft - specs: - - https://anaconda.org/bioconda/goleft - - package: hts-nim-tools - specs: - - https://anaconda.org/bioconda/hts-nim-tools - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: picard - specs: - - https://anaconda.org/bioconda/picard - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: qsignature - specs: - - https://anaconda.org/bioconda/qsignature - - package: qualimap - specs: - - https://anaconda.org/bioconda/qualimap - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools -- class: arv:RuntimeConstraints - keep_cache: 4096 -inputs: -- id: qc_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__rnaseq__transcripts - type: File - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: File - name: qc_rec - type: record -outputs: -- id: qcout_rec - type: - fields: - - name: summary__qc - type: - - File - - 'null' - - name: summary__metrics - type: - - string - - 'null' - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__rnaseq__transcripts - type: File - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: File - name: qcout_rec - type: record -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/prepare_sample.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/prepare_sample.cwl deleted file mode 100644 index e5b8cdf645e..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/prepare_sample.cwl +++ /dev/null @@ -1,146 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=prep_rec:resources;description;files;reference__fasta__base;config__algorithm__expression_caller;rgnames__lb;rgnames__rg;reference__hisat2__indexes;config__algorithm__fusion_caller;config__algorithm__aligner;rgnames__pl;genome_build;rgnames__pu;genome_resources__rnaseq__transcripts;config__algorithm__quality_format;analysis;rgnames__sample;rgnames__lane -- sentinel_inputs=files:var,rgnames__sample:var,reference__fasta__base:var,genome_build:var,genome_resources__rnaseq__transcripts:var,analysis:var,rgnames__pl:var,rgnames__pu:var,rgnames__lane:var,rgnames__rg:var,rgnames__lb:var,reference__hisat2__indexes:var,config__algorithm__aligner:var,config__algorithm__expression_caller:var,config__algorithm__fusion_caller:var,config__algorithm__quality_format:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- prepare_sample -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1024 - ramMin: 4096 - tmpdirMin: 0 -- class: dx:InputResourceRequirement - indirMin: 4 -- class: SoftwareRequirement - packages: - - package: picard - specs: - - https://anaconda.org/bioconda/picard - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: pysam> - specs: - - https://anaconda.org/bioconda/pysam> - version: - - 0.13.0 -inputs: -- id: files - type: - items: File - type: array -- id: rgnames__sample - type: string -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: File -- id: genome_build - type: string -- id: genome_resources__rnaseq__transcripts - secondaryFiles: - - .db - type: File -- id: analysis - type: string -- id: rgnames__pl - type: string -- id: rgnames__pu - type: string -- id: rgnames__lane - type: string -- id: rgnames__rg - type: string -- id: rgnames__lb - type: - - 'null' - - string -- id: reference__hisat2__indexes - type: File -- id: config__algorithm__aligner - type: string -- id: config__algorithm__expression_caller - type: - items: string - type: array -- id: config__algorithm__fusion_caller - type: - items: string - type: array -- id: config__algorithm__quality_format - type: string -- id: resources - type: string -- id: description - type: string -outputs: -- id: prep_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: files - type: - items: File - type: array - - name: reference__fasta__base - type: File - - name: config__algorithm__expression_caller - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: reference__hisat2__indexes - type: File - - name: config__algorithm__fusion_caller - type: - items: string - type: array - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: genome_build - type: string - - name: rgnames__pu - type: string - - name: genome_resources__rnaseq__transcripts - type: File - - name: config__algorithm__quality_format - type: string - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: rgnames__lane - type: string - name: prep_rec - type: record -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/process_alignment.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/process_alignment.cwl deleted file mode 100644 index 88af1224f95..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/process_alignment.cwl +++ /dev/null @@ -1,108 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=align_bam -- sentinel_inputs=trim_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- process_alignment -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1028 - ramMin: 4096 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 4 -- class: SoftwareRequirement - packages: - - package: star - specs: - - https://anaconda.org/bioconda/star - - package: hisat2 - specs: - - https://anaconda.org/bioconda/hisat2 - - package: tophat - specs: - - https://anaconda.org/bioconda/tophat - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: seqtk - specs: - - https://anaconda.org/bioconda/seqtk -inputs: -- id: trim_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: files - type: - items: File - type: array - - name: reference__fasta__base - type: File - - name: config__algorithm__expression_caller - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: reference__hisat2__indexes - type: File - - name: config__algorithm__fusion_caller - type: - items: string - type: array - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: genome_build - type: string - - name: rgnames__pu - type: string - - name: genome_resources__rnaseq__transcripts - type: File - - name: config__algorithm__quality_format - type: string - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: rgnames__lane - type: string - name: trim_rec - type: record -outputs: -- id: align_bam - secondaryFiles: - - .bai - type: File -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/qc_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/qc_to_rec.cwl deleted file mode 100644 index 5ff5b2f10e0..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/qc_to_rec.cwl +++ /dev/null @@ -1,135 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=qc_rec:resources;description;reference__fasta__base;config__algorithm__coverage_interval;genome_build;genome_resources__rnaseq__transcripts;config__algorithm__tools_off;config__algorithm__qc;analysis;config__algorithm__tools_on;align_bam -- sentinel_inputs=align_bam:var,analysis:var,reference__fasta__base:var,genome_resources__rnaseq__transcripts:var,genome_build:var,config__algorithm__coverage_interval:var,config__algorithm__tools_on:var,config__algorithm__tools_off:var,config__algorithm__qc:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- qc_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 2048 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: align_bam - secondaryFiles: - - .bai - type: - items: File - type: array -- id: analysis - type: - items: string - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: genome_resources__rnaseq__transcripts - secondaryFiles: - - .db - type: - items: File - type: array -- id: genome_build - type: - items: string - type: array -- id: config__algorithm__coverage_interval - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__tools_on - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__qc - type: - items: - items: string - type: array - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: qc_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__rnaseq__transcripts - type: File - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: File - name: qc_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/rnaseq_quantitate.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/rnaseq_quantitate.cwl deleted file mode 100644 index 05d2ec813ae..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/rnaseq_quantitate.cwl +++ /dev/null @@ -1,123 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=count_file,quant__tsv,quant__hdf5,quant__fusion -- sentinel_inputs=trim_rec:record,align_bam:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- rnaseq_quantitate -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1025 - ramMin: 4096 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 4 -- class: SoftwareRequirement - packages: - - package: sailfish - specs: - - https://anaconda.org/bioconda/sailfish - - package: salmon - specs: - - https://anaconda.org/bioconda/salmon - - package: kallisto> - specs: - - https://anaconda.org/bioconda/kallisto> - version: - - 0.43.1 - - package: subread - specs: - - https://anaconda.org/bioconda/subread - - package: gffread - specs: - - https://anaconda.org/bioconda/gffread - - package: r - specs: - - https://anaconda.org/bioconda/r - version: - - 3.4.1 - - package: r-wasabi - specs: - - https://anaconda.org/bioconda/r-wasabi -inputs: -- id: trim_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: files - type: - items: File - type: array - - name: reference__fasta__base - type: File - - name: config__algorithm__expression_caller - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: reference__hisat2__indexes - type: File - - name: config__algorithm__fusion_caller - type: - items: string - type: array - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: genome_build - type: string - - name: rgnames__pu - type: string - - name: genome_resources__rnaseq__transcripts - type: File - - name: config__algorithm__quality_format - type: string - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: rgnames__lane - type: string - name: trim_rec - type: record -- id: align_bam - secondaryFiles: - - .bai - type: File -outputs: -- id: count_file - type: File -- id: quant__tsv - type: File -- id: quant__hdf5 - type: File -- id: quant__fusion - type: File -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/trim_sample.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/trim_sample.cwl deleted file mode 100644 index 14124636cec..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/rnaseq-workflow/steps/trim_sample.cwl +++ /dev/null @@ -1,140 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=trim_rec:resources;description;files;reference__fasta__base;config__algorithm__expression_caller;rgnames__lb;rgnames__rg;reference__hisat2__indexes;config__algorithm__fusion_caller;config__algorithm__aligner;rgnames__pl;genome_build;rgnames__pu;genome_resources__rnaseq__transcripts;config__algorithm__quality_format;analysis;rgnames__sample;rgnames__lane -- sentinel_inputs=prep_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- trim_sample -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-rnaseq:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1024 - ramMin: 4096 - tmpdirMin: 0 -- class: dx:InputResourceRequirement - indirMin: 4 -- class: SoftwareRequirement - packages: - - package: atropos;env - specs: - - https://anaconda.org/bioconda/atropos;env - version: - - python3 -inputs: -- id: prep_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: files - type: - items: File - type: array - - name: reference__fasta__base - type: File - - name: config__algorithm__expression_caller - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: reference__hisat2__indexes - type: File - - name: config__algorithm__fusion_caller - type: - items: string - type: array - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: genome_build - type: string - - name: rgnames__pu - type: string - - name: genome_resources__rnaseq__transcripts - type: File - - name: config__algorithm__quality_format - type: string - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: rgnames__lane - type: string - name: prep_rec - type: record -outputs: -- id: trim_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: files - type: - items: File - type: array - - name: reference__fasta__base - type: File - - name: config__algorithm__expression_caller - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: reference__hisat2__indexes - type: File - - name: config__algorithm__fusion_caller - type: - items: string - type: array - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: genome_build - type: string - - name: rgnames__pu - type: string - - name: genome_resources__rnaseq__transcripts - type: File - - name: config__algorithm__quality_format - type: string - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: rgnames__lane - type: string - name: trim_rec - type: record -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/main-somatic-samples.json b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/main-somatic-samples.json deleted file mode 100644 index cbafda3641d..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/main-somatic-samples.json +++ /dev/null @@ -1,680 +0,0 @@ -{ - "analysis": [ - "variant2", - "variant2" - ], - "config__algorithm__adapters": [ - [], - [] - ], - "config__algorithm__align_split_size": [ - null, - null - ], - "config__algorithm__aligner": [ - "bwa", - "bwa" - ], - "config__algorithm__bam_clean": [ - "False", - "False" - ], - "config__algorithm__coverage_interval": [ - null, - null - ], - "config__algorithm__effects": [ - "snpeff", - "snpeff" - ], - "config__algorithm__ensemble": [ - null, - null - ], - "config__algorithm__exclude_regions": [ - [], - [] - ], - "config__algorithm__mark_duplicates": [ - "True", - "True" - ], - "config__algorithm__min_allele_fraction": [ - 10.0, - 10.0 - ], - "config__algorithm__nomap_split_size": [ - 250, - 250 - ], - "config__algorithm__nomap_split_targets": [ - 20, - 20 - ], - "config__algorithm__qc": [ - [ - "contamination", - "coverage", - "fastqc", - "peddy", - "picard", - "qsignature", - "samtools", - "variants", - "viral" - ], - [ - "contamination", - "coverage", - "fastqc", - "peddy", - "picard", - "qsignature", - "samtools", - "variants", - "viral" - ] - ], - "config__algorithm__recalibrate": [ - "False", - "False" - ], - "config__algorithm__svcaller": [ - [ - "manta" - ], - [ - "manta" - ] - ], - "config__algorithm__svprioritize": [ - null, - null - ], - "config__algorithm__svvalidate": [ - null, - null - ], - "config__algorithm__tools_off": [ - [], - [] - ], - "config__algorithm__tools_on": [ - [], - [] - ], - "config__algorithm__trim_reads": [ - "False", - "False" - ], - "config__algorithm__validate": [ - null, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/reference_material/Test1-grade.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/reference_material/Test1-grade.vcf.gz.tbi" - } - ] - } - ], - "config__algorithm__validate_regions": [ - null, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed" - } - ], - "config__algorithm__variant_regions": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed" - } - ], - "config__algorithm__variantcaller": [ - [ - "vardict" - ], - [ - "vardict" - ] - ], - "config__algorithm__vcfanno": [ - [], - [] - ], - "description": [ - "Test2", - "Test1" - ], - "files": [ - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam" - } - ], - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam.bai" - } - ] - } - ] - ], - "genome_build": [ - "hg19", - "hg19" - ], - "genome_resources__aliases__ensembl": [ - "homo_sapiens_vep_83_GRCh37", - "homo_sapiens_vep_83_GRCh37" - ], - "genome_resources__aliases__human": [ - "True", - "True" - ], - "genome_resources__aliases__snpeff": [ - "hg19", - "hg19" - ], - "genome_resources__rnaseq__gene_bed": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rnaseq/ref-transcripts.bed" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rnaseq/ref-transcripts.bed" - } - ], - "genome_resources__variation__1000g": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__clinvar": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__cosmic": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__dbsnp": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__encode_blacklist": [ - null, - null - ], - "genome_resources__variation__esp": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__exac": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__gc_profile": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/gc/GC_profile.1000bp.cnp" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/gc/GC_profile.1000bp.cnp" - } - ], - "genome_resources__variation__germline_het_pon": [ - null, - null - ], - "genome_resources__variation__gnomad_exome": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__lcr": [ - null, - null - ], - "genome_resources__variation__polyx": [ - null, - null - ], - "genome_resources__variation__train_hapmap": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__train_indels": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz.tbi" - } - ] - } - ], - "metadata__batch": [ - "b1", - "b1" - ], - "metadata__phenotype": [ - "normal", - "tumor" - ], - "reference__bwa__indexes": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.ann" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.bwt" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.pac" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.sa" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.ann" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.bwt" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.pac" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.sa" - } - ] - } - ], - "reference__fasta__base": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa.fai" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.dict" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa.fai" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.dict" - } - ] - } - ], - "reference__genome_context": [ - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz.tbi" - } - ] - } - ], - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz.tbi" - } - ] - } - ] - ], - "reference__rtg": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rtg--hg19.sdf-wf.tar.gz" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rtg--hg19.sdf-wf.tar.gz" - } - ], - "reference__snpeff__hg19": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/snpeff--hg19-wf.tar.gz" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/snpeff--hg19-wf.tar.gz" - } - ], - "reference__versions": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/versions.csv" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/versions.csv" - } - ], - "reference__viral": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.dict" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.amb" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.ann" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.bwt" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.pac" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.sa" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.dict" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.amb" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.ann" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.bwt" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.pac" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.sa" - } - ] - } - ], - "resources": [ - "{}", - "{}" - ], - "rgnames__lane": [ - "Test2", - "Test1" - ], - "rgnames__lb": [ - null, - null - ], - "rgnames__pl": [ - "illumina", - "illumina" - ], - "rgnames__pu": [ - "Test2", - "Test1" - ], - "rgnames__rg": [ - "Test2", - "Test1" - ], - "rgnames__sample": [ - "Test2", - "Test1" - ], - "vrn_file": [ - null, - null - ] -} diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/main-somatic.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/main-somatic.cwl deleted file mode 100644 index 699d99a4690..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/main-somatic.cwl +++ /dev/null @@ -1,973 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: config__algorithm__align_split_size - type: - items: - - 'null' - - string - type: array -- id: files - type: - items: - items: File - type: array - type: array -- id: config__algorithm__trim_reads - type: - items: - - string - - 'null' - - boolean - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: config__algorithm__vcfanno - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__svprioritize - type: - items: - - 'null' - - string - type: array -- id: resources - type: - items: string - type: array -- id: config__algorithm__variantcaller - type: - items: - items: string - type: array - type: array -- id: config__algorithm__adapters - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__svcaller - type: - items: - items: string - type: array - type: array -- id: config__algorithm__coverage_interval - type: - items: - - 'null' - - string - type: array -- id: genome_resources__rnaseq__gene_bed - type: - items: File - type: array -- id: genome_resources__variation__gc_profile - type: - items: File - type: array -- id: rgnames__lb - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__clinvar - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__esp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: rgnames__rg - type: - items: string - type: array -- id: metadata__batch - type: - items: string - type: array -- id: rgnames__lane - type: - items: string - type: array -- id: genome_resources__variation__lcr - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__1000g - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__min_allele_fraction - type: - items: double - type: array -- id: config__algorithm__nomap_split_targets - type: - items: long - type: array -- id: reference__versions - type: - items: File - type: array -- id: reference__bwa__indexes - secondaryFiles: - - ^.ann - - ^.pac - - ^.sa - - ^.bwt - type: - items: File - type: array -- id: vrn_file - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__train_hapmap - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: reference__genome_context - secondaryFiles: - - .tbi - type: - items: - items: File - type: array - type: array -- id: config__algorithm__bam_clean - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__nomap_split_size - type: - items: long - type: array -- id: config__algorithm__validate - secondaryFiles: - - .tbi - type: - items: - - 'null' - - File - type: array -- id: reference__snpeff__hg19 - type: - items: File - type: array -- id: description - type: - items: string - type: array -- id: config__algorithm__validate_regions - type: - items: - - 'null' - - File - type: array -- id: config__algorithm__aligner - type: - items: string - type: array -- id: rgnames__pl - type: - items: string - type: array -- id: genome_build - type: - items: string - type: array -- id: rgnames__pu - type: - items: string - type: array -- id: genome_resources__variation__germline_het_pon - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__exac - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__gnomad_exome - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__recalibrate - type: - items: - - string - - 'null' - - boolean - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: genome_resources__aliases__human - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: genome_resources__variation__dbsnp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__mark_duplicates - type: - items: - - string - - 'null' - - boolean - type: array -- id: genome_resources__variation__polyx - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__encode_blacklist - type: - items: - - 'null' - - string - type: array -- id: reference__viral - secondaryFiles: - - .amb - - .ann - - .sa - - .pac - - ^.dict - - .bwt - type: - items: File - type: array -- id: genome_resources__variation__cosmic - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__ensemble - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__qc - type: - items: - items: string - type: array - type: array -- id: analysis - type: - items: string - type: array -- id: rgnames__sample - type: - items: string - type: array -- id: config__algorithm__tools_on - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__effects - type: - items: string - type: array -- id: config__algorithm__variant_regions - type: - items: File - type: array -- id: config__algorithm__svvalidate - type: - items: - - 'null' - - string - type: array -- id: genome_resources__aliases__ensembl - type: - items: string - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: reference__rtg - type: - items: File - type: array -- id: genome_resources__variation__train_indels - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__aliases__snpeff - type: - items: string - type: array -outputs: -- id: rgnames__sample_out - outputSource: prep_samples/rgnames__sample - type: - items: string - type: array -- id: align_bam - outputSource: postprocess_alignment/align_bam - type: - items: - - File - - 'null' - type: array -- id: regions__sample_callable - outputSource: postprocess_alignment/regions__sample_callable - type: - items: - - File - - 'null' - type: array -- id: validate__grading_summary - outputSource: summarize_vc/validate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: variants__calls - outputSource: summarize_vc/variants__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: variants__gvcf - outputSource: summarize_vc/variants__gvcf - type: - items: - - 'null' - - items: - - File - - 'null' - type: array - type: array -- id: sv__calls - outputSource: summarize_sv/sv__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: svvalidate__grading_summary - outputSource: summarize_sv/svvalidate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: sv__prioritize__tsv - outputSource: summarize_sv/sv__prioritize__tsv - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: sv__prioritize__raw - outputSource: summarize_sv/sv__prioritize__raw - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: sv__supplemental - outputSource: summarize_sv/sv__supplemental - type: - items: - items: - - File - type: array - type: array -- id: summary__multiqc - outputSource: multiqc_summary/summary__multiqc - type: - items: - - File - - 'null' - type: array -- id: versions__tools - outputSource: multiqc_summary/versions__tools - type: - items: - - File - - 'null' - type: array -- id: versions__data - outputSource: multiqc_summary/versions__data - type: - items: - - File - - 'null' - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: alignment_to_rec - in: - - id: files - source: files - - id: analysis - source: analysis - - id: config__algorithm__align_split_size - source: config__algorithm__align_split_size - - id: reference__fasta__base - source: reference__fasta__base - - id: rgnames__pl - source: rgnames__pl - - id: rgnames__sample - source: rgnames__sample - - id: rgnames__pu - source: rgnames__pu - - id: rgnames__lane - source: rgnames__lane - - id: rgnames__rg - source: rgnames__rg - - id: rgnames__lb - source: rgnames__lb - - id: reference__bwa__indexes - source: reference__bwa__indexes - - id: config__algorithm__aligner - source: config__algorithm__aligner - - id: config__algorithm__trim_reads - source: config__algorithm__trim_reads - - id: config__algorithm__adapters - source: config__algorithm__adapters - - id: config__algorithm__bam_clean - source: config__algorithm__bam_clean - - id: config__algorithm__variant_regions - source: config__algorithm__variant_regions - - id: config__algorithm__mark_duplicates - source: config__algorithm__mark_duplicates - - id: resources - source: resources - - id: description - source: description - out: - - id: alignment_rec - run: steps/alignment_to_rec.cwl -- id: alignment - in: - - id: alignment_rec - source: alignment_to_rec/alignment_rec - out: - - id: align_bam - - id: work_bam_plus__disc - - id: work_bam_plus__sr - - id: hla__fastq - run: wf-alignment.cwl - scatter: - - alignment_rec - scatterMethod: dotproduct -- id: prep_samples_to_rec - in: - - id: rgnames__sample - source: rgnames__sample - - id: config__algorithm__svcaller - source: config__algorithm__svcaller - - id: config__algorithm__variant_regions - source: config__algorithm__variant_regions - - id: reference__fasta__base - source: reference__fasta__base - - id: resources - source: resources - - id: description - source: description - out: - - id: prep_samples_rec - run: steps/prep_samples_to_rec.cwl -- id: prep_samples - in: - - id: prep_samples_rec - source: prep_samples_to_rec/prep_samples_rec - out: - - id: rgnames__sample - - id: config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - - id: config__algorithm__variant_regions_orig - - id: config__algorithm__coverage - - id: config__algorithm__coverage_merged - - id: config__algorithm__coverage_orig - - id: config__algorithm__seq2c_bed_ready - run: steps/prep_samples.cwl - scatter: - - prep_samples_rec - scatterMethod: dotproduct -- id: postprocess_alignment_to_rec - in: - - id: align_bam - source: alignment/align_bam - - id: config__algorithm__coverage_interval - source: config__algorithm__coverage_interval - - id: config__algorithm__exclude_regions - source: config__algorithm__exclude_regions - - id: config__algorithm__variant_regions - source: prep_samples/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: prep_samples/config__algorithm__variant_regions_merged - - id: config__algorithm__variant_regions_orig - source: prep_samples/config__algorithm__variant_regions_orig - - id: config__algorithm__coverage - source: prep_samples/config__algorithm__coverage - - id: config__algorithm__coverage_merged - source: prep_samples/config__algorithm__coverage_merged - - id: config__algorithm__coverage_orig - source: prep_samples/config__algorithm__coverage_orig - - id: config__algorithm__seq2c_bed_ready - source: prep_samples/config__algorithm__seq2c_bed_ready - - id: config__algorithm__recalibrate - source: config__algorithm__recalibrate - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: genome_resources__rnaseq__gene_bed - source: genome_resources__rnaseq__gene_bed - - id: genome_resources__variation__dbsnp - source: genome_resources__variation__dbsnp - - id: genome_resources__variation__lcr - source: genome_resources__variation__lcr - - id: genome_resources__variation__polyx - source: genome_resources__variation__polyx - - id: genome_resources__variation__encode_blacklist - source: genome_resources__variation__encode_blacklist - - id: reference__fasta__base - source: reference__fasta__base - - id: resources - source: resources - - id: description - source: description - out: - - id: postprocess_alignment_rec - run: steps/postprocess_alignment_to_rec.cwl -- id: postprocess_alignment - in: - - id: postprocess_alignment_rec - source: postprocess_alignment_to_rec/postprocess_alignment_rec - out: - - id: config__algorithm__coverage_interval - - id: config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - - id: config__algorithm__variant_regions_orig - - id: config__algorithm__coverage - - id: config__algorithm__coverage_merged - - id: config__algorithm__coverage_orig - - id: config__algorithm__seq2c_bed_ready - - id: regions__callable - - id: regions__sample_callable - - id: regions__nblock - - id: depth__samtools__stats - - id: depth__samtools__idxstats - - id: depth__variant_regions__regions - - id: depth__variant_regions__dist - - id: depth__sv_regions__regions - - id: depth__sv_regions__dist - - id: depth__coverage__regions - - id: depth__coverage__dist - - id: depth__coverage__thresholds - - id: align_bam - run: steps/postprocess_alignment.cwl - scatter: - - postprocess_alignment_rec - scatterMethod: dotproduct -- id: combine_sample_regions - in: - - id: regions__callable - source: postprocess_alignment/regions__callable - - id: regions__nblock - source: postprocess_alignment/regions__nblock - - id: metadata__batch - source: metadata__batch - - id: config__algorithm__nomap_split_size - source: config__algorithm__nomap_split_size - - id: config__algorithm__nomap_split_targets - source: config__algorithm__nomap_split_targets - - id: reference__fasta__base - source: reference__fasta__base - - id: resources - source: resources - - id: description - source: description - out: - - id: config__algorithm__callable_regions - - id: config__algorithm__non_callable_regions - - id: config__algorithm__callable_count - run: steps/combine_sample_regions.cwl -- id: batch_for_variantcall - in: - - id: analysis - source: analysis - - id: genome_build - source: genome_build - - id: align_bam - source: postprocess_alignment/align_bam - - id: vrn_file - source: vrn_file - - id: metadata__batch - source: metadata__batch - - id: metadata__phenotype - source: metadata__phenotype - - id: config__algorithm__callable_regions - source: combine_sample_regions/config__algorithm__callable_regions - - id: regions__sample_callable - source: postprocess_alignment/regions__sample_callable - - id: config__algorithm__variantcaller - source: config__algorithm__variantcaller - - id: config__algorithm__ensemble - source: config__algorithm__ensemble - - id: config__algorithm__vcfanno - source: config__algorithm__vcfanno - - id: config__algorithm__coverage_interval - source: postprocess_alignment/config__algorithm__coverage_interval - - id: config__algorithm__effects - source: config__algorithm__effects - - id: config__algorithm__min_allele_fraction - source: config__algorithm__min_allele_fraction - - id: config__algorithm__exclude_regions - source: config__algorithm__exclude_regions - - id: config__algorithm__variant_regions - source: postprocess_alignment/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: postprocess_alignment/config__algorithm__variant_regions_merged - - id: config__algorithm__validate - source: config__algorithm__validate - - id: config__algorithm__validate_regions - source: config__algorithm__validate_regions - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: config__algorithm__tools_off - source: config__algorithm__tools_off - - id: reference__fasta__base - source: reference__fasta__base - - id: reference__rtg - source: reference__rtg - - id: reference__genome_context - source: reference__genome_context - - id: genome_resources__variation__clinvar - source: genome_resources__variation__clinvar - - id: genome_resources__variation__cosmic - source: genome_resources__variation__cosmic - - id: genome_resources__variation__dbsnp - source: genome_resources__variation__dbsnp - - id: genome_resources__variation__esp - source: genome_resources__variation__esp - - id: genome_resources__variation__exac - source: genome_resources__variation__exac - - id: genome_resources__variation__gnomad_exome - source: genome_resources__variation__gnomad_exome - - id: genome_resources__variation__1000g - source: genome_resources__variation__1000g - - id: genome_resources__variation__lcr - source: genome_resources__variation__lcr - - id: genome_resources__variation__polyx - source: genome_resources__variation__polyx - - id: genome_resources__variation__encode_blacklist - source: genome_resources__variation__encode_blacklist - - id: genome_resources__aliases__ensembl - source: genome_resources__aliases__ensembl - - id: genome_resources__aliases__human - source: genome_resources__aliases__human - - id: genome_resources__aliases__snpeff - source: genome_resources__aliases__snpeff - - id: reference__snpeff__hg19 - source: reference__snpeff__hg19 - - id: genome_resources__variation__train_hapmap - source: genome_resources__variation__train_hapmap - - id: genome_resources__variation__train_indels - source: genome_resources__variation__train_indels - - id: resources - source: resources - - id: description - source: description - out: - - id: batch_rec - run: steps/batch_for_variantcall.cwl -- id: variantcall - in: - - id: batch_rec - source: batch_for_variantcall/batch_rec - out: - - id: vc_rec - run: wf-variantcall.cwl - scatter: - - batch_rec - scatterMethod: dotproduct -- id: summarize_vc - in: - - id: vc_rec - source: variantcall/vc_rec - out: - - id: variants__calls - - id: variants__gvcf - - id: variants__samples - - id: validate__grading_summary - - id: validate__grading_plots - run: steps/summarize_vc.cwl -- id: calculate_sv_bins - in: - - id: align_bam - source: postprocess_alignment/align_bam - - id: reference__fasta__base - source: reference__fasta__base - - id: metadata__batch - source: metadata__batch - - id: metadata__phenotype - source: metadata__phenotype - - id: config__algorithm__callable_regions - source: combine_sample_regions/config__algorithm__callable_regions - - id: config__algorithm__coverage_interval - source: postprocess_alignment/config__algorithm__coverage_interval - - id: config__algorithm__exclude_regions - source: config__algorithm__exclude_regions - - id: config__algorithm__variant_regions - source: postprocess_alignment/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: postprocess_alignment/config__algorithm__variant_regions_merged - - id: config__algorithm__seq2c_bed_ready - source: postprocess_alignment/config__algorithm__seq2c_bed_ready - - id: config__algorithm__svcaller - source: config__algorithm__svcaller - - id: depth__variant_regions__regions - source: postprocess_alignment/depth__variant_regions__regions - - id: genome_resources__variation__lcr - source: genome_resources__variation__lcr - - id: genome_resources__variation__polyx - source: genome_resources__variation__polyx - - id: genome_resources__variation__encode_blacklist - source: genome_resources__variation__encode_blacklist - - id: genome_resources__rnaseq__gene_bed - source: genome_resources__rnaseq__gene_bed - - id: resources - source: resources - - id: description - source: description - out: - - id: sv_bin_rec - run: steps/calculate_sv_bins.cwl -- id: calculate_sv_coverage - in: - - id: sv_bin_rec - source: calculate_sv_bins/sv_bin_rec - out: - - id: sv_rawcoverage_rec - run: steps/calculate_sv_coverage.cwl - scatter: - - sv_bin_rec - scatterMethod: dotproduct -- id: normalize_sv_coverage - in: - - id: sv_rawcoverage_rec - source: calculate_sv_coverage/sv_rawcoverage_rec - out: - - id: sv_coverage_rec - run: steps/normalize_sv_coverage.cwl -- id: batch_for_sv - in: - - id: analysis - source: analysis - - id: genome_build - source: genome_build - - id: work_bam_plus__disc - source: alignment/work_bam_plus__disc - - id: work_bam_plus__sr - source: alignment/work_bam_plus__sr - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: config__algorithm__tools_off - source: config__algorithm__tools_off - - id: config__algorithm__svprioritize - source: config__algorithm__svprioritize - - id: config__algorithm__svvalidate - source: config__algorithm__svvalidate - - id: regions__sample_callable - source: postprocess_alignment/regions__sample_callable - - id: genome_resources__variation__gc_profile - source: genome_resources__variation__gc_profile - - id: genome_resources__variation__germline_het_pon - source: genome_resources__variation__germline_het_pon - - id: genome_resources__aliases__snpeff - source: genome_resources__aliases__snpeff - - id: reference__snpeff__hg19 - source: reference__snpeff__hg19 - - id: sv_coverage_rec - source: normalize_sv_coverage/sv_coverage_rec - - id: variants__samples - source: summarize_vc/variants__samples - out: - - id: sv_batch_rec - run: steps/batch_for_sv.cwl -- id: svcall - in: - - id: sv_batch_rec - source: batch_for_sv/sv_batch_rec - out: - - id: sv_rec - run: wf-svcall.cwl - scatter: - - sv_batch_rec - scatterMethod: dotproduct -- id: summarize_sv - in: - - id: sv_rec - source: svcall/sv_rec - out: - - id: sv__calls - - id: sv__supplemental - - id: sv__prioritize__tsv - - id: sv__prioritize__raw - - id: svvalidate__grading_summary - - id: svvalidate__grading_plots - run: steps/summarize_sv.cwl -- id: qc_to_rec - in: - - id: align_bam - source: postprocess_alignment/align_bam - - id: analysis - source: analysis - - id: reference__fasta__base - source: reference__fasta__base - - id: reference__versions - source: reference__versions - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: config__algorithm__tools_off - source: config__algorithm__tools_off - - id: genome_build - source: genome_build - - id: config__algorithm__qc - source: config__algorithm__qc - - id: metadata__batch - source: metadata__batch - - id: metadata__phenotype - source: metadata__phenotype - - id: config__algorithm__coverage_interval - source: postprocess_alignment/config__algorithm__coverage_interval - - id: depth__variant_regions__regions - source: postprocess_alignment/depth__variant_regions__regions - - id: depth__variant_regions__dist - source: postprocess_alignment/depth__variant_regions__dist - - id: depth__samtools__stats - source: postprocess_alignment/depth__samtools__stats - - id: depth__samtools__idxstats - source: postprocess_alignment/depth__samtools__idxstats - - id: depth__sv_regions__regions - source: postprocess_alignment/depth__sv_regions__regions - - id: depth__sv_regions__dist - source: postprocess_alignment/depth__sv_regions__dist - - id: depth__coverage__regions - source: postprocess_alignment/depth__coverage__regions - - id: depth__coverage__dist - source: postprocess_alignment/depth__coverage__dist - - id: depth__coverage__thresholds - source: postprocess_alignment/depth__coverage__thresholds - - id: config__algorithm__variant_regions - source: postprocess_alignment/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: postprocess_alignment/config__algorithm__variant_regions_merged - - id: config__algorithm__coverage - source: postprocess_alignment/config__algorithm__coverage - - id: config__algorithm__coverage_merged - source: postprocess_alignment/config__algorithm__coverage_merged - - id: variants__samples - source: summarize_vc/variants__samples - - id: reference__viral - source: reference__viral - - id: resources - source: resources - - id: description - source: description - out: - - id: qc_rec - run: steps/qc_to_rec.cwl -- id: pipeline_summary - in: - - id: qc_rec - source: qc_to_rec/qc_rec - out: - - id: qcout_rec - run: steps/pipeline_summary.cwl - scatter: - - qc_rec - scatterMethod: dotproduct -- id: multiqc_summary - in: - - id: qcout_rec - source: pipeline_summary/qcout_rec - out: - - id: summary__multiqc - - id: versions__tools - - id: versions__data - run: steps/multiqc_summary.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/alignment_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/alignment_to_rec.cwl deleted file mode 100644 index 11cd9fc8d40..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/alignment_to_rec.cwl +++ /dev/null @@ -1,200 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=alignment_rec:resources;description;config__algorithm__align_split_size;files;config__algorithm__trim_reads;reference__fasta__base;config__algorithm__adapters;rgnames__lb;rgnames__rg;rgnames__lane;reference__bwa__indexes;config__algorithm__bam_clean;config__algorithm__aligner;rgnames__pl;rgnames__pu;config__algorithm__mark_duplicates;analysis;rgnames__sample;config__algorithm__variant_regions -- sentinel_inputs=files:var,analysis:var,config__algorithm__align_split_size:var,reference__fasta__base:var,rgnames__pl:var,rgnames__sample:var,rgnames__pu:var,rgnames__lane:var,rgnames__rg:var,rgnames__lb:var,reference__bwa__indexes:var,config__algorithm__aligner:var,config__algorithm__trim_reads:var,config__algorithm__adapters:var,config__algorithm__bam_clean:var,config__algorithm__variant_regions:var,config__algorithm__mark_duplicates:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- alignment_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10244 - ramMin: 3072 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: files - type: - items: - items: File - type: array - type: array -- id: analysis - type: - items: string - type: array -- id: config__algorithm__align_split_size - type: - items: - - 'null' - - string - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: rgnames__pl - type: - items: string - type: array -- id: rgnames__sample - type: - items: string - type: array -- id: rgnames__pu - type: - items: string - type: array -- id: rgnames__lane - type: - items: string - type: array -- id: rgnames__rg - type: - items: string - type: array -- id: rgnames__lb - type: - items: - - 'null' - - string - type: array -- id: reference__bwa__indexes - secondaryFiles: - - ^.ann - - ^.pac - - ^.sa - - ^.bwt - type: - items: File - type: array -- id: config__algorithm__aligner - type: - items: string - type: array -- id: config__algorithm__trim_reads - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__adapters - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__bam_clean - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__variant_regions - type: - items: File - type: array -- id: config__algorithm__mark_duplicates - type: - items: - - string - - 'null' - - boolean - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: alignment_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: - - string - - 'null' - - boolean - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - - 'null' - - items: 'null' - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: rgnames__pu - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: alignment_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/batch_for_sv.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/batch_for_sv.cwl deleted file mode 100644 index 16b905acacf..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/batch_for_sv.cwl +++ /dev/null @@ -1,375 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-batch -- sentinel_outputs=sv_batch_rec:resources;description;config__algorithm__svprioritize;genome_resources__variation__gc_profile;reference__snpeff__hg19;genome_build;genome_resources__variation__germline_het_pon;config__algorithm__tools_off;analysis;config__algorithm__tools_on;config__algorithm__svvalidate;genome_resources__aliases__snpeff;work_bam_plus__disc;work_bam_plus__sr;regions__sample_callable;variants__samples;depth__bins__normalized;depth__bins__background;depth__bins__target;depth__bins__antitarget;depth__bins__seq2c;regions__bins__target;regions__bins__antitarget;regions__bins__gcannotated;regions__bins__group;reference__fasta__base;config__algorithm__svcaller;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;metadata__batch;genome_resources__variation__lcr;metadata__phenotype;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__seq2c_bed_ready;depth__variant_regions__regions;config__algorithm__callable_regions -- sentinel_inputs=analysis:var,genome_build:var,work_bam_plus__disc:var,work_bam_plus__sr:var,config__algorithm__tools_on:var,config__algorithm__tools_off:var,config__algorithm__svprioritize:var,config__algorithm__svvalidate:var,regions__sample_callable:var,genome_resources__variation__gc_profile:var,genome_resources__variation__germline_het_pon:var,genome_resources__aliases__snpeff:var,reference__snpeff__hg19:var,sv_coverage_rec:record,variants__samples:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- batch_for_sv -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 10240 - ramMin: 6144 - tmpdirMin: 0 -- class: dx:InputResourceRequirement - indirMin: 1 -inputs: -- id: analysis - type: - items: string - type: array -- id: genome_build - type: - items: string - type: array -- id: work_bam_plus__disc - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: work_bam_plus__sr - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__tools_on - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__svprioritize - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__svvalidate - type: - items: - - 'null' - - string - type: array -- id: regions__sample_callable - type: - items: - - File - - 'null' - type: array -- id: genome_resources__variation__gc_profile - type: - items: File - type: array -- id: genome_resources__variation__germline_het_pon - type: - items: - - 'null' - - string - type: array -- id: genome_resources__aliases__snpeff - type: - items: string - type: array -- id: reference__snpeff__hg19 - type: - items: File - type: array -- id: sv_coverage_rec - type: - items: - fields: - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_coverage_rec - type: record - type: array -- id: variants__samples - type: - items: - items: - items: - - File - - 'null' - type: array - type: array - type: array -outputs: -- id: sv_batch_rec - type: - items: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: - - 'null' - - string - - name: genome_resources__variation__gc_profile - type: File - - name: reference__snpeff__hg19 - type: File - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__svvalidate - type: - - 'null' - - string - - name: genome_resources__aliases__snpeff - type: string - - name: work_bam_plus__disc - type: - - File - - 'null' - - name: work_bam_plus__sr - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_batch_rec - type: record - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/batch_for_variantcall.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/batch_for_variantcall.cwl deleted file mode 100644 index 60af6c4469e..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/batch_for_variantcall.cwl +++ /dev/null @@ -1,407 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-batch -- sentinel_outputs=batch_rec:resources;description;reference__fasta__base;config__algorithm__vcfanno;config__algorithm__variantcaller;config__algorithm__coverage_interval;genome_resources__variation__clinvar;genome_resources__variation__esp;metadata__batch;genome_resources__variation__lcr;genome_resources__variation__1000g;config__algorithm__min_allele_fraction;vrn_file;genome_resources__variation__train_hapmap;reference__genome_context;config__algorithm__validate;reference__snpeff__hg19;config__algorithm__validate_regions;genome_build;genome_resources__variation__exac;genome_resources__variation__gnomad_exome;metadata__phenotype;genome_resources__aliases__human;config__algorithm__tools_off;genome_resources__variation__dbsnp;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;genome_resources__variation__cosmic;config__algorithm__ensemble;analysis;config__algorithm__tools_on;config__algorithm__effects;config__algorithm__variant_regions;genome_resources__aliases__ensembl;config__algorithm__exclude_regions;reference__rtg;genome_resources__variation__train_indels;genome_resources__aliases__snpeff;align_bam;config__algorithm__variant_regions_merged;regions__sample_callable;config__algorithm__callable_regions -- sentinel_inputs=analysis:var,genome_build:var,align_bam:var,vrn_file:var,metadata__batch:var,metadata__phenotype:var,config__algorithm__callable_regions:var,regions__sample_callable:var,config__algorithm__variantcaller:var,config__algorithm__ensemble:var,config__algorithm__vcfanno:var,config__algorithm__coverage_interval:var,config__algorithm__effects:var,config__algorithm__min_allele_fraction:var,config__algorithm__exclude_regions:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__validate:var,config__algorithm__validate_regions:var,config__algorithm__tools_on:var,config__algorithm__tools_off:var,reference__fasta__base:var,reference__rtg:var,reference__genome_context:var,genome_resources__variation__clinvar:var,genome_resources__variation__cosmic:var,genome_resources__variation__dbsnp:var,genome_resources__variation__esp:var,genome_resources__variation__exac:var,genome_resources__variation__gnomad_exome:var,genome_resources__variation__1000g:var,genome_resources__variation__lcr:var,genome_resources__variation__polyx:var,genome_resources__variation__encode_blacklist:var,genome_resources__aliases__ensembl:var,genome_resources__aliases__human:var,genome_resources__aliases__snpeff:var,reference__snpeff__hg19:var,genome_resources__variation__train_hapmap:var,genome_resources__variation__train_indels:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- batch_for_variantcall -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10246 - ramMin: 3072 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: analysis - type: - items: string - type: array -- id: genome_build - type: - items: string - type: array -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: vrn_file - type: - items: - - 'null' - - string - type: array -- id: metadata__batch - type: - items: string - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: config__algorithm__callable_regions - type: - items: File - type: array -- id: regions__sample_callable - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variantcaller - type: - items: - items: string - type: array - type: array -- id: config__algorithm__ensemble - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__vcfanno - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__coverage_interval - type: - items: - - string - - 'null' - type: array -- id: config__algorithm__effects - type: - items: string - type: array -- id: config__algorithm__min_allele_fraction - type: - items: double - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__validate - secondaryFiles: - - .tbi - type: - items: - - 'null' - - File - type: array -- id: config__algorithm__validate_regions - type: - items: - - 'null' - - File - type: array -- id: config__algorithm__tools_on - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: reference__rtg - type: - items: File - type: array -- id: reference__genome_context - secondaryFiles: - - .tbi - type: - items: - items: File - type: array - type: array -- id: genome_resources__variation__clinvar - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__cosmic - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__dbsnp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__esp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__exac - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__gnomad_exome - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__1000g - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__lcr - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__polyx - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__encode_blacklist - type: - items: - - 'null' - - string - type: array -- id: genome_resources__aliases__ensembl - type: - items: string - type: array -- id: genome_resources__aliases__human - type: - items: - - string - - 'null' - - boolean - type: array -- id: genome_resources__aliases__snpeff - type: - items: string - type: array -- id: reference__snpeff__hg19 - type: - items: File - type: array -- id: genome_resources__variation__train_hapmap - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__train_indels - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: batch_rec - type: - items: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - File - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - File - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/calculate_sv_bins.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/calculate_sv_bins.cwl deleted file mode 100644 index 40bd035478a..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/calculate_sv_bins.cwl +++ /dev/null @@ -1,225 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=sv_bin_rec:regions__bins__target;regions__bins__antitarget;regions__bins__gcannotated;regions__bins__group;resources;description;reference__fasta__base;config__algorithm__svcaller;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;metadata__batch;genome_resources__variation__lcr;metadata__phenotype;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__seq2c_bed_ready;depth__variant_regions__regions;config__algorithm__callable_regions -- sentinel_inputs=align_bam:var,reference__fasta__base:var,metadata__batch:var,metadata__phenotype:var,config__algorithm__callable_regions:var,config__algorithm__coverage_interval:var,config__algorithm__exclude_regions:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__seq2c_bed_ready:var,config__algorithm__svcaller:var,depth__variant_regions__regions:var,genome_resources__variation__lcr:var,genome_resources__variation__polyx:var,genome_resources__variation__encode_blacklist:var,genome_resources__rnaseq__gene_bed:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- calculate_sv_bins -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10244 - ramMin: 3072 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: cnvkit - specs: - - https://anaconda.org/bioconda/cnvkit -inputs: -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: metadata__batch - type: - items: string - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: config__algorithm__callable_regions - type: - items: File - type: array -- id: config__algorithm__coverage_interval - type: - items: - - string - - 'null' - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__seq2c_bed_ready - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__svcaller - type: - items: - items: string - type: array - type: array -- id: depth__variant_regions__regions - type: - items: - - File - - 'null' - type: array -- id: genome_resources__variation__lcr - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__polyx - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__encode_blacklist - type: - items: - - 'null' - - string - type: array -- id: genome_resources__rnaseq__gene_bed - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: sv_bin_rec - type: - items: - fields: - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_bin_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/calculate_sv_coverage.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/calculate_sv_coverage.cwl deleted file mode 100644 index fc75d2c1b71..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/calculate_sv_coverage.cwl +++ /dev/null @@ -1,218 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=sv_rawcoverage_rec:depth__bins__target;depth__bins__antitarget;depth__bins__seq2c;resources;description;regions__bins__target;regions__bins__antitarget;regions__bins__gcannotated;regions__bins__group;reference__fasta__base;config__algorithm__svcaller;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;metadata__batch;genome_resources__variation__lcr;metadata__phenotype;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__seq2c_bed_ready;depth__variant_regions__regions;config__algorithm__callable_regions -- sentinel_inputs=sv_bin_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- calculate_sv_coverage -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 10244 - ramMin: 6144 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: cnvkit - specs: - - https://anaconda.org/bioconda/cnvkit - - package: seq2c - specs: - - https://anaconda.org/bioconda/seq2c -inputs: -- id: sv_bin_rec - type: - fields: - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_bin_rec - type: record -outputs: -- id: sv_rawcoverage_rec - type: - fields: - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_rawcoverage_rec - type: record -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/combine_sample_regions.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/combine_sample_regions.cwl deleted file mode 100644 index 5a8f8a3c6cb..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/combine_sample_regions.cwl +++ /dev/null @@ -1,99 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=config__algorithm__callable_regions,config__algorithm__non_callable_regions,config__algorithm__callable_count -- sentinel_inputs=regions__callable:var,regions__nblock:var,metadata__batch:var,config__algorithm__nomap_split_size:var,config__algorithm__nomap_split_targets:var,reference__fasta__base:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- combine_sample_regions -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10241 - ramMin: 3072 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 -- class: arv:APIRequirement -inputs: -- id: regions__callable - type: - items: - - File - - 'null' - type: array -- id: regions__nblock - type: - items: - - File - - 'null' - type: array -- id: metadata__batch - type: - items: string - type: array -- id: config__algorithm__nomap_split_size - type: - items: long - type: array -- id: config__algorithm__nomap_split_targets - type: - items: long - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: config__algorithm__callable_regions - type: - items: File - type: array -- id: config__algorithm__non_callable_regions - type: - items: File - type: array -- id: config__algorithm__callable_count - type: - items: int - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/compare_to_rm.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/compare_to_rm.cwl deleted file mode 100644 index d6b910b7f9e..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/compare_to_rm.cwl +++ /dev/null @@ -1,311 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-single -- sentinel_outputs=vc_rec:batch_samples;validate__summary;validate__tp;validate__fp;validate__fn;resources;description;vrn_file;reference__fasta__base;config__algorithm__vcfanno;config__algorithm__variantcaller;config__algorithm__coverage_interval;metadata__batch;config__algorithm__min_allele_fraction;reference__genome_context;config__algorithm__validate;reference__snpeff__hg19;config__algorithm__validate_regions;genome_build;metadata__phenotype;genome_resources__aliases__human;config__algorithm__tools_off;config__algorithm__ensemble;analysis;config__algorithm__tools_on;config__algorithm__effects;config__algorithm__variant_regions;genome_resources__aliases__ensembl;config__algorithm__exclude_regions;reference__rtg;genome_resources__aliases__snpeff;config__algorithm__variant_regions_merged;regions__sample_callable;config__algorithm__callable_regions -- sentinel_inputs=batch_rec:record,vrn_file:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- compare_to_rm -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 10244 - ramMin: 6144 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: gvcf-regions - specs: - - https://anaconda.org/bioconda/gvcf-regions - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: rtg-tools - specs: - - https://anaconda.org/bioconda/rtg-tools - - package: vcfanno - specs: - - https://anaconda.org/bioconda/vcfanno -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - File - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - File - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: vrn_file - secondaryFiles: - - .tbi - type: File -outputs: -- id: vc_rec - type: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - File - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - File - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/concat_batch_variantcalls.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/concat_batch_variantcalls.cwl deleted file mode 100644 index ef31b2ffe4f..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/concat_batch_variantcalls.cwl +++ /dev/null @@ -1,198 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-merge -- sentinel_outputs=vrn_file -- sentinel_inputs=batch_rec:record,region_block:var,vrn_file_region:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- concat_batch_variantcalls -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10244 - ramMin: 3072 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 -- class: arv:APIRequirement -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - File - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - File - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: region_block - type: - items: - items: string - type: array - type: array -- id: vrn_file_region - secondaryFiles: - - .tbi - type: - items: - - File - - 'null' - type: array -outputs: -- id: vrn_file - secondaryFiles: - - .tbi - type: File -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/detect_sv.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/detect_sv.cwl deleted file mode 100644 index 234bcc60d69..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/detect_sv.cwl +++ /dev/null @@ -1,414 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-single -- sentinel_outputs=sv_rec:sv__variantcaller;sv__vrn_file;sv__supplemental;svvalidate__summary;resources;description;config__algorithm__svprioritize;genome_resources__variation__gc_profile;genome_build;genome_resources__variation__germline_het_pon;config__algorithm__tools_off;analysis;config__algorithm__tools_on;config__algorithm__svvalidate;genome_resources__aliases__snpeff;regions__sample_callable;variants__samples;depth__bins__normalized;depth__bins__background;depth__bins__target;depth__bins__antitarget;depth__bins__seq2c;regions__bins__target;regions__bins__antitarget;regions__bins__gcannotated;regions__bins__group;reference__fasta__base;config__algorithm__svcaller;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;metadata__batch;genome_resources__variation__lcr;metadata__phenotype;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__variant_regions;config__algorithm__exclude_regions;config__algorithm__variant_regions_merged;config__algorithm__seq2c_bed_ready;depth__variant_regions__regions;config__algorithm__callable_regions -- sentinel_inputs=sv_batch_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- detect_sv -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 10246 - ramMin: 6144 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: cnvkit - specs: - - https://anaconda.org/bioconda/cnvkit - - package: delly - specs: - - https://anaconda.org/bioconda/delly - - package: duphold - specs: - - https://anaconda.org/bioconda/duphold - - package: extract-sv-reads - specs: - - https://anaconda.org/bioconda/extract-sv-reads - - package: lumpy-sv - specs: - - https://anaconda.org/bioconda/lumpy-sv - - package: manta - specs: - - https://anaconda.org/bioconda/manta - - package: break-point-inspector - specs: - - https://anaconda.org/bioconda/break-point-inspector - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: smoove - specs: - - https://anaconda.org/bioconda/smoove - - package: pysam> - specs: - - https://anaconda.org/bioconda/pysam> - version: - - 0.13.0 - - package: seq2c - specs: - - https://anaconda.org/bioconda/seq2c - - package: simple_sv_annotation - specs: - - https://anaconda.org/bioconda/simple_sv_annotation - - package: survivor - specs: - - https://anaconda.org/bioconda/survivor - - package: svtools - specs: - - https://anaconda.org/bioconda/svtools - - package: svtyper - specs: - - https://anaconda.org/bioconda/svtyper - - package: r - specs: - - https://anaconda.org/bioconda/r - version: - - 3.4.1 - - package: r-base=3.4.1=h4fe35fd_8 - specs: - - https://anaconda.org/bioconda/r-base=3.4.1=h4fe35fd_8 - - package: xorg-libxt - specs: - - https://anaconda.org/bioconda/xorg-libxt - - package: vawk - specs: - - https://anaconda.org/bioconda/vawk -- class: arv:RuntimeConstraints - keep_cache: 4096 -inputs: -- id: sv_batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: - - 'null' - - string - - name: genome_resources__variation__gc_profile - type: File - - name: reference__snpeff__hg19 - type: File - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__svvalidate - type: - - 'null' - - string - - name: genome_resources__aliases__snpeff - type: string - - name: work_bam_plus__disc - type: - - File - - 'null' - - name: work_bam_plus__sr - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_batch_rec - type: record - type: array -outputs: -- id: sv_rec - type: - items: - fields: - - name: sv__variantcaller - type: - - string - - 'null' - - name: sv__vrn_file - type: - - File - - 'null' - - name: sv__supplemental - type: - items: - - File - type: array - - name: svvalidate__summary - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: - - 'null' - - string - - name: genome_resources__variation__gc_profile - type: File - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__svvalidate - type: - - 'null' - - string - - name: genome_resources__aliases__snpeff - type: string - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/get_parallel_regions.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/get_parallel_regions.cwl deleted file mode 100644 index 48e5f050eb3..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/get_parallel_regions.cwl +++ /dev/null @@ -1,173 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-split -- sentinel_outputs=region_block -- sentinel_inputs=batch_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- get_parallel_regions -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10246 - ramMin: 3072 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - File - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - File - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -outputs: -- id: region_block - type: - items: - items: string - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/merge_split_alignments.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/merge_split_alignments.cwl deleted file mode 100644 index 029b704bbb2..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/merge_split_alignments.cwl +++ /dev/null @@ -1,169 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=single-merge -- sentinel_outputs=align_bam,work_bam_plus__disc,work_bam_plus__sr,hla__fastq -- sentinel_inputs=alignment_rec:record,work_bam:var,align_bam:var,work_bam_plus__disc:var,work_bam_plus__sr:var,hla__fastq:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- merge_split_alignments -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 10251 - ramMin: 6144 - tmpdirMin: 6 -- class: dx:InputResourceRequirement - indirMin: 4 -- class: SoftwareRequirement - packages: - - package: biobambam - specs: - - https://anaconda.org/bioconda/biobambam - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: variantbam - specs: - - https://anaconda.org/bioconda/variantbam -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: - - string - - 'null' - - boolean - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - - 'null' - - items: 'null' - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: rgnames__pu - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: alignment_rec - type: record -- id: work_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: align_bam_toolinput - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: work_bam_plus__disc_toolinput - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: work_bam_plus__sr_toolinput - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: hla__fastq_toolinput - type: - items: - - 'null' - - items: File - type: array - type: array -outputs: -- id: align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__disc - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__sr - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: hla__fastq - type: - - 'null' - - items: File - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/multiqc_summary.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/multiqc_summary.cwl deleted file mode 100644 index 9d65504ba6b..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/multiqc_summary.cwl +++ /dev/null @@ -1,96 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=summary__multiqc,versions__tools,versions__data -- sentinel_inputs=qcout_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- multiqc_summary -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10246 - ramMin: 3072 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: multiqc - specs: - - https://anaconda.org/bioconda/multiqc - - package: multiqc-bcbio - specs: - - https://anaconda.org/bioconda/multiqc-bcbio -inputs: -- id: qcout_rec - type: - items: - fields: - - name: summary__qc - type: - - File - - 'null' - - name: summary__metrics - type: - - string - - 'null' - - name: description - type: string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - name: qcout_rec - type: record - type: array -outputs: -- id: summary__multiqc - type: - items: - - File - - 'null' - type: array -- id: versions__tools - type: - items: - - File - - 'null' - type: array -- id: versions__data - type: - items: - - File - - 'null' - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/normalize_sv_coverage.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/normalize_sv_coverage.cwl deleted file mode 100644 index 1bdc8dbc3b8..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/normalize_sv_coverage.cwl +++ /dev/null @@ -1,236 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=sv_coverage_rec:depth__bins__normalized;depth__bins__background;resources;description;depth__bins__target;depth__bins__antitarget;depth__bins__seq2c;regions__bins__target;regions__bins__antitarget;regions__bins__gcannotated;regions__bins__group;reference__fasta__base;config__algorithm__svcaller;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;metadata__batch;genome_resources__variation__lcr;metadata__phenotype;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__seq2c_bed_ready;depth__variant_regions__regions;config__algorithm__callable_regions -- sentinel_inputs=sv_rawcoverage_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- normalize_sv_coverage -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 10244 - ramMin: 6144 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: cnvkit - specs: - - https://anaconda.org/bioconda/cnvkit -inputs: -- id: sv_rawcoverage_rec - type: - items: - fields: - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_rawcoverage_rec - type: record - type: array -outputs: -- id: sv_coverage_rec - type: - items: - fields: - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_coverage_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/pipeline_summary.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/pipeline_summary.cwl deleted file mode 100644 index 89f6372949c..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/pipeline_summary.cwl +++ /dev/null @@ -1,223 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=qcout_rec:summary__qc;summary__metrics;description;reference__versions;genome_build;config__algorithm__tools_off;config__algorithm__qc;config__algorithm__tools_on -- sentinel_inputs=qc_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- pipeline_summary -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 10246 - ramMin: 6144 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: fastqc=0.11.7=5 - specs: - - https://anaconda.org/bioconda/fastqc=0.11.7=5 - - package: goleft - specs: - - https://anaconda.org/bioconda/goleft - - package: hts-nim-tools - specs: - - https://anaconda.org/bioconda/hts-nim-tools - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: picard - specs: - - https://anaconda.org/bioconda/picard - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: qsignature - specs: - - https://anaconda.org/bioconda/qsignature - - package: qualimap - specs: - - https://anaconda.org/bioconda/qualimap - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: preseq - specs: - - https://anaconda.org/bioconda/preseq - - package: peddy - specs: - - https://anaconda.org/bioconda/peddy - - package: verifybamid2 - specs: - - https://anaconda.org/bioconda/verifybamid2 -- class: arv:RuntimeConstraints - keep_cache: 4096 -inputs: -- id: qc_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: reference__viral - type: File - - name: config__algorithm__qc - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: depth__samtools__stats - type: - - File - - 'null' - - name: depth__samtools__idxstats - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: depth__variant_regions__dist - type: - - File - - 'null' - - name: depth__sv_regions__regions - type: - - File - - 'null' - - name: depth__sv_regions__dist - type: - - File - - 'null' - - name: depth__coverage__regions - type: - - File - - 'null' - - name: depth__coverage__dist - type: - - File - - 'null' - - name: depth__coverage__thresholds - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - name: qc_rec - type: record -outputs: -- id: qcout_rec - type: - fields: - - name: summary__qc - type: - - File - - 'null' - - name: summary__metrics - type: - - string - - 'null' - - name: description - type: string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - name: qcout_rec - type: record -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/postprocess_alignment.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/postprocess_alignment.cwl deleted file mode 100644 index 8585bfb3074..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/postprocess_alignment.cwl +++ /dev/null @@ -1,224 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=config__algorithm__coverage_interval,config__algorithm__variant_regions,config__algorithm__variant_regions_merged,config__algorithm__variant_regions_orig,config__algorithm__coverage,config__algorithm__coverage_merged,config__algorithm__coverage_orig,config__algorithm__seq2c_bed_ready,regions__callable,regions__sample_callable,regions__nblock,depth__samtools__stats,depth__samtools__idxstats,depth__variant_regions__regions,depth__variant_regions__dist,depth__sv_regions__regions,depth__sv_regions__dist,depth__coverage__regions,depth__coverage__dist,depth__coverage__thresholds,align_bam -- sentinel_inputs=postprocess_alignment_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- postprocess_alignment -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 10249 - ramMin: 6144 - tmpdirMin: 5 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: goleft - specs: - - https://anaconda.org/bioconda/goleft - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: sentieon - specs: - - https://anaconda.org/bioconda/sentieon -- class: arv:APIRequirement -inputs: -- id: postprocess_alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - 'null' - - string - - name: genome_resources__rnaseq__gene_bed - type: File - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__recalibrate - type: - - string - - 'null' - - boolean - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__variant_regions_orig - type: - - File - - 'null' - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: config__algorithm__coverage_orig - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - name: postprocess_alignment_rec - type: record -outputs: -- id: config__algorithm__coverage_interval - type: - - string - - 'null' -- id: config__algorithm__variant_regions - type: - - File - - 'null' -- id: config__algorithm__variant_regions_merged - type: - - File - - 'null' -- id: config__algorithm__variant_regions_orig - type: - - File - - 'null' -- id: config__algorithm__coverage - type: - - File - - 'null' -- id: config__algorithm__coverage_merged - type: - - File - - 'null' -- id: config__algorithm__coverage_orig - type: - - File - - 'null' -- id: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' -- id: regions__callable - type: - - File - - 'null' -- id: regions__sample_callable - type: - - File - - 'null' -- id: regions__nblock - type: - - File - - 'null' -- id: depth__samtools__stats - type: - - File - - 'null' -- id: depth__samtools__idxstats - type: - - File - - 'null' -- id: depth__variant_regions__regions - type: - - File - - 'null' -- id: depth__variant_regions__dist - type: - - File - - 'null' -- id: depth__sv_regions__regions - type: - - File - - 'null' -- id: depth__sv_regions__dist - type: - - File - - 'null' -- id: depth__coverage__regions - type: - - File - - 'null' -- id: depth__coverage__dist - type: - - File - - 'null' -- id: depth__coverage__thresholds - type: - - File - - 'null' -- id: align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/postprocess_alignment_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/postprocess_alignment_to_rec.cwl deleted file mode 100644 index 9d93ac8f1db..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/postprocess_alignment_to_rec.cwl +++ /dev/null @@ -1,235 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=postprocess_alignment_rec:resources;description;reference__fasta__base;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;genome_resources__variation__lcr;config__algorithm__recalibrate;genome_resources__variation__dbsnp;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__tools_on;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__variant_regions_orig;config__algorithm__coverage;config__algorithm__coverage_merged;config__algorithm__coverage_orig;config__algorithm__seq2c_bed_ready -- sentinel_inputs=align_bam:var,config__algorithm__coverage_interval:var,config__algorithm__exclude_regions:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__variant_regions_orig:var,config__algorithm__coverage:var,config__algorithm__coverage_merged:var,config__algorithm__coverage_orig:var,config__algorithm__seq2c_bed_ready:var,config__algorithm__recalibrate:var,config__algorithm__tools_on:var,genome_resources__rnaseq__gene_bed:var,genome_resources__variation__dbsnp:var,genome_resources__variation__lcr:var,genome_resources__variation__polyx:var,genome_resources__variation__encode_blacklist:var,reference__fasta__base:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- postprocess_alignment_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10244 - ramMin: 3072 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_interval - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_orig - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_orig - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__seq2c_bed_ready - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__recalibrate - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__tools_on - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: genome_resources__rnaseq__gene_bed - type: - items: File - type: array -- id: genome_resources__variation__dbsnp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__lcr - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__polyx - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__encode_blacklist - type: - items: - - 'null' - - string - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: postprocess_alignment_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - 'null' - - string - - name: genome_resources__rnaseq__gene_bed - type: File - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__recalibrate - type: - - string - - 'null' - - boolean - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__variant_regions_orig - type: - - File - - 'null' - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: config__algorithm__coverage_orig - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - name: postprocess_alignment_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/postprocess_variants.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/postprocess_variants.cwl deleted file mode 100644 index 6fba055bcd7..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/postprocess_variants.cwl +++ /dev/null @@ -1,182 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-single -- sentinel_outputs=vrn_file -- sentinel_inputs=batch_rec:record,vrn_file:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- postprocess_variants -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 10241 - ramMin: 6144 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: snpeff - specs: - - https://anaconda.org/bioconda/snpeff - version: - - 4.3.1t -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - File - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - File - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: vrn_file_toolinput - secondaryFiles: - - .tbi - type: File -outputs: -- id: vrn_file - secondaryFiles: - - .tbi - type: File -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/prep_align_inputs.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/prep_align_inputs.cwl deleted file mode 100644 index c40a38fc01a..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/prep_align_inputs.cwl +++ /dev/null @@ -1,143 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=single-split -- sentinel_outputs=process_alignment_rec:files;config__algorithm__quality_format;align_split -- sentinel_inputs=alignment_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- prep_align_inputs -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 10244 - ramMin: 6144 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 4 -- class: SoftwareRequirement - packages: - - package: grabix - specs: - - https://anaconda.org/bioconda/grabix - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: biobambam - specs: - - https://anaconda.org/bioconda/biobambam - - package: atropos;env - specs: - - https://anaconda.org/bioconda/atropos;env - version: - - python3 - - package: optitype - specs: - - https://anaconda.org/bioconda/optitype - - package: razers3 - specs: - - https://anaconda.org/bioconda/razers3 - version: - - 3.5.0 - - package: coincbc - specs: - - https://anaconda.org/bioconda/coincbc -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: - - string - - 'null' - - boolean - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - - 'null' - - items: 'null' - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: rgnames__pu - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: alignment_rec - type: record -outputs: -- id: process_alignment_rec - type: - items: - fields: - - name: files - type: - - 'null' - - items: File - type: array - - name: config__algorithm__quality_format - type: - - string - - 'null' - - name: align_split - type: - - string - - 'null' - name: process_alignment_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/prep_samples.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/prep_samples.cwl deleted file mode 100644 index 4e62820988d..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/prep_samples.cwl +++ /dev/null @@ -1,95 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=rgnames__sample,config__algorithm__variant_regions,config__algorithm__variant_regions_merged,config__algorithm__variant_regions_orig,config__algorithm__coverage,config__algorithm__coverage_merged,config__algorithm__coverage_orig,config__algorithm__seq2c_bed_ready -- sentinel_inputs=prep_samples_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- prep_samples -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10241 - ramMin: 3072 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy -inputs: -- id: prep_samples_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: prep_samples_rec - type: record -outputs: -- id: rgnames__sample - type: string -- id: config__algorithm__variant_regions - type: - - File - - 'null' -- id: config__algorithm__variant_regions_merged - type: - - File - - 'null' -- id: config__algorithm__variant_regions_orig - type: - - File - - 'null' -- id: config__algorithm__coverage - type: - - File - - 'null' -- id: config__algorithm__coverage_merged - type: - - File - - 'null' -- id: config__algorithm__coverage_orig - type: - - File - - 'null' -- id: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/prep_samples_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/prep_samples_to_rec.cwl deleted file mode 100644 index 9268efdafcb..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/prep_samples_to_rec.cwl +++ /dev/null @@ -1,85 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=prep_samples_rec:resources;description;reference__fasta__base;config__algorithm__svcaller;rgnames__sample;config__algorithm__variant_regions -- sentinel_inputs=rgnames__sample:var,config__algorithm__svcaller:var,config__algorithm__variant_regions:var,reference__fasta__base:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- prep_samples_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10241 - ramMin: 3072 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: rgnames__sample - type: - items: string - type: array -- id: config__algorithm__svcaller - type: - items: - items: string - type: array - type: array -- id: config__algorithm__variant_regions - type: - items: File - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: prep_samples_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: prep_samples_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/process_alignment.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/process_alignment.cwl deleted file mode 100644 index 45b58d1ab98..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/process_alignment.cwl +++ /dev/null @@ -1,199 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=single-parallel -- sentinel_outputs=work_bam,align_bam,hla__fastq,work_bam_plus__disc,work_bam_plus__sr -- sentinel_inputs=alignment_rec:record,process_alignment_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- process_alignment -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 10246 - ramMin: 6144 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 7 -- class: SoftwareRequirement - packages: - - package: bwa - specs: - - https://anaconda.org/bioconda/bwa - - package: bwakit - specs: - - https://anaconda.org/bioconda/bwakit - - package: grabix - specs: - - https://anaconda.org/bioconda/grabix - - package: minimap2 - specs: - - https://anaconda.org/bioconda/minimap2 - - package: novoalign - specs: - - https://anaconda.org/bioconda/novoalign - - package: snap-aligner - specs: - - https://anaconda.org/bioconda/snap-aligner - version: - - 1.0dev.97 - - package: sentieon - specs: - - https://anaconda.org/bioconda/sentieon - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: pysam> - specs: - - https://anaconda.org/bioconda/pysam> - version: - - 0.13.0 - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: fgbio - specs: - - https://anaconda.org/bioconda/fgbio - - package: umis - specs: - - https://anaconda.org/bioconda/umis - - package: biobambam - specs: - - https://anaconda.org/bioconda/biobambam - - package: seqtk - specs: - - https://anaconda.org/bioconda/seqtk - - package: samblaster - specs: - - https://anaconda.org/bioconda/samblaster - - package: variantbam - specs: - - https://anaconda.org/bioconda/variantbam -- class: arv:APIRequirement -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: - - string - - 'null' - - boolean - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - - 'null' - - items: 'null' - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: rgnames__pu - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: alignment_rec - type: record -- id: process_alignment_rec - type: - fields: - - name: files - type: - - 'null' - - items: File - type: array - - name: config__algorithm__quality_format - type: - - string - - 'null' - - name: align_split - type: - - string - - 'null' - name: process_alignment_rec - type: record -outputs: -- id: work_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: hla__fastq - type: - - 'null' - - items: File - type: array -- id: work_bam_plus__disc - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__sr - secondaryFiles: - - .bai - type: - - File - - 'null' -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/qc_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/qc_to_rec.cwl deleted file mode 100644 index eccc8330618..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/qc_to_rec.cwl +++ /dev/null @@ -1,310 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=qc_rec:resources;description;reference__fasta__base;config__algorithm__coverage_interval;metadata__batch;reference__versions;genome_build;metadata__phenotype;config__algorithm__tools_off;reference__viral;config__algorithm__qc;analysis;config__algorithm__tools_on;config__algorithm__variant_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__coverage;config__algorithm__coverage_merged;depth__samtools__stats;depth__samtools__idxstats;depth__variant_regions__regions;depth__variant_regions__dist;depth__sv_regions__regions;depth__sv_regions__dist;depth__coverage__regions;depth__coverage__dist;depth__coverage__thresholds;variants__samples -- sentinel_inputs=align_bam:var,analysis:var,reference__fasta__base:var,reference__versions:var,config__algorithm__tools_on:var,config__algorithm__tools_off:var,genome_build:var,config__algorithm__qc:var,metadata__batch:var,metadata__phenotype:var,config__algorithm__coverage_interval:var,depth__variant_regions__regions:var,depth__variant_regions__dist:var,depth__samtools__stats:var,depth__samtools__idxstats:var,depth__sv_regions__regions:var,depth__sv_regions__dist:var,depth__coverage__regions:var,depth__coverage__dist:var,depth__coverage__thresholds:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__coverage:var,config__algorithm__coverage_merged:var,variants__samples:var,reference__viral:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- qc_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10244 - ramMin: 3072 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: analysis - type: - items: string - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: reference__versions - type: - items: File - type: array -- id: config__algorithm__tools_on - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: genome_build - type: - items: string - type: array -- id: config__algorithm__qc - type: - items: - items: string - type: array - type: array -- id: metadata__batch - type: - items: string - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: config__algorithm__coverage_interval - type: - items: - - string - - 'null' - type: array -- id: depth__variant_regions__regions - type: - items: - - File - - 'null' - type: array -- id: depth__variant_regions__dist - type: - items: - - File - - 'null' - type: array -- id: depth__samtools__stats - type: - items: - - File - - 'null' - type: array -- id: depth__samtools__idxstats - type: - items: - - File - - 'null' - type: array -- id: depth__sv_regions__regions - type: - items: - - File - - 'null' - type: array -- id: depth__sv_regions__dist - type: - items: - - File - - 'null' - type: array -- id: depth__coverage__regions - type: - items: - - File - - 'null' - type: array -- id: depth__coverage__dist - type: - items: - - File - - 'null' - type: array -- id: depth__coverage__thresholds - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_merged - type: - items: - - File - - 'null' - type: array -- id: variants__samples - type: - items: - items: - items: - - File - - 'null' - type: array - type: array - type: array -- id: reference__viral - secondaryFiles: - - .amb - - .ann - - .sa - - .pac - - ^.dict - - .bwt - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: qc_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: reference__viral - type: File - - name: config__algorithm__qc - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: depth__samtools__stats - type: - - File - - 'null' - - name: depth__samtools__idxstats - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: depth__variant_regions__dist - type: - - File - - 'null' - - name: depth__sv_regions__regions - type: - - File - - 'null' - - name: depth__sv_regions__dist - type: - - File - - 'null' - - name: depth__coverage__regions - type: - - File - - 'null' - - name: depth__coverage__dist - type: - - File - - 'null' - - name: depth__coverage__thresholds - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - name: qc_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/summarize_sv.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/summarize_sv.cwl deleted file mode 100644 index be7b4a8c87f..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/summarize_sv.cwl +++ /dev/null @@ -1,242 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=sv__calls,sv__supplemental,sv__prioritize__tsv,sv__prioritize__raw,svvalidate__grading_summary,svvalidate__grading_plots -- sentinel_inputs=sv_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- summarize_sv -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10243 - ramMin: 3072 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcbio-prioritize - specs: - - https://anaconda.org/bioconda/bcbio-prioritize -inputs: -- id: sv_rec - type: - items: - items: - fields: - - name: sv__variantcaller - type: - - string - - 'null' - - name: sv__vrn_file - type: - - File - - 'null' - - name: sv__supplemental - type: - items: - - File - type: array - - name: svvalidate__summary - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: - - 'null' - - string - - name: genome_resources__variation__gc_profile - type: File - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__svvalidate - type: - - 'null' - - string - - name: genome_resources__aliases__snpeff - type: string - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_rec - type: record - type: array - type: array -outputs: -- id: sv__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: sv__supplemental - type: - items: - items: - - File - type: array - type: array -- id: sv__prioritize__tsv - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: sv__prioritize__raw - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: svvalidate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: svvalidate__grading_plots - type: - items: - items: - - File - - 'null' - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/summarize_vc.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/summarize_vc.cwl deleted file mode 100644 index 62c64099b31..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/summarize_vc.cwl +++ /dev/null @@ -1,195 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=variants__calls,variants__gvcf,variants__samples,validate__grading_summary,validate__grading_plots -- sentinel_inputs=vc_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- summarize_vc -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10241 - ramMin: 3072 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 1 -inputs: -- id: vc_rec - type: - items: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - File - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - File - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array - type: array -outputs: -- id: variants__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: variants__gvcf - type: - items: - - 'null' - - items: - - File - - 'null' - type: array - type: array -- id: variants__samples - type: - items: - items: - items: - - File - - 'null' - type: array - type: array - type: array -- id: validate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: validate__grading_plots - type: - items: - items: - - File - - 'null' - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/variantcall_batch_region.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/variantcall_batch_region.cwl deleted file mode 100644 index 1dc7a6e7128..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/steps/variantcall_batch_region.cwl +++ /dev/null @@ -1,268 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-parallel -- sentinel_outputs=vrn_file_region,region_block -- sentinel_inputs=batch_rec:record,region_block:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- variantcall_batch_region -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 10246 - ramMin: 6144 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: freebayes - specs: - - https://anaconda.org/bioconda/freebayes - version: - - 1.1.0.46 - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 - - package: vqsr_cnn - specs: - - https://anaconda.org/bioconda/vqsr_cnn - - package: deepvariant - specs: - - https://anaconda.org/bioconda/deepvariant - - package: sentieon - specs: - - https://anaconda.org/bioconda/sentieon - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: octopus - specs: - - https://anaconda.org/bioconda/octopus - - package: picard - specs: - - https://anaconda.org/bioconda/picard - - package: platypus-variant - specs: - - https://anaconda.org/bioconda/platypus-variant - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: pysam> - specs: - - https://anaconda.org/bioconda/pysam> - version: - - 0.13.0 - - package: strelka - specs: - - https://anaconda.org/bioconda/strelka - - package: vardict - specs: - - https://anaconda.org/bioconda/vardict - - package: vardict-java - specs: - - https://anaconda.org/bioconda/vardict-java - - package: varscan - specs: - - https://anaconda.org/bioconda/varscan - - package: moreutils - specs: - - https://anaconda.org/bioconda/moreutils - - package: vcfanno - specs: - - https://anaconda.org/bioconda/vcfanno - - package: vcflib - specs: - - https://anaconda.org/bioconda/vcflib - - package: vt - specs: - - https://anaconda.org/bioconda/vt - - package: r - specs: - - https://anaconda.org/bioconda/r - version: - - 3.4.1 - - package: r-base=3.4.1=h4fe35fd_8 - specs: - - https://anaconda.org/bioconda/r-base=3.4.1=h4fe35fd_8 - - package: perl - specs: - - https://anaconda.org/bioconda/perl -- class: arv:APIRequirement -- class: arv:RuntimeConstraints - keep_cache: 4096 -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - File - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - File - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: region_block_toolinput - type: - items: string - type: array -outputs: -- id: vrn_file_region - secondaryFiles: - - .tbi - type: - - File - - 'null' -- id: region_block - type: - items: string - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/wf-alignment.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/wf-alignment.cwl deleted file mode 100644 index c2ce73cfc61..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/wf-alignment.cwl +++ /dev/null @@ -1,144 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: - - string - - 'null' - - boolean - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - - 'null' - - items: 'null' - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: rgnames__pu - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: alignment_rec - type: record -outputs: -- id: align_bam - outputSource: merge_split_alignments/align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__disc - outputSource: merge_split_alignments/work_bam_plus__disc - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__sr - outputSource: merge_split_alignments/work_bam_plus__sr - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: hla__fastq - outputSource: merge_split_alignments/hla__fastq - type: - - 'null' - - items: File - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: prep_align_inputs - in: - - id: alignment_rec - source: alignment_rec - out: - - id: process_alignment_rec - run: steps/prep_align_inputs.cwl -- id: process_alignment - in: - - id: alignment_rec - source: alignment_rec - - id: process_alignment_rec - source: prep_align_inputs/process_alignment_rec - out: - - id: work_bam - - id: align_bam - - id: hla__fastq - - id: work_bam_plus__disc - - id: work_bam_plus__sr - run: steps/process_alignment.cwl - scatter: - - process_alignment_rec - scatterMethod: dotproduct -- id: merge_split_alignments - in: - - id: alignment_rec - source: alignment_rec - - id: work_bam - source: process_alignment/work_bam - - id: align_bam_toolinput - source: process_alignment/align_bam - - id: work_bam_plus__disc_toolinput - source: process_alignment/work_bam_plus__disc - - id: work_bam_plus__sr_toolinput - source: process_alignment/work_bam_plus__sr - - id: hla__fastq_toolinput - source: process_alignment/hla__fastq - out: - - id: align_bam - - id: work_bam_plus__disc - - id: work_bam_plus__sr - - id: hla__fastq - run: steps/merge_split_alignments.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/wf-svcall.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/wf-svcall.cwl deleted file mode 100644 index db9522935a1..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/wf-svcall.cwl +++ /dev/null @@ -1,328 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: sv_batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: - - 'null' - - string - - name: genome_resources__variation__gc_profile - type: File - - name: reference__snpeff__hg19 - type: File - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__svvalidate - type: - - 'null' - - string - - name: genome_resources__aliases__snpeff - type: string - - name: work_bam_plus__disc - type: - - File - - 'null' - - name: work_bam_plus__sr - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_batch_rec - type: record - type: array -outputs: -- id: sv_rec - outputSource: detect_sv/sv_rec - type: - items: - fields: - - name: sv__variantcaller - type: - - string - - 'null' - - name: sv__vrn_file - type: - - File - - 'null' - - name: sv__supplemental - type: - items: - - File - type: array - - name: svvalidate__summary - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: - - 'null' - - string - - name: genome_resources__variation__gc_profile - type: File - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__svvalidate - type: - - 'null' - - string - - name: genome_resources__aliases__snpeff - type: string - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_rec - type: record - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: detect_sv - in: - - id: sv_batch_rec - source: sv_batch_rec - out: - - id: sv_rec - run: steps/detect_sv.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/wf-variantcall.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/wf-variantcall.cwl deleted file mode 100644 index da6bab7ac21..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/somatic-workflow/wf-variantcall.cwl +++ /dev/null @@ -1,312 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - File - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - File - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -outputs: -- id: vc_rec - outputSource: compare_to_rm/vc_rec - type: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - File - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - File - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: get_parallel_regions - in: - - id: batch_rec - source: batch_rec - out: - - id: region_block - run: steps/get_parallel_regions.cwl -- id: variantcall_batch_region - in: - - id: batch_rec - source: batch_rec - - id: region_block_toolinput - source: get_parallel_regions/region_block - out: - - id: vrn_file_region - - id: region_block - run: steps/variantcall_batch_region.cwl - scatter: - - region_block_toolinput - scatterMethod: dotproduct -- id: concat_batch_variantcalls - in: - - id: batch_rec - source: batch_rec - - id: region_block - source: variantcall_batch_region/region_block - - id: vrn_file_region - source: variantcall_batch_region/vrn_file_region - out: - - id: vrn_file - run: steps/concat_batch_variantcalls.cwl -- id: postprocess_variants - in: - - id: batch_rec - source: batch_rec - - id: vrn_file_toolinput - source: concat_batch_variantcalls/vrn_file - out: - - id: vrn_file - run: steps/postprocess_variants.cwl -- id: compare_to_rm - in: - - id: batch_rec - source: batch_rec - - id: vrn_file - source: postprocess_variants/vrn_file - out: - - id: vc_rec - run: steps/compare_to_rm.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/main-svcall-samples.json b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/main-svcall-samples.json deleted file mode 100644 index 28826670c5c..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/main-svcall-samples.json +++ /dev/null @@ -1,707 +0,0 @@ -{ - "analysis": [ - "variant", - "variant" - ], - "config__algorithm__adapters": [ - [], - [] - ], - "config__algorithm__align_split_size": [ - null, - null - ], - "config__algorithm__aligner": [ - "bwa", - "bwa" - ], - "config__algorithm__bam_clean": [ - "False", - "False" - ], - "config__algorithm__coverage_interval": [ - null, - null - ], - "config__algorithm__effects": [ - "snpeff", - "snpeff" - ], - "config__algorithm__ensemble": [ - null, - null - ], - "config__algorithm__exclude_regions": [ - [], - [] - ], - "config__algorithm__mark_duplicates": [ - "True", - "True" - ], - "config__algorithm__min_allele_fraction": [ - 10.0, - 10.0 - ], - "config__algorithm__nomap_split_size": [ - 250, - 250 - ], - "config__algorithm__nomap_split_targets": [ - 20, - 20 - ], - "config__algorithm__qc": [ - [ - "samtools" - ], - [ - "samtools" - ] - ], - "config__algorithm__recalibrate": [ - "False", - "False" - ], - "config__algorithm__sv_regions": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed" - } - ], - "config__algorithm__svcaller": [ - [ - "cnvkit", - "manta", - "lumpy" - ], - [ - "cnvkit", - "manta", - "lumpy" - ] - ], - "config__algorithm__svprioritize": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed" - } - ], - "config__algorithm__svvalidate": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/reference_material/Test1-grade.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/reference_material/Test1-grade.vcf.gz.tbi" - } - ] - }, - null - ], - "config__algorithm__tools_off": [ - [], - [] - ], - "config__algorithm__tools_on": [ - [ - "coverage_perbase" - ], - [ - "coverage_perbase" - ] - ], - "config__algorithm__trim_reads": [ - "False", - "False" - ], - "config__algorithm__validate": [ - null, - null - ], - "config__algorithm__validate_regions": [ - null, - null - ], - "config__algorithm__variant_regions": [ - null, - null - ], - "config__algorithm__variantcaller": [ - [ - "vardict" - ], - [ - "vardict" - ] - ], - "config__algorithm__vcfanno": [ - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/config/vcfanno/gemini.conf" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/config/vcfanno/gemini.lua" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/config/vcfanno/somatic.conf" - } - ], - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/config/vcfanno/gemini.conf" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/config/vcfanno/gemini.lua" - } - ] - ], - "description": [ - "Test1", - "Test2" - ], - "files": [ - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam.bai" - } - ] - } - ], - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam.bai" - } - ] - } - ] - ], - "genome_build": [ - "hg19", - "hg19" - ], - "genome_resources__aliases__ensembl": [ - "homo_sapiens_vep_83_GRCh37", - "homo_sapiens_vep_83_GRCh37" - ], - "genome_resources__aliases__human": [ - "True", - "True" - ], - "genome_resources__aliases__snpeff": [ - "hg19", - "hg19" - ], - "genome_resources__rnaseq__gene_bed": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rnaseq/ref-transcripts.bed" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rnaseq/ref-transcripts.bed" - } - ], - "genome_resources__variation__1000g": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/1000g.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__clinvar": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/clinvar.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__cosmic": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/cosmic.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__dbsnp": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/dbsnp.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__encode_blacklist": [ - null, - null - ], - "genome_resources__variation__esp": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/esp.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__exac": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/exac.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__gc_profile": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/gc/GC_profile.1000bp.cnp" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/gc/GC_profile.1000bp.cnp" - } - ], - "genome_resources__variation__germline_het_pon": [ - null, - null - ], - "genome_resources__variation__gnomad_exome": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/gnomad_exome.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__lcr": [ - null, - null - ], - "genome_resources__variation__polyx": [ - null, - null - ], - "genome_resources__variation__train_hapmap": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__train_indels": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz.tbi" - } - ] - } - ], - "metadata__batch": [ - "b1", - "b1" - ], - "metadata__phenotype": [ - "tumor", - "normal" - ], - "reference__bwa__indexes": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.ann" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.bwt" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.pac" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.sa" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.ann" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.bwt" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.pac" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.sa" - } - ] - } - ], - "reference__fasta__base": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa.fai" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.dict" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa.fai" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.dict" - } - ] - } - ], - "reference__genome_context": [ - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz.tbi" - } - ] - } - ], - [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz.tbi" - } - ] - } - ] - ], - "reference__rtg": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rtg--hg19.sdf-wf.tar.gz" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/rtg--hg19.sdf-wf.tar.gz" - } - ], - "reference__snpeff__hg19": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/snpeff--hg19-wf.tar.gz" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/snpeff--hg19-wf.tar.gz" - } - ], - "reference__versions": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/versions.csv" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/versions.csv" - } - ], - "reference__viral": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.pac" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.amb" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.sa" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.dict" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.ann" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.bwt" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.pac" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.amb" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.sa" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.dict" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.ann" - }, - { - "class": "File", - "path": "gs://bcbiodata/test_bcbio_cwl/testdata/genomes/hg19/viral/gdc-viral.fa.bwt" - } - ] - } - ], - "resources": [ - "{}", - "{}" - ], - "rgnames__lane": [ - "Test1", - "Test2" - ], - "rgnames__lb": [ - null, - null - ], - "rgnames__pl": [ - "illumina", - "illumina" - ], - "rgnames__pu": [ - "Test1", - "Test2" - ], - "rgnames__rg": [ - "Test1", - "Test2" - ], - "rgnames__sample": [ - "Test1", - "Test2" - ], - "vrn_file": [ - null, - null - ] -} diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/main-svcall.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/main-svcall.cwl deleted file mode 100644 index ec310cc54cd..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/main-svcall.cwl +++ /dev/null @@ -1,975 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: config__algorithm__align_split_size - type: - items: - - 'null' - - string - type: array -- id: files - secondaryFiles: - - .bai - type: - items: - items: File - type: array - type: array -- id: config__algorithm__trim_reads - type: - items: - - string - - 'null' - - boolean - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: config__algorithm__vcfanno - type: - items: - items: File - type: array - type: array -- id: config__algorithm__svprioritize - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: config__algorithm__variantcaller - type: - items: - items: string - type: array - type: array -- id: config__algorithm__adapters - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__svcaller - type: - items: - items: string - type: array - type: array -- id: config__algorithm__coverage_interval - type: - items: - - 'null' - - string - type: array -- id: genome_resources__rnaseq__gene_bed - type: - items: File - type: array -- id: genome_resources__variation__gc_profile - type: - items: File - type: array -- id: rgnames__lb - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__clinvar - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__esp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: rgnames__rg - type: - items: string - type: array -- id: metadata__batch - type: - items: string - type: array -- id: rgnames__lane - type: - items: string - type: array -- id: genome_resources__variation__lcr - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__1000g - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__min_allele_fraction - type: - items: double - type: array -- id: config__algorithm__nomap_split_targets - type: - items: long - type: array -- id: reference__versions - type: - items: File - type: array -- id: reference__bwa__indexes - secondaryFiles: - - ^.ann - - ^.pac - - ^.sa - - ^.bwt - type: - items: File - type: array -- id: vrn_file - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__train_hapmap - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: reference__genome_context - secondaryFiles: - - .tbi - type: - items: - items: File - type: array - type: array -- id: config__algorithm__bam_clean - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__nomap_split_size - type: - items: long - type: array -- id: config__algorithm__validate - type: - items: - - 'null' - - string - type: array -- id: reference__snpeff__hg19 - type: - items: File - type: array -- id: description - type: - items: string - type: array -- id: config__algorithm__validate_regions - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__aligner - type: - items: string - type: array -- id: rgnames__pl - type: - items: string - type: array -- id: genome_build - type: - items: string - type: array -- id: rgnames__pu - type: - items: string - type: array -- id: genome_resources__variation__germline_het_pon - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__exac - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__gnomad_exome - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__recalibrate - type: - items: - - string - - 'null' - - boolean - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: genome_resources__aliases__human - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: genome_resources__variation__dbsnp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__mark_duplicates - type: - items: - - string - - 'null' - - boolean - type: array -- id: genome_resources__variation__polyx - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__encode_blacklist - type: - items: - - 'null' - - string - type: array -- id: reference__viral - secondaryFiles: - - .amb - - .ann - - .sa - - .pac - - ^.dict - - .bwt - type: - items: File - type: array -- id: genome_resources__variation__cosmic - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__ensemble - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__qc - type: - items: - items: string - type: array - type: array -- id: config__algorithm__sv_regions - type: - items: File - type: array -- id: analysis - type: - items: string - type: array -- id: rgnames__sample - type: - items: string - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: config__algorithm__effects - type: - items: string - type: array -- id: config__algorithm__variant_regions - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__svvalidate - secondaryFiles: - - .tbi - type: - items: - - File - - 'null' - type: array -- id: genome_resources__aliases__ensembl - type: - items: string - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: reference__rtg - type: - items: File - type: array -- id: genome_resources__variation__train_indels - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__aliases__snpeff - type: - items: string - type: array -outputs: -- id: rgnames__sample_out - outputSource: prep_samples/rgnames__sample - type: - items: string - type: array -- id: align_bam - outputSource: postprocess_alignment/align_bam - type: - items: - - File - - 'null' - type: array -- id: regions__sample_callable - outputSource: postprocess_alignment/regions__sample_callable - type: - items: - - File - - 'null' - type: array -- id: validate__grading_summary - outputSource: summarize_vc/validate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: variants__calls - outputSource: summarize_vc/variants__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: variants__gvcf - outputSource: summarize_vc/variants__gvcf - type: - items: - - 'null' - - items: - - File - - 'null' - type: array - type: array -- id: sv__calls - outputSource: summarize_sv/sv__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: svvalidate__grading_summary - outputSource: summarize_sv/svvalidate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: sv__prioritize__tsv - outputSource: summarize_sv/sv__prioritize__tsv - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: sv__prioritize__raw - outputSource: summarize_sv/sv__prioritize__raw - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: sv__supplemental - outputSource: summarize_sv/sv__supplemental - type: - items: - items: - - File - type: array - type: array -- id: summary__multiqc - outputSource: multiqc_summary/summary__multiqc - type: - items: - - File - - 'null' - type: array -- id: versions__tools - outputSource: multiqc_summary/versions__tools - type: - items: - - File - - 'null' - type: array -- id: versions__data - outputSource: multiqc_summary/versions__data - type: - items: - - File - - 'null' - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: alignment_to_rec - in: - - id: files - source: files - - id: analysis - source: analysis - - id: config__algorithm__align_split_size - source: config__algorithm__align_split_size - - id: reference__fasta__base - source: reference__fasta__base - - id: rgnames__pl - source: rgnames__pl - - id: rgnames__sample - source: rgnames__sample - - id: rgnames__pu - source: rgnames__pu - - id: rgnames__lane - source: rgnames__lane - - id: rgnames__rg - source: rgnames__rg - - id: rgnames__lb - source: rgnames__lb - - id: reference__bwa__indexes - source: reference__bwa__indexes - - id: config__algorithm__aligner - source: config__algorithm__aligner - - id: config__algorithm__trim_reads - source: config__algorithm__trim_reads - - id: config__algorithm__adapters - source: config__algorithm__adapters - - id: config__algorithm__bam_clean - source: config__algorithm__bam_clean - - id: config__algorithm__variant_regions - source: config__algorithm__variant_regions - - id: config__algorithm__mark_duplicates - source: config__algorithm__mark_duplicates - - id: resources - source: resources - - id: description - source: description - out: - - id: alignment_rec - run: steps/alignment_to_rec.cwl -- id: alignment - in: - - id: alignment_rec - source: alignment_to_rec/alignment_rec - out: - - id: align_bam - - id: work_bam_plus__disc - - id: work_bam_plus__sr - - id: hla__fastq - run: wf-alignment.cwl - scatter: - - alignment_rec - scatterMethod: dotproduct -- id: prep_samples_to_rec - in: - - id: rgnames__sample - source: rgnames__sample - - id: config__algorithm__variant_regions - source: config__algorithm__variant_regions - - id: reference__fasta__base - source: reference__fasta__base - - id: resources - source: resources - - id: description - source: description - out: - - id: prep_samples_rec - run: steps/prep_samples_to_rec.cwl -- id: prep_samples - in: - - id: prep_samples_rec - source: prep_samples_to_rec/prep_samples_rec - out: - - id: rgnames__sample - - id: config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - - id: config__algorithm__variant_regions_orig - - id: config__algorithm__coverage - - id: config__algorithm__coverage_merged - - id: config__algorithm__coverage_orig - - id: config__algorithm__seq2c_bed_ready - run: steps/prep_samples.cwl - scatter: - - prep_samples_rec - scatterMethod: dotproduct -- id: postprocess_alignment_to_rec - in: - - id: align_bam - source: alignment/align_bam - - id: config__algorithm__coverage_interval - source: config__algorithm__coverage_interval - - id: config__algorithm__exclude_regions - source: config__algorithm__exclude_regions - - id: config__algorithm__variant_regions - source: prep_samples/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: prep_samples/config__algorithm__variant_regions_merged - - id: config__algorithm__variant_regions_orig - source: prep_samples/config__algorithm__variant_regions_orig - - id: config__algorithm__coverage - source: prep_samples/config__algorithm__coverage - - id: config__algorithm__coverage_merged - source: prep_samples/config__algorithm__coverage_merged - - id: config__algorithm__coverage_orig - source: prep_samples/config__algorithm__coverage_orig - - id: config__algorithm__seq2c_bed_ready - source: prep_samples/config__algorithm__seq2c_bed_ready - - id: config__algorithm__recalibrate - source: config__algorithm__recalibrate - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: genome_resources__rnaseq__gene_bed - source: genome_resources__rnaseq__gene_bed - - id: genome_resources__variation__dbsnp - source: genome_resources__variation__dbsnp - - id: genome_resources__variation__lcr - source: genome_resources__variation__lcr - - id: genome_resources__variation__polyx - source: genome_resources__variation__polyx - - id: genome_resources__variation__encode_blacklist - source: genome_resources__variation__encode_blacklist - - id: reference__fasta__base - source: reference__fasta__base - - id: resources - source: resources - - id: description - source: description - out: - - id: postprocess_alignment_rec - run: steps/postprocess_alignment_to_rec.cwl -- id: postprocess_alignment - in: - - id: postprocess_alignment_rec - source: postprocess_alignment_to_rec/postprocess_alignment_rec - out: - - id: config__algorithm__coverage_interval - - id: config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - - id: config__algorithm__variant_regions_orig - - id: config__algorithm__coverage - - id: config__algorithm__coverage_merged - - id: config__algorithm__coverage_orig - - id: config__algorithm__seq2c_bed_ready - - id: regions__callable - - id: regions__sample_callable - - id: regions__nblock - - id: depth__samtools__stats - - id: depth__samtools__idxstats - - id: depth__variant_regions__regions - - id: depth__variant_regions__dist - - id: depth__sv_regions__regions - - id: depth__sv_regions__dist - - id: depth__coverage__regions - - id: depth__coverage__dist - - id: depth__coverage__thresholds - - id: align_bam - run: steps/postprocess_alignment.cwl - scatter: - - postprocess_alignment_rec - scatterMethod: dotproduct -- id: combine_sample_regions - in: - - id: regions__callable - source: postprocess_alignment/regions__callable - - id: regions__nblock - source: postprocess_alignment/regions__nblock - - id: metadata__batch - source: metadata__batch - - id: config__algorithm__nomap_split_size - source: config__algorithm__nomap_split_size - - id: config__algorithm__nomap_split_targets - source: config__algorithm__nomap_split_targets - - id: reference__fasta__base - source: reference__fasta__base - - id: resources - source: resources - - id: description - source: description - out: - - id: config__algorithm__callable_regions - - id: config__algorithm__non_callable_regions - - id: config__algorithm__callable_count - run: steps/combine_sample_regions.cwl -- id: batch_for_variantcall - in: - - id: analysis - source: analysis - - id: genome_build - source: genome_build - - id: align_bam - source: postprocess_alignment/align_bam - - id: vrn_file - source: vrn_file - - id: metadata__batch - source: metadata__batch - - id: metadata__phenotype - source: metadata__phenotype - - id: config__algorithm__callable_regions - source: combine_sample_regions/config__algorithm__callable_regions - - id: regions__sample_callable - source: postprocess_alignment/regions__sample_callable - - id: config__algorithm__variantcaller - source: config__algorithm__variantcaller - - id: config__algorithm__ensemble - source: config__algorithm__ensemble - - id: config__algorithm__vcfanno - source: config__algorithm__vcfanno - - id: config__algorithm__coverage_interval - source: postprocess_alignment/config__algorithm__coverage_interval - - id: config__algorithm__effects - source: config__algorithm__effects - - id: config__algorithm__min_allele_fraction - source: config__algorithm__min_allele_fraction - - id: config__algorithm__exclude_regions - source: config__algorithm__exclude_regions - - id: config__algorithm__variant_regions - source: postprocess_alignment/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: postprocess_alignment/config__algorithm__variant_regions_merged - - id: config__algorithm__validate - source: config__algorithm__validate - - id: config__algorithm__validate_regions - source: config__algorithm__validate_regions - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: config__algorithm__tools_off - source: config__algorithm__tools_off - - id: reference__fasta__base - source: reference__fasta__base - - id: reference__rtg - source: reference__rtg - - id: reference__genome_context - source: reference__genome_context - - id: genome_resources__variation__clinvar - source: genome_resources__variation__clinvar - - id: genome_resources__variation__cosmic - source: genome_resources__variation__cosmic - - id: genome_resources__variation__dbsnp - source: genome_resources__variation__dbsnp - - id: genome_resources__variation__esp - source: genome_resources__variation__esp - - id: genome_resources__variation__exac - source: genome_resources__variation__exac - - id: genome_resources__variation__gnomad_exome - source: genome_resources__variation__gnomad_exome - - id: genome_resources__variation__1000g - source: genome_resources__variation__1000g - - id: genome_resources__variation__lcr - source: genome_resources__variation__lcr - - id: genome_resources__variation__polyx - source: genome_resources__variation__polyx - - id: genome_resources__variation__encode_blacklist - source: genome_resources__variation__encode_blacklist - - id: genome_resources__aliases__ensembl - source: genome_resources__aliases__ensembl - - id: genome_resources__aliases__human - source: genome_resources__aliases__human - - id: genome_resources__aliases__snpeff - source: genome_resources__aliases__snpeff - - id: reference__snpeff__hg19 - source: reference__snpeff__hg19 - - id: genome_resources__variation__train_hapmap - source: genome_resources__variation__train_hapmap - - id: genome_resources__variation__train_indels - source: genome_resources__variation__train_indels - - id: resources - source: resources - - id: description - source: description - out: - - id: batch_rec - run: steps/batch_for_variantcall.cwl -- id: variantcall - in: - - id: batch_rec - source: batch_for_variantcall/batch_rec - out: - - id: vc_rec - run: wf-variantcall.cwl - scatter: - - batch_rec - scatterMethod: dotproduct -- id: summarize_vc - in: - - id: vc_rec - source: variantcall/vc_rec - out: - - id: variants__calls - - id: variants__gvcf - - id: variants__samples - - id: validate__grading_summary - - id: validate__grading_plots - run: steps/summarize_vc.cwl -- id: calculate_sv_bins - in: - - id: align_bam - source: postprocess_alignment/align_bam - - id: reference__fasta__base - source: reference__fasta__base - - id: metadata__batch - source: metadata__batch - - id: metadata__phenotype - source: metadata__phenotype - - id: config__algorithm__callable_regions - source: combine_sample_regions/config__algorithm__callable_regions - - id: config__algorithm__coverage_interval - source: postprocess_alignment/config__algorithm__coverage_interval - - id: config__algorithm__exclude_regions - source: config__algorithm__exclude_regions - - id: config__algorithm__sv_regions - source: config__algorithm__sv_regions - - id: config__algorithm__variant_regions - source: postprocess_alignment/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: postprocess_alignment/config__algorithm__variant_regions_merged - - id: config__algorithm__svcaller - source: config__algorithm__svcaller - - id: depth__variant_regions__regions - source: postprocess_alignment/depth__variant_regions__regions - - id: genome_resources__variation__lcr - source: genome_resources__variation__lcr - - id: genome_resources__variation__polyx - source: genome_resources__variation__polyx - - id: genome_resources__variation__encode_blacklist - source: genome_resources__variation__encode_blacklist - - id: genome_resources__rnaseq__gene_bed - source: genome_resources__rnaseq__gene_bed - - id: resources - source: resources - - id: description - source: description - out: - - id: sv_bin_rec - run: steps/calculate_sv_bins.cwl -- id: calculate_sv_coverage - in: - - id: sv_bin_rec - source: calculate_sv_bins/sv_bin_rec - out: - - id: sv_rawcoverage_rec - run: steps/calculate_sv_coverage.cwl - scatter: - - sv_bin_rec - scatterMethod: dotproduct -- id: normalize_sv_coverage - in: - - id: sv_rawcoverage_rec - source: calculate_sv_coverage/sv_rawcoverage_rec - out: - - id: sv_coverage_rec - run: steps/normalize_sv_coverage.cwl -- id: batch_for_sv - in: - - id: analysis - source: analysis - - id: genome_build - source: genome_build - - id: work_bam_plus__disc - source: alignment/work_bam_plus__disc - - id: work_bam_plus__sr - source: alignment/work_bam_plus__sr - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: config__algorithm__tools_off - source: config__algorithm__tools_off - - id: config__algorithm__svprioritize - source: config__algorithm__svprioritize - - id: config__algorithm__svvalidate - source: config__algorithm__svvalidate - - id: regions__sample_callable - source: postprocess_alignment/regions__sample_callable - - id: genome_resources__variation__gc_profile - source: genome_resources__variation__gc_profile - - id: genome_resources__variation__germline_het_pon - source: genome_resources__variation__germline_het_pon - - id: genome_resources__aliases__snpeff - source: genome_resources__aliases__snpeff - - id: reference__snpeff__hg19 - source: reference__snpeff__hg19 - - id: sv_coverage_rec - source: normalize_sv_coverage/sv_coverage_rec - - id: variants__samples - source: summarize_vc/variants__samples - out: - - id: sv_batch_rec - run: steps/batch_for_sv.cwl -- id: svcall - in: - - id: sv_batch_rec - source: batch_for_sv/sv_batch_rec - out: - - id: sv_rec - run: wf-svcall.cwl - scatter: - - sv_batch_rec - scatterMethod: dotproduct -- id: summarize_sv - in: - - id: sv_rec - source: svcall/sv_rec - out: - - id: sv__calls - - id: sv__supplemental - - id: sv__prioritize__tsv - - id: sv__prioritize__raw - - id: svvalidate__grading_summary - - id: svvalidate__grading_plots - run: steps/summarize_sv.cwl -- id: qc_to_rec - in: - - id: align_bam - source: postprocess_alignment/align_bam - - id: analysis - source: analysis - - id: reference__fasta__base - source: reference__fasta__base - - id: reference__versions - source: reference__versions - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: config__algorithm__tools_off - source: config__algorithm__tools_off - - id: genome_build - source: genome_build - - id: config__algorithm__qc - source: config__algorithm__qc - - id: metadata__batch - source: metadata__batch - - id: metadata__phenotype - source: metadata__phenotype - - id: config__algorithm__coverage_interval - source: postprocess_alignment/config__algorithm__coverage_interval - - id: depth__variant_regions__regions - source: postprocess_alignment/depth__variant_regions__regions - - id: depth__variant_regions__dist - source: postprocess_alignment/depth__variant_regions__dist - - id: depth__samtools__stats - source: postprocess_alignment/depth__samtools__stats - - id: depth__samtools__idxstats - source: postprocess_alignment/depth__samtools__idxstats - - id: depth__sv_regions__regions - source: postprocess_alignment/depth__sv_regions__regions - - id: depth__sv_regions__dist - source: postprocess_alignment/depth__sv_regions__dist - - id: depth__coverage__regions - source: postprocess_alignment/depth__coverage__regions - - id: depth__coverage__dist - source: postprocess_alignment/depth__coverage__dist - - id: depth__coverage__thresholds - source: postprocess_alignment/depth__coverage__thresholds - - id: config__algorithm__variant_regions - source: postprocess_alignment/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: postprocess_alignment/config__algorithm__variant_regions_merged - - id: config__algorithm__coverage - source: postprocess_alignment/config__algorithm__coverage - - id: config__algorithm__coverage_merged - source: postprocess_alignment/config__algorithm__coverage_merged - - id: variants__samples - source: summarize_vc/variants__samples - - id: reference__viral - source: reference__viral - - id: resources - source: resources - - id: description - source: description - out: - - id: qc_rec - run: steps/qc_to_rec.cwl -- id: pipeline_summary - in: - - id: qc_rec - source: qc_to_rec/qc_rec - out: - - id: qcout_rec - run: steps/pipeline_summary.cwl - scatter: - - qc_rec - scatterMethod: dotproduct -- id: multiqc_summary - in: - - id: qcout_rec - source: pipeline_summary/qcout_rec - out: - - id: summary__multiqc - - id: versions__tools - - id: versions__data - run: steps/multiqc_summary.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/alignment_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/alignment_to_rec.cwl deleted file mode 100644 index 8208a1f1eed..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/alignment_to_rec.cwl +++ /dev/null @@ -1,206 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=alignment_rec:resources;description;config__algorithm__align_split_size;files;config__algorithm__trim_reads;reference__fasta__base;config__algorithm__adapters;rgnames__lb;rgnames__rg;rgnames__lane;reference__bwa__indexes;config__algorithm__bam_clean;config__algorithm__aligner;rgnames__pl;rgnames__pu;config__algorithm__mark_duplicates;analysis;rgnames__sample;config__algorithm__variant_regions -- sentinel_inputs=files:var,analysis:var,config__algorithm__align_split_size:var,reference__fasta__base:var,rgnames__pl:var,rgnames__sample:var,rgnames__pu:var,rgnames__lane:var,rgnames__rg:var,rgnames__lb:var,reference__bwa__indexes:var,config__algorithm__aligner:var,config__algorithm__trim_reads:var,config__algorithm__adapters:var,config__algorithm__bam_clean:var,config__algorithm__variant_regions:var,config__algorithm__mark_duplicates:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- alignment_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 3072 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: files - secondaryFiles: - - .bai - type: - items: - items: File - type: array - type: array -- id: analysis - type: - items: string - type: array -- id: config__algorithm__align_split_size - type: - items: - - 'null' - - string - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: rgnames__pl - type: - items: string - type: array -- id: rgnames__sample - type: - items: string - type: array -- id: rgnames__pu - type: - items: string - type: array -- id: rgnames__lane - type: - items: string - type: array -- id: rgnames__rg - type: - items: string - type: array -- id: rgnames__lb - type: - items: - - 'null' - - string - type: array -- id: reference__bwa__indexes - secondaryFiles: - - ^.ann - - ^.pac - - ^.sa - - ^.bwt - type: - items: File - type: array -- id: config__algorithm__aligner - type: - items: string - type: array -- id: config__algorithm__trim_reads - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__adapters - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__bam_clean - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__variant_regions - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__mark_duplicates - type: - items: - - string - - 'null' - - boolean - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: alignment_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: - - string - - 'null' - - boolean - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - - 'null' - - items: 'null' - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: rgnames__pu - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: - - 'null' - - string - name: alignment_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/batch_for_sv.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/batch_for_sv.cwl deleted file mode 100644 index 5c725369def..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/batch_for_sv.cwl +++ /dev/null @@ -1,359 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-batch -- sentinel_outputs=sv_batch_rec:resources;description;config__algorithm__svprioritize;genome_resources__variation__gc_profile;reference__snpeff__hg19;genome_build;genome_resources__variation__germline_het_pon;config__algorithm__tools_off;analysis;config__algorithm__tools_on;config__algorithm__svvalidate;genome_resources__aliases__snpeff;work_bam_plus__disc;work_bam_plus__sr;regions__sample_callable;variants__samples;depth__bins__normalized;depth__bins__background;depth__bins__target;depth__bins__antitarget;regions__bins__target;regions__bins__antitarget;regions__bins__gcannotated;regions__bins__group;reference__fasta__base;config__algorithm__svcaller;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;metadata__batch;genome_resources__variation__lcr;metadata__phenotype;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__sv_regions;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;depth__variant_regions__regions;config__algorithm__callable_regions -- sentinel_inputs=analysis:var,genome_build:var,work_bam_plus__disc:var,work_bam_plus__sr:var,config__algorithm__tools_on:var,config__algorithm__tools_off:var,config__algorithm__svprioritize:var,config__algorithm__svvalidate:var,regions__sample_callable:var,genome_resources__variation__gc_profile:var,genome_resources__variation__germline_het_pon:var,genome_resources__aliases__snpeff:var,reference__snpeff__hg19:var,sv_coverage_rec:record,variants__samples:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- batch_for_sv -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1024 - ramMin: 6144 - tmpdirMin: 0 -- class: dx:InputResourceRequirement - indirMin: 1 -inputs: -- id: analysis - type: - items: string - type: array -- id: genome_build - type: - items: string - type: array -- id: work_bam_plus__disc - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: work_bam_plus__sr - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__svprioritize - type: - items: File - type: array -- id: config__algorithm__svvalidate - secondaryFiles: - - .tbi - type: - items: - - File - - 'null' - type: array -- id: regions__sample_callable - type: - items: - - File - - 'null' - type: array -- id: genome_resources__variation__gc_profile - type: - items: File - type: array -- id: genome_resources__variation__germline_het_pon - type: - items: - - 'null' - - string - type: array -- id: genome_resources__aliases__snpeff - type: - items: string - type: array -- id: reference__snpeff__hg19 - type: - items: File - type: array -- id: sv_coverage_rec - type: - items: - fields: - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__sv_regions - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_coverage_rec - type: record - type: array -- id: variants__samples - type: - items: - items: - items: - - File - - 'null' - type: array - type: array - type: array -outputs: -- id: sv_batch_rec - type: - items: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: File - - name: genome_resources__variation__gc_profile - type: File - - name: reference__snpeff__hg19 - type: File - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__svvalidate - type: - - File - - 'null' - - name: genome_resources__aliases__snpeff - type: string - - name: work_bam_plus__disc - type: - - File - - 'null' - - name: work_bam_plus__sr - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__sv_regions - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_batch_rec - type: record - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/batch_for_variantcall.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/batch_for_variantcall.cwl deleted file mode 100644 index 1a04f1e6887..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/batch_for_variantcall.cwl +++ /dev/null @@ -1,401 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-batch -- sentinel_outputs=batch_rec:resources;description;reference__fasta__base;config__algorithm__vcfanno;config__algorithm__variantcaller;config__algorithm__coverage_interval;genome_resources__variation__clinvar;genome_resources__variation__esp;metadata__batch;genome_resources__variation__lcr;genome_resources__variation__1000g;config__algorithm__min_allele_fraction;vrn_file;genome_resources__variation__train_hapmap;reference__genome_context;config__algorithm__validate;reference__snpeff__hg19;config__algorithm__validate_regions;genome_build;genome_resources__variation__exac;genome_resources__variation__gnomad_exome;metadata__phenotype;genome_resources__aliases__human;config__algorithm__tools_off;genome_resources__variation__dbsnp;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;genome_resources__variation__cosmic;config__algorithm__ensemble;analysis;config__algorithm__tools_on;config__algorithm__effects;config__algorithm__variant_regions;genome_resources__aliases__ensembl;config__algorithm__exclude_regions;reference__rtg;genome_resources__variation__train_indels;genome_resources__aliases__snpeff;align_bam;config__algorithm__variant_regions_merged;regions__sample_callable;config__algorithm__callable_regions -- sentinel_inputs=analysis:var,genome_build:var,align_bam:var,vrn_file:var,metadata__batch:var,metadata__phenotype:var,config__algorithm__callable_regions:var,regions__sample_callable:var,config__algorithm__variantcaller:var,config__algorithm__ensemble:var,config__algorithm__vcfanno:var,config__algorithm__coverage_interval:var,config__algorithm__effects:var,config__algorithm__min_allele_fraction:var,config__algorithm__exclude_regions:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__validate:var,config__algorithm__validate_regions:var,config__algorithm__tools_on:var,config__algorithm__tools_off:var,reference__fasta__base:var,reference__rtg:var,reference__genome_context:var,genome_resources__variation__clinvar:var,genome_resources__variation__cosmic:var,genome_resources__variation__dbsnp:var,genome_resources__variation__esp:var,genome_resources__variation__exac:var,genome_resources__variation__gnomad_exome:var,genome_resources__variation__1000g:var,genome_resources__variation__lcr:var,genome_resources__variation__polyx:var,genome_resources__variation__encode_blacklist:var,genome_resources__aliases__ensembl:var,genome_resources__aliases__human:var,genome_resources__aliases__snpeff:var,reference__snpeff__hg19:var,genome_resources__variation__train_hapmap:var,genome_resources__variation__train_indels:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- batch_for_variantcall -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1030 - ramMin: 3072 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: analysis - type: - items: string - type: array -- id: genome_build - type: - items: string - type: array -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: vrn_file - type: - items: - - 'null' - - string - type: array -- id: metadata__batch - type: - items: string - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: config__algorithm__callable_regions - type: - items: File - type: array -- id: regions__sample_callable - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variantcaller - type: - items: - items: string - type: array - type: array -- id: config__algorithm__ensemble - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__vcfanno - type: - items: - items: File - type: array - type: array -- id: config__algorithm__coverage_interval - type: - items: - - string - - 'null' - type: array -- id: config__algorithm__effects - type: - items: string - type: array -- id: config__algorithm__min_allele_fraction - type: - items: double - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__validate - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__validate_regions - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: reference__rtg - type: - items: File - type: array -- id: reference__genome_context - secondaryFiles: - - .tbi - type: - items: - items: File - type: array - type: array -- id: genome_resources__variation__clinvar - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__cosmic - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__dbsnp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__esp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__exac - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__gnomad_exome - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__1000g - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__lcr - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__polyx - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__encode_blacklist - type: - items: - - 'null' - - string - type: array -- id: genome_resources__aliases__ensembl - type: - items: string - type: array -- id: genome_resources__aliases__human - type: - items: - - string - - 'null' - - boolean - type: array -- id: genome_resources__aliases__snpeff - type: - items: string - type: array -- id: reference__snpeff__hg19 - type: - items: File - type: array -- id: genome_resources__variation__train_hapmap - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__train_indels - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: batch_rec - type: - items: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/calculate_sv_bins.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/calculate_sv_bins.cwl deleted file mode 100644 index 34683452f0d..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/calculate_sv_bins.cwl +++ /dev/null @@ -1,221 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=sv_bin_rec:regions__bins__target;regions__bins__antitarget;regions__bins__gcannotated;regions__bins__group;resources;description;reference__fasta__base;config__algorithm__svcaller;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;metadata__batch;genome_resources__variation__lcr;metadata__phenotype;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__sv_regions;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;depth__variant_regions__regions;config__algorithm__callable_regions -- sentinel_inputs=align_bam:var,reference__fasta__base:var,metadata__batch:var,metadata__phenotype:var,config__algorithm__callable_regions:var,config__algorithm__coverage_interval:var,config__algorithm__exclude_regions:var,config__algorithm__sv_regions:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__svcaller:var,depth__variant_regions__regions:var,genome_resources__variation__lcr:var,genome_resources__variation__polyx:var,genome_resources__variation__encode_blacklist:var,genome_resources__rnaseq__gene_bed:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- calculate_sv_bins -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 3072 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: cnvkit - specs: - - https://anaconda.org/bioconda/cnvkit -inputs: -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: metadata__batch - type: - items: string - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: config__algorithm__callable_regions - type: - items: File - type: array -- id: config__algorithm__coverage_interval - type: - items: - - string - - 'null' - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__sv_regions - type: - items: File - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__svcaller - type: - items: - items: string - type: array - type: array -- id: depth__variant_regions__regions - type: - items: - - File - - 'null' - type: array -- id: genome_resources__variation__lcr - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__polyx - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__encode_blacklist - type: - items: - - 'null' - - string - type: array -- id: genome_resources__rnaseq__gene_bed - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: sv_bin_rec - type: - items: - fields: - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__sv_regions - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_bin_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/calculate_sv_coverage.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/calculate_sv_coverage.cwl deleted file mode 100644 index 79ba8e7eaf5..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/calculate_sv_coverage.cwl +++ /dev/null @@ -1,207 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=sv_rawcoverage_rec:depth__bins__target;depth__bins__antitarget;resources;description;regions__bins__target;regions__bins__antitarget;regions__bins__gcannotated;regions__bins__group;reference__fasta__base;config__algorithm__svcaller;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;metadata__batch;genome_resources__variation__lcr;metadata__phenotype;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__sv_regions;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;depth__variant_regions__regions;config__algorithm__callable_regions -- sentinel_inputs=sv_bin_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- calculate_sv_coverage -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1028 - ramMin: 6144 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: cnvkit - specs: - - https://anaconda.org/bioconda/cnvkit -inputs: -- id: sv_bin_rec - type: - fields: - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__sv_regions - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_bin_rec - type: record -outputs: -- id: sv_rawcoverage_rec - type: - fields: - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__sv_regions - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_rawcoverage_rec - type: record -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/combine_sample_regions.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/combine_sample_regions.cwl deleted file mode 100644 index 434161e89da..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/combine_sample_regions.cwl +++ /dev/null @@ -1,99 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=config__algorithm__callable_regions,config__algorithm__non_callable_regions,config__algorithm__callable_count -- sentinel_inputs=regions__callable:var,regions__nblock:var,metadata__batch:var,config__algorithm__nomap_split_size:var,config__algorithm__nomap_split_targets:var,reference__fasta__base:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- combine_sample_regions -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1025 - ramMin: 3072 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 -- class: arv:APIRequirement -inputs: -- id: regions__callable - type: - items: - - File - - 'null' - type: array -- id: regions__nblock - type: - items: - - File - - 'null' - type: array -- id: metadata__batch - type: - items: string - type: array -- id: config__algorithm__nomap_split_size - type: - items: long - type: array -- id: config__algorithm__nomap_split_targets - type: - items: long - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: config__algorithm__callable_regions - type: - items: File - type: array -- id: config__algorithm__non_callable_regions - type: - items: File - type: array -- id: config__algorithm__callable_count - type: - items: int - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/compare_to_rm.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/compare_to_rm.cwl deleted file mode 100644 index 142a0d44ce9..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/compare_to_rm.cwl +++ /dev/null @@ -1,307 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-single -- sentinel_outputs=vc_rec:batch_samples;validate__summary;validate__tp;validate__fp;validate__fn;resources;description;vrn_file;reference__fasta__base;config__algorithm__vcfanno;config__algorithm__variantcaller;config__algorithm__coverage_interval;metadata__batch;config__algorithm__min_allele_fraction;reference__genome_context;config__algorithm__validate;reference__snpeff__hg19;config__algorithm__validate_regions;genome_build;metadata__phenotype;genome_resources__aliases__human;config__algorithm__tools_off;config__algorithm__ensemble;analysis;config__algorithm__tools_on;config__algorithm__effects;config__algorithm__variant_regions;genome_resources__aliases__ensembl;config__algorithm__exclude_regions;reference__rtg;genome_resources__aliases__snpeff;config__algorithm__variant_regions_merged;regions__sample_callable;config__algorithm__callable_regions -- sentinel_inputs=batch_rec:record,vrn_file:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- compare_to_rm -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1028 - ramMin: 6144 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: gvcf-regions - specs: - - https://anaconda.org/bioconda/gvcf-regions - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: rtg-tools - specs: - - https://anaconda.org/bioconda/rtg-tools - - package: vcfanno - specs: - - https://anaconda.org/bioconda/vcfanno -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: vrn_file - secondaryFiles: - - .tbi - type: File -outputs: -- id: vc_rec - type: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/concat_batch_variantcalls.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/concat_batch_variantcalls.cwl deleted file mode 100644 index b8cd24d68bc..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/concat_batch_variantcalls.cwl +++ /dev/null @@ -1,196 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-merge -- sentinel_outputs=vrn_file -- sentinel_inputs=batch_rec:record,region_block:var,vrn_file_region:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- concat_batch_variantcalls -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 3072 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 -- class: arv:APIRequirement -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: region_block - type: - items: - items: string - type: array - type: array -- id: vrn_file_region - secondaryFiles: - - .tbi - type: - items: - - File - - 'null' - type: array -outputs: -- id: vrn_file - secondaryFiles: - - .tbi - type: File -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/detect_sv.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/detect_sv.cwl deleted file mode 100644 index 4aab2c1343b..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/detect_sv.cwl +++ /dev/null @@ -1,396 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-single -- sentinel_outputs=sv_rec:sv__variantcaller;sv__vrn_file;sv__supplemental;svvalidate__summary;resources;description;config__algorithm__svprioritize;genome_resources__variation__gc_profile;genome_build;genome_resources__variation__germline_het_pon;config__algorithm__tools_off;analysis;config__algorithm__tools_on;config__algorithm__svvalidate;genome_resources__aliases__snpeff;regions__sample_callable;variants__samples;depth__bins__normalized;depth__bins__background;depth__bins__target;depth__bins__antitarget;regions__bins__target;regions__bins__antitarget;regions__bins__gcannotated;regions__bins__group;reference__fasta__base;config__algorithm__svcaller;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;metadata__batch;genome_resources__variation__lcr;metadata__phenotype;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__sv_regions;config__algorithm__variant_regions;config__algorithm__exclude_regions;config__algorithm__variant_regions_merged;depth__variant_regions__regions;config__algorithm__callable_regions -- sentinel_inputs=sv_batch_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- detect_sv -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1030 - ramMin: 6144 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: cnvkit - specs: - - https://anaconda.org/bioconda/cnvkit - - package: delly - specs: - - https://anaconda.org/bioconda/delly - - package: duphold - specs: - - https://anaconda.org/bioconda/duphold - - package: extract-sv-reads - specs: - - https://anaconda.org/bioconda/extract-sv-reads - - package: lumpy-sv - specs: - - https://anaconda.org/bioconda/lumpy-sv - - package: manta - specs: - - https://anaconda.org/bioconda/manta - - package: break-point-inspector - specs: - - https://anaconda.org/bioconda/break-point-inspector - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: smoove - specs: - - https://anaconda.org/bioconda/smoove - - package: pysam> - specs: - - https://anaconda.org/bioconda/pysam> - version: - - 0.13.0 - - package: seq2c - specs: - - https://anaconda.org/bioconda/seq2c - - package: simple_sv_annotation - specs: - - https://anaconda.org/bioconda/simple_sv_annotation - - package: survivor - specs: - - https://anaconda.org/bioconda/survivor - - package: svtools - specs: - - https://anaconda.org/bioconda/svtools - - package: svtyper - specs: - - https://anaconda.org/bioconda/svtyper - - package: r - specs: - - https://anaconda.org/bioconda/r - version: - - 3.4.1 - - package: r-base=3.4.1=h4fe35fd_8 - specs: - - https://anaconda.org/bioconda/r-base=3.4.1=h4fe35fd_8 - - package: xorg-libxt - specs: - - https://anaconda.org/bioconda/xorg-libxt - - package: vawk - specs: - - https://anaconda.org/bioconda/vawk -- class: arv:RuntimeConstraints - keep_cache: 4096 -inputs: -- id: sv_batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: File - - name: genome_resources__variation__gc_profile - type: File - - name: reference__snpeff__hg19 - type: File - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__svvalidate - type: - - File - - 'null' - - name: genome_resources__aliases__snpeff - type: string - - name: work_bam_plus__disc - type: - - File - - 'null' - - name: work_bam_plus__sr - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__sv_regions - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_batch_rec - type: record - type: array -outputs: -- id: sv_rec - type: - items: - fields: - - name: sv__variantcaller - type: - - string - - 'null' - - name: sv__vrn_file - type: - - File - - 'null' - - name: sv__supplemental - type: - items: - - File - type: array - - name: svvalidate__summary - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: File - - name: genome_resources__variation__gc_profile - type: File - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__svvalidate - type: - - File - - 'null' - - name: genome_resources__aliases__snpeff - type: string - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__sv_regions - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/get_parallel_regions.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/get_parallel_regions.cwl deleted file mode 100644 index 71ac417a4ce..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/get_parallel_regions.cwl +++ /dev/null @@ -1,171 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-split -- sentinel_outputs=region_block -- sentinel_inputs=batch_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- get_parallel_regions -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1030 - ramMin: 3072 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -outputs: -- id: region_block - type: - items: - items: string - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/merge_split_alignments.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/merge_split_alignments.cwl deleted file mode 100644 index afe6f73cfc2..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/merge_split_alignments.cwl +++ /dev/null @@ -1,171 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=single-merge -- sentinel_outputs=align_bam,work_bam_plus__disc,work_bam_plus__sr,hla__fastq -- sentinel_inputs=alignment_rec:record,work_bam:var,align_bam:var,work_bam_plus__disc:var,work_bam_plus__sr:var,hla__fastq:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- merge_split_alignments -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1035 - ramMin: 6144 - tmpdirMin: 6 -- class: dx:InputResourceRequirement - indirMin: 4 -- class: SoftwareRequirement - packages: - - package: biobambam - specs: - - https://anaconda.org/bioconda/biobambam - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: variantbam - specs: - - https://anaconda.org/bioconda/variantbam -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: - - string - - 'null' - - boolean - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - - 'null' - - items: 'null' - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: rgnames__pu - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: - - 'null' - - string - name: alignment_rec - type: record -- id: work_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: align_bam_toolinput - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: work_bam_plus__disc_toolinput - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: work_bam_plus__sr_toolinput - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: hla__fastq_toolinput - type: - items: - - 'null' - - items: File - type: array - type: array -outputs: -- id: align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__disc - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__sr - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: hla__fastq - type: - - 'null' - - items: File - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/multiqc_summary.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/multiqc_summary.cwl deleted file mode 100644 index 82ce9f97a43..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/multiqc_summary.cwl +++ /dev/null @@ -1,95 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=summary__multiqc,versions__tools,versions__data -- sentinel_inputs=qcout_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- multiqc_summary -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1030 - ramMin: 3072 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: multiqc - specs: - - https://anaconda.org/bioconda/multiqc - - package: multiqc-bcbio - specs: - - https://anaconda.org/bioconda/multiqc-bcbio -inputs: -- id: qcout_rec - type: - items: - fields: - - name: summary__qc - type: - - File - - 'null' - - name: summary__metrics - type: - - string - - 'null' - - name: description - type: string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: config__algorithm__tools_on - type: - items: string - type: array - name: qcout_rec - type: record - type: array -outputs: -- id: summary__multiqc - type: - items: - - File - - 'null' - type: array -- id: versions__tools - type: - items: - - File - - 'null' - type: array -- id: versions__data - type: - items: - - File - - 'null' - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/normalize_sv_coverage.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/normalize_sv_coverage.cwl deleted file mode 100644 index 813a2a21a63..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/normalize_sv_coverage.cwl +++ /dev/null @@ -1,224 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=sv_coverage_rec:depth__bins__normalized;depth__bins__background;resources;description;depth__bins__target;depth__bins__antitarget;regions__bins__target;regions__bins__antitarget;regions__bins__gcannotated;regions__bins__group;reference__fasta__base;config__algorithm__svcaller;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;metadata__batch;genome_resources__variation__lcr;metadata__phenotype;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__sv_regions;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;depth__variant_regions__regions;config__algorithm__callable_regions -- sentinel_inputs=sv_rawcoverage_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- normalize_sv_coverage -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1028 - ramMin: 6144 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: cnvkit - specs: - - https://anaconda.org/bioconda/cnvkit -inputs: -- id: sv_rawcoverage_rec - type: - items: - fields: - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__sv_regions - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_rawcoverage_rec - type: record - type: array -outputs: -- id: sv_coverage_rec - type: - items: - fields: - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__sv_regions - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_coverage_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/pipeline_summary.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/pipeline_summary.cwl deleted file mode 100644 index d320e1afb86..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/pipeline_summary.cwl +++ /dev/null @@ -1,221 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=qcout_rec:summary__qc;summary__metrics;description;reference__versions;genome_build;config__algorithm__tools_off;config__algorithm__qc;config__algorithm__tools_on -- sentinel_inputs=qc_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- pipeline_summary -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1030 - ramMin: 6144 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: fastqc=0.11.7=5 - specs: - - https://anaconda.org/bioconda/fastqc=0.11.7=5 - - package: goleft - specs: - - https://anaconda.org/bioconda/goleft - - package: hts-nim-tools - specs: - - https://anaconda.org/bioconda/hts-nim-tools - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: picard - specs: - - https://anaconda.org/bioconda/picard - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: qsignature - specs: - - https://anaconda.org/bioconda/qsignature - - package: qualimap - specs: - - https://anaconda.org/bioconda/qualimap - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: preseq - specs: - - https://anaconda.org/bioconda/preseq - - package: peddy - specs: - - https://anaconda.org/bioconda/peddy - - package: verifybamid2 - specs: - - https://anaconda.org/bioconda/verifybamid2 -- class: arv:RuntimeConstraints - keep_cache: 4096 -inputs: -- id: qc_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: reference__viral - type: File - - name: config__algorithm__qc - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: depth__samtools__stats - type: - - File - - 'null' - - name: depth__samtools__idxstats - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: depth__variant_regions__dist - type: - - File - - 'null' - - name: depth__sv_regions__regions - type: - - File - - 'null' - - name: depth__sv_regions__dist - type: - - File - - 'null' - - name: depth__coverage__regions - type: - - File - - 'null' - - name: depth__coverage__dist - type: - - File - - 'null' - - name: depth__coverage__thresholds - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - name: qc_rec - type: record -outputs: -- id: qcout_rec - type: - fields: - - name: summary__qc - type: - - File - - 'null' - - name: summary__metrics - type: - - string - - 'null' - - name: description - type: string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__qc - type: - items: string - type: array - - name: config__algorithm__tools_on - type: - items: string - type: array - name: qcout_rec - type: record -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/postprocess_alignment.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/postprocess_alignment.cwl deleted file mode 100644 index ccdfc8e177e..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/postprocess_alignment.cwl +++ /dev/null @@ -1,223 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=config__algorithm__coverage_interval,config__algorithm__variant_regions,config__algorithm__variant_regions_merged,config__algorithm__variant_regions_orig,config__algorithm__coverage,config__algorithm__coverage_merged,config__algorithm__coverage_orig,config__algorithm__seq2c_bed_ready,regions__callable,regions__sample_callable,regions__nblock,depth__samtools__stats,depth__samtools__idxstats,depth__variant_regions__regions,depth__variant_regions__dist,depth__sv_regions__regions,depth__sv_regions__dist,depth__coverage__regions,depth__coverage__dist,depth__coverage__thresholds,align_bam -- sentinel_inputs=postprocess_alignment_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- postprocess_alignment -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1033 - ramMin: 6144 - tmpdirMin: 5 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: goleft - specs: - - https://anaconda.org/bioconda/goleft - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: sentieon - specs: - - https://anaconda.org/bioconda/sentieon -- class: arv:APIRequirement -inputs: -- id: postprocess_alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - 'null' - - string - - name: genome_resources__rnaseq__gene_bed - type: File - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__recalibrate - type: - - string - - 'null' - - boolean - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__variant_regions_orig - type: - - File - - 'null' - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: config__algorithm__coverage_orig - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - name: postprocess_alignment_rec - type: record -outputs: -- id: config__algorithm__coverage_interval - type: - - string - - 'null' -- id: config__algorithm__variant_regions - type: - - File - - 'null' -- id: config__algorithm__variant_regions_merged - type: - - File - - 'null' -- id: config__algorithm__variant_regions_orig - type: - - File - - 'null' -- id: config__algorithm__coverage - type: - - File - - 'null' -- id: config__algorithm__coverage_merged - type: - - File - - 'null' -- id: config__algorithm__coverage_orig - type: - - File - - 'null' -- id: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' -- id: regions__callable - type: - - File - - 'null' -- id: regions__sample_callable - type: - - File - - 'null' -- id: regions__nblock - type: - - File - - 'null' -- id: depth__samtools__stats - type: - - File - - 'null' -- id: depth__samtools__idxstats - type: - - File - - 'null' -- id: depth__variant_regions__regions - type: - - File - - 'null' -- id: depth__variant_regions__dist - type: - - File - - 'null' -- id: depth__sv_regions__regions - type: - - File - - 'null' -- id: depth__sv_regions__dist - type: - - File - - 'null' -- id: depth__coverage__regions - type: - - File - - 'null' -- id: depth__coverage__dist - type: - - File - - 'null' -- id: depth__coverage__thresholds - type: - - File - - 'null' -- id: align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/postprocess_alignment_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/postprocess_alignment_to_rec.cwl deleted file mode 100644 index 1f033c9d3c7..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/postprocess_alignment_to_rec.cwl +++ /dev/null @@ -1,233 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=postprocess_alignment_rec:resources;description;reference__fasta__base;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;genome_resources__variation__lcr;config__algorithm__recalibrate;genome_resources__variation__dbsnp;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__tools_on;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__variant_regions_orig;config__algorithm__coverage;config__algorithm__coverage_merged;config__algorithm__coverage_orig;config__algorithm__seq2c_bed_ready -- sentinel_inputs=align_bam:var,config__algorithm__coverage_interval:var,config__algorithm__exclude_regions:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__variant_regions_orig:var,config__algorithm__coverage:var,config__algorithm__coverage_merged:var,config__algorithm__coverage_orig:var,config__algorithm__seq2c_bed_ready:var,config__algorithm__recalibrate:var,config__algorithm__tools_on:var,genome_resources__rnaseq__gene_bed:var,genome_resources__variation__dbsnp:var,genome_resources__variation__lcr:var,genome_resources__variation__polyx:var,genome_resources__variation__encode_blacklist:var,reference__fasta__base:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- postprocess_alignment_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 3072 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_interval - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_orig - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_orig - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__seq2c_bed_ready - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__recalibrate - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: genome_resources__rnaseq__gene_bed - type: - items: File - type: array -- id: genome_resources__variation__dbsnp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__lcr - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__polyx - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__encode_blacklist - type: - items: - - 'null' - - string - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: postprocess_alignment_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - 'null' - - string - - name: genome_resources__rnaseq__gene_bed - type: File - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: config__algorithm__recalibrate - type: - - string - - 'null' - - boolean - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__variant_regions_orig - type: - - File - - 'null' - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: config__algorithm__coverage_orig - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - name: postprocess_alignment_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/postprocess_variants.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/postprocess_variants.cwl deleted file mode 100644 index dd40cf1d9a4..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/postprocess_variants.cwl +++ /dev/null @@ -1,180 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-single -- sentinel_outputs=vrn_file -- sentinel_inputs=batch_rec:record,vrn_file:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- postprocess_variants -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1025 - ramMin: 6144 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: snpeff - specs: - - https://anaconda.org/bioconda/snpeff - version: - - 4.3.1t -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: vrn_file_toolinput - secondaryFiles: - - .tbi - type: File -outputs: -- id: vrn_file - secondaryFiles: - - .tbi - type: File -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/prep_align_inputs.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/prep_align_inputs.cwl deleted file mode 100644 index cd325c5f774..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/prep_align_inputs.cwl +++ /dev/null @@ -1,145 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=single-split -- sentinel_outputs=process_alignment_rec:files;config__algorithm__quality_format;align_split -- sentinel_inputs=alignment_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- prep_align_inputs -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1028 - ramMin: 6144 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 4 -- class: SoftwareRequirement - packages: - - package: grabix - specs: - - https://anaconda.org/bioconda/grabix - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: biobambam - specs: - - https://anaconda.org/bioconda/biobambam - - package: atropos;env - specs: - - https://anaconda.org/bioconda/atropos;env - version: - - python3 - - package: optitype - specs: - - https://anaconda.org/bioconda/optitype - - package: razers3 - specs: - - https://anaconda.org/bioconda/razers3 - version: - - 3.5.0 - - package: coincbc - specs: - - https://anaconda.org/bioconda/coincbc -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: - - string - - 'null' - - boolean - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - - 'null' - - items: 'null' - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: rgnames__pu - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: - - 'null' - - string - name: alignment_rec - type: record -outputs: -- id: process_alignment_rec - type: - items: - fields: - - name: files - type: - - 'null' - - items: File - type: array - - name: config__algorithm__quality_format - type: - - string - - 'null' - - name: align_split - type: - - string - - 'null' - name: process_alignment_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/prep_samples.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/prep_samples.cwl deleted file mode 100644 index 63d24367261..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/prep_samples.cwl +++ /dev/null @@ -1,93 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=rgnames__sample,config__algorithm__variant_regions,config__algorithm__variant_regions_merged,config__algorithm__variant_regions_orig,config__algorithm__coverage,config__algorithm__coverage_merged,config__algorithm__coverage_orig,config__algorithm__seq2c_bed_ready -- sentinel_inputs=prep_samples_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- prep_samples -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1025 - ramMin: 3072 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy -inputs: -- id: prep_samples_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: - - 'null' - - string - name: prep_samples_rec - type: record -outputs: -- id: rgnames__sample - type: string -- id: config__algorithm__variant_regions - type: - - File - - 'null' -- id: config__algorithm__variant_regions_merged - type: - - File - - 'null' -- id: config__algorithm__variant_regions_orig - type: - - File - - 'null' -- id: config__algorithm__coverage - type: - - File - - 'null' -- id: config__algorithm__coverage_merged - type: - - File - - 'null' -- id: config__algorithm__coverage_orig - type: - - File - - 'null' -- id: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/prep_samples_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/prep_samples_to_rec.cwl deleted file mode 100644 index da7b1d7f2b0..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/prep_samples_to_rec.cwl +++ /dev/null @@ -1,79 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=prep_samples_rec:resources;description;reference__fasta__base;rgnames__sample;config__algorithm__variant_regions -- sentinel_inputs=rgnames__sample:var,config__algorithm__variant_regions:var,reference__fasta__base:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- prep_samples_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1025 - ramMin: 3072 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: rgnames__sample - type: - items: string - type: array -- id: config__algorithm__variant_regions - type: - items: - - 'null' - - string - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: prep_samples_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: - - 'null' - - string - name: prep_samples_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/process_alignment.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/process_alignment.cwl deleted file mode 100644 index af6768e2f88..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/process_alignment.cwl +++ /dev/null @@ -1,201 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=single-parallel -- sentinel_outputs=work_bam,align_bam,hla__fastq,work_bam_plus__disc,work_bam_plus__sr -- sentinel_inputs=alignment_rec:record,process_alignment_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- process_alignment -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1030 - ramMin: 6144 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 7 -- class: SoftwareRequirement - packages: - - package: bwa - specs: - - https://anaconda.org/bioconda/bwa - - package: bwakit - specs: - - https://anaconda.org/bioconda/bwakit - - package: grabix - specs: - - https://anaconda.org/bioconda/grabix - - package: minimap2 - specs: - - https://anaconda.org/bioconda/minimap2 - - package: novoalign - specs: - - https://anaconda.org/bioconda/novoalign - - package: snap-aligner - specs: - - https://anaconda.org/bioconda/snap-aligner - version: - - 1.0dev.97 - - package: sentieon - specs: - - https://anaconda.org/bioconda/sentieon - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: pysam> - specs: - - https://anaconda.org/bioconda/pysam> - version: - - 0.13.0 - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: fgbio - specs: - - https://anaconda.org/bioconda/fgbio - - package: umis - specs: - - https://anaconda.org/bioconda/umis - - package: biobambam - specs: - - https://anaconda.org/bioconda/biobambam - - package: seqtk - specs: - - https://anaconda.org/bioconda/seqtk - - package: samblaster - specs: - - https://anaconda.org/bioconda/samblaster - - package: variantbam - specs: - - https://anaconda.org/bioconda/variantbam -- class: arv:APIRequirement -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: - - string - - 'null' - - boolean - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - - 'null' - - items: 'null' - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: rgnames__pu - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: - - 'null' - - string - name: alignment_rec - type: record -- id: process_alignment_rec - type: - fields: - - name: files - type: - - 'null' - - items: File - type: array - - name: config__algorithm__quality_format - type: - - string - - 'null' - - name: align_split - type: - - string - - 'null' - name: process_alignment_rec - type: record -outputs: -- id: work_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: hla__fastq - type: - - 'null' - - items: File - type: array -- id: work_bam_plus__disc - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__sr - secondaryFiles: - - .bai - type: - - File - - 'null' -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/qc_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/qc_to_rec.cwl deleted file mode 100644 index 4565b513de7..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/qc_to_rec.cwl +++ /dev/null @@ -1,308 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=qc_rec:resources;description;reference__fasta__base;config__algorithm__coverage_interval;metadata__batch;reference__versions;genome_build;metadata__phenotype;config__algorithm__tools_off;reference__viral;config__algorithm__qc;analysis;config__algorithm__tools_on;config__algorithm__variant_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__coverage;config__algorithm__coverage_merged;depth__samtools__stats;depth__samtools__idxstats;depth__variant_regions__regions;depth__variant_regions__dist;depth__sv_regions__regions;depth__sv_regions__dist;depth__coverage__regions;depth__coverage__dist;depth__coverage__thresholds;variants__samples -- sentinel_inputs=align_bam:var,analysis:var,reference__fasta__base:var,reference__versions:var,config__algorithm__tools_on:var,config__algorithm__tools_off:var,genome_build:var,config__algorithm__qc:var,metadata__batch:var,metadata__phenotype:var,config__algorithm__coverage_interval:var,depth__variant_regions__regions:var,depth__variant_regions__dist:var,depth__samtools__stats:var,depth__samtools__idxstats:var,depth__sv_regions__regions:var,depth__sv_regions__dist:var,depth__coverage__regions:var,depth__coverage__dist:var,depth__coverage__thresholds:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__coverage:var,config__algorithm__coverage_merged:var,variants__samples:var,reference__viral:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- qc_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1028 - ramMin: 3072 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: analysis - type: - items: string - type: array -- id: reference__fasta__base - secondaryFiles: - - .fai - - ^.dict - type: - items: File - type: array -- id: reference__versions - type: - items: File - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: config__algorithm__tools_off - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: genome_build - type: - items: string - type: array -- id: config__algorithm__qc - type: - items: - items: string - type: array - type: array -- id: metadata__batch - type: - items: string - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: config__algorithm__coverage_interval - type: - items: - - string - - 'null' - type: array -- id: depth__variant_regions__regions - type: - items: - - File - - 'null' - type: array -- id: depth__variant_regions__dist - type: - items: - - File - - 'null' - type: array -- id: depth__samtools__stats - type: - items: - - File - - 'null' - type: array -- id: depth__samtools__idxstats - type: - items: - - File - - 'null' - type: array -- id: depth__sv_regions__regions - type: - items: - - File - - 'null' - type: array -- id: depth__sv_regions__dist - type: - items: - - File - - 'null' - type: array -- id: depth__coverage__regions - type: - items: - - File - - 'null' - type: array -- id: depth__coverage__dist - type: - items: - - File - - 'null' - type: array -- id: depth__coverage__thresholds - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_merged - type: - items: - - File - - 'null' - type: array -- id: variants__samples - type: - items: - items: - items: - - File - - 'null' - type: array - type: array - type: array -- id: reference__viral - secondaryFiles: - - .amb - - .ann - - .sa - - .pac - - ^.dict - - .bwt - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: qc_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: reference__versions - type: File - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: reference__viral - type: File - - name: config__algorithm__qc - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: depth__samtools__stats - type: - - File - - 'null' - - name: depth__samtools__idxstats - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: depth__variant_regions__dist - type: - - File - - 'null' - - name: depth__sv_regions__regions - type: - - File - - 'null' - - name: depth__sv_regions__dist - type: - - File - - 'null' - - name: depth__coverage__regions - type: - - File - - 'null' - - name: depth__coverage__dist - type: - - File - - 'null' - - name: depth__coverage__thresholds - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - name: qc_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/summarize_sv.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/summarize_sv.cwl deleted file mode 100644 index 87f503d9117..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/summarize_sv.cwl +++ /dev/null @@ -1,228 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=sv__calls,sv__supplemental,sv__prioritize__tsv,sv__prioritize__raw,svvalidate__grading_summary,svvalidate__grading_plots -- sentinel_inputs=sv_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- summarize_sv -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1027 - ramMin: 3072 - tmpdirMin: 2 -- class: dx:InputResourceRequirement - indirMin: 1 -inputs: -- id: sv_rec - type: - items: - items: - fields: - - name: sv__variantcaller - type: - - string - - 'null' - - name: sv__vrn_file - type: - - File - - 'null' - - name: sv__supplemental - type: - items: - - File - type: array - - name: svvalidate__summary - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: File - - name: genome_resources__variation__gc_profile - type: File - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__svvalidate - type: - - File - - 'null' - - name: genome_resources__aliases__snpeff - type: string - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__sv_regions - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_rec - type: record - type: array - type: array -outputs: -- id: sv__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: sv__supplemental - type: - items: - items: - - File - type: array - type: array -- id: sv__prioritize__tsv - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: sv__prioritize__raw - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: svvalidate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: svvalidate__grading_plots - type: - items: - items: - - File - - 'null' - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/summarize_vc.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/summarize_vc.cwl deleted file mode 100644 index c560affeded..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/summarize_vc.cwl +++ /dev/null @@ -1,193 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=variants__calls,variants__gvcf,variants__samples,validate__grading_summary,validate__grading_plots -- sentinel_inputs=vc_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- summarize_vc -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1025 - ramMin: 3072 - tmpdirMin: 1 -- class: dx:InputResourceRequirement - indirMin: 1 -inputs: -- id: vc_rec - type: - items: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array - type: array -outputs: -- id: variants__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: variants__gvcf - type: - items: - - 'null' - - items: - - File - - 'null' - type: array - type: array -- id: variants__samples - type: - items: - items: - items: - - File - - 'null' - type: array - type: array - type: array -- id: validate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: validate__grading_plots - type: - items: - items: - - File - - 'null' - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/variantcall_batch_region.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/variantcall_batch_region.cwl deleted file mode 100644 index 7e8f1574bfc..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/steps/variantcall_batch_region.cwl +++ /dev/null @@ -1,266 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-parallel -- sentinel_outputs=vrn_file_region,region_block -- sentinel_inputs=batch_rec:record,region_block:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- variantcall_batch_region -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 2 - outdirMin: 1030 - ramMin: 6144 - tmpdirMin: 3 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: freebayes - specs: - - https://anaconda.org/bioconda/freebayes - version: - - 1.1.0.46 - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 - - package: vqsr_cnn - specs: - - https://anaconda.org/bioconda/vqsr_cnn - - package: deepvariant - specs: - - https://anaconda.org/bioconda/deepvariant - - package: sentieon - specs: - - https://anaconda.org/bioconda/sentieon - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: octopus - specs: - - https://anaconda.org/bioconda/octopus - - package: picard - specs: - - https://anaconda.org/bioconda/picard - - package: platypus-variant - specs: - - https://anaconda.org/bioconda/platypus-variant - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: pysam> - specs: - - https://anaconda.org/bioconda/pysam> - version: - - 0.13.0 - - package: strelka - specs: - - https://anaconda.org/bioconda/strelka - - package: vardict - specs: - - https://anaconda.org/bioconda/vardict - - package: vardict-java - specs: - - https://anaconda.org/bioconda/vardict-java - - package: varscan - specs: - - https://anaconda.org/bioconda/varscan - - package: moreutils - specs: - - https://anaconda.org/bioconda/moreutils - - package: vcfanno - specs: - - https://anaconda.org/bioconda/vcfanno - - package: vcflib - specs: - - https://anaconda.org/bioconda/vcflib - - package: vt - specs: - - https://anaconda.org/bioconda/vt - - package: r - specs: - - https://anaconda.org/bioconda/r - version: - - 3.4.1 - - package: r-base=3.4.1=h4fe35fd_8 - specs: - - https://anaconda.org/bioconda/r-base=3.4.1=h4fe35fd_8 - - package: perl - specs: - - https://anaconda.org/bioconda/perl -- class: arv:APIRequirement -- class: arv:RuntimeConstraints - keep_cache: 4096 -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: region_block_toolinput - type: - items: string - type: array -outputs: -- id: vrn_file_region - secondaryFiles: - - .tbi - type: - - File - - 'null' -- id: region_block - type: - items: string - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/wf-alignment.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/wf-alignment.cwl deleted file mode 100644 index 3785183a8df..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/wf-alignment.cwl +++ /dev/null @@ -1,146 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: - - string - - 'null' - - boolean - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - - 'null' - - items: 'null' - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: rgnames__pu - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: - - 'null' - - string - name: alignment_rec - type: record -outputs: -- id: align_bam - outputSource: merge_split_alignments/align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__disc - outputSource: merge_split_alignments/work_bam_plus__disc - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__sr - outputSource: merge_split_alignments/work_bam_plus__sr - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: hla__fastq - outputSource: merge_split_alignments/hla__fastq - type: - - 'null' - - items: File - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: prep_align_inputs - in: - - id: alignment_rec - source: alignment_rec - out: - - id: process_alignment_rec - run: steps/prep_align_inputs.cwl -- id: process_alignment - in: - - id: alignment_rec - source: alignment_rec - - id: process_alignment_rec - source: prep_align_inputs/process_alignment_rec - out: - - id: work_bam - - id: align_bam - - id: hla__fastq - - id: work_bam_plus__disc - - id: work_bam_plus__sr - run: steps/process_alignment.cwl - scatter: - - process_alignment_rec - scatterMethod: dotproduct -- id: merge_split_alignments - in: - - id: alignment_rec - source: alignment_rec - - id: work_bam - source: process_alignment/work_bam - - id: align_bam_toolinput - source: process_alignment/align_bam - - id: work_bam_plus__disc_toolinput - source: process_alignment/work_bam_plus__disc - - id: work_bam_plus__sr_toolinput - source: process_alignment/work_bam_plus__sr - - id: hla__fastq_toolinput - source: process_alignment/hla__fastq - out: - - id: align_bam - - id: work_bam_plus__disc - - id: work_bam_plus__sr - - id: hla__fastq - run: steps/merge_split_alignments.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/wf-svcall.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/wf-svcall.cwl deleted file mode 100644 index 01ae7b07f7a..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/wf-svcall.cwl +++ /dev/null @@ -1,310 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: sv_batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: File - - name: genome_resources__variation__gc_profile - type: File - - name: reference__snpeff__hg19 - type: File - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__svvalidate - type: - - File - - 'null' - - name: genome_resources__aliases__snpeff - type: string - - name: work_bam_plus__disc - type: - - File - - 'null' - - name: work_bam_plus__sr - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__sv_regions - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_batch_rec - type: record - type: array -outputs: -- id: sv_rec - outputSource: detect_sv/sv_rec - type: - items: - fields: - - name: sv__variantcaller - type: - - string - - 'null' - - name: sv__vrn_file - type: - - File - - 'null' - - name: sv__supplemental - type: - items: - - File - type: array - - name: svvalidate__summary - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: File - - name: genome_resources__variation__gc_profile - type: File - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__svvalidate - type: - - File - - 'null' - - name: genome_resources__aliases__snpeff - type: string - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: config__algorithm__sv_regions - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_rec - type: record - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: detect_sv - in: - - id: sv_batch_rec - source: sv_batch_rec - out: - - id: sv_rec - run: steps/detect_sv.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/wf-variantcall.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/wf-variantcall.cwl deleted file mode 100644 index f7dd3955627..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/svcall-workflow/wf-variantcall.cwl +++ /dev/null @@ -1,308 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: - - 'null' - - string - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: - - 'null' - - string - - name: genome_resources__variation__encode_blacklist - type: - - 'null' - - string - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -outputs: -- id: vc_rec - outputSource: compare_to_rm/vc_rec - type: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - items: File - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__genome_context - type: - items: File - type: array - - name: config__algorithm__validate - type: - - 'null' - - string - - name: reference__snpeff__hg19 - type: File - - name: config__algorithm__validate_regions - type: - - 'null' - - string - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__ensemble - type: - - 'null' - - string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: reference__rtg - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: get_parallel_regions - in: - - id: batch_rec - source: batch_rec - out: - - id: region_block - run: steps/get_parallel_regions.cwl -- id: variantcall_batch_region - in: - - id: batch_rec - source: batch_rec - - id: region_block_toolinput - source: get_parallel_regions/region_block - out: - - id: vrn_file_region - - id: region_block - run: steps/variantcall_batch_region.cwl - scatter: - - region_block_toolinput - scatterMethod: dotproduct -- id: concat_batch_variantcalls - in: - - id: batch_rec - source: batch_rec - - id: region_block - source: variantcall_batch_region/region_block - - id: vrn_file_region - source: variantcall_batch_region/vrn_file_region - out: - - id: vrn_file - run: steps/concat_batch_variantcalls.cwl -- id: postprocess_variants - in: - - id: batch_rec - source: batch_rec - - id: vrn_file_toolinput - source: concat_batch_variantcalls/vrn_file - out: - - id: vrn_file - run: steps/postprocess_variants.cwl -- id: compare_to_rm - in: - - id: batch_rec - source: batch_rec - - id: vrn_file - source: postprocess_variants/vrn_file - out: - - id: vc_rec - run: steps/compare_to_rm.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/main-wes_chr21_test-samples.json b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/main-wes_chr21_test-samples.json deleted file mode 100644 index 1439ccade50..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/main-wes_chr21_test-samples.json +++ /dev/null @@ -1,1059 +0,0 @@ -{ - "expected_validation": - { - "class": "File", - "path": "gs://bcbiodata/wes_agha_test/grading-summary-combined-expected.csv" - }, - "analysis": [ - "variant2", - "variant2" - ], - "config__algorithm__adapters": [ - [ - "polyx" - ], - [ - "polyx" - ] - ], - "config__algorithm__align_split_size": [ - null, - null - ], - "config__algorithm__aligner": [ - "bwa", - "bwa" - ], - "config__algorithm__archive": [ - null, - null - ], - "config__algorithm__bam_clean": [ - "False", - "False" - ], - "config__algorithm__coverage_interval": [ - null, - null - ], - "config__algorithm__effects": [ - "snpeff", - "snpeff" - ], - "config__algorithm__ensemble": [ - "{\"numpass\": 2}", - "{\"numpass\": 2}" - ], - "config__algorithm__exclude_regions": [ - [], - [] - ], - "config__algorithm__mark_duplicates": [ - "True", - "True" - ], - "config__algorithm__min_allele_fraction": [ - 10.0, - 10.0 - ], - "config__algorithm__nomap_split_size": [ - 250, - 250 - ], - "config__algorithm__nomap_split_targets": [ - 20, - 20 - ], - "config__algorithm__qc": [ - [ - "contamination", - "coverage", - "fastqc", - "peddy", - "picard", - "qsignature", - "samtools", - "variants", - "viral" - ], - [ - "contamination", - "coverage", - "fastqc", - "peddy", - "picard", - "qsignature", - "samtools", - "variants", - "viral" - ] - ], - "config__algorithm__recalibrate": [ - "False", - "False" - ], - "config__algorithm__svcaller": [ - [ - "cnvkit" - ], - [ - "cnvkit" - ] - ], - "config__algorithm__svprioritize": [ - null, - null - ], - "config__algorithm__svvalidate": [ - null, - null - ], - "config__algorithm__tools_off": [ - [ - "gemini" - ], - [ - "gemini" - ] - ], - "config__algorithm__tools_on": [ - [ - "gatk4", - "break-point-inspector", - "noalt_calling" - ], - [ - "gatk4", - "break-point-inspector", - "noalt_calling" - ] - ], - "config__algorithm__trim_reads": [ - "atropos", - "atropos" - ], - "config__algorithm__validate": [ - { - "class": "File", - "path": "gs://bcbiodata/wes_agha_test/wes-ensemble-annotated_exome_chr21_noAnnot_somatic.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/wes_agha_test/wes-ensemble-annotated_exome_chr21_noAnnot_somatic.vcf.gz.tbi" - } - ] - }, - null - ], - "config__algorithm__validate_regions": [ - { - "class": "File", - "path": "gs://bcbiodata/wes_agha_test/Exome-Agilent_V6_chr21.bed" - }, - null - ], - "config__algorithm__variant_regions": [ - { - "class": "File", - "path": "gs://bcbiodata/wes_agha_test/Exome-Agilent_V6_chr21.bed" - }, - { - "class": "File", - "path": "gs://bcbiodata/wes_agha_test/Exome-Agilent_V6_chr21.bed" - } - ], - "config__algorithm__variantcaller": [ - [ - "germline:vardict", - "germline:strelka2", - "germline:gatk-haplotype", - "somatic:vardict", - "somatic:strelka2", - "somatic:mutect2" - ], - [ - "germline:vardict", - "germline:strelka2", - "germline:gatk-haplotype", - "somatic:vardict", - "somatic:strelka2", - "somatic:mutect2" - ] - ], - "config__algorithm__vcfanno": [ - [], - [] - ], - "description": [ - "NA12878_chr21", - "NA24385_chr21" - ], - "files": [ - [ - { - "class": "File", - "path": "gs://bcbiodata/wes_agha_test/NA12878-ready_exome21.bam", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/wes_agha_test/NA12878-ready_exome21.bam.bai" - } - ] - } - ], - [ - { - "class": "File", - "path": "gs://bcbiodata/wes_agha_test/NA24385-ready_exome21.bam", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/wes_agha_test/NA24385-ready_exome21.bam.bai" - } - ] - } - ] - ], - "genome_build": [ - "GRCh37", - "GRCh37" - ], - "genome_resources__aliases__ensembl": [ - "homo_sapiens_merged_vep_93_GRCh37", - "homo_sapiens_merged_vep_93_GRCh37" - ], - "genome_resources__aliases__human": [ - "True", - "True" - ], - "genome_resources__aliases__snpeff": [ - "GRCh37.75", - "GRCh37.75" - ], - "genome_resources__rnaseq__gene_bed": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/rnaseq/ref-transcripts.bed" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/rnaseq/ref-transcripts.bed" - } - ], - "genome_resources__variation__1000g": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/1000g.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/1000g.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/1000g.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/1000g.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__clinvar": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/clinvar.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/clinvar.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/clinvar.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/clinvar.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__cosmic": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/cosmic.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/cosmic.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/cosmic.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/cosmic.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__dbsnp": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/dbsnp-151.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/dbsnp-151.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/dbsnp-151.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/dbsnp-151.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__encode_blacklist": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/ENCODE/wgEncodeDacMapabilityConsensusExcludable.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/ENCODE/wgEncodeDacMapabilityConsensusExcludable.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/ENCODE/wgEncodeDacMapabilityConsensusExcludable.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/ENCODE/wgEncodeDacMapabilityConsensusExcludable.bed.gz.tbi" - } - ] - } - ], - "genome_resources__variation__esp": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/esp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/esp.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/esp.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/esp.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__exac": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/exac.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/exac.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/exac.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/exac.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__gc_profile": [ - null, - null - ], - "genome_resources__variation__germline_het_pon": [ - null, - null - ], - "genome_resources__variation__gnomad_exome": [ - null, - null - ], - "genome_resources__variation__lcr": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/LCR.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/LCR.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/LCR.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/LCR.bed.gz.tbi" - } - ] - } - ], - "genome_resources__variation__polyx": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/polyx.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/polyx.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/polyx.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/polyx.bed.gz.tbi" - } - ] - } - ], - "genome_resources__variation__train_hapmap": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/hapmap_3.3.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/hapmap_3.3.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/hapmap_3.3.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/hapmap_3.3.vcf.gz.tbi" - } - ] - } - ], - "genome_resources__variation__train_indels": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/Mills_and_1000G_gold_standard.indels.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/Mills_and_1000G_gold_standard.indels.vcf.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/Mills_and_1000G_gold_standard.indels.vcf.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/variation/Mills_and_1000G_gold_standard.indels.vcf.gz.tbi" - } - ] - } - ], - "metadata__batch": [ - "wes", - "wes" - ], - "metadata__phenotype": [ - "tumor", - "normal" - ], - "reference__bwa__indexes": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/bwa/GRCh37.fa.amb", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/bwa/GRCh37.fa.ann" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/bwa/GRCh37.fa.bwt" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/bwa/GRCh37.fa.pac" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/bwa/GRCh37.fa.sa" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/bwa/GRCh37.fa.amb", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/bwa/GRCh37.fa.ann" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/bwa/GRCh37.fa.bwt" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/bwa/GRCh37.fa.pac" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/bwa/GRCh37.fa.sa" - } - ] - } - ], - "reference__fasta__base": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/seq/GRCh37.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/seq/GRCh37.dict" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/seq/GRCh37.fa.fai" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/seq/GRCh37.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/seq/GRCh37.dict" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/seq/GRCh37.fa.fai" - } - ] - } - ], - "reference__genome_context": [ - [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/LCR.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/LCR.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/bad_promoter.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/bad_promoter.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc15.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc15.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc15to20.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc15to20.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc20to25.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc20to25.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc25to30.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc25to30.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc65to70.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc65to70.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc70to75.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc70to75.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc75to80.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc75to80.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc80to85.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc80to85.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc85.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc85.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/heng_um75-hs37d5.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/heng_um75-hs37d5.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/low_complexity_51to200bp.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/low_complexity_51to200bp.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/low_complexity_gt200bp.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/low_complexity_gt200bp.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/polyx.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/polyx.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/self_chain.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/self_chain.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/ENCODE/wgEncodeDacMapabilityConsensusExcludable.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/ENCODE/wgEncodeDacMapabilityConsensusExcludable.bed.gz.tbi" - } - ] - } - ], - [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/LCR.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/LCR.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/bad_promoter.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/bad_promoter.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc15.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc15.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc15to20.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc15to20.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc20to25.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc20to25.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc25to30.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc25to30.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc65to70.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc65to70.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc70to75.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc70to75.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc75to80.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc75to80.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc80to85.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc80to85.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc85.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/gc85.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/heng_um75-hs37d5.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/heng_um75-hs37d5.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/low_complexity_51to200bp.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/low_complexity_51to200bp.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/low_complexity_gt200bp.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/low_complexity_gt200bp.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/polyx.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/repeats/polyx.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/self_chain.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/GA4GH/self_chain.bed.gz.tbi" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/ENCODE/wgEncodeDacMapabilityConsensusExcludable.bed.gz", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/coverage/problem_regions/ENCODE/wgEncodeDacMapabilityConsensusExcludable.bed.gz.tbi" - } - ] - } - ] - ], - "reference__rtg": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/rtg--GRCh37.sdf-wf.tar.gz" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/rtg--GRCh37.sdf-wf.tar.gz" - } - ], - "reference__snpeff__GRCh37_75": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/snpeff--GRCh37.75-wf.tar.gz" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/snpeff--GRCh37.75-wf.tar.gz" - } - ], - "reference__versions": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/versions.csv" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/versions.csv" - } - ], - "reference__viral": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.dict" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.fa.amb" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.fa.ann" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.fa.bwt" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.fa.fai" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.fa.pac" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.fa.sa" - } - ] - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.fa", - "secondaryFiles": [ - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.dict" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.fa.amb" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.fa.ann" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.fa.bwt" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.fa.fai" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.fa.pac" - }, - { - "class": "File", - "path": "gs://bcbiodata/collections/GRCh37/viral/gdc-viral.fa.sa" - } - ] - } - ], - "resources": [ - "{}", - "{}" - ], - "rgnames__lane": [ - "NA12878_chr21", - "NA24385_chr21" - ], - "rgnames__lb": [ - null, - null - ], - "rgnames__pl": [ - "illumina", - "illumina" - ], - "rgnames__pu": [ - "NA12878_chr21", - "NA24385_chr21" - ], - "rgnames__rg": [ - "NA12878_chr21", - "NA24385_chr21" - ], - "rgnames__sample": [ - "NA12878_chr21", - "NA24385_chr21" - ], - "vrn_file": [ - null, - null - ] -} diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/main-wes_chr21_test.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/main-wes_chr21_test.cwl deleted file mode 100644 index c325ecade0b..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/main-wes_chr21_test.cwl +++ /dev/null @@ -1,997 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: config__algorithm__align_split_size - type: - items: - - 'null' - - string - type: array -- id: files - secondaryFiles: - - .bai - type: - items: - items: File - type: array - type: array -- id: config__algorithm__trim_reads - type: - items: string - type: array -- id: reference__fasta__base - secondaryFiles: - - ^.dict - - .fai - type: - items: File - type: array -- id: config__algorithm__vcfanno - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__svprioritize - type: - items: - - 'null' - - string - type: array -- id: resources - type: - items: string - type: array -- id: config__algorithm__variantcaller - type: - items: - items: string - type: array - type: array -- id: config__algorithm__adapters - type: - items: - items: string - type: array - type: array -- id: config__algorithm__svcaller - type: - items: - items: string - type: array - type: array -- id: config__algorithm__coverage_interval - type: - items: - - 'null' - - string - type: array -- id: genome_resources__rnaseq__gene_bed - type: - items: File - type: array -- id: genome_resources__variation__gc_profile - type: - items: - - 'null' - - string - type: array -- id: rgnames__lb - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__clinvar - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__esp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: rgnames__rg - type: - items: string - type: array -- id: metadata__batch - type: - items: string - type: array -- id: rgnames__lane - type: - items: string - type: array -- id: genome_resources__variation__lcr - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__min_allele_fraction - type: - items: double - type: array -- id: config__algorithm__nomap_split_targets - type: - items: long - type: array -- id: reference__bwa__indexes - secondaryFiles: - - ^.ann - - ^.pac - - ^.sa - - ^.bwt - type: - items: File - type: array -- id: reference__snpeff__GRCh37_75 - type: - items: File - type: array -- id: vrn_file - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__train_hapmap - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: reference__genome_context - secondaryFiles: - - .tbi - type: - items: - items: File - type: array - type: array -- id: config__algorithm__bam_clean - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__nomap_split_size - type: - items: long - type: array -- id: reference__rtg - type: - items: File - type: array -- id: config__algorithm__validate - secondaryFiles: - - .tbi - type: - items: - - File - - 'null' - type: array -- id: genome_resources__variation__1000g - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: description - type: - items: string - type: array -- id: config__algorithm__validate_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__aligner - type: - items: string - type: array -- id: rgnames__pl - type: - items: string - type: array -- id: genome_build - type: - items: string - type: array -- id: genome_resources__variation__germline_het_pon - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__exac - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__gnomad_exome - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__recalibrate - type: - items: - - string - - 'null' - - boolean - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: genome_resources__aliases__human - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__tools_off - type: - items: - items: string - type: array - type: array -- id: genome_resources__variation__dbsnp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__mark_duplicates - type: - items: - - string - - 'null' - - boolean - type: array -- id: genome_resources__variation__polyx - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: reference__versions - type: - items: File - type: array -- id: genome_resources__variation__encode_blacklist - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: reference__viral - secondaryFiles: - - .fai - - .ann - - .sa - - .pac - - .amb - - ^.dict - - .bwt - type: - items: File - type: array -- id: genome_resources__variation__cosmic - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: config__algorithm__ensemble - type: - items: string - type: array -- id: config__algorithm__qc - type: - items: - items: string - type: array - type: array -- id: analysis - type: - items: string - type: array -- id: rgnames__sample - type: - items: string - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: config__algorithm__effects - type: - items: string - type: array -- id: config__algorithm__variant_regions - type: - items: File - type: array -- id: config__algorithm__svvalidate - type: - items: - - 'null' - - string - type: array -- id: genome_resources__aliases__ensembl - type: - items: string - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: genome_resources__variation__train_indels - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__aliases__snpeff - type: - items: string - type: array -- id: rgnames__pu - type: - items: string - type: array -- id: config__algorithm__archive - type: - items: - - 'null' - - string - type: array -outputs: -- id: rgnames__sample_out - outputSource: prep_samples/rgnames__sample - type: - items: string - type: array -- id: align_bam - outputSource: postprocess_alignment/align_bam - type: - items: - - File - - 'null' - type: array -- id: regions__sample_callable - outputSource: postprocess_alignment/regions__sample_callable - type: - items: - - File - - 'null' - type: array -- id: validate__grading_summary - outputSource: summarize_vc/validate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: variants__calls - outputSource: summarize_vc/variants__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: variants__gvcf - outputSource: summarize_vc/variants__gvcf - type: - items: - - 'null' - - items: - - File - - 'null' - type: array - type: array -- id: sv__calls - outputSource: summarize_sv/sv__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: svvalidate__grading_summary - outputSource: summarize_sv/svvalidate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: sv__prioritize__tsv - outputSource: summarize_sv/sv__prioritize__tsv - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: sv__prioritize__raw - outputSource: summarize_sv/sv__prioritize__raw - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: sv__supplemental - outputSource: summarize_sv/sv__supplemental - type: - items: - items: - - File - type: array - type: array -- id: summary__multiqc - outputSource: multiqc_summary/summary__multiqc - type: - items: - - File - - 'null' - type: array -- id: versions__tools - outputSource: multiqc_summary/versions__tools - type: - items: - - File - - 'null' - type: array -- id: versions__data - outputSource: multiqc_summary/versions__data - type: - items: - - File - - 'null' - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: alignment_to_rec - in: - - id: files - source: files - - id: analysis - source: analysis - - id: config__algorithm__align_split_size - source: config__algorithm__align_split_size - - id: reference__fasta__base - source: reference__fasta__base - - id: rgnames__pl - source: rgnames__pl - - id: rgnames__sample - source: rgnames__sample - - id: rgnames__pu - source: rgnames__pu - - id: rgnames__lane - source: rgnames__lane - - id: rgnames__rg - source: rgnames__rg - - id: rgnames__lb - source: rgnames__lb - - id: reference__bwa__indexes - source: reference__bwa__indexes - - id: config__algorithm__aligner - source: config__algorithm__aligner - - id: config__algorithm__trim_reads - source: config__algorithm__trim_reads - - id: config__algorithm__adapters - source: config__algorithm__adapters - - id: config__algorithm__bam_clean - source: config__algorithm__bam_clean - - id: config__algorithm__variant_regions - source: config__algorithm__variant_regions - - id: config__algorithm__mark_duplicates - source: config__algorithm__mark_duplicates - - id: resources - source: resources - - id: description - source: description - out: - - id: alignment_rec - run: steps/alignment_to_rec.cwl -- id: alignment - in: - - id: alignment_rec - source: alignment_to_rec/alignment_rec - out: - - id: align_bam - - id: work_bam_plus__disc - - id: work_bam_plus__sr - - id: hla__fastq - run: wf-alignment.cwl - scatter: - - alignment_rec - scatterMethod: dotproduct -- id: prep_samples_to_rec - in: - - id: rgnames__sample - source: rgnames__sample - - id: config__algorithm__svcaller - source: config__algorithm__svcaller - - id: config__algorithm__variant_regions - source: config__algorithm__variant_regions - - id: reference__fasta__base - source: reference__fasta__base - - id: resources - source: resources - - id: description - source: description - out: - - id: prep_samples_rec - run: steps/prep_samples_to_rec.cwl -- id: prep_samples - in: - - id: prep_samples_rec - source: prep_samples_to_rec/prep_samples_rec - out: - - id: rgnames__sample - - id: config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - - id: config__algorithm__variant_regions_orig - - id: config__algorithm__coverage - - id: config__algorithm__coverage_merged - - id: config__algorithm__coverage_orig - - id: config__algorithm__seq2c_bed_ready - run: steps/prep_samples.cwl - scatter: - - prep_samples_rec - scatterMethod: dotproduct -- id: postprocess_alignment_to_rec - in: - - id: align_bam - source: alignment/align_bam - - id: config__algorithm__archive - source: config__algorithm__archive - - id: config__algorithm__coverage_interval - source: config__algorithm__coverage_interval - - id: config__algorithm__exclude_regions - source: config__algorithm__exclude_regions - - id: config__algorithm__variant_regions - source: prep_samples/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: prep_samples/config__algorithm__variant_regions_merged - - id: config__algorithm__variant_regions_orig - source: prep_samples/config__algorithm__variant_regions_orig - - id: config__algorithm__coverage - source: prep_samples/config__algorithm__coverage - - id: config__algorithm__coverage_merged - source: prep_samples/config__algorithm__coverage_merged - - id: config__algorithm__coverage_orig - source: prep_samples/config__algorithm__coverage_orig - - id: config__algorithm__seq2c_bed_ready - source: prep_samples/config__algorithm__seq2c_bed_ready - - id: config__algorithm__recalibrate - source: config__algorithm__recalibrate - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: genome_resources__rnaseq__gene_bed - source: genome_resources__rnaseq__gene_bed - - id: genome_resources__variation__dbsnp - source: genome_resources__variation__dbsnp - - id: genome_resources__variation__lcr - source: genome_resources__variation__lcr - - id: genome_resources__variation__polyx - source: genome_resources__variation__polyx - - id: genome_resources__variation__encode_blacklist - source: genome_resources__variation__encode_blacklist - - id: reference__fasta__base - source: reference__fasta__base - - id: resources - source: resources - - id: description - source: description - out: - - id: postprocess_alignment_rec - run: steps/postprocess_alignment_to_rec.cwl -- id: postprocess_alignment - in: - - id: postprocess_alignment_rec - source: postprocess_alignment_to_rec/postprocess_alignment_rec - out: - - id: config__algorithm__coverage_interval - - id: config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - - id: config__algorithm__variant_regions_orig - - id: config__algorithm__coverage - - id: config__algorithm__coverage_merged - - id: config__algorithm__coverage_orig - - id: config__algorithm__seq2c_bed_ready - - id: regions__callable - - id: regions__sample_callable - - id: regions__nblock - - id: depth__samtools__stats - - id: depth__samtools__idxstats - - id: depth__variant_regions__regions - - id: depth__variant_regions__dist - - id: depth__sv_regions__regions - - id: depth__sv_regions__dist - - id: depth__coverage__regions - - id: depth__coverage__dist - - id: depth__coverage__thresholds - - id: align_bam - run: steps/postprocess_alignment.cwl - scatter: - - postprocess_alignment_rec - scatterMethod: dotproduct -- id: combine_sample_regions - in: - - id: regions__callable - source: postprocess_alignment/regions__callable - - id: regions__nblock - source: postprocess_alignment/regions__nblock - - id: metadata__batch - source: metadata__batch - - id: config__algorithm__nomap_split_size - source: config__algorithm__nomap_split_size - - id: config__algorithm__nomap_split_targets - source: config__algorithm__nomap_split_targets - - id: reference__fasta__base - source: reference__fasta__base - - id: resources - source: resources - - id: description - source: description - out: - - id: config__algorithm__callable_regions - - id: config__algorithm__non_callable_regions - - id: config__algorithm__callable_count - run: steps/combine_sample_regions.cwl -- id: batch_for_variantcall - in: - - id: analysis - source: analysis - - id: genome_build - source: genome_build - - id: align_bam - source: postprocess_alignment/align_bam - - id: vrn_file - source: vrn_file - - id: metadata__batch - source: metadata__batch - - id: metadata__phenotype - source: metadata__phenotype - - id: config__algorithm__callable_regions - source: combine_sample_regions/config__algorithm__callable_regions - - id: regions__sample_callable - source: postprocess_alignment/regions__sample_callable - - id: config__algorithm__variantcaller - source: config__algorithm__variantcaller - - id: config__algorithm__ensemble - source: config__algorithm__ensemble - - id: config__algorithm__vcfanno - source: config__algorithm__vcfanno - - id: config__algorithm__coverage_interval - source: postprocess_alignment/config__algorithm__coverage_interval - - id: config__algorithm__effects - source: config__algorithm__effects - - id: config__algorithm__min_allele_fraction - source: config__algorithm__min_allele_fraction - - id: config__algorithm__exclude_regions - source: config__algorithm__exclude_regions - - id: config__algorithm__variant_regions - source: postprocess_alignment/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: postprocess_alignment/config__algorithm__variant_regions_merged - - id: config__algorithm__validate - source: config__algorithm__validate - - id: config__algorithm__validate_regions - source: config__algorithm__validate_regions - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: config__algorithm__tools_off - source: config__algorithm__tools_off - - id: reference__fasta__base - source: reference__fasta__base - - id: reference__rtg - source: reference__rtg - - id: reference__genome_context - source: reference__genome_context - - id: genome_resources__variation__clinvar - source: genome_resources__variation__clinvar - - id: genome_resources__variation__cosmic - source: genome_resources__variation__cosmic - - id: genome_resources__variation__dbsnp - source: genome_resources__variation__dbsnp - - id: genome_resources__variation__esp - source: genome_resources__variation__esp - - id: genome_resources__variation__exac - source: genome_resources__variation__exac - - id: genome_resources__variation__gnomad_exome - source: genome_resources__variation__gnomad_exome - - id: genome_resources__variation__1000g - source: genome_resources__variation__1000g - - id: genome_resources__variation__lcr - source: genome_resources__variation__lcr - - id: genome_resources__variation__polyx - source: genome_resources__variation__polyx - - id: genome_resources__variation__encode_blacklist - source: genome_resources__variation__encode_blacklist - - id: genome_resources__aliases__ensembl - source: genome_resources__aliases__ensembl - - id: genome_resources__aliases__human - source: genome_resources__aliases__human - - id: genome_resources__aliases__snpeff - source: genome_resources__aliases__snpeff - - id: reference__snpeff__GRCh37_75 - source: reference__snpeff__GRCh37_75 - - id: genome_resources__variation__train_hapmap - source: genome_resources__variation__train_hapmap - - id: genome_resources__variation__train_indels - source: genome_resources__variation__train_indels - - id: resources - source: resources - - id: description - source: description - out: - - id: batch_rec - run: steps/batch_for_variantcall.cwl -- id: variantcall - in: - - id: batch_rec - source: batch_for_variantcall/batch_rec - out: - - id: vc_rec - run: wf-variantcall.cwl - scatter: - - batch_rec - scatterMethod: dotproduct -- id: batch_for_ensemble - in: - - id: vc_rec - source: variantcall/vc_rec - out: - - id: ensemble_prep_rec - run: steps/batch_for_ensemble.cwl -- id: combine_calls - in: - - id: ensemble_prep_rec - source: batch_for_ensemble/ensemble_prep_rec - out: - - id: ensemble_rec - run: steps/combine_calls.cwl - scatter: - - ensemble_prep_rec - scatterMethod: dotproduct -- id: summarize_vc - in: - - id: vc_rec - source: variantcall/vc_rec - - id: ensemble_rec - source: combine_calls/ensemble_rec - out: - - id: variants__calls - - id: variants__gvcf - - id: variants__samples - - id: validate__grading_summary - - id: validate__grading_plots - run: steps/summarize_vc.cwl -- id: calculate_sv_bins - in: - - id: align_bam - source: postprocess_alignment/align_bam - - id: reference__fasta__base - source: reference__fasta__base - - id: metadata__batch - source: metadata__batch - - id: metadata__phenotype - source: metadata__phenotype - - id: config__algorithm__callable_regions - source: combine_sample_regions/config__algorithm__callable_regions - - id: config__algorithm__coverage_interval - source: postprocess_alignment/config__algorithm__coverage_interval - - id: config__algorithm__exclude_regions - source: config__algorithm__exclude_regions - - id: config__algorithm__variant_regions - source: postprocess_alignment/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: postprocess_alignment/config__algorithm__variant_regions_merged - - id: config__algorithm__seq2c_bed_ready - source: postprocess_alignment/config__algorithm__seq2c_bed_ready - - id: config__algorithm__svcaller - source: config__algorithm__svcaller - - id: depth__variant_regions__regions - source: postprocess_alignment/depth__variant_regions__regions - - id: genome_resources__variation__lcr - source: genome_resources__variation__lcr - - id: genome_resources__variation__polyx - source: genome_resources__variation__polyx - - id: genome_resources__variation__encode_blacklist - source: genome_resources__variation__encode_blacklist - - id: genome_resources__rnaseq__gene_bed - source: genome_resources__rnaseq__gene_bed - - id: resources - source: resources - - id: description - source: description - out: - - id: sv_bin_rec - run: steps/calculate_sv_bins.cwl -- id: calculate_sv_coverage - in: - - id: sv_bin_rec - source: calculate_sv_bins/sv_bin_rec - out: - - id: sv_rawcoverage_rec - run: steps/calculate_sv_coverage.cwl - scatter: - - sv_bin_rec - scatterMethod: dotproduct -- id: normalize_sv_coverage - in: - - id: sv_rawcoverage_rec - source: calculate_sv_coverage/sv_rawcoverage_rec - out: - - id: sv_coverage_rec - run: steps/normalize_sv_coverage.cwl -- id: batch_for_sv - in: - - id: analysis - source: analysis - - id: genome_build - source: genome_build - - id: work_bam_plus__disc - source: alignment/work_bam_plus__disc - - id: work_bam_plus__sr - source: alignment/work_bam_plus__sr - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: config__algorithm__tools_off - source: config__algorithm__tools_off - - id: config__algorithm__svprioritize - source: config__algorithm__svprioritize - - id: config__algorithm__svvalidate - source: config__algorithm__svvalidate - - id: regions__sample_callable - source: postprocess_alignment/regions__sample_callable - - id: genome_resources__variation__gc_profile - source: genome_resources__variation__gc_profile - - id: genome_resources__variation__germline_het_pon - source: genome_resources__variation__germline_het_pon - - id: genome_resources__aliases__snpeff - source: genome_resources__aliases__snpeff - - id: reference__snpeff__GRCh37_75 - source: reference__snpeff__GRCh37_75 - - id: sv_coverage_rec - source: normalize_sv_coverage/sv_coverage_rec - - id: variants__samples - source: summarize_vc/variants__samples - out: - - id: sv_batch_rec - run: steps/batch_for_sv.cwl -- id: svcall - in: - - id: sv_batch_rec - source: batch_for_sv/sv_batch_rec - out: - - id: sv_rec - run: wf-svcall.cwl - scatter: - - sv_batch_rec - scatterMethod: dotproduct -- id: summarize_sv - in: - - id: sv_rec - source: svcall/sv_rec - out: - - id: sv__calls - - id: sv__supplemental - - id: sv__prioritize__tsv - - id: sv__prioritize__raw - - id: svvalidate__grading_summary - - id: svvalidate__grading_plots - run: steps/summarize_sv.cwl -- id: qc_to_rec - in: - - id: align_bam - source: postprocess_alignment/align_bam - - id: analysis - source: analysis - - id: reference__fasta__base - source: reference__fasta__base - - id: reference__versions - source: reference__versions - - id: config__algorithm__tools_on - source: config__algorithm__tools_on - - id: config__algorithm__tools_off - source: config__algorithm__tools_off - - id: genome_build - source: genome_build - - id: config__algorithm__qc - source: config__algorithm__qc - - id: metadata__batch - source: metadata__batch - - id: metadata__phenotype - source: metadata__phenotype - - id: config__algorithm__coverage_interval - source: postprocess_alignment/config__algorithm__coverage_interval - - id: depth__variant_regions__regions - source: postprocess_alignment/depth__variant_regions__regions - - id: depth__variant_regions__dist - source: postprocess_alignment/depth__variant_regions__dist - - id: depth__samtools__stats - source: postprocess_alignment/depth__samtools__stats - - id: depth__samtools__idxstats - source: postprocess_alignment/depth__samtools__idxstats - - id: depth__sv_regions__regions - source: postprocess_alignment/depth__sv_regions__regions - - id: depth__sv_regions__dist - source: postprocess_alignment/depth__sv_regions__dist - - id: depth__coverage__regions - source: postprocess_alignment/depth__coverage__regions - - id: depth__coverage__dist - source: postprocess_alignment/depth__coverage__dist - - id: depth__coverage__thresholds - source: postprocess_alignment/depth__coverage__thresholds - - id: config__algorithm__variant_regions - source: postprocess_alignment/config__algorithm__variant_regions - - id: config__algorithm__variant_regions_merged - source: postprocess_alignment/config__algorithm__variant_regions_merged - - id: config__algorithm__coverage - source: postprocess_alignment/config__algorithm__coverage - - id: config__algorithm__coverage_merged - source: postprocess_alignment/config__algorithm__coverage_merged - - id: variants__samples - source: summarize_vc/variants__samples - - id: reference__viral - source: reference__viral - - id: resources - source: resources - - id: description - source: description - out: - - id: qc_rec - run: steps/qc_to_rec.cwl -- id: pipeline_summary - in: - - id: qc_rec - source: qc_to_rec/qc_rec - out: - - id: qcout_rec - run: steps/pipeline_summary.cwl - scatter: - - qc_rec - scatterMethod: dotproduct -- id: multiqc_summary - in: - - id: qcout_rec - source: pipeline_summary/qcout_rec - out: - - id: summary__multiqc - - id: versions__tools - - id: versions__data - run: steps/multiqc_summary.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/alignment_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/alignment_to_rec.cwl deleted file mode 100644 index 506764917f5..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/alignment_to_rec.cwl +++ /dev/null @@ -1,194 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=alignment_rec:resources;description;config__algorithm__align_split_size;files;config__algorithm__trim_reads;reference__fasta__base;config__algorithm__adapters;rgnames__lb;rgnames__rg;rgnames__lane;reference__bwa__indexes;config__algorithm__bam_clean;config__algorithm__aligner;rgnames__pl;config__algorithm__mark_duplicates;analysis;rgnames__sample;config__algorithm__variant_regions;rgnames__pu -- sentinel_inputs=files:var,analysis:var,config__algorithm__align_split_size:var,reference__fasta__base:var,rgnames__pl:var,rgnames__sample:var,rgnames__pu:var,rgnames__lane:var,rgnames__rg:var,rgnames__lb:var,reference__bwa__indexes:var,config__algorithm__aligner:var,config__algorithm__trim_reads:var,config__algorithm__adapters:var,config__algorithm__bam_clean:var,config__algorithm__variant_regions:var,config__algorithm__mark_duplicates:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- alignment_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10418 - ramMin: 3072 - tmpdirMin: 89 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: files - secondaryFiles: - - .bai - type: - items: - items: File - type: array - type: array -- id: analysis - type: - items: string - type: array -- id: config__algorithm__align_split_size - type: - items: - - 'null' - - string - type: array -- id: reference__fasta__base - secondaryFiles: - - ^.dict - - .fai - type: - items: File - type: array -- id: rgnames__pl - type: - items: string - type: array -- id: rgnames__sample - type: - items: string - type: array -- id: rgnames__pu - type: - items: string - type: array -- id: rgnames__lane - type: - items: string - type: array -- id: rgnames__rg - type: - items: string - type: array -- id: rgnames__lb - type: - items: - - 'null' - - string - type: array -- id: reference__bwa__indexes - secondaryFiles: - - ^.ann - - ^.pac - - ^.sa - - ^.bwt - type: - items: File - type: array -- id: config__algorithm__aligner - type: - items: string - type: array -- id: config__algorithm__trim_reads - type: - items: string - type: array -- id: config__algorithm__adapters - type: - items: - items: string - type: array - type: array -- id: config__algorithm__bam_clean - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__variant_regions - type: - items: File - type: array -- id: config__algorithm__mark_duplicates - type: - items: - - string - - 'null' - - boolean - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: alignment_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - - name: rgnames__pu - type: string - name: alignment_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/batch_for_ensemble.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/batch_for_ensemble.cwl deleted file mode 100644 index df553e9e92f..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/batch_for_ensemble.cwl +++ /dev/null @@ -1,273 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=ensemble_prep_rec:batch_id;variants__calls;variants__variantcallers;resources;description;batch_samples;validate__summary;validate__tp;validate__fp;validate__fn;vrn_file;reference__fasta__base;config__algorithm__vcfanno;config__algorithm__variantcaller;config__algorithm__coverage_interval;metadata__batch;config__algorithm__min_allele_fraction;reference__snpeff__GRCh37_75;reference__genome_context;reference__rtg;config__algorithm__validate;config__algorithm__validate_regions;genome_build;metadata__phenotype;genome_resources__aliases__human;config__algorithm__tools_off;config__algorithm__ensemble;analysis;config__algorithm__tools_on;config__algorithm__effects;config__algorithm__variant_regions;genome_resources__aliases__ensembl;config__algorithm__exclude_regions;genome_resources__aliases__snpeff;config__algorithm__variant_regions_merged;regions__sample_callable;config__algorithm__callable_regions -- sentinel_inputs=vc_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- batch_for_ensemble -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10240 - ramMin: 3072 - tmpdirMin: 0 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: vc_rec - type: - items: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__snpeff__GRCh37_75 - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: reference__rtg - type: File - - name: config__algorithm__validate - type: - - File - - 'null' - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: config__algorithm__ensemble - type: string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array - type: array -outputs: -- id: ensemble_prep_rec - type: - items: - fields: - - name: batch_id - type: string - - name: variants__calls - type: - items: File - type: array - - name: variants__variantcallers - type: - items: string - type: array - - name: resources - type: string - - name: description - type: string - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__snpeff__GRCh37_75 - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: reference__rtg - type: File - - name: config__algorithm__validate - type: - - File - - 'null' - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: config__algorithm__ensemble - type: string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: ensemble_prep_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/batch_for_sv.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/batch_for_sv.cwl deleted file mode 100644 index c25d67e007b..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/batch_for_sv.cwl +++ /dev/null @@ -1,363 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-batch -- sentinel_outputs=sv_batch_rec:resources;description;config__algorithm__svprioritize;genome_resources__variation__gc_profile;reference__snpeff__GRCh37_75;genome_build;genome_resources__variation__germline_het_pon;config__algorithm__tools_off;analysis;config__algorithm__tools_on;config__algorithm__svvalidate;genome_resources__aliases__snpeff;work_bam_plus__disc;work_bam_plus__sr;regions__sample_callable;variants__samples;depth__bins__normalized;depth__bins__background;depth__bins__target;depth__bins__antitarget;depth__bins__seq2c;regions__bins__target;regions__bins__antitarget;regions__bins__gcannotated;regions__bins__group;reference__fasta__base;config__algorithm__svcaller;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;metadata__batch;genome_resources__variation__lcr;metadata__phenotype;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__seq2c_bed_ready;depth__variant_regions__regions;config__algorithm__callable_regions -- sentinel_inputs=analysis:var,genome_build:var,work_bam_plus__disc:var,work_bam_plus__sr:var,config__algorithm__tools_on:var,config__algorithm__tools_off:var,config__algorithm__svprioritize:var,config__algorithm__svvalidate:var,regions__sample_callable:var,genome_resources__variation__gc_profile:var,genome_resources__variation__germline_het_pon:var,genome_resources__aliases__snpeff:var,reference__snpeff__GRCh37_75:var,sv_coverage_rec:record,variants__samples:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- batch_for_sv -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 16 - outdirMin: 10240 - ramMin: 49152 - tmpdirMin: 0 -- class: dx:InputResourceRequirement - indirMin: 3656 -inputs: -- id: analysis - type: - items: string - type: array -- id: genome_build - type: - items: string - type: array -- id: work_bam_plus__disc - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: work_bam_plus__sr - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: config__algorithm__tools_off - type: - items: - items: string - type: array - type: array -- id: config__algorithm__svprioritize - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__svvalidate - type: - items: - - 'null' - - string - type: array -- id: regions__sample_callable - type: - items: - - File - - 'null' - type: array -- id: genome_resources__variation__gc_profile - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__germline_het_pon - type: - items: - - 'null' - - string - type: array -- id: genome_resources__aliases__snpeff - type: - items: string - type: array -- id: reference__snpeff__GRCh37_75 - type: - items: File - type: array -- id: sv_coverage_rec - type: - items: - fields: - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_coverage_rec - type: record - type: array -- id: variants__samples - type: - items: - items: - items: - - File - - 'null' - type: array - type: array - type: array -outputs: -- id: sv_batch_rec - type: - items: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: - - 'null' - - string - - name: genome_resources__variation__gc_profile - type: - - 'null' - - string - - name: reference__snpeff__GRCh37_75 - type: File - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__svvalidate - type: - - 'null' - - string - - name: genome_resources__aliases__snpeff - type: string - - name: work_bam_plus__disc - type: - - File - - 'null' - - name: work_bam_plus__sr - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_batch_rec - type: record - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/batch_for_variantcall.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/batch_for_variantcall.cwl deleted file mode 100644 index 6504ed26811..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/batch_for_variantcall.cwl +++ /dev/null @@ -1,395 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-batch -- sentinel_outputs=batch_rec:resources;description;reference__fasta__base;config__algorithm__vcfanno;config__algorithm__variantcaller;config__algorithm__coverage_interval;genome_resources__variation__clinvar;genome_resources__variation__esp;metadata__batch;genome_resources__variation__lcr;config__algorithm__min_allele_fraction;reference__snpeff__GRCh37_75;vrn_file;genome_resources__variation__train_hapmap;reference__genome_context;reference__rtg;config__algorithm__validate;genome_resources__variation__1000g;config__algorithm__validate_regions;genome_build;genome_resources__variation__exac;genome_resources__variation__gnomad_exome;metadata__phenotype;genome_resources__aliases__human;config__algorithm__tools_off;genome_resources__variation__dbsnp;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;genome_resources__variation__cosmic;config__algorithm__ensemble;analysis;config__algorithm__tools_on;config__algorithm__effects;config__algorithm__variant_regions;genome_resources__aliases__ensembl;config__algorithm__exclude_regions;genome_resources__variation__train_indels;genome_resources__aliases__snpeff;align_bam;config__algorithm__variant_regions_merged;regions__sample_callable;config__algorithm__callable_regions -- sentinel_inputs=analysis:var,genome_build:var,align_bam:var,vrn_file:var,metadata__batch:var,metadata__phenotype:var,config__algorithm__callable_regions:var,regions__sample_callable:var,config__algorithm__variantcaller:var,config__algorithm__ensemble:var,config__algorithm__vcfanno:var,config__algorithm__coverage_interval:var,config__algorithm__effects:var,config__algorithm__min_allele_fraction:var,config__algorithm__exclude_regions:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__validate:var,config__algorithm__validate_regions:var,config__algorithm__tools_on:var,config__algorithm__tools_off:var,reference__fasta__base:var,reference__rtg:var,reference__genome_context:var,genome_resources__variation__clinvar:var,genome_resources__variation__cosmic:var,genome_resources__variation__dbsnp:var,genome_resources__variation__esp:var,genome_resources__variation__exac:var,genome_resources__variation__gnomad_exome:var,genome_resources__variation__1000g:var,genome_resources__variation__lcr:var,genome_resources__variation__polyx:var,genome_resources__variation__encode_blacklist:var,genome_resources__aliases__ensembl:var,genome_resources__aliases__human:var,genome_resources__aliases__snpeff:var,reference__snpeff__GRCh37_75:var,genome_resources__variation__train_hapmap:var,genome_resources__variation__train_indels:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- batch_for_variantcall -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10477 - ramMin: 3072 - tmpdirMin: 119 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: analysis - type: - items: string - type: array -- id: genome_build - type: - items: string - type: array -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: vrn_file - type: - items: - - 'null' - - string - type: array -- id: metadata__batch - type: - items: string - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: config__algorithm__callable_regions - type: - items: File - type: array -- id: regions__sample_callable - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variantcaller - type: - items: - items: string - type: array - type: array -- id: config__algorithm__ensemble - type: - items: string - type: array -- id: config__algorithm__vcfanno - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__coverage_interval - type: - items: - - string - - 'null' - type: array -- id: config__algorithm__effects - type: - items: string - type: array -- id: config__algorithm__min_allele_fraction - type: - items: double - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__validate - secondaryFiles: - - .tbi - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__validate_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: config__algorithm__tools_off - type: - items: - items: string - type: array - type: array -- id: reference__fasta__base - secondaryFiles: - - ^.dict - - .fai - type: - items: File - type: array -- id: reference__rtg - type: - items: File - type: array -- id: reference__genome_context - secondaryFiles: - - .tbi - type: - items: - items: File - type: array - type: array -- id: genome_resources__variation__clinvar - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__cosmic - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__dbsnp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__esp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__exac - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__gnomad_exome - type: - items: - - 'null' - - string - type: array -- id: genome_resources__variation__1000g - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__lcr - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__polyx - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__encode_blacklist - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__aliases__ensembl - type: - items: string - type: array -- id: genome_resources__aliases__human - type: - items: - - string - - 'null' - - boolean - type: array -- id: genome_resources__aliases__snpeff - type: - items: string - type: array -- id: reference__snpeff__GRCh37_75 - type: - items: File - type: array -- id: genome_resources__variation__train_hapmap - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__train_indels - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: batch_rec - type: - items: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__snpeff__GRCh37_75 - type: File - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: reference__rtg - type: File - - name: config__algorithm__validate - type: - - File - - 'null' - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/calculate_sv_bins.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/calculate_sv_bins.cwl deleted file mode 100644 index bee219164db..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/calculate_sv_bins.cwl +++ /dev/null @@ -1,219 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=sv_bin_rec:regions__bins__target;regions__bins__antitarget;regions__bins__gcannotated;regions__bins__group;resources;description;reference__fasta__base;config__algorithm__svcaller;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;metadata__batch;genome_resources__variation__lcr;metadata__phenotype;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__seq2c_bed_ready;depth__variant_regions__regions;config__algorithm__callable_regions -- sentinel_inputs=align_bam:var,reference__fasta__base:var,metadata__batch:var,metadata__phenotype:var,config__algorithm__callable_regions:var,config__algorithm__coverage_interval:var,config__algorithm__exclude_regions:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__seq2c_bed_ready:var,config__algorithm__svcaller:var,depth__variant_regions__regions:var,genome_resources__variation__lcr:var,genome_resources__variation__polyx:var,genome_resources__variation__encode_blacklist:var,genome_resources__rnaseq__gene_bed:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- calculate_sv_bins -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10418 - ramMin: 3072 - tmpdirMin: 89 -- class: dx:InputResourceRequirement - indirMin: 3024 -- class: SoftwareRequirement - packages: - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: cnvkit - specs: - - https://anaconda.org/bioconda/cnvkit -inputs: -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: reference__fasta__base - secondaryFiles: - - ^.dict - - .fai - type: - items: File - type: array -- id: metadata__batch - type: - items: string - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: config__algorithm__callable_regions - type: - items: File - type: array -- id: config__algorithm__coverage_interval - type: - items: - - string - - 'null' - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__seq2c_bed_ready - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__svcaller - type: - items: - items: string - type: array - type: array -- id: depth__variant_regions__regions - type: - items: - - File - - 'null' - type: array -- id: genome_resources__variation__lcr - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__polyx - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__encode_blacklist - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__rnaseq__gene_bed - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: sv_bin_rec - type: - items: - fields: - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_bin_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/calculate_sv_coverage.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/calculate_sv_coverage.cwl deleted file mode 100644 index 330224b83db..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/calculate_sv_coverage.cwl +++ /dev/null @@ -1,206 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=sv_rawcoverage_rec:depth__bins__target;depth__bins__antitarget;depth__bins__seq2c;resources;description;regions__bins__target;regions__bins__antitarget;regions__bins__gcannotated;regions__bins__group;reference__fasta__base;config__algorithm__svcaller;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;metadata__batch;genome_resources__variation__lcr;metadata__phenotype;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__seq2c_bed_ready;depth__variant_regions__regions;config__algorithm__callable_regions -- sentinel_inputs=sv_bin_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- calculate_sv_coverage -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 16 - outdirMin: 10418 - ramMin: 49152 - tmpdirMin: 89 -- class: dx:InputResourceRequirement - indirMin: 3024 -- class: SoftwareRequirement - packages: - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: cnvkit - specs: - - https://anaconda.org/bioconda/cnvkit - - package: seq2c - specs: - - https://anaconda.org/bioconda/seq2c -inputs: -- id: sv_bin_rec - type: - fields: - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_bin_rec - type: record -outputs: -- id: sv_rawcoverage_rec - type: - fields: - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_rawcoverage_rec - type: record -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/combine_calls.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/combine_calls.cwl deleted file mode 100644 index 7730a2a3223..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/combine_calls.cwl +++ /dev/null @@ -1,180 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=ensemble_rec:ensemble__vrn_file;ensemble__validate__summary;ensemble__batch_samples;ensemble__batch_id -- sentinel_inputs=ensemble_prep_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- combine_calls -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 16 - outdirMin: 10240 - ramMin: 49152 - tmpdirMin: 0 -- class: dx:InputResourceRequirement - indirMin: 4687 -- class: SoftwareRequirement - packages: - - package: bcbio-variation-recall - specs: - - https://anaconda.org/bioconda/bcbio-variation-recall -inputs: -- id: ensemble_prep_rec - type: - fields: - - name: batch_id - type: string - - name: variants__calls - type: - items: File - type: array - - name: variants__variantcallers - type: - items: string - type: array - - name: resources - type: string - - name: description - type: string - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__snpeff__GRCh37_75 - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: reference__rtg - type: File - - name: config__algorithm__validate - type: - - File - - 'null' - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: config__algorithm__ensemble - type: string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: ensemble_prep_rec - type: record -outputs: -- id: ensemble_rec - type: - fields: - - name: ensemble__vrn_file - type: - - File - - 'null' - - name: ensemble__validate__summary - type: - - File - - 'null' - - name: ensemble__batch_samples - type: - items: string - type: array - - name: ensemble__batch_id - type: string - name: ensemble_rec - type: record -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/combine_sample_regions.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/combine_sample_regions.cwl deleted file mode 100644 index e59fc89683e..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/combine_sample_regions.cwl +++ /dev/null @@ -1,99 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=config__algorithm__callable_regions,config__algorithm__non_callable_regions,config__algorithm__callable_count -- sentinel_inputs=regions__callable:var,regions__nblock:var,metadata__batch:var,config__algorithm__nomap_split_size:var,config__algorithm__nomap_split_targets:var,reference__fasta__base:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- combine_sample_regions -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10299 - ramMin: 3072 - tmpdirMin: 30 -- class: dx:InputResourceRequirement - indirMin: 3008 -- class: SoftwareRequirement - packages: - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 -- class: arv:APIRequirement -inputs: -- id: regions__callable - type: - items: - - File - - 'null' - type: array -- id: regions__nblock - type: - items: - - File - - 'null' - type: array -- id: metadata__batch - type: - items: string - type: array -- id: config__algorithm__nomap_split_size - type: - items: long - type: array -- id: config__algorithm__nomap_split_targets - type: - items: long - type: array -- id: reference__fasta__base - secondaryFiles: - - ^.dict - - .fai - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: config__algorithm__callable_regions - type: - items: File - type: array -- id: config__algorithm__non_callable_regions - type: - items: File - type: array -- id: config__algorithm__callable_count - type: - items: int - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/compare_to_rm.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/compare_to_rm.cwl deleted file mode 100644 index ed7c9b1e0b1..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/compare_to_rm.cwl +++ /dev/null @@ -1,299 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-single -- sentinel_outputs=vc_rec:batch_samples;validate__summary;validate__tp;validate__fp;validate__fn;resources;description;vrn_file;reference__fasta__base;config__algorithm__vcfanno;config__algorithm__variantcaller;config__algorithm__coverage_interval;metadata__batch;config__algorithm__min_allele_fraction;reference__snpeff__GRCh37_75;reference__genome_context;reference__rtg;config__algorithm__validate;config__algorithm__validate_regions;genome_build;metadata__phenotype;genome_resources__aliases__human;config__algorithm__tools_off;config__algorithm__ensemble;analysis;config__algorithm__tools_on;config__algorithm__effects;config__algorithm__variant_regions;genome_resources__aliases__ensembl;config__algorithm__exclude_regions;genome_resources__aliases__snpeff;config__algorithm__variant_regions_merged;regions__sample_callable;config__algorithm__callable_regions -- sentinel_inputs=batch_rec:record,vrn_file:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- compare_to_rm -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 16 - outdirMin: 10418 - ramMin: 49152 - tmpdirMin: 89 -- class: dx:InputResourceRequirement - indirMin: 24883 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: gvcf-regions - specs: - - https://anaconda.org/bioconda/gvcf-regions - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: rtg-tools - specs: - - https://anaconda.org/bioconda/rtg-tools - - package: vcfanno - specs: - - https://anaconda.org/bioconda/vcfanno -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__snpeff__GRCh37_75 - type: File - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: reference__rtg - type: File - - name: config__algorithm__validate - type: - - File - - 'null' - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: vrn_file - secondaryFiles: - - .tbi - type: File -outputs: -- id: vc_rec - type: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__snpeff__GRCh37_75 - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: reference__rtg - type: File - - name: config__algorithm__validate - type: - - File - - 'null' - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: config__algorithm__ensemble - type: string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/concat_batch_variantcalls.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/concat_batch_variantcalls.cwl deleted file mode 100644 index a5d59d661a0..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/concat_batch_variantcalls.cwl +++ /dev/null @@ -1,190 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-merge -- sentinel_outputs=vrn_file -- sentinel_inputs=batch_rec:record,region_block:var,vrn_file_region:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- concat_batch_variantcalls -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10418 - ramMin: 3072 - tmpdirMin: 89 -- class: dx:InputResourceRequirement - indirMin: 24883 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 -- class: arv:APIRequirement -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__snpeff__GRCh37_75 - type: File - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: reference__rtg - type: File - - name: config__algorithm__validate - type: - - File - - 'null' - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: region_block - type: - items: - items: string - type: array - type: array -- id: vrn_file_region - secondaryFiles: - - .tbi - type: - items: - - File - - 'null' - type: array -outputs: -- id: vrn_file - secondaryFiles: - - .tbi - type: File -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/detect_sv.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/detect_sv.cwl deleted file mode 100644 index 66d592642d7..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/detect_sv.cwl +++ /dev/null @@ -1,402 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-single -- sentinel_outputs=sv_rec:sv__variantcaller;sv__vrn_file;sv__supplemental;svvalidate__summary;resources;description;config__algorithm__svprioritize;genome_resources__variation__gc_profile;genome_build;genome_resources__variation__germline_het_pon;config__algorithm__tools_off;analysis;config__algorithm__tools_on;config__algorithm__svvalidate;genome_resources__aliases__snpeff;regions__sample_callable;variants__samples;depth__bins__normalized;depth__bins__background;depth__bins__target;depth__bins__antitarget;depth__bins__seq2c;regions__bins__target;regions__bins__antitarget;regions__bins__gcannotated;regions__bins__group;reference__fasta__base;config__algorithm__svcaller;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;metadata__batch;genome_resources__variation__lcr;metadata__phenotype;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__variant_regions;config__algorithm__exclude_regions;config__algorithm__variant_regions_merged;config__algorithm__seq2c_bed_ready;depth__variant_regions__regions;config__algorithm__callable_regions -- sentinel_inputs=sv_batch_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- detect_sv -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 16 - outdirMin: 10477 - ramMin: 49152 - tmpdirMin: 119 -- class: dx:InputResourceRequirement - indirMin: 3656 -- class: SoftwareRequirement - packages: - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: cnvkit - specs: - - https://anaconda.org/bioconda/cnvkit - - package: delly - specs: - - https://anaconda.org/bioconda/delly - - package: duphold - specs: - - https://anaconda.org/bioconda/duphold - - package: extract-sv-reads - specs: - - https://anaconda.org/bioconda/extract-sv-reads - - package: lumpy-sv - specs: - - https://anaconda.org/bioconda/lumpy-sv - - package: manta - specs: - - https://anaconda.org/bioconda/manta - - package: break-point-inspector - specs: - - https://anaconda.org/bioconda/break-point-inspector - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: smoove - specs: - - https://anaconda.org/bioconda/smoove - - package: pysam> - specs: - - https://anaconda.org/bioconda/pysam> - version: - - 0.13.0 - - package: seq2c - specs: - - https://anaconda.org/bioconda/seq2c - - package: simple_sv_annotation - specs: - - https://anaconda.org/bioconda/simple_sv_annotation - - package: survivor - specs: - - https://anaconda.org/bioconda/survivor - - package: svtools - specs: - - https://anaconda.org/bioconda/svtools - - package: svtyper - specs: - - https://anaconda.org/bioconda/svtyper - - package: r - specs: - - https://anaconda.org/bioconda/r - version: - - 3.4.1 - - package: r-base=3.4.1=h4fe35fd_8 - specs: - - https://anaconda.org/bioconda/r-base=3.4.1=h4fe35fd_8 - - package: xorg-libxt - specs: - - https://anaconda.org/bioconda/xorg-libxt - - package: vawk - specs: - - https://anaconda.org/bioconda/vawk -- class: arv:RuntimeConstraints - keep_cache: 4096 -inputs: -- id: sv_batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: - - 'null' - - string - - name: genome_resources__variation__gc_profile - type: - - 'null' - - string - - name: reference__snpeff__GRCh37_75 - type: File - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__svvalidate - type: - - 'null' - - string - - name: genome_resources__aliases__snpeff - type: string - - name: work_bam_plus__disc - type: - - File - - 'null' - - name: work_bam_plus__sr - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_batch_rec - type: record - type: array -outputs: -- id: sv_rec - type: - items: - fields: - - name: sv__variantcaller - type: - - string - - 'null' - - name: sv__vrn_file - type: - - File - - 'null' - - name: sv__supplemental - type: - items: - - File - type: array - - name: svvalidate__summary - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: - - 'null' - - string - - name: genome_resources__variation__gc_profile - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__svvalidate - type: - - 'null' - - string - - name: genome_resources__aliases__snpeff - type: string - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/get_parallel_regions.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/get_parallel_regions.cwl deleted file mode 100644 index 71c08911610..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/get_parallel_regions.cwl +++ /dev/null @@ -1,165 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-split -- sentinel_outputs=region_block -- sentinel_inputs=batch_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- get_parallel_regions -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10477 - ramMin: 3072 - tmpdirMin: 119 -- class: dx:InputResourceRequirement - indirMin: 24883 -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__snpeff__GRCh37_75 - type: File - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: reference__rtg - type: File - - name: config__algorithm__validate - type: - - File - - 'null' - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -outputs: -- id: region_block - type: - items: - items: string - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/merge_split_alignments.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/merge_split_alignments.cwl deleted file mode 100644 index 4e139a4d748..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/merge_split_alignments.cwl +++ /dev/null @@ -1,165 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=single-merge -- sentinel_outputs=align_bam,work_bam_plus__disc,work_bam_plus__sr,hla__fastq -- sentinel_inputs=alignment_rec:record,work_bam:var,align_bam:var,work_bam_plus__disc:var,work_bam_plus__sr:var,hla__fastq:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- merge_split_alignments -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 16 - outdirMin: 10655 - ramMin: 49152 - tmpdirMin: 208 -- class: dx:InputResourceRequirement - indirMin: 3127 -- class: SoftwareRequirement - packages: - - package: biobambam - specs: - - https://anaconda.org/bioconda/biobambam - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: variantbam - specs: - - https://anaconda.org/bioconda/variantbam -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - - name: rgnames__pu - type: string - name: alignment_rec - type: record -- id: work_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: align_bam_toolinput - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: work_bam_plus__disc_toolinput - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: work_bam_plus__sr_toolinput - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: hla__fastq_toolinput - type: - items: - - 'null' - - items: File - type: array - type: array -outputs: -- id: align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__disc - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__sr - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: hla__fastq - type: - - 'null' - - items: File - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/multiqc_summary.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/multiqc_summary.cwl deleted file mode 100644 index 96036a07fc2..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/multiqc_summary.cwl +++ /dev/null @@ -1,94 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=summary__multiqc,versions__tools,versions__data -- sentinel_inputs=qcout_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- multiqc_summary -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10477 - ramMin: 3072 - tmpdirMin: 119 -- class: dx:InputResourceRequirement - indirMin: 1 -- class: SoftwareRequirement - packages: - - package: multiqc - specs: - - https://anaconda.org/bioconda/multiqc - - package: multiqc-bcbio - specs: - - https://anaconda.org/bioconda/multiqc-bcbio -inputs: -- id: qcout_rec - type: - items: - fields: - - name: summary__qc - type: - - File - - 'null' - - name: summary__metrics - type: - - string - - 'null' - - name: description - type: string - - name: genome_build - type: string - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: reference__versions - type: File - - name: config__algorithm__qc - type: - items: string - type: array - - name: config__algorithm__tools_on - type: - items: string - type: array - name: qcout_rec - type: record - type: array -outputs: -- id: summary__multiqc - type: - items: - - File - - 'null' - type: array -- id: versions__tools - type: - items: - - File - - 'null' - type: array -- id: versions__data - type: - items: - - File - - 'null' - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/normalize_sv_coverage.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/normalize_sv_coverage.cwl deleted file mode 100644 index af33573637d..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/normalize_sv_coverage.cwl +++ /dev/null @@ -1,224 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=sv_coverage_rec:depth__bins__normalized;depth__bins__background;resources;description;depth__bins__target;depth__bins__antitarget;depth__bins__seq2c;regions__bins__target;regions__bins__antitarget;regions__bins__gcannotated;regions__bins__group;reference__fasta__base;config__algorithm__svcaller;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;metadata__batch;genome_resources__variation__lcr;metadata__phenotype;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__variant_regions;config__algorithm__exclude_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__seq2c_bed_ready;depth__variant_regions__regions;config__algorithm__callable_regions -- sentinel_inputs=sv_rawcoverage_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- normalize_sv_coverage -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 16 - outdirMin: 10418 - ramMin: 49152 - tmpdirMin: 89 -- class: dx:InputResourceRequirement - indirMin: 3024 -- class: SoftwareRequirement - packages: - - package: cnvkit - specs: - - https://anaconda.org/bioconda/cnvkit -inputs: -- id: sv_rawcoverage_rec - type: - items: - fields: - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_rawcoverage_rec - type: record - type: array -outputs: -- id: sv_coverage_rec - type: - items: - fields: - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_coverage_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/pipeline_summary.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/pipeline_summary.cwl deleted file mode 100644 index 4ffe2646ec9..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/pipeline_summary.cwl +++ /dev/null @@ -1,219 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=qcout_rec:summary__qc;summary__metrics;description;genome_build;config__algorithm__tools_off;reference__versions;config__algorithm__qc;config__algorithm__tools_on -- sentinel_inputs=qc_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- pipeline_summary -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 16 - outdirMin: 10477 - ramMin: 49152 - tmpdirMin: 119 -- class: dx:InputResourceRequirement - indirMin: 3010 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: fastqc=0.11.7=5 - specs: - - https://anaconda.org/bioconda/fastqc=0.11.7=5 - - package: goleft - specs: - - https://anaconda.org/bioconda/goleft - - package: hts-nim-tools - specs: - - https://anaconda.org/bioconda/hts-nim-tools - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: picard - specs: - - https://anaconda.org/bioconda/picard - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: qsignature - specs: - - https://anaconda.org/bioconda/qsignature - - package: qualimap - specs: - - https://anaconda.org/bioconda/qualimap - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: preseq - specs: - - https://anaconda.org/bioconda/preseq - - package: peddy - specs: - - https://anaconda.org/bioconda/peddy - - package: verifybamid2 - specs: - - https://anaconda.org/bioconda/verifybamid2 -- class: arv:RuntimeConstraints - keep_cache: 4096 -inputs: -- id: qc_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: reference__versions - type: File - - name: reference__viral - type: File - - name: config__algorithm__qc - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: depth__samtools__stats - type: - - File - - 'null' - - name: depth__samtools__idxstats - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: depth__variant_regions__dist - type: - - File - - 'null' - - name: depth__sv_regions__regions - type: - - File - - 'null' - - name: depth__sv_regions__dist - type: - - File - - 'null' - - name: depth__coverage__regions - type: - - File - - 'null' - - name: depth__coverage__dist - type: - - File - - 'null' - - name: depth__coverage__thresholds - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - name: qc_rec - type: record -outputs: -- id: qcout_rec - type: - fields: - - name: summary__qc - type: - - File - - 'null' - - name: summary__metrics - type: - - string - - 'null' - - name: description - type: string - - name: genome_build - type: string - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: reference__versions - type: File - - name: config__algorithm__qc - type: - items: string - type: array - - name: config__algorithm__tools_on - type: - items: string - type: array - name: qcout_rec - type: record -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/postprocess_alignment.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/postprocess_alignment.cwl deleted file mode 100644 index 92f4acb8c7f..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/postprocess_alignment.cwl +++ /dev/null @@ -1,221 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=config__algorithm__coverage_interval,config__algorithm__variant_regions,config__algorithm__variant_regions_merged,config__algorithm__variant_regions_orig,config__algorithm__coverage,config__algorithm__coverage_merged,config__algorithm__coverage_orig,config__algorithm__seq2c_bed_ready,regions__callable,regions__sample_callable,regions__nblock,depth__samtools__stats,depth__samtools__idxstats,depth__variant_regions__regions,depth__variant_regions__dist,depth__sv_regions__regions,depth__sv_regions__dist,depth__coverage__regions,depth__coverage__dist,depth__coverage__thresholds,align_bam -- sentinel_inputs=postprocess_alignment_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- postprocess_alignment -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 16 - outdirMin: 10596 - ramMin: 49152 - tmpdirMin: 178 -- class: dx:InputResourceRequirement - indirMin: 18018 -- class: SoftwareRequirement - packages: - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: goleft - specs: - - https://anaconda.org/bioconda/goleft - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 - - package: mosdepth - specs: - - https://anaconda.org/bioconda/mosdepth - - package: sentieon - specs: - - https://anaconda.org/bioconda/sentieon -- class: arv:APIRequirement -inputs: -- id: postprocess_alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - 'null' - - string - - name: genome_resources__rnaseq__gene_bed - type: File - - name: genome_resources__variation__lcr - type: File - - name: config__algorithm__recalibrate - type: - - string - - 'null' - - boolean - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__archive - type: - - 'null' - - string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__variant_regions_orig - type: - - File - - 'null' - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: config__algorithm__coverage_orig - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - name: postprocess_alignment_rec - type: record -outputs: -- id: config__algorithm__coverage_interval - type: - - string - - 'null' -- id: config__algorithm__variant_regions - type: - - File - - 'null' -- id: config__algorithm__variant_regions_merged - type: - - File - - 'null' -- id: config__algorithm__variant_regions_orig - type: - - File - - 'null' -- id: config__algorithm__coverage - type: - - File - - 'null' -- id: config__algorithm__coverage_merged - type: - - File - - 'null' -- id: config__algorithm__coverage_orig - type: - - File - - 'null' -- id: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' -- id: regions__callable - type: - - File - - 'null' -- id: regions__sample_callable - type: - - File - - 'null' -- id: regions__nblock - type: - - File - - 'null' -- id: depth__samtools__stats - type: - - File - - 'null' -- id: depth__samtools__idxstats - type: - - File - - 'null' -- id: depth__variant_regions__regions - type: - - File - - 'null' -- id: depth__variant_regions__dist - type: - - File - - 'null' -- id: depth__sv_regions__regions - type: - - File - - 'null' -- id: depth__sv_regions__dist - type: - - File - - 'null' -- id: depth__coverage__regions - type: - - File - - 'null' -- id: depth__coverage__dist - type: - - File - - 'null' -- id: depth__coverage__thresholds - type: - - File - - 'null' -- id: align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/postprocess_alignment_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/postprocess_alignment_to_rec.cwl deleted file mode 100644 index 926d37e91df..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/postprocess_alignment_to_rec.cwl +++ /dev/null @@ -1,237 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=postprocess_alignment_rec:resources;description;reference__fasta__base;config__algorithm__coverage_interval;genome_resources__rnaseq__gene_bed;genome_resources__variation__lcr;config__algorithm__recalibrate;genome_resources__variation__dbsnp;genome_resources__variation__polyx;genome_resources__variation__encode_blacklist;config__algorithm__tools_on;config__algorithm__variant_regions;config__algorithm__exclude_regions;config__algorithm__archive;align_bam;config__algorithm__variant_regions_merged;config__algorithm__variant_regions_orig;config__algorithm__coverage;config__algorithm__coverage_merged;config__algorithm__coverage_orig;config__algorithm__seq2c_bed_ready -- sentinel_inputs=align_bam:var,config__algorithm__archive:var,config__algorithm__coverage_interval:var,config__algorithm__exclude_regions:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__variant_regions_orig:var,config__algorithm__coverage:var,config__algorithm__coverage_merged:var,config__algorithm__coverage_orig:var,config__algorithm__seq2c_bed_ready:var,config__algorithm__recalibrate:var,config__algorithm__tools_on:var,genome_resources__rnaseq__gene_bed:var,genome_resources__variation__dbsnp:var,genome_resources__variation__lcr:var,genome_resources__variation__polyx:var,genome_resources__variation__encode_blacklist:var,reference__fasta__base:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- postprocess_alignment_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10418 - ramMin: 3072 - tmpdirMin: 89 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__archive - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__coverage_interval - type: - items: - - 'null' - - string - type: array -- id: config__algorithm__exclude_regions - type: - items: - - 'null' - - items: 'null' - type: array - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_orig - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_orig - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__seq2c_bed_ready - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__recalibrate - type: - items: - - string - - 'null' - - boolean - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: genome_resources__rnaseq__gene_bed - type: - items: File - type: array -- id: genome_resources__variation__dbsnp - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__lcr - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__polyx - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: genome_resources__variation__encode_blacklist - secondaryFiles: - - .tbi - type: - items: File - type: array -- id: reference__fasta__base - secondaryFiles: - - ^.dict - - .fai - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: postprocess_alignment_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - 'null' - - string - - name: genome_resources__rnaseq__gene_bed - type: File - - name: genome_resources__variation__lcr - type: File - - name: config__algorithm__recalibrate - type: - - string - - 'null' - - boolean - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__archive - type: - - 'null' - - string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__variant_regions_orig - type: - - File - - 'null' - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: config__algorithm__coverage_orig - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - name: postprocess_alignment_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/postprocess_variants.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/postprocess_variants.cwl deleted file mode 100644 index 5151f1520c6..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/postprocess_variants.cwl +++ /dev/null @@ -1,174 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-single -- sentinel_outputs=vrn_file -- sentinel_inputs=batch_rec:record,vrn_file:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- postprocess_variants -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 16 - outdirMin: 10299 - ramMin: 49152 - tmpdirMin: 30 -- class: dx:InputResourceRequirement - indirMin: 24883 -- class: SoftwareRequirement - packages: - - package: snpeff - specs: - - https://anaconda.org/bioconda/snpeff - version: - - 4.3.1t -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__snpeff__GRCh37_75 - type: File - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: reference__rtg - type: File - - name: config__algorithm__validate - type: - - File - - 'null' - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: vrn_file_toolinput - secondaryFiles: - - .tbi - type: File -outputs: -- id: vrn_file - secondaryFiles: - - .tbi - type: File -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/prep_align_inputs.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/prep_align_inputs.cwl deleted file mode 100644 index 5c392bdb350..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/prep_align_inputs.cwl +++ /dev/null @@ -1,139 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=single-split -- sentinel_outputs=process_alignment_rec:files;config__algorithm__quality_format;align_split -- sentinel_inputs=alignment_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- prep_align_inputs -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 16 - outdirMin: 10418 - ramMin: 49152 - tmpdirMin: 89 -- class: dx:InputResourceRequirement - indirMin: 3127 -- class: SoftwareRequirement - packages: - - package: grabix - specs: - - https://anaconda.org/bioconda/grabix - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: biobambam - specs: - - https://anaconda.org/bioconda/biobambam - - package: atropos;env - specs: - - https://anaconda.org/bioconda/atropos;env - version: - - python3 - - package: optitype - specs: - - https://anaconda.org/bioconda/optitype - - package: razers3 - specs: - - https://anaconda.org/bioconda/razers3 - version: - - 3.5.0 - - package: coincbc - specs: - - https://anaconda.org/bioconda/coincbc -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - - name: rgnames__pu - type: string - name: alignment_rec - type: record -outputs: -- id: process_alignment_rec - type: - items: - fields: - - name: files - type: - - 'null' - - items: File - type: array - - name: config__algorithm__quality_format - type: - - string - - 'null' - - name: align_split - type: - - string - - 'null' - name: process_alignment_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/prep_samples.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/prep_samples.cwl deleted file mode 100644 index 8e6d9500406..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/prep_samples.cwl +++ /dev/null @@ -1,95 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-parallel -- sentinel_outputs=rgnames__sample,config__algorithm__variant_regions,config__algorithm__variant_regions_merged,config__algorithm__variant_regions_orig,config__algorithm__coverage,config__algorithm__coverage_merged,config__algorithm__coverage_orig,config__algorithm__seq2c_bed_ready -- sentinel_inputs=prep_samples_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- prep_samples -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10299 - ramMin: 3072 - tmpdirMin: 30 -- class: dx:InputResourceRequirement - indirMin: 3008 -- class: SoftwareRequirement - packages: - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy -inputs: -- id: prep_samples_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: prep_samples_rec - type: record -outputs: -- id: rgnames__sample - type: string -- id: config__algorithm__variant_regions - type: - - File - - 'null' -- id: config__algorithm__variant_regions_merged - type: - - File - - 'null' -- id: config__algorithm__variant_regions_orig - type: - - File - - 'null' -- id: config__algorithm__coverage - type: - - File - - 'null' -- id: config__algorithm__coverage_merged - type: - - File - - 'null' -- id: config__algorithm__coverage_orig - type: - - File - - 'null' -- id: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/prep_samples_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/prep_samples_to_rec.cwl deleted file mode 100644 index 9ccaa591fde..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/prep_samples_to_rec.cwl +++ /dev/null @@ -1,85 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=prep_samples_rec:resources;description;reference__fasta__base;config__algorithm__svcaller;rgnames__sample;config__algorithm__variant_regions -- sentinel_inputs=rgnames__sample:var,config__algorithm__svcaller:var,config__algorithm__variant_regions:var,reference__fasta__base:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- prep_samples_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10299 - ramMin: 3072 - tmpdirMin: 30 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: rgnames__sample - type: - items: string - type: array -- id: config__algorithm__svcaller - type: - items: - items: string - type: array - type: array -- id: config__algorithm__variant_regions - type: - items: File - type: array -- id: reference__fasta__base - secondaryFiles: - - ^.dict - - .fai - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: prep_samples_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: - items: string - type: array - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - name: prep_samples_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/process_alignment.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/process_alignment.cwl deleted file mode 100644 index 718e2cd14e0..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/process_alignment.cwl +++ /dev/null @@ -1,195 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=single-parallel -- sentinel_outputs=work_bam,align_bam,hla__fastq,work_bam_plus__disc,work_bam_plus__sr -- sentinel_inputs=alignment_rec:record,process_alignment_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- process_alignment -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 16 - outdirMin: 10477 - ramMin: 49152 - tmpdirMin: 119 -- class: dx:InputResourceRequirement - indirMin: 3246 -- class: SoftwareRequirement - packages: - - package: bwa - specs: - - https://anaconda.org/bioconda/bwa - - package: bwakit - specs: - - https://anaconda.org/bioconda/bwakit - - package: grabix - specs: - - https://anaconda.org/bioconda/grabix - - package: minimap2 - specs: - - https://anaconda.org/bioconda/minimap2 - - package: novoalign - specs: - - https://anaconda.org/bioconda/novoalign - - package: snap-aligner - specs: - - https://anaconda.org/bioconda/snap-aligner - version: - - 1.0dev.97 - - package: sentieon - specs: - - https://anaconda.org/bioconda/sentieon - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: pysam> - specs: - - https://anaconda.org/bioconda/pysam> - version: - - 0.13.0 - - package: sambamba - specs: - - https://anaconda.org/bioconda/sambamba - - package: fgbio - specs: - - https://anaconda.org/bioconda/fgbio - - package: umis - specs: - - https://anaconda.org/bioconda/umis - - package: biobambam - specs: - - https://anaconda.org/bioconda/biobambam - - package: seqtk - specs: - - https://anaconda.org/bioconda/seqtk - - package: samblaster - specs: - - https://anaconda.org/bioconda/samblaster - - package: variantbam - specs: - - https://anaconda.org/bioconda/variantbam -- class: arv:APIRequirement -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - - name: rgnames__pu - type: string - name: alignment_rec - type: record -- id: process_alignment_rec - type: - fields: - - name: files - type: - - 'null' - - items: File - type: array - - name: config__algorithm__quality_format - type: - - string - - 'null' - - name: align_split - type: - - string - - 'null' - name: process_alignment_rec - type: record -outputs: -- id: work_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: hla__fastq - type: - - 'null' - - items: File - type: array -- id: work_bam_plus__disc - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__sr - secondaryFiles: - - .bai - type: - - File - - 'null' -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/qc_to_rec.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/qc_to_rec.cwl deleted file mode 100644 index 0336669ad83..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/qc_to_rec.cwl +++ /dev/null @@ -1,307 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=qc_rec:resources;description;reference__fasta__base;config__algorithm__coverage_interval;metadata__batch;genome_build;metadata__phenotype;config__algorithm__tools_off;reference__versions;reference__viral;config__algorithm__qc;analysis;config__algorithm__tools_on;config__algorithm__variant_regions;align_bam;config__algorithm__variant_regions_merged;config__algorithm__coverage;config__algorithm__coverage_merged;depth__samtools__stats;depth__samtools__idxstats;depth__variant_regions__regions;depth__variant_regions__dist;depth__sv_regions__regions;depth__sv_regions__dist;depth__coverage__regions;depth__coverage__dist;depth__coverage__thresholds;variants__samples -- sentinel_inputs=align_bam:var,analysis:var,reference__fasta__base:var,reference__versions:var,config__algorithm__tools_on:var,config__algorithm__tools_off:var,genome_build:var,config__algorithm__qc:var,metadata__batch:var,metadata__phenotype:var,config__algorithm__coverage_interval:var,depth__variant_regions__regions:var,depth__variant_regions__dist:var,depth__samtools__stats:var,depth__samtools__idxstats:var,depth__sv_regions__regions:var,depth__sv_regions__dist:var,depth__coverage__regions:var,depth__coverage__dist:var,depth__coverage__thresholds:var,config__algorithm__variant_regions:var,config__algorithm__variant_regions_merged:var,config__algorithm__coverage:var,config__algorithm__coverage_merged:var,variants__samples:var,reference__viral:var,resources:var,description:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- qc_to_rec -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10418 - ramMin: 3072 - tmpdirMin: 89 -- class: dx:InputResourceRequirement - indirMin: 0 -inputs: -- id: align_bam - secondaryFiles: - - .bai - type: - items: - - File - - 'null' - type: array -- id: analysis - type: - items: string - type: array -- id: reference__fasta__base - secondaryFiles: - - ^.dict - - .fai - type: - items: File - type: array -- id: reference__versions - type: - items: File - type: array -- id: config__algorithm__tools_on - type: - items: - items: string - type: array - type: array -- id: config__algorithm__tools_off - type: - items: - items: string - type: array - type: array -- id: genome_build - type: - items: string - type: array -- id: config__algorithm__qc - type: - items: - items: string - type: array - type: array -- id: metadata__batch - type: - items: string - type: array -- id: metadata__phenotype - type: - items: string - type: array -- id: config__algorithm__coverage_interval - type: - items: - - string - - 'null' - type: array -- id: depth__variant_regions__regions - type: - items: - - File - - 'null' - type: array -- id: depth__variant_regions__dist - type: - items: - - File - - 'null' - type: array -- id: depth__samtools__stats - type: - items: - - File - - 'null' - type: array -- id: depth__samtools__idxstats - type: - items: - - File - - 'null' - type: array -- id: depth__sv_regions__regions - type: - items: - - File - - 'null' - type: array -- id: depth__sv_regions__dist - type: - items: - - File - - 'null' - type: array -- id: depth__coverage__regions - type: - items: - - File - - 'null' - type: array -- id: depth__coverage__dist - type: - items: - - File - - 'null' - type: array -- id: depth__coverage__thresholds - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__variant_regions_merged - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage - type: - items: - - File - - 'null' - type: array -- id: config__algorithm__coverage_merged - type: - items: - - File - - 'null' - type: array -- id: variants__samples - type: - items: - items: - items: - - File - - 'null' - type: array - type: array - type: array -- id: reference__viral - secondaryFiles: - - .fai - - .ann - - .sa - - .pac - - .amb - - ^.dict - - .bwt - type: - items: File - type: array -- id: resources - type: - items: string - type: array -- id: description - type: - items: string - type: array -outputs: -- id: qc_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: reference__versions - type: File - - name: reference__viral - type: File - - name: config__algorithm__qc - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__coverage - type: - - File - - 'null' - - name: config__algorithm__coverage_merged - type: - - File - - 'null' - - name: depth__samtools__stats - type: - - File - - 'null' - - name: depth__samtools__idxstats - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: depth__variant_regions__dist - type: - - File - - 'null' - - name: depth__sv_regions__regions - type: - - File - - 'null' - - name: depth__sv_regions__dist - type: - - File - - 'null' - - name: depth__coverage__regions - type: - - File - - 'null' - - name: depth__coverage__dist - type: - - File - - 'null' - - name: depth__coverage__thresholds - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - name: qc_rec - type: record - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/summarize_sv.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/summarize_sv.cwl deleted file mode 100644 index c5f8047d69a..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/summarize_sv.cwl +++ /dev/null @@ -1,236 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=sv__calls,sv__supplemental,sv__prioritize__tsv,sv__prioritize__raw,svvalidate__grading_summary,svvalidate__grading_plots -- sentinel_inputs=sv_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- summarize_sv -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 10358 - ramMin: 3072 - tmpdirMin: 59 -- class: dx:InputResourceRequirement - indirMin: 3024 -- class: SoftwareRequirement - packages: - - package: bcbio-prioritize - specs: - - https://anaconda.org/bioconda/bcbio-prioritize -inputs: -- id: sv_rec - type: - items: - items: - fields: - - name: sv__variantcaller - type: - - string - - 'null' - - name: sv__vrn_file - type: - - File - - 'null' - - name: sv__supplemental - type: - items: - - File - type: array - - name: svvalidate__summary - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: - - 'null' - - string - - name: genome_resources__variation__gc_profile - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__svvalidate - type: - - 'null' - - string - - name: genome_resources__aliases__snpeff - type: string - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_rec - type: record - type: array - type: array -outputs: -- id: sv__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: sv__supplemental - type: - items: - items: - - File - type: array - type: array -- id: sv__prioritize__tsv - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: sv__prioritize__raw - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: svvalidate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: svvalidate__grading_plots - type: - items: - items: - - File - - 'null' - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/summarize_vc.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/summarize_vc.cwl deleted file mode 100644 index 48ec1cd601a..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/summarize_vc.cwl +++ /dev/null @@ -1,212 +0,0 @@ -$namespaces: - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=multi-combined -- sentinel_outputs=variants__calls,variants__gvcf,variants__samples,validate__grading_summary,validate__grading_plots -- sentinel_inputs=vc_rec:record,ensemble_rec:record -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- summarize_vc -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 1 - outdirMin: 1000000 - ramMin: 3072 - tmpdirMin: 1000000 -- class: dx:InputResourceRequirement - indirMin: 1000000 -inputs: -- id: vc_rec - type: - items: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__snpeff__GRCh37_75 - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: reference__rtg - type: File - - name: config__algorithm__validate - type: - - File - - 'null' - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: config__algorithm__ensemble - type: string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array - type: array -- id: ensemble_rec - type: - items: - fields: - - name: ensemble__vrn_file - type: - - File - - 'null' - - name: ensemble__validate__summary - type: - - File - - 'null' - - name: ensemble__batch_samples - type: - items: string - type: array - - name: ensemble__batch_id - type: string - name: ensemble_rec - type: record - type: array -outputs: -- id: variants__calls - type: - items: - items: - - File - - 'null' - type: array - type: array -- id: variants__gvcf - type: - items: - - 'null' - - items: - - File - - 'null' - type: array - type: array -- id: variants__samples - type: - items: - items: - items: - - File - - 'null' - type: array - type: array - type: array -- id: validate__grading_summary - type: - items: - - File - - 'null' - type: array -- id: validate__grading_plots - type: - items: - items: - - File - - 'null' - type: array - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/variantcall_batch_region.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/variantcall_batch_region.cwl deleted file mode 100644 index a4fadb20991..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/steps/variantcall_batch_region.cwl +++ /dev/null @@ -1,260 +0,0 @@ -$namespaces: - arv: http://arvados.org/cwl# - dx: https://www.dnanexus.com/cwl# -arguments: -- position: 0 - valueFrom: sentinel_runtime=cores,$(runtime['cores']),ram,$(runtime['ram']) -- sentinel_parallel=batch-parallel -- sentinel_outputs=vrn_file_region,region_block -- sentinel_inputs=batch_rec:record,region_block:var -- run_number=0 -baseCommand: -- bcbio_nextgen.py -- runfn -- variantcall_batch_region -- cwl -class: CommandLineTool -cwlVersion: v1.0 -hints: -- class: DockerRequirement - dockerImageId: quay.io/bcbio/bcbio-vc:1.1.4a-741877e - dockerPull: quay.io/bcbio/bcbio-vc:1.1.4a-741877e -- class: ResourceRequirement - coresMin: 16 - outdirMin: 10477 - ramMin: 49152 - tmpdirMin: 119 -- class: dx:InputResourceRequirement - indirMin: 24883 -- class: SoftwareRequirement - packages: - - package: bcftools - specs: - - https://anaconda.org/bioconda/bcftools - - package: bedtools - specs: - - https://anaconda.org/bioconda/bedtools - - package: freebayes - specs: - - https://anaconda.org/bioconda/freebayes - version: - - 1.1.0.46 - - package: gatk4 - specs: - - https://anaconda.org/bioconda/gatk4 - - package: vqsr_cnn - specs: - - https://anaconda.org/bioconda/vqsr_cnn - - package: deepvariant - specs: - - https://anaconda.org/bioconda/deepvariant - - package: sentieon - specs: - - https://anaconda.org/bioconda/sentieon - - package: htslib - specs: - - https://anaconda.org/bioconda/htslib - - package: octopus - specs: - - https://anaconda.org/bioconda/octopus - - package: picard - specs: - - https://anaconda.org/bioconda/picard - - package: platypus-variant - specs: - - https://anaconda.org/bioconda/platypus-variant - - package: pythonpy - specs: - - https://anaconda.org/bioconda/pythonpy - - package: samtools - specs: - - https://anaconda.org/bioconda/samtools - - package: pysam> - specs: - - https://anaconda.org/bioconda/pysam> - version: - - 0.13.0 - - package: strelka - specs: - - https://anaconda.org/bioconda/strelka - - package: vardict - specs: - - https://anaconda.org/bioconda/vardict - - package: vardict-java - specs: - - https://anaconda.org/bioconda/vardict-java - - package: varscan - specs: - - https://anaconda.org/bioconda/varscan - - package: moreutils - specs: - - https://anaconda.org/bioconda/moreutils - - package: vcfanno - specs: - - https://anaconda.org/bioconda/vcfanno - - package: vcflib - specs: - - https://anaconda.org/bioconda/vcflib - - package: vt - specs: - - https://anaconda.org/bioconda/vt - - package: r - specs: - - https://anaconda.org/bioconda/r - version: - - 3.4.1 - - package: r-base=3.4.1=h4fe35fd_8 - specs: - - https://anaconda.org/bioconda/r-base=3.4.1=h4fe35fd_8 - - package: perl - specs: - - https://anaconda.org/bioconda/perl -- class: arv:APIRequirement -- class: arv:RuntimeConstraints - keep_cache: 4096 -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__snpeff__GRCh37_75 - type: File - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: reference__rtg - type: File - - name: config__algorithm__validate - type: - - File - - 'null' - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -- id: region_block_toolinput - type: - items: string - type: array -outputs: -- id: vrn_file_region - secondaryFiles: - - .tbi - type: - - File - - 'null' -- id: region_block - type: - items: string - type: array -requirements: -- class: InlineJavascriptRequirement -- class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/wf-alignment.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/wf-alignment.cwl deleted file mode 100644 index 0f1dfd3dd81..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/wf-alignment.cwl +++ /dev/null @@ -1,140 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: alignment_rec - type: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__align_split_size - type: - - 'null' - - string - - name: files - type: - items: File - type: array - - name: config__algorithm__trim_reads - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__adapters - type: - items: string - type: array - - name: rgnames__lb - type: - - 'null' - - string - - name: rgnames__rg - type: string - - name: rgnames__lane - type: string - - name: reference__bwa__indexes - type: File - - name: config__algorithm__bam_clean - type: - - string - - 'null' - - boolean - - name: config__algorithm__aligner - type: string - - name: rgnames__pl - type: string - - name: config__algorithm__mark_duplicates - type: - - string - - 'null' - - boolean - - name: analysis - type: string - - name: rgnames__sample - type: string - - name: config__algorithm__variant_regions - type: File - - name: rgnames__pu - type: string - name: alignment_rec - type: record -outputs: -- id: align_bam - outputSource: merge_split_alignments/align_bam - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__disc - outputSource: merge_split_alignments/work_bam_plus__disc - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: work_bam_plus__sr - outputSource: merge_split_alignments/work_bam_plus__sr - secondaryFiles: - - .bai - type: - - File - - 'null' -- id: hla__fastq - outputSource: merge_split_alignments/hla__fastq - type: - - 'null' - - items: File - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: prep_align_inputs - in: - - id: alignment_rec - source: alignment_rec - out: - - id: process_alignment_rec - run: steps/prep_align_inputs.cwl -- id: process_alignment - in: - - id: alignment_rec - source: alignment_rec - - id: process_alignment_rec - source: prep_align_inputs/process_alignment_rec - out: - - id: work_bam - - id: align_bam - - id: hla__fastq - - id: work_bam_plus__disc - - id: work_bam_plus__sr - run: steps/process_alignment.cwl - scatter: - - process_alignment_rec - scatterMethod: dotproduct -- id: merge_split_alignments - in: - - id: alignment_rec - source: alignment_rec - - id: work_bam - source: process_alignment/work_bam - - id: align_bam_toolinput - source: process_alignment/align_bam - - id: work_bam_plus__disc_toolinput - source: process_alignment/work_bam_plus__disc - - id: work_bam_plus__sr_toolinput - source: process_alignment/work_bam_plus__sr - - id: hla__fastq_toolinput - source: process_alignment/hla__fastq - out: - - id: align_bam - - id: work_bam_plus__disc - - id: work_bam_plus__sr - - id: hla__fastq - run: steps/merge_split_alignments.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/wf-svcall.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/wf-svcall.cwl deleted file mode 100644 index f8724f9b0b9..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/wf-svcall.cwl +++ /dev/null @@ -1,316 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: sv_batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: - - 'null' - - string - - name: genome_resources__variation__gc_profile - type: - - 'null' - - string - - name: reference__snpeff__GRCh37_75 - type: File - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__svvalidate - type: - - 'null' - - string - - name: genome_resources__aliases__snpeff - type: string - - name: work_bam_plus__disc - type: - - File - - 'null' - - name: work_bam_plus__sr - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_batch_rec - type: record - type: array -outputs: -- id: sv_rec - outputSource: detect_sv/sv_rec - type: - items: - fields: - - name: sv__variantcaller - type: - - string - - 'null' - - name: sv__vrn_file - type: - - File - - 'null' - - name: sv__supplemental - type: - items: - - File - type: array - - name: svvalidate__summary - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: config__algorithm__svprioritize - type: - - 'null' - - string - - name: genome_resources__variation__gc_profile - type: - - 'null' - - string - - name: genome_build - type: string - - name: genome_resources__variation__germline_het_pon - type: - - 'null' - - string - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__svvalidate - type: - - 'null' - - string - - name: genome_resources__aliases__snpeff - type: string - - name: regions__sample_callable - type: - - File - - 'null' - - name: variants__samples - type: - items: - items: - - File - - 'null' - type: array - type: array - - name: depth__bins__normalized - type: - - File - - 'null' - - name: depth__bins__background - type: - - File - - 'null' - - name: depth__bins__target - type: - - File - - 'null' - - name: depth__bins__antitarget - type: - - File - - 'null' - - name: depth__bins__seq2c - type: - - File - - 'null' - - name: regions__bins__target - type: - - File - - 'null' - - name: regions__bins__antitarget - type: - - File - - 'null' - - name: regions__bins__gcannotated - type: - - File - - 'null' - - name: regions__bins__group - type: - - string - - 'null' - - name: reference__fasta__base - type: File - - name: config__algorithm__svcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__rnaseq__gene_bed - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: metadata__phenotype - type: string - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: config__algorithm__seq2c_bed_ready - type: - - File - - 'null' - - name: depth__variant_regions__regions - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: sv_rec - type: record - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: detect_sv - in: - - id: sv_batch_rec - source: sv_batch_rec - out: - - id: sv_rec - run: steps/detect_sv.cwl diff --git a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/wf-variantcall.cwl b/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/wf-variantcall.cwl deleted file mode 100644 index fc5066d386e..00000000000 --- a/centaur/src/main/resources/integrationTestCases/cwl/bcbio/wes_chr21_test-workflow-gcp/wf-variantcall.cwl +++ /dev/null @@ -1,300 +0,0 @@ -class: Workflow -cwlVersion: v1.0 -hints: [] -inputs: -- id: batch_rec - type: - items: - fields: - - name: resources - type: string - - name: description - type: string - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: genome_resources__variation__clinvar - type: File - - name: genome_resources__variation__esp - type: File - - name: metadata__batch - type: string - - name: genome_resources__variation__lcr - type: File - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__snpeff__GRCh37_75 - type: File - - name: vrn_file - type: - - 'null' - - string - - name: genome_resources__variation__train_hapmap - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: reference__rtg - type: File - - name: config__algorithm__validate - type: - - File - - 'null' - - name: genome_resources__variation__1000g - type: File - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: genome_resources__variation__exac - type: File - - name: genome_resources__variation__gnomad_exome - type: - - 'null' - - string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: genome_resources__variation__dbsnp - type: File - - name: genome_resources__variation__polyx - type: File - - name: genome_resources__variation__encode_blacklist - type: File - - name: genome_resources__variation__cosmic - type: File - - name: config__algorithm__ensemble - type: string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__variation__train_indels - type: File - - name: genome_resources__aliases__snpeff - type: string - - name: align_bam - type: - - File - - 'null' - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: batch_rec - type: record - type: array -outputs: -- id: vc_rec - outputSource: compare_to_rm/vc_rec - type: - items: - fields: - - name: batch_samples - type: - - 'null' - - items: string - type: array - - name: validate__summary - type: - - File - - 'null' - - name: validate__tp - type: - - File - - 'null' - - name: validate__fp - type: - - File - - 'null' - - name: validate__fn - type: - - File - - 'null' - - name: resources - type: string - - name: description - type: string - - name: vrn_file - type: File - - name: reference__fasta__base - type: File - - name: config__algorithm__vcfanno - type: - - 'null' - - items: 'null' - type: array - - name: config__algorithm__variantcaller - type: string - - name: config__algorithm__coverage_interval - type: - - string - - 'null' - - name: metadata__batch - type: string - - name: config__algorithm__min_allele_fraction - type: double - - name: reference__snpeff__GRCh37_75 - type: File - - name: reference__genome_context - type: - items: File - type: array - - name: reference__rtg - type: File - - name: config__algorithm__validate - type: - - File - - 'null' - - name: config__algorithm__validate_regions - type: - - File - - 'null' - - name: genome_build - type: string - - name: metadata__phenotype - type: string - - name: genome_resources__aliases__human - type: - - string - - 'null' - - boolean - - name: config__algorithm__tools_off - type: - items: string - type: array - - name: config__algorithm__ensemble - type: string - - name: analysis - type: string - - name: config__algorithm__tools_on - type: - items: string - type: array - - name: config__algorithm__effects - type: string - - name: config__algorithm__variant_regions - type: - - File - - 'null' - - name: genome_resources__aliases__ensembl - type: string - - name: config__algorithm__exclude_regions - type: - - 'null' - - items: 'null' - type: array - - name: genome_resources__aliases__snpeff - type: string - - name: config__algorithm__variant_regions_merged - type: - - File - - 'null' - - name: regions__sample_callable - type: - - File - - 'null' - - name: config__algorithm__callable_regions - type: File - name: vc_rec - type: record - type: array -requirements: -- class: EnvVarRequirement - envDef: - - envName: MPLCONFIGDIR - envValue: . -- class: ScatterFeatureRequirement -- class: SubworkflowFeatureRequirement -steps: -- id: get_parallel_regions - in: - - id: batch_rec - source: batch_rec - out: - - id: region_block - run: steps/get_parallel_regions.cwl -- id: variantcall_batch_region - in: - - id: batch_rec - source: batch_rec - - id: region_block_toolinput - source: get_parallel_regions/region_block - out: - - id: vrn_file_region - - id: region_block - run: steps/variantcall_batch_region.cwl - scatter: - - region_block_toolinput - scatterMethod: dotproduct -- id: concat_batch_variantcalls - in: - - id: batch_rec - source: batch_rec - - id: region_block - source: variantcall_batch_region/region_block - - id: vrn_file_region - source: variantcall_batch_region/vrn_file_region - out: - - id: vrn_file - run: steps/concat_batch_variantcalls.cwl -- id: postprocess_variants - in: - - id: batch_rec - source: batch_rec - - id: vrn_file_toolinput - source: concat_batch_variantcalls/vrn_file - out: - - id: vrn_file - run: steps/postprocess_variants.cwl -- id: compare_to_rm - in: - - id: batch_rec - source: batch_rec - - id: vrn_file - source: postprocess_variants/vrn_file - out: - - id: vc_rec - run: steps/compare_to_rm.cwl diff --git a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.inline_file.test b/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.inline_file.test deleted file mode 100644 index 98ddd1a832e..00000000000 --- a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.inline_file.test +++ /dev/null @@ -1,16 +0,0 @@ -name: inline_file -testFormat: workflowsuccess -workflowType: CWL -workflowRoot: iwdr_inline_file -workflowTypeVersion: v1.0 -skipDescribeEndpointValidation: true - -files { - workflow: InitialWorkDirRequirement/inline_file.cwl -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "outputs.iwdr_inline_file.prime_list": "[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]" -} diff --git a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.inline_file_custom_entryname.test b/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.inline_file_custom_entryname.test deleted file mode 100644 index 1164b6fdf2f..00000000000 --- a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.inline_file_custom_entryname.test +++ /dev/null @@ -1,16 +0,0 @@ -name: inline_file_custom_entryname -testFormat: workflowsuccess -workflowType: CWL -workflowRoot: iwdr_inline_file -workflowTypeVersion: v1.0 -skipDescribeEndpointValidation: true - -files { - workflow: InitialWorkDirRequirement/inline_file_custom_entryname.cwl -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "outputs.iwdr_inline_file.prime_list": "[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]" -} diff --git a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.input_string.test b/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.input_string.test deleted file mode 100644 index 4ec70064ffc..00000000000 --- a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.input_string.test +++ /dev/null @@ -1,16 +0,0 @@ -name: iwdr_input_string -testFormat: workflowsuccess -workflowType: CWL -workflowRoot: iwdr_input_string -workflowTypeVersion: v1.0 -skipDescribeEndpointValidation: true - -files { - workflow: InitialWorkDirRequirement/input_string.cwl -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "outputs.iwdr_input_string.prime_list": "[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]" -} diff --git a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.input_string_function.test b/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.input_string_function.test deleted file mode 100644 index 1c7973b96d6..00000000000 --- a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement.input_string_function.test +++ /dev/null @@ -1,16 +0,0 @@ -name: iwdr_input_string_function -testFormat: workflowsuccess -workflowType: CWL -workflowRoot: iwdr_input_string_function -workflowTypeVersion: v1.0 -skipDescribeEndpointValidation: true - -files { - workflow: InitialWorkDirRequirement/input_string_function.cwl -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "outputs.iwdr_input_string_function.prime_list": "[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]" -} diff --git a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/file_array_listing.cwl b/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/file_array_listing.cwl deleted file mode 100644 index 86fb7c060a3..00000000000 --- a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/file_array_listing.cwl +++ /dev/null @@ -1,26 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: iwdr_array_listing - class: CommandLineTool - baseCommand: ['ls'] - requirements: - - class: DockerRequirement - dockerPull: "python:3.5.0" - - class: InitialWorkDirRequirement - listing: $(inputs.files) - - stdout: file_list - inputs: - - id: files - type: - type: array - items: File - default: [ "/Users/chrisl/Downloads/cwl/allRequirements.txt" ] - arguments: [ "*.txt" ] - outputs: - - id: file_list - type: string - outputBinding: - glob: file_list - loadContents: true - outputEval: $(self[0].contents.trim()) diff --git a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/inline_file.cwl b/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/inline_file.cwl deleted file mode 100644 index 1980beb52d8..00000000000 --- a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/inline_file.cwl +++ /dev/null @@ -1,43 +0,0 @@ -# Little tool that uses a prime sieve to generate primes up to 100 - -cwlVersion: v1.0 -$graph: -- id: iwdr_inline_file - class: CommandLineTool - baseCommand: ['python3', 'prime_sieve.py', '100'] - requirements: - - class: DockerRequirement - dockerPull: "python:3.5.0" - - class: InitialWorkDirRequirement - listing: - - entryname: 'prime_sieve.py' - entry: | - import sys - import math - - limit = int(sys.argv[1]) - sieve = [True for i in range(limit)] - for i in range(2, math.floor(limit / 2)): - if sieve[i]: - for j in range(i * 2, limit, i): - sieve[j] = False - - result = "[" - for i in range(2, limit): - if sieve[i]: - if result != "[": - result += ", " - result += str(i) - result += "]" - - print(result) - - stdout: "primes" - inputs: [] - outputs: - - id: prime_list - type: string - outputBinding: - glob: primes - loadContents: true - outputEval: $(self[0].contents.trim()) diff --git a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/inline_file_custom_entryname.cwl b/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/inline_file_custom_entryname.cwl deleted file mode 100644 index bff672a7e8a..00000000000 --- a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/inline_file_custom_entryname.cwl +++ /dev/null @@ -1,47 +0,0 @@ -# Little tool that uses a prime sieve to generate primes up to 100 - -cwlVersion: v1.0 -$graph: -- id: iwdr_inline_file - class: CommandLineTool - baseCommand: ['python3'] - requirements: - - class: DockerRequirement - dockerPull: "python:3.5.0" - - class: InitialWorkDirRequirement - listing: - - entryname: $(inputs.script_name) - entry: | - import sys - import math - - limit = int(sys.argv[1]) - sieve = [True for i in range(limit)] - for i in range(2, math.floor(limit / 2)): - if sieve[i]: - for j in range(i * 2, limit, i): - sieve[j] = False - - result = "[" - for i in range(2, limit): - if sieve[i]: - if result != "[": - result += ", " - result += str(i) - result += "]" - - print(result) - - stdout: "primes" - inputs: - - id: script_name - type: string - default: "astounding.py" - arguments: [ $(inputs.script_name), '100' ] - outputs: - - id: prime_list - type: string - outputBinding: - glob: primes - loadContents: true - outputEval: $(self[0].contents.trim()) diff --git a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/input_string.cwl b/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/input_string.cwl deleted file mode 100644 index 286f6b7aedc..00000000000 --- a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/input_string.cwl +++ /dev/null @@ -1,50 +0,0 @@ -# Little tool that uses a prime sieve to generate primes up to 100 - -cwlVersion: v1.0 -$graph: -- id: iwdr_input_string - class: CommandLineTool - baseCommand: ['python3', 'prime_sieve.py', '100'] - requirements: - - class: DockerRequirement - dockerPull: "python:3.5.0" - - class: InitialWorkDirRequirement - listing: - - entryname: $(inputs.script_name) - entry: $(inputs.script) - - stdout: "primes" - inputs: - - id: script - type: string - default: | - import sys - import math - - limit = int(sys.argv[1]) - sieve = [True for i in range(limit)] - for i in range(2, math.floor(limit / 2)): - if sieve[i]: - for j in range(i * 2, limit, i): - sieve[j] = False - - result = "[" - for i in range(2, limit): - if sieve[i]: - if result != "[": - result += ", " - result += str(i) - result += "]" - - print(result) - - id: script_name - type: string - default: "prime_sieve.py" - arguments: [ $(inputs.script_name), '100' ] - outputs: - - id: prime_list - type: string - outputBinding: - glob: primes - loadContents: true - outputEval: $(self[0].contents.trim()) diff --git a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/input_string_function.cwl b/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/input_string_function.cwl deleted file mode 100644 index f70cd7bf8b9..00000000000 --- a/centaur/src/main/resources/standardTestCases/InitialWorkDirRequirement/input_string_function.cwl +++ /dev/null @@ -1,49 +0,0 @@ -# Little tool that uses a prime sieve to generate primes up to 100 - -cwlVersion: v1.0 -$graph: -- id: iwdr_input_string_function - class: CommandLineTool - baseCommand: ['python3'] - requirements: - - class: DockerRequirement - dockerPull: "python:3.5.0" - - class: InitialWorkDirRequirement - listing: - - entryname: ${var result = inputs.script_name + ".py"; return result;} - entry: ${var scr = inputs.script; return scr + "print(result);"} - - stdout: "primes" - inputs: - - id: script - type: string - default: | - import sys - import math - - limit = int(sys.argv[1]) - sieve = [True for i in range(limit)] - for i in range(2, math.floor(limit / 2)): - if sieve[i]: - for j in range(i * 2, limit, i): - sieve[j] = False - - result = "[" - for i in range(2, limit): - if sieve[i]: - if result != "[": - result += ", " - result += str(i) - result += "]" - - - id: script_name - type: string - default: "prime_sieve.py" - arguments: [ '${var result = inputs.script_name + ".py"; return result;}', '100' ] - outputs: - - id: prime_list - type: string - outputBinding: - glob: primes - loadContents: true - outputEval: $(self[0].contents.trim()) diff --git a/centaur/src/main/resources/standardTestCases/ad_hoc_file_test.aws.test b/centaur/src/main/resources/standardTestCases/ad_hoc_file_test.aws.test deleted file mode 100644 index 8016f90f665..00000000000 --- a/centaur/src/main/resources/standardTestCases/ad_hoc_file_test.aws.test +++ /dev/null @@ -1,14 +0,0 @@ -name: ad_hoc_file_test.aws -testFormat: workflowsuccess -backends: [ AWSBATCH ] - -files { - workflow: ad_hoc_file_test/workflow.cwl - imports: [ - ad_hoc_file_test/cwl-test.cwl - ] -} - -metadata { - status: Succeeded -} diff --git a/centaur/src/main/resources/standardTestCases/ad_hoc_file_test/cwl-test.cwl b/centaur/src/main/resources/standardTestCases/ad_hoc_file_test/cwl-test.cwl deleted file mode 100644 index 7ac312f6f3a..00000000000 --- a/centaur/src/main/resources/standardTestCases/ad_hoc_file_test/cwl-test.cwl +++ /dev/null @@ -1,21 +0,0 @@ -class: CommandLineTool -cwlVersion: v1.0 -baseCommand: ["sh", "example.sh"] -hints: - DockerRequirement: - dockerPull: ubuntu:latest -inputs: [] - -requirements: - InitialWorkDirRequirement: - listing: - - entryname: example.sh - entry: |- - PREFIX='Message is:' - MSG="\${PREFIX} Hello world!" - echo \${MSG} - -outputs: - example_out: - type: stdout -stdout: output.txt diff --git a/centaur/src/main/resources/standardTestCases/ad_hoc_file_test/workflow.cwl b/centaur/src/main/resources/standardTestCases/ad_hoc_file_test/workflow.cwl deleted file mode 100644 index 678132165b2..00000000000 --- a/centaur/src/main/resources/standardTestCases/ad_hoc_file_test/workflow.cwl +++ /dev/null @@ -1,10 +0,0 @@ -cwlVersion: v1.0 -class: Workflow -inputs: [] -outputs: [] - -steps: - test: - run: cwl-test.cwl - in: [] - out: [] diff --git a/centaur/src/main/resources/standardTestCases/blob_md5/blob_md5.inputs.json b/centaur/src/main/resources/standardTestCases/blob_md5/blob_md5.inputs.json new file mode 100644 index 00000000000..f004c8be647 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/blob_md5/blob_md5.inputs.json @@ -0,0 +1,3 @@ +{ + "fileChecksum.inputFile": "https://.blob.core.windows.net/cromwell/user-inputs/inputFile.txt" +} diff --git a/centaur/src/main/resources/standardTestCases/blob_md5/blob_md5.wdl b/centaur/src/main/resources/standardTestCases/blob_md5/blob_md5.wdl new file mode 100644 index 00000000000..b482f80d6e4 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/blob_md5/blob_md5.wdl @@ -0,0 +1,20 @@ +task md5 { + File inputFile + command { + echo "`date`: Running checksum on ${inputFile}..." + md5sum ${inputFile} > md5sum.txt + echo "`date`: Checksum is complete." + } + output { + File result = "md5sum.txt" + } + runtime { + docker: 'ubuntu:18.04' + preemptible: true + } +} + +workflow fileChecksum { + File inputFile + call md5 { input: inputFile=inputFile} +} diff --git a/centaur/src/main/resources/standardTestCases/cwl_cache_between_workflows.test b/centaur/src/main/resources/standardTestCases/cwl_cache_between_workflows.test deleted file mode 100644 index e5381017829..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_cache_between_workflows.test +++ /dev/null @@ -1,19 +0,0 @@ -name: cwl_cache_between_workflows -testFormat: runtwiceexpectingcallcaching -workflowRoot: cwl-cache-between-workflows -workflowType: CWL -workflowTypeVersion: v1.0 -skipDescribeEndpointValidation: true - -files { - workflow: cwl_cache_between_workflows/cwl_cache_between_workflows.cwl - inputs: cwl_cache_between_workflows/cwl_cache_between_workflows.json -} - -metadata { - status: Succeeded - "calls.cwl-cache-between-workflows.step-average.callCaching.result": "Cache Hit: <>:cwl-cache-between-workflows.step-average:-1" - "calls.cwl-cache-between-workflows.step-product.callCaching.result": "Cache Hit: <>:cwl-cache-between-workflows.step-product:-1" - "outputs.cwl-cache-between-workflows.trapezoidalArea": 77.0, - "outputs.cwl-cache-between-workflows.baseAverage": 11.0 -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_cache_between_workflows/cwl_cache_between_workflows.cwl b/centaur/src/main/resources/standardTestCases/cwl_cache_between_workflows/cwl_cache_between_workflows.cwl deleted file mode 100644 index d48e74cd935..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_cache_between_workflows/cwl_cache_between_workflows.cwl +++ /dev/null @@ -1,79 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: average - class: CommandLineTool - - inputs: - base1: - type: int - default: 9 - base2: - type: int - default: 13 - - outputs: - baseAverage: - type: float - outputBinding: - glob: stdout.txt - loadContents: true - outputEval: $(parseFloat(self[0].contents)) - - arguments: - - valueFrom: "echo $((inputs.base1 + inputs.base2) / 2)" - shellQuote: false - - stdout: stdout.txt - -- id: height-product - class: CommandLineTool - - inputs: - baseAverage: float - height: - type: float - default: 7.0 - - outputs: - trapezoidalArea: - type: long - outputBinding: - glob: stdout.txt - loadContents: true - outputEval: $(parseFloat(self[0].contents)) - - arguments: - - valueFrom: "echo $(inputs.baseAverage * inputs.height)" - shellQuote: false - - stdout: stdout.txt - -- id: cwl-cache-between-workflows - class: Workflow - - requirements: - - class: DockerRequirement - dockerPull: "ubuntu:latest" - - class: ShellCommandRequirement - - inputs: [] - - outputs: - baseAverage: - type: float - outputSource: step-average/baseAverage - trapezoidalArea: - type: long - outputSource: step-product/trapezoidalArea - - steps: - step-average: - run: "#average" - in: [] - out: [baseAverage] - - step-product: - run: "#height-product" - in: - baseAverage: step-average/baseAverage - out: [trapezoidalArea] diff --git a/centaur/src/main/resources/standardTestCases/cwl_cache_between_workflows/cwl_cache_between_workflows.json b/centaur/src/main/resources/standardTestCases/cwl_cache_between_workflows/cwl_cache_between_workflows.json deleted file mode 100644 index 0967ef424bc..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_cache_between_workflows/cwl_cache_between_workflows.json +++ /dev/null @@ -1 +0,0 @@ -{} diff --git a/centaur/src/main/resources/standardTestCases/cwl_cache_within_workflow.test b/centaur/src/main/resources/standardTestCases/cwl_cache_within_workflow.test deleted file mode 100644 index 8ef753b494d..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_cache_within_workflow.test +++ /dev/null @@ -1,26 +0,0 @@ -name: cwl_cache_within_workflow -testFormat: workflowsuccess -workflowRoot: cwl-cache-within-workflow -workflowType: CWL -workflowTypeVersion: v1.0 -skipDescribeEndpointValidation: true - -files { - workflow: cwl_cache_within_workflow/cwl_cache_within_workflow.cwl - inputs: cwl_cache_within_workflow/cwl_cache_within_workflow.json -} - -metadata { - status: Succeeded - "calls.cwl-cache-within-workflow.foo.callCaching.result": "Cache Miss" - "calls.cwl-cache-within-workflow.bar.callCaching.result": "Cache Miss" - "calls.cwl-cache-within-workflow.re-bar.callCaching.result": "Cache Hit: <>:cwl-cache-within-workflow.bar:-1" - "calls.cwl-cache-within-workflow.bar.inputs.pi": 3.14159 - "calls.cwl-cache-within-workflow.bar.inputs.rSquared": 16.23381791021144 - "calls.cwl-cache-within-workflow.bar.outputs.area": 51 - "calls.cwl-cache-within-workflow.bar.outputs.rSquaredCopy": 16.23381791021144 - "calls.cwl-cache-within-workflow.re-bar.inputs.pi": 3.14159 - "calls.cwl-cache-within-workflow.re-bar.inputs.rSquared": 16.23381791021144 - "calls.cwl-cache-within-workflow.re-bar.outputs.area": 51 - "calls.cwl-cache-within-workflow.re-bar.outputs.rSquaredCopy": 16.23381791021144 -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_cache_within_workflow/cwl_cache_within_workflow.cwl b/centaur/src/main/resources/standardTestCases/cwl_cache_within_workflow/cwl_cache_within_workflow.cwl deleted file mode 100644 index 15b1fd79a01..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_cache_within_workflow/cwl_cache_within_workflow.cwl +++ /dev/null @@ -1,129 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: one - class: CommandLineTool - - inputs: - r: - type: float - - outputs: - rSquared: - type: float - outputBinding: - glob: stdout.txt - loadContents: true - outputEval: $(parseFloat(self[0].contents)) - rCopy: - type: float - outputBinding: - outputEval: $(inputs.r) - - arguments: - - valueFrom: "echo $(inputs.r * inputs.r)" - shellQuote: false - - stdout: stdout.txt - -- id: two - class: CommandLineTool - - inputs: - rSquared: float - pi: float - - outputs: - area: - outputBinding: - glob: stdout.txt - loadContents: true - outputEval: $(parseInt(self[0].contents)) - type: int - rSquaredCopy: - outputBinding: - outputEval: $(inputs.rSquared) - type: float - - arguments: - - valueFrom: "echo $(inputs.rSquared * inputs.pi)" - shellQuote: false - - stdout: stdout.txt - -# Throwing in some ExpressionTools for fun. Currently these not only don't cache but aren't even listed in the metadata -# except in the original workflow blob (see #3499). The current thinking is that these should appear in metadata with a -# call-like presentation but that they don't need to be eligible for caching since ExpressionTools are supposed to be -# very lightweight and not worth the expense of hashing and potential cache hit copying. -- id: three - class: ExpressionTool - - inputs: - rSquared: float - pi: float - - outputs: - area: int - rSquaredCopy: float - - expression: | - ${ - return {"area": parseInt(inputs.pi * inputs.rSquared), - "rSquaredCopy": inputs.rSquared }; - } - - -- id: cwl-cache-within-workflow - class: Workflow - - requirements: - - class: DockerRequirement - dockerPull: "ubuntu:latest" - - class: ShellCommandRequirement - - inputs: - radius: float - pi: float - - outputs: - area: - type: int - outputSource: re-bar/area - area-expression: - type: int - outputSource: re-baz/area - - steps: - foo: - run: "#one" - in: - r: radius - pi: pi - out: [rSquared, rCopy] - - bar: - run: "#two" - in: - rSquared: "foo/rSquared" - pi: pi - out: [area, rSquaredCopy] - - re-bar: - run: "#two" - in: - rSquared: "bar/rSquaredCopy" - pi: pi - out: [area, rSquaredCopy] - - baz: - run: "#three" - in: - rSquared: "foo/rSquared" - pi: pi - out: [area, rSquaredCopy] - - re-baz: - run: "#three" - in: - rSquared: "baz/rSquaredCopy" - pi: pi - out: [area, rSquaredCopy] diff --git a/centaur/src/main/resources/standardTestCases/cwl_cache_within_workflow/cwl_cache_within_workflow.json b/centaur/src/main/resources/standardTestCases/cwl_cache_within_workflow/cwl_cache_within_workflow.json deleted file mode 100644 index 563f2413547..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_cache_within_workflow/cwl_cache_within_workflow.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "radius": 4.029121233, - "pi": 3.14159 -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_disk_resources_papiv2.test b/centaur/src/main/resources/standardTestCases/cwl_disk_resources_papiv2.test deleted file mode 100644 index 8677bcf711e..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_disk_resources_papiv2.test +++ /dev/null @@ -1,17 +0,0 @@ -name: cwl_disk_resources_papiv2 -testFormat: workflowsuccess -backends: [Papiv2] -skipDescribeEndpointValidation: true - -files { - workflow: cwl_resources/cwl_disk_resources.cwl -} - -metadata { - status: Succeeded - "outputs.diskSizeTool.disk_size": 30 -} - -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: diskSizeTool diff --git a/centaur/src/main/resources/standardTestCases/cwl_docker_size.test b/centaur/src/main/resources/standardTestCases/cwl_docker_size.test deleted file mode 100644 index ca700d5cd9d..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_docker_size.test +++ /dev/null @@ -1,21 +0,0 @@ -name: cwl_docker_files -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: main -backendsMode: "any" -backends: [Local, LocalNoDocker] -tags: [localdockertest] -skipDescribeEndpointValidation: true - -files { - workflow: cwl_docker_size/cwl_docker_size.cwl - inputs: cwl_docker_size/cwl_docker_size.yaml - options: cwl_docker_size/cwl_docker_size.options -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "outputs.main.stdout_output": "execute order=66" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size.cwl b/centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size.cwl deleted file mode 100644 index 5383bccb484..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size.cwl +++ /dev/null @@ -1,40 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: main - class: CommandLineTool - hints: - DockerRequirement: - dockerPull: "debian:stretch-slim" - inputs: - - id: INPUT - type: File[] - stdout: stdout - outputs: - stdout_output: - type: string - outputBinding: - glob: stdout - loadContents: true - outputEval: $(self[0].contents.trim()) - arguments: - - valueFrom: | - ${ - if (inputs.INPUT.length == 0) { - var cmd = ['echo', "no inputs"]; - return cmd - } - else { - var cmd = ["echo", "execute"]; - var use_input = []; - for (var i = 0; i < inputs.INPUT.length; i++) { - var filesize = inputs.INPUT[i].size; - use_input.push("order=".concat(filesize)); - } - - var run_cmd = cmd.concat(use_input); - return run_cmd - } - - } - out: stdout - baseCommand: [] diff --git a/centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size.options b/centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size.options deleted file mode 100644 index 71d245d2bf6..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size.options +++ /dev/null @@ -1,3 +0,0 @@ -{ - "backend": "Local" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size.yaml b/centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size.yaml deleted file mode 100644 index 097e69e8590..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size.yaml +++ /dev/null @@ -1,3 +0,0 @@ -INPUT: - - class: File - path: "centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size_input.txt" diff --git a/centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size_input.txt b/centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size_input.txt deleted file mode 100644 index 1a6374205aa..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_docker_size/cwl_docker_size_input.txt +++ /dev/null @@ -1 +0,0 @@ -And you people, you're all astronauts, on some kind of star trek. diff --git a/centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir.test b/centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir.test deleted file mode 100644 index 57a69084c62..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir.test +++ /dev/null @@ -1,20 +0,0 @@ -name: cwl_dynamic_initial_workdir -testFormat: workflowsuccess -ignore: false -workflowRoot: main -workflowType: CWL -workflowTypeVersion: v1.0 -backendsMode: "only" -backends: [Local, LocalNoDocker] -tags: [localdockertest] -skipDescribeEndpointValidation: true - -files { - workflow: cwl_dynamic_initial_workdir/cwl_dynamic_initial_workdir.cwl - inputs: cwl_dynamic_initial_workdir/cwl_dynamic_initial_workdir.inputs -} - -metadata { - status: Succeeded - "outputs.main.sha": "13cda8661796ae241da3a18668fb552161a72592\n" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/cwl_dynamic_initial_workdir.cwl b/centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/cwl_dynamic_initial_workdir.cwl deleted file mode 100644 index 6012854d0b7..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/cwl_dynamic_initial_workdir.cwl +++ /dev/null @@ -1,29 +0,0 @@ -cwlVersion: v1.0 -$graph: - id: main - class: CommandLineTool - cwlVersion: v1.0 - requirements: - - class: ShellCommandRequirement - - class: InitialWorkDirRequirement - listing: $(inputs.indir.listing) - - class: DockerRequirement - dockerPull: "ubuntu:latest" - inputs: - indir: Directory - outputs: - sha: - type: string - outputBinding: - glob: output.txt - loadContents: true - outputEval: $(self[0].contents) - arguments: ["find", "-L", ".", "-name", "?", - {shellQuote: false, valueFrom: "|"}, - "sort", - {shellQuote: false, valueFrom: "|"}, - "sha1sum", - {shellQuote: false, valueFrom: "|"}, - "cut", "-c", "1-40" - ] - stdout: output.txt diff --git a/centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/cwl_dynamic_initial_workdir.inputs b/centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/cwl_dynamic_initial_workdir.inputs deleted file mode 100644 index f37fed14367..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/cwl_dynamic_initial_workdir.inputs +++ /dev/null @@ -1,3 +0,0 @@ -indir: - class: Directory - location: centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/testdir diff --git a/centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/testdir/a b/centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/testdir/a deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/testdir/b b/centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/testdir/b deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/testdir/c/d b/centaur/src/main/resources/standardTestCases/cwl_dynamic_initial_workdir/testdir/c/d deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format.cwl b/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format.cwl deleted file mode 100644 index c2079a82721..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format.cwl +++ /dev/null @@ -1,25 +0,0 @@ -cwlVersion: v1.0 -$namespaces: - edam: http://edamontology.org/ - gx: http://galaxyproject.org/formats/ -$schemas: - - file:./cwl/src/test/resources/cwl/ontology/EDAM.owl - - file:./cwl/src/test/resources/cwl/ontology/gx_edam.ttl -$graph: -- id: main - class: CommandLineTool - cwlVersion: v1.0 - inputs: - - id: reference - type: File - format: gx:fasta - inputBinding: { position: 2 } - outputs: - - id: lineCount - type: int - outputBinding: - glob: stdout - loadContents: true - outputEval: "$(parseInt(self[0].contents))" - baseCommand: ["wc", "-l"] - stdout: stdout diff --git a/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format.options b/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format.options deleted file mode 100644 index 71d245d2bf6..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format.options +++ /dev/null @@ -1,3 +0,0 @@ -{ - "backend": "Local" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_invalid.yaml b/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_invalid.yaml deleted file mode 100644 index 6e75ae34f28..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_invalid.yaml +++ /dev/null @@ -1,5 +0,0 @@ -# Try to pass a fastq in as a fasta -reference: - class: File - format: edam:format_1930 - path: "centaur/src/main/resources/standardTestCases/cwl_format/foo.fastq" diff --git a/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_missing.yaml b/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_missing.yaml deleted file mode 100644 index 4e8a35cb2e9..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_missing.yaml +++ /dev/null @@ -1,5 +0,0 @@ -# Pass a fastq in as a fasta, but without passing the format -reference: - class: File - #format: edam:format_1930 - path: "centaur/src/main/resources/standardTestCases/cwl_format/foo.fastq" diff --git a/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_url.cwl b/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_url.cwl deleted file mode 100644 index b85bf8d2434..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_url.cwl +++ /dev/null @@ -1,27 +0,0 @@ -cwlVersion: v1.0 -$namespaces: - edam: http://edamontology.org/ - gx: http://galaxyproject.org/formats/ -$schemas: - # Relative path: - - ../../../../../../cwl/src/test/resources/cwl/ontology/EDAM.owl - # Absolute (remote) path: - - https://raw.githubusercontent.com/broadinstitute/cromwell/develop/cwl/src/test/resources/cwl/ontology/gx_edam.ttl -$graph: -- id: main - class: CommandLineTool - cwlVersion: v1.0 - inputs: - - id: reference - type: File - format: gx:fasta - inputBinding: { position: 2 } - outputs: - - id: lineCount - type: int - outputBinding: - glob: stdout - loadContents: true - outputEval: "$(parseInt(self[0].contents))" - baseCommand: ["wc", "-l"] - stdout: stdout diff --git a/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_valid.yaml b/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_valid.yaml deleted file mode 100644 index b598f92b7e1..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_valid.yaml +++ /dev/null @@ -1,5 +0,0 @@ -# Pass in a valid input file format -reference: - class: File - format: edam:format_1929 - path: "centaur/src/main/resources/standardTestCases/cwl_format/foo.fasta" diff --git a/centaur/src/main/resources/standardTestCases/cwl_format/foo.fasta b/centaur/src/main/resources/standardTestCases/cwl_format/foo.fasta deleted file mode 100644 index e91d1521810..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_format/foo.fasta +++ /dev/null @@ -1,2 +0,0 @@ ->sequence #1 -GATTTGGGGTTCAAAGCAGTATCGATCAAATAGTAAATCCATTTGTTCAACTCACAGTTT diff --git a/centaur/src/main/resources/standardTestCases/cwl_format/foo.fastq b/centaur/src/main/resources/standardTestCases/cwl_format/foo.fastq deleted file mode 100644 index 5ecec0ebec8..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_format/foo.fastq +++ /dev/null @@ -1,4 +0,0 @@ -@SEQ_ID -GATTTGGGGTTCAAAGCAGTATCGATCAAATAGTAAATCCATTTGTTCAACTCACAGTTT -+ -!''*((((***+))%%%++)(%%%%).1***-+*''))**55CCF>>>>>>CCCCCCC65 diff --git a/centaur/src/main/resources/standardTestCases/cwl_format_invalid.test b/centaur/src/main/resources/standardTestCases/cwl_format_invalid.test deleted file mode 100644 index 43962bb5bfc..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_format_invalid.test +++ /dev/null @@ -1,22 +0,0 @@ -name: cwl_format_invalid -testFormat: workflowfailure -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: main -backendsMode: "any" -backends: [Local, LocalNoDocker] -tags: [localdockertest] -skipDescribeEndpointValidation: true - -files { - workflow: cwl_format/cwl_format.cwl - inputs: cwl_format/cwl_format_invalid.yaml - options: cwl_format/cwl_format.options -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "failures.0.message": "Workflow input processing failed" - "failures.0.causedBy.0.message": "Failed to evaluate input 'reference' (reason 1 of 1): edam:format_1930 is not compatible with http://galaxyproject.org/formats/fasta" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_format_missing.test b/centaur/src/main/resources/standardTestCases/cwl_format_missing.test deleted file mode 100644 index a7b626f69e4..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_format_missing.test +++ /dev/null @@ -1,22 +0,0 @@ -name: cwl_format_missing -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: main -backendsMode: "any" -backends: [Local, LocalNoDocker] -tags: [localdockertest] -skipDescribeEndpointValidation: true - -files { - workflow: cwl_format/cwl_format.cwl - inputs: cwl_format/cwl_format_missing.yaml - options: cwl_format/cwl_format.options -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "workflowName": "main" - "outputs.main.lineCount": "4" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_format_valid.test b/centaur/src/main/resources/standardTestCases/cwl_format_valid.test deleted file mode 100644 index 6a6452deff5..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_format_valid.test +++ /dev/null @@ -1,22 +0,0 @@ -name: cwl_format_valid -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: main -backendsMode: "any" -backends: [Local, LocalNoDocker] -tags: [localdockertest] -skipDescribeEndpointValidation: true - -files { - workflow: cwl_format/cwl_format.cwl - inputs: cwl_format/cwl_format_valid.yaml - options: cwl_format/cwl_format.options -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "workflowName": "main" - "outputs.main.lineCount": "2" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_format_valid_with_workflow_url.test b/centaur/src/main/resources/standardTestCases/cwl_format_valid_with_workflow_url.test deleted file mode 100644 index e587dbfe6d6..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_format_valid_with_workflow_url.test +++ /dev/null @@ -1,22 +0,0 @@ -name: cwl_format_valid_with_workflow_url -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: main -backendsMode: "any" -backends: [Local, LocalNoDocker] -tags: [localdockertest] -skipDescribeEndpointValidation: true - -files { - workflowUrl: "https://raw.githubusercontent.com/broadinstitute/cromwell/develop/centaur/src/main/resources/standardTestCases/cwl_format/cwl_format_url.cwl" - inputs: cwl_format/cwl_format_valid.yaml - options: cwl_format/cwl_format.options -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "workflowName": "main" - "outputs.main.lineCount": "2" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_glob_sort.test b/centaur/src/main/resources/standardTestCases/cwl_glob_sort.test deleted file mode 100644 index 04856ab66b9..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_glob_sort.test +++ /dev/null @@ -1,16 +0,0 @@ -name: cwl_glob_sort -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: globSort -skipDescribeEndpointValidation: true - -files { - workflow: cwl_glob_sort/cwl_glob_sort.cwl -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "outputs.globSort.letters": "a b c w x y z" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_glob_sort/cwl_glob_sort.cwl b/centaur/src/main/resources/standardTestCases/cwl_glob_sort/cwl_glob_sort.cwl deleted file mode 100644 index 8d54af1c040..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_glob_sort/cwl_glob_sort.cwl +++ /dev/null @@ -1,19 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: globSort - cwlVersion: v1.0 - class: CommandLineTool - requirements: - - class: InlineJavascriptRequirement - hints: - DockerRequirement: - dockerPull: "debian:stretch-slim" - inputs: [] - baseCommand: [touch, z, y, x, w, c, b, a] - outputs: - letters: - type: string - outputBinding: - glob: '?' - outputEval: | - ${ return self.sort(function(a,b) { return a.location > b.location ? 1 : (a.location < b.location ? -1 : 0) }).map(f => f.basename).join(" ") } diff --git a/centaur/src/main/resources/standardTestCases/cwl_glob_sort_with_workflow_url.test b/centaur/src/main/resources/standardTestCases/cwl_glob_sort_with_workflow_url.test deleted file mode 100644 index 6e7ed2b1dd2..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_glob_sort_with_workflow_url.test +++ /dev/null @@ -1,16 +0,0 @@ -name: cwl_glob_sort_with_workflow_url -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: globSort -skipDescribeEndpointValidation: true - -files { - workflowUrl: "https://raw.githubusercontent.com/broadinstitute/cromwell/develop/centaur/src/main/resources/standardTestCases/cwl_glob_sort/cwl_glob_sort.cwl" -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "outputs.globSort.letters": "a b c w x y z" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_import_type.test b/centaur/src/main/resources/standardTestCases/cwl_import_type.test deleted file mode 100644 index 9e530a95469..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_import_type.test +++ /dev/null @@ -1,14 +0,0 @@ -name: cwl_import_type -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -skipDescribeEndpointValidation: true - -files { - workflow: cwl_import_type/test_wf.cwl - inputs: cwl_import_type/test_wf_inputs.json - imports: [ - cwl_import_type/capture_kit.yml, - cwl_import_type/touch.cwl - ] -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_import_type/capture_kit.yml b/centaur/src/main/resources/standardTestCases/cwl_import_type/capture_kit.yml deleted file mode 100644 index 4323dd90f6d..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_import_type/capture_kit.yml +++ /dev/null @@ -1,5 +0,0 @@ -- name: capture_kit - type: record - fields: - - name: bait - type: string diff --git a/centaur/src/main/resources/standardTestCases/cwl_import_type/test_wf.cwl b/centaur/src/main/resources/standardTestCases/cwl_import_type/test_wf.cwl deleted file mode 100644 index 3658ab127d0..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_import_type/test_wf.cwl +++ /dev/null @@ -1,30 +0,0 @@ -#!/usr/bin/env cwl-runner - -cwlVersion: v1.0 - -class: Workflow - -requirements: - - class: SchemaDefRequirement - types: - - $import: capture_kit.yml - -inputs: - - id: bam - type: string - - id: capture_kit - type: capture_kit.yml#capture_kit - -outputs: - - id: output_bam - type: File - outputSource: touch_bam/output - -steps: - - id: touch_bam - run: touch.cwl - in: - - id: input - source: bam - out: - - id: output diff --git a/centaur/src/main/resources/standardTestCases/cwl_import_type/test_wf_inputs.json b/centaur/src/main/resources/standardTestCases/cwl_import_type/test_wf_inputs.json deleted file mode 100644 index 55141261811..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_import_type/test_wf_inputs.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "bam": "a.bam", - "capture_kit": { - "bait": "abait" - } -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_import_type/touch.cwl b/centaur/src/main/resources/standardTestCases/cwl_import_type/touch.cwl deleted file mode 100644 index 8a0be057c42..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_import_type/touch.cwl +++ /dev/null @@ -1,23 +0,0 @@ -#!/usr/bin/env cwl-runner - -cwlVersion: v1.0 - -requirements: - - class: DockerRequirement - dockerPull: ubuntu:bionic-20180426 - -class: CommandLineTool - -inputs: - - id: input - type: string - inputBinding: - position: 0 - -outputs: - - id: output - type: File - outputBinding: - glob: $(inputs.input) - -baseCommand: [touch] diff --git a/centaur/src/main/resources/standardTestCases/cwl_import_type_packed.test b/centaur/src/main/resources/standardTestCases/cwl_import_type_packed.test deleted file mode 100644 index 0a3fd7c20a4..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_import_type_packed.test +++ /dev/null @@ -1,11 +0,0 @@ -name: cwl_import_type_packed -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: main -skipDescribeEndpointValidation: true - -files { - workflow: cwl_import_type_packed/test_pack.cwl - inputs: cwl_import_type_packed/test_wf_inputs.json -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_import_type_packed/test_pack.cwl b/centaur/src/main/resources/standardTestCases/cwl_import_type_packed/test_pack.cwl deleted file mode 100644 index 4150b5a5796..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_import_type_packed/test_pack.cwl +++ /dev/null @@ -1,91 +0,0 @@ -{ - "cwlVersion": "v1.0", - "$graph": [ - { - "inputs": [ - { - "type": "string", - "id": "#main/bam" - }, - { - "type": "#capture_kit.yml/capture_kit", - "id": "#main/capture_kit" - } - ], - "requirements": [ - { - "class": "SchemaDefRequirement", - "types": [ - { - "fields": [ - { - "type": "string", - "name": "#capture_kit.yml/capture_kit/bait" - } - ], - "type": "record", - "name": "#capture_kit.yml/capture_kit" - } - ] - } - ], - "outputs": [ - { - "outputSource": "#main/touch_bam/output", - "type": "File", - "id": "#main/output_bam" - } - ], - "class": "Workflow", - "steps": [ - { - "out": [ - { - "id": "#main/touch_bam/output" - } - ], - "run": "#touch.cwl", - "id": "#main/touch_bam", - "in": [ - { - "source": "#main/bam", - "id": "#main/touch_bam/input" - } - ] - } - ], - "id": "#main" - }, - { - "inputs": [ - { - "inputBinding": { - "position": 0 - }, - "type": "string", - "id": "#touch.cwl/input" - } - ], - "requirements": [ - { - "dockerPull": "ubuntu:bionic-20180426", - "class": "DockerRequirement" - } - ], - "outputs": [ - { - "outputBinding": { - "glob": "$(inputs.input)" - }, - "type": "File", - "id": "#touch.cwl/output" - } - ], - "baseCommand": [ - "touch" - ], - "class": "CommandLineTool", - "id": "#touch.cwl" - } - ] -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_import_type_packed/test_wf_inputs.json b/centaur/src/main/resources/standardTestCases/cwl_import_type_packed/test_wf_inputs.json deleted file mode 100644 index 55141261811..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_import_type_packed/test_wf_inputs.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "bam": "a.bam", - "capture_kit": { - "bait": "abait" - } -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_input_binding_expression.test b/centaur/src/main/resources/standardTestCases/cwl_input_binding_expression.test deleted file mode 100644 index 0e39f78ce78..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_input_binding_expression.test +++ /dev/null @@ -1,17 +0,0 @@ -name: cwl_input_binding_expression -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: cwl_input_binding_expression -skipDescribeEndpointValidation: true - -files { - workflow: cwl_input_binding_expression/cwl_input_binding_expression.cwl - inputs: cwl_input_binding_expression/cwl_input_binding_expression.json -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "outputs.cwl_input_binding_expression.b": "hello world" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_input_binding_expression/cwl_input_binding_expression.cwl b/centaur/src/main/resources/standardTestCases/cwl_input_binding_expression/cwl_input_binding_expression.cwl deleted file mode 100644 index f534adfaa36..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_input_binding_expression/cwl_input_binding_expression.cwl +++ /dev/null @@ -1,25 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: cwl_input_binding_expression - class: CommandLineTool - requirements: - - class: InlineJavascriptRequirement - hints: - DockerRequirement: - dockerPull: "debian:stretch-slim" - inputs: - - id: hello - type: string - inputBinding: - valueFrom: $(self + " world") - position: 1 - outputs: - b: - type: string - outputBinding: - loadContents: true - glob: stdout - outputEval: $(self[0].contents.trim()) - stdout: stdout - baseCommand: [] - arguments: [echo] diff --git a/centaur/src/main/resources/standardTestCases/cwl_input_binding_expression/cwl_input_binding_expression.json b/centaur/src/main/resources/standardTestCases/cwl_input_binding_expression/cwl_input_binding_expression.json deleted file mode 100644 index 8920ee8d630..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_input_binding_expression/cwl_input_binding_expression.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "hello": "hello" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_input_json.test b/centaur/src/main/resources/standardTestCases/cwl_input_json.test deleted file mode 100644 index 1dd81ba8260..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_input_json.test +++ /dev/null @@ -1,18 +0,0 @@ -name: cwl_input_json -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -backendsMode: "only" -workflowRoot: cwl_input_json -backends: [Local, LocalNoDocker] -tags: [localdockertest] -skipDescribeEndpointValidation: true - -files { - workflow: cwl_input_json/cwl_input_json.yaml -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_input_json/cwl_input_json.yaml b/centaur/src/main/resources/standardTestCases/cwl_input_json/cwl_input_json.yaml deleted file mode 100644 index f6e5616c6c4..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_input_json/cwl_input_json.yaml +++ /dev/null @@ -1,70 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: cwl_input_json - class: Workflow - inputs: [] - outputs: - final_output: - type: File - outputSource: round/output_file - steps: - make: - run: "#makefile" - in: [] - out: [fileoutput] - round: - run: "#roundtrip" - in: - input_record: - source: "make/fileoutput" - out: [output_file] - -- id: makefile - class: CommandLineTool - requirements: - - class: ShellCommandRequirement - - class: InlineJavascriptRequirement - - class: DockerRequirement - dockerPull: "ubuntu:latest" - inputs: [] - outputs: - fileoutput: - type: - fields: - - name: input_file - type: File - name: input_record - type: record - arguments: - - valueFrom: > - echo foo > foo && echo '{ "fileoutput": { "input_file": {"path": "$(runtime.outdir)/foo", "class": "File"} } }' > cwl.output.json - shellQuote: false - -- id: roundtrip - class: CommandLineTool - hints: - - class: DockerRequirement - dockerPull: "stedolan/jq:latest" - inputs: - - id: input_record - type: - fields: - - name: input_file - type: File - name: input_record - type: record - outputs: - - id: output_file - type: File - requirements: - - class: ShellCommandRequirement - - class: InlineJavascriptRequirement - - class: InitialWorkDirRequirement - listing: - - entry: $(JSON.stringify(inputs)) - entryname: cwl.inputs.json - arguments: - # Round-trips the file referenced in cwl.input.json to cwl.output.json. Also ls it in the command to make sure it's there. - - valueFrom: > - INPUT_FILE=\$(cat cwl.inputs.json | jq -r '.. | .path? // empty') && ls $INPUT_FILE && echo "{\"output_file\": {\"path\": \"\$INPUT_FILE\", \"class\": \"File\"} }" > cwl.output.json - shellQuote: false diff --git a/centaur/src/main/resources/standardTestCases/cwl_input_typearray.test b/centaur/src/main/resources/standardTestCases/cwl_input_typearray.test deleted file mode 100644 index a87bcbf9f65..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_input_typearray.test +++ /dev/null @@ -1,18 +0,0 @@ -name: cwl_input_typearray -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: input_typearray -skipDescribeEndpointValidation: true - -files { - workflow: cwl_input_typearray/input_typearray.cwl - inputs: cwl_input_typearray/input_typearray.yml -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "outputs.input_typearray.response_f": "input.txt" - "outputs.input_typearray.response_s": "nonexistent_path.txt" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_input_typearray/input.txt b/centaur/src/main/resources/standardTestCases/cwl_input_typearray/input.txt deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/centaur/src/main/resources/standardTestCases/cwl_input_typearray/input_typearray.cwl b/centaur/src/main/resources/standardTestCases/cwl_input_typearray/input_typearray.cwl deleted file mode 100644 index 897b74822e8..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_input_typearray/input_typearray.cwl +++ /dev/null @@ -1,42 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: input_typearray - cwlVersion: v1.0 - class: CommandLineTool - baseCommand: ['/bin/echo'] - stdout: "response.txt" - requirements: - - class: DockerRequirement - dockerPull: "ubuntu:latest" - - class: InlineJavascriptRequirement - arguments: - - position: 3 - valueFrom: "sentinel" - inputs: - value_f: - type: - - string - - File - inputBinding: - position: 1 - doc: "an input to test with a File value" - value_s: - type: - - string - - File - inputBinding: - position: 2 - doc: "an input to test with a string value" - outputs: - response_f: - type: string - outputBinding: - glob: response.txt - loadContents: true - outputEval: $(self[0].contents.split(" ")[0].split("/").slice(-1)[0]) - response_s: - type: string - outputBinding: - glob: response.txt - loadContents: true - outputEval: $(self[0].contents.split(" ")[1].split("/").slice(-1)[0]) diff --git a/centaur/src/main/resources/standardTestCases/cwl_input_typearray/input_typearray.yml b/centaur/src/main/resources/standardTestCases/cwl_input_typearray/input_typearray.yml deleted file mode 100644 index 421402d9289..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_input_typearray/input_typearray.yml +++ /dev/null @@ -1,4 +0,0 @@ -value_f: - class: File - path: "centaur/src/main/resources/standardTestCases/cwl_input_typearray/input.txt" -value_s: "centaur/src/main/resources/standardTestCases/cwl_input_typearray/nonexistent_path.txt" diff --git a/centaur/src/main/resources/standardTestCases/cwl_inputdir_zero_doesnt_localize_papiv2.test b/centaur/src/main/resources/standardTestCases/cwl_inputdir_zero_doesnt_localize_papiv2.test deleted file mode 100644 index 564633779a7..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_inputdir_zero_doesnt_localize_papiv2.test +++ /dev/null @@ -1,17 +0,0 @@ -name: cwl_inputdir_zero_doesnt_localize_papiv2 -testFormat: workflowsuccess -backends: [Papiv2] -skipDescribeEndpointValidation: true - -files { - workflow: cwl_resources/cwl_inputdir_zero_doesnt_localize.cwl -} - -metadata { - status: Succeeded - "outputs.inputdir_zero_doesnt_localize.errors": "" -} - -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: inputdir_zero_doesnt_localize diff --git a/centaur/src/main/resources/standardTestCases/cwl_interpolated_strings.test b/centaur/src/main/resources/standardTestCases/cwl_interpolated_strings.test deleted file mode 100644 index 89399e2e8fc..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_interpolated_strings.test +++ /dev/null @@ -1,17 +0,0 @@ -name: cwl_interpolated_strings -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: interpolatedStrings -skipDescribeEndpointValidation: true - -files { - workflow: cwl_interpolated_strings/cwl_interpolated_strings.cwl - inputs: cwl_interpolated_strings/cwl_interpolated_strings.json -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "outputs.interpolatedStrings.rfc3092": "foo bar baz qux quux" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_interpolated_strings/cwl_interpolated_strings.cwl b/centaur/src/main/resources/standardTestCases/cwl_interpolated_strings/cwl_interpolated_strings.cwl deleted file mode 100644 index e650061a5cc..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_interpolated_strings/cwl_interpolated_strings.cwl +++ /dev/null @@ -1,25 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: interpolatedStrings - class: CommandLineTool - cwlVersion: v1.0 - requirements: - - class: ShellCommandRequirement - hints: - DockerRequirement: - dockerPull: "debian:stretch-slim" - - inputs: - - id: bar - type: string - - id: qux - type: string - - outputs: - - id: rfc3092 - type: string - - arguments: - - valueFrom: > - echo '{ "rfc3092": "foo $(inputs.bar) baz $(inputs.qux) quux" }' > cwl.output.json - shellQuote: false diff --git a/centaur/src/main/resources/standardTestCases/cwl_interpolated_strings/cwl_interpolated_strings.json b/centaur/src/main/resources/standardTestCases/cwl_interpolated_strings/cwl_interpolated_strings.json deleted file mode 100644 index 3f1bcedd11f..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_interpolated_strings/cwl_interpolated_strings.json +++ /dev/null @@ -1 +0,0 @@ -{"bar": "bar", "qux": "qux"} diff --git a/centaur/src/main/resources/standardTestCases/cwl_optionals.test b/centaur/src/main/resources/standardTestCases/cwl_optionals.test deleted file mode 100644 index 2d945f36ee0..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_optionals.test +++ /dev/null @@ -1,13 +0,0 @@ -name: cwl_optionals -testFormat: workflowsuccess -skipDescribeEndpointValidation: true - -files { - workflow: cwl_optionals/cwl_optionals.cwl - inputs: cwl_optionals/cwl_optionals.json -} - -metadata { - "actualWorkflowLanguage": CWL - "actualWorkflowLanguageVersion": v1.0 -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_optionals/cwl_optionals.cwl b/centaur/src/main/resources/standardTestCases/cwl_optionals/cwl_optionals.cwl deleted file mode 100755 index 592227bf025..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_optionals/cwl_optionals.cwl +++ /dev/null @@ -1,12 +0,0 @@ -cwlVersion: v1.0 -class: CommandLineTool -hints: - DockerRequirement: - dockerPull: "ubuntu:latest" -baseCommand: echo -inputs: - message: - type: string[]? - unsupplied_optional: - type: string[]? -outputs: [] diff --git a/centaur/src/main/resources/standardTestCases/cwl_optionals/cwl_optionals.json b/centaur/src/main/resources/standardTestCases/cwl_optionals/cwl_optionals.json deleted file mode 100644 index c76e98d49c3..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_optionals/cwl_optionals.json +++ /dev/null @@ -1 +0,0 @@ -{"message": ["foo","bar"]} diff --git a/centaur/src/main/resources/standardTestCases/cwl_output_json/cwl_output_json.cwl b/centaur/src/main/resources/standardTestCases/cwl_output_json/cwl_output_json.cwl deleted file mode 100644 index 406301e7ecb..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_output_json/cwl_output_json.cwl +++ /dev/null @@ -1,34 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: outputJson - # slightly modified version of conformance test #2 to remove the default File that doesn't work on PAPI - # It is instead being passed as an input through the input json file - class: CommandLineTool - cwlVersion: v1.0 - hints: - - class: DockerRequirement - dockerPull: python:2-slim - inputs: - - id: reference - type: File - inputBinding: { position: 2 } - - - id: reads - type: - type: array - items: File - inputBinding: { prefix: "-YYY" } - inputBinding: { position: 3, prefix: "-XXX" } - - - id: "args.py" - type: File - inputBinding: - position: -1 - - outputs: - # note the absence of any sort of valueFrom. - # The output value is generated from the "cwl.output.json" file created by the python script - args: string[] - - baseCommand: python - arguments: ["bwa", "mem"] diff --git a/centaur/src/main/resources/standardTestCases/cwl_output_json/cwl_output_json.inputs b/centaur/src/main/resources/standardTestCases/cwl_output_json/cwl_output_json.inputs deleted file mode 100644 index d83a30872a9..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_output_json/cwl_output_json.inputs +++ /dev/null @@ -1,22 +0,0 @@ -{ - "args.py": { - "class": "File", - "location": "gs://centaur-cwl-conformance-1f501e3/cwl-inputs/args.py" - }, - "reference": { - "class": "File", - "location": "gs://centaur-cwl-conformance-1f501e3/cwl-inputs/chr20.fa", - "size": 123, - "checksum": "sha1$hash" - }, - "reads": [ - { - "class": "File", - "location": "gs://centaur-cwl-conformance-1f501e3/cwl-inputs/example_human_Illumina.pe_1.fastq" - }, - { - "class": "File", - "location": "gs://centaur-cwl-conformance-1f501e3/cwl-inputs/example_human_Illumina.pe_2.fastq" - } - ] -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_output_json_jes.test b/centaur/src/main/resources/standardTestCases/cwl_output_json_jes.test deleted file mode 100644 index c7ddf584bd4..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_output_json_jes.test +++ /dev/null @@ -1,25 +0,0 @@ -name: cwl_output_json_jes -testFormat: workflowsuccess -backends: [Papi] -skipDescribeEndpointValidation: true - -files { - workflow: cwl_output_json/cwl_output_json.cwl - inputs: cwl_output_json/cwl_output_json.inputs -} - -metadata { - status: Succeeded - "outputs.outputJson.args.0": "bwa" - "outputs.outputJson.args.1": "mem" - "outputs.outputJson.args.2": "chr20.fa" - "outputs.outputJson.args.3": "-XXX" - "outputs.outputJson.args.4": "-YYY" - "outputs.outputJson.args.5": "example_human_Illumina.pe_1.fastq" - "outputs.outputJson.args.6": "-YYY" - "outputs.outputJson.args.7": "example_human_Illumina.pe_2.fastq" -} - -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: outputJson diff --git a/centaur/src/main/resources/standardTestCases/cwl_prefix_for_array.test b/centaur/src/main/resources/standardTestCases/cwl_prefix_for_array.test deleted file mode 100644 index cea66a37026..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_prefix_for_array.test +++ /dev/null @@ -1,17 +0,0 @@ -name: cwl_prefix_for_array -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: prefix-for-array -skipDescribeEndpointValidation: true - -files { - workflow: cwl_prefix_for_array/prefix_for_array.cwl - inputs: cwl_prefix_for_array/prefix_for_array.yml -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "outputs.prefix-for-array.out": "--bonus first second\n" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_prefix_for_array/prefix_for_array.cwl b/centaur/src/main/resources/standardTestCases/cwl_prefix_for_array/prefix_for_array.cwl deleted file mode 100644 index ca9044e5c61..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_prefix_for_array/prefix_for_array.cwl +++ /dev/null @@ -1,23 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: prefix-for-array - cwlVersion: v1.0 - class: CommandLineTool - baseCommand: ['/bin/echo'] - stdout: "hello.txt" - requirements: - - class: DockerRequirement - dockerPull: "ubuntu:latest" - inputs: - bonus: - type: string[] - inputBinding: - prefix: "--bonus" - outputs: - out: - type: string - outputBinding: - glob: hello.txt - loadContents: true - outputEval: $(self[0].contents) - diff --git a/centaur/src/main/resources/standardTestCases/cwl_prefix_for_array/prefix_for_array.yml b/centaur/src/main/resources/standardTestCases/cwl_prefix_for_array/prefix_for_array.yml deleted file mode 100644 index 6a3badb824c..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_prefix_for_array/prefix_for_array.yml +++ /dev/null @@ -1 +0,0 @@ -bonus: ["first", "second"] diff --git a/centaur/src/main/resources/standardTestCases/cwl_recursive_link_directories.test b/centaur/src/main/resources/standardTestCases/cwl_recursive_link_directories.test deleted file mode 100644 index 390c1d66409..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_recursive_link_directories.test +++ /dev/null @@ -1,18 +0,0 @@ -name: cwl_recursive_link_directories -testFormat: workflowsuccess -backends: [Local] -backendsMode: only -tags: [localdockertest] -skipDescribeEndpointValidation: true - -files { - workflow: cwl_recursive_link_directories/cwl_recursive_link_directories.cwl -} - -metadata { - status: Succeeded -} - -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: cwl_recursive_link_directories diff --git a/centaur/src/main/resources/standardTestCases/cwl_recursive_link_directories/cwl_recursive_link_directories.cwl b/centaur/src/main/resources/standardTestCases/cwl_recursive_link_directories/cwl_recursive_link_directories.cwl deleted file mode 100644 index a4a19fd6a5d..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_recursive_link_directories/cwl_recursive_link_directories.cwl +++ /dev/null @@ -1,19 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: cwl_recursive_link_directories - class: CommandLineTool - requirements: - - class: ShellCommandRequirement - cwlVersion: v1.0 - hints: - - class: DockerRequirement - dockerPull: ubuntu:latest - inputs: [] - outputs: - output_dir: - type: Directory - outputBinding: - glob: work_dir - arguments: - - shellQuote: false - valueFrom: "mkdir work_dir && ln -s .. work_dir/link" diff --git a/centaur/src/main/resources/standardTestCases/cwl_relative_imports/cwl_glob_sort.cwl b/centaur/src/main/resources/standardTestCases/cwl_relative_imports/cwl_glob_sort.cwl deleted file mode 100644 index 330c896eea6..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_relative_imports/cwl_glob_sort.cwl +++ /dev/null @@ -1,16 +0,0 @@ -cwlVersion: v1.0 -class: CommandLineTool -requirements: - - class: InlineJavascriptRequirement -hints: - DockerRequirement: - dockerPull: "debian:stretch-slim" -inputs: [] -baseCommand: [touch, z, y, x, w, c, b, a] -outputs: - letters: - type: string - outputBinding: - glob: '?' - outputEval: | - ${ return self.sort(function(a,b) { return a.location > b.location ? 1 : (a.location < b.location ? -1 : 0) }).map(f => f.basename).join(" ") } diff --git a/centaur/src/main/resources/standardTestCases/cwl_relative_imports/workflow.cwl b/centaur/src/main/resources/standardTestCases/cwl_relative_imports/workflow.cwl deleted file mode 100644 index 0ba8fc7236d..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_relative_imports/workflow.cwl +++ /dev/null @@ -1,16 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: relative_imports - class: Workflow - - inputs: [] - outputs: - letters: - type: string - outputSource: globSort/letters - - steps: - globSort: - in: [] - run: cwl_glob_sort.cwl - out: [letters] diff --git a/centaur/src/main/resources/standardTestCases/cwl_relative_imports_url.test b/centaur/src/main/resources/standardTestCases/cwl_relative_imports_url.test deleted file mode 100644 index eabd10754f6..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_relative_imports_url.test +++ /dev/null @@ -1,20 +0,0 @@ -name: cwl_relative_imports_url -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: relative_imports -skipDescribeEndpointValidation: true -# This test has experienced sporadic failures, and CWL is not fully supported. Therefore the test is being disabled. -ignore: true - -files { - workflowUrl: "https://raw.githubusercontent.com/broadinstitute/cromwell/develop/centaur/src/main/resources/standardTestCases/cwl_relative_imports/workflow.cwl" -} - -metadata { - status: Succeeded - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "calls.relative_imports.globSort.outputs.letters": "a b c w x y z" - "outputs.relative_imports.letters": "a b c w x y z" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_relative_imports_zip.test b/centaur/src/main/resources/standardTestCases/cwl_relative_imports_zip.test deleted file mode 100644 index deccb8d73a3..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_relative_imports_zip.test +++ /dev/null @@ -1,22 +0,0 @@ -name: cwl_relative_imports_zip -testFormat: workflowsuccess -skipDescribeEndpointValidation: true - -files { - workflow: cwl_relative_imports/workflow.cwl - imports: [ - cwl_relative_imports/cwl_glob_sort.cwl - ] -} - -metadata { - status: Succeeded - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "calls.relative_imports.globSort.outputs.letters": "a b c w x y z" - "outputs.relative_imports.letters": "a b c w x y z" -} - -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: relative_imports diff --git a/centaur/src/main/resources/standardTestCases/cwl_resources/cwl_disk_resources.cwl b/centaur/src/main/resources/standardTestCases/cwl_resources/cwl_disk_resources.cwl deleted file mode 100644 index 683dd5b17e6..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_resources/cwl_disk_resources.cwl +++ /dev/null @@ -1,38 +0,0 @@ -cwlVersion: v1.0 -# Asks for 10 GB (~=10240 MB) of output dir, tmp dir and input dir. -# Unfortunately the internal metadata from the VM, despite having some information about the disks doesn't give their size. -# To avoid possibly unreliable computations based on free or other tool, instead use the GCE API directly to look up the disk info. -# To do that we need project name, zone and name of the disk which we all obtain via the internal metadata -# Of importance is the assumption that == -1 -# Note the "-1" after the instance name. That is because the disk just named is the boot disk. The working disk has a "-1" suffix -# The test expectation ensures that the output is 30, which means all 3 resources have been taken into account. -$namespaces: - dx: https://www.dnanexus.com/cwl# -$graph: -- id: diskSizeTool - class: CommandLineTool - cwlVersion: v1.0 - doc: "Asks for disk minimums" - requirements: - ResourceRequirement: - outdirMin: 10240 - tmpdirMin: 10240 - hints: - - class: ShellCommandRequirement - - class: DockerRequirement - dockerPull: gcr.io/google.com/cloudsdktool/cloud-sdk:slim - - class: dx:InputResourceRequirement - indirMin: 10240 - inputs: [] - outputs: - disk_size: - type: int - outputBinding: - glob: stdout - loadContents: true - outputEval: $(parseInt(self[0].contents.trim())) - arguments: - - valueFrom: > - apt-get install --assume-yes jq > /dev/null && NAME=`curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/name` && ZONE=`basename \`curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/zone\`` && PROJECT=`curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/project/project-id` && curl -s -H "Authorization: Bearer `gcloud auth print-access-token`" "https://www.googleapis.com/compute/v1/projects/${PROJECT}/zones/${ZONE}/disks/${NAME}-1?fields=sizeGb" | jq -r '.sizeGb' - shellQuote: false - stdout: stdout diff --git a/centaur/src/main/resources/standardTestCases/cwl_resources/cwl_inputdir_zero_doesnt_localize.cwl b/centaur/src/main/resources/standardTestCases/cwl_resources/cwl_inputdir_zero_doesnt_localize.cwl deleted file mode 100644 index d874ee0bffb..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_resources/cwl_inputdir_zero_doesnt_localize.cwl +++ /dev/null @@ -1,129 +0,0 @@ -cwlVersion: v1.0 -$namespaces: - dx: https://www.dnanexus.com/cwl# -$graph: -- id: inputdir_zero_doesnt_localize - class: Workflow - hints: - - class: ShellCommandRequirement - - class: DockerRequirement - dockerPull: gcr.io/google.com/cloudsdktool/cloud-sdk:slim - - class: dx:InputResourceRequirement - indirMin: 0 - inputs: [] - outputs: - - id: errors - type: File - outputSource: findFile/errors - steps: - - id: echo - in: [] - out: - - id: echoOut - - id: echoArrayOut - - id: echoRecordOut - - id: echoMaybeOut - run: - class: CommandLineTool - inputs: [] - outputs: - - id: echoOut - outputBinding: - glob: lonely_file - type: File - secondaryFiles: [.also] - - id: echoArrayOut - outputBinding: - glob: '*.txt*' - outputEval: $([self[0]]) - type: - items: File - type: array - - id: echoRecordOut - secondaryFiles: [.also] - type: - type: record - name: foo - fields: - - name: a - type: File - outputBinding: - glob: file_in_object - - id: echoMaybeOut - type: - - File - - 'null' - secondaryFiles: [.also] - outputBinding: - glob: maybe_file - arguments: - - valueFrom: | - echo "lonely_file" > lonely_file - echo "lonely_file.also" > lonely_file.also - echo "file_in_array.txt" > file_in_array.txt - echo "file_in_array.txt.also" > file_in_array.txt.also - echo "file_in_object" > file_in_object - echo "file_in_object.also" > file_in_object.also - echo "maybe_file" > maybe_file - echo "maybe_file.also" > maybe_file.also - shellQuote: false - - id: findFile - in: - - id: f - source: "#inputdir_zero_doesnt_localize/echo/echoOut" - - id: g - source: "#inputdir_zero_doesnt_localize/echo/echoArrayOut" - - id: h - source: "#inputdir_zero_doesnt_localize/echo/echoRecordOut" - - id: i - source: "#inputdir_zero_doesnt_localize/echo/echoMaybeOut" - out: - - id: errors - run: - inputs: - - id: f - type: File - secondaryFiles: [.also] - - id: i - type: File? - secondaryFiles: [.also] - - id: g - type: - type: array - items: File - secondaryFiles: [.also] - - id: h - secondaryFiles: [.also] - type: - type: record - name: foo - fields: - - name: a - type: File - outputs: - - id: errors - type: string - outputBinding: - glob: errors.txt - loadContents: true - outputEval: $(self[0].contents) - class: CommandLineTool - requirements: - - class: ShellCommandRequirement - - class: InlineJavascriptRequirement - arguments: - # Unfortunately we can't use the same trick as in draft3_nio_file_papi1 because in papi 2 the instance metadata doesn't seem to contain - # the PAPI operation ID. So instead simply check that the input files are not there. This is not as neat as the PAPI1 version - # but should check the right thing as long as the files are localized (in general, but not here) somewhere in the same directory as where the command - # runs. If that changes then the directory where "find" searches should be updated - - valueFrom: | - touch errors.txt - find . -name lonely_file >> errors.txt - find . -name lonely_file.also >> errors.txt - find . -name file_in_array >> errors.txt - find . -name file_in_array.also >> errors.txt - find . -name file_in_object >> errors.txt - find . -name file_in_object.also >> errors.txt - find . -name maybe_file >> errors.txt - find . -name maybe_file.also >> errors.txt - shellQuote: false diff --git a/centaur/src/main/resources/standardTestCases/cwl_resources/cwl_resources.cwl b/centaur/src/main/resources/standardTestCases/cwl_resources/cwl_resources.cwl deleted file mode 100644 index 295302acc2e..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_resources/cwl_resources.cwl +++ /dev/null @@ -1,27 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: machineTypeTool - class: CommandLineTool - cwlVersion: v1.0 - doc: "Asks for CPU and Memory minimums" - requirements: - ResourceRequirement: - coresMin: 2 - coresMax: 2 - ramMin: 7GB - ramMax: 7GB - hints: - DockerRequirement: - dockerPull: python:latest - inputs: [] - outputs: - machine_type: - type: string - outputBinding: - glob: stdout - loadContents: true - outputEval: $(self[0].contents.trim()) - baseCommand: ['curl', 'http://metadata.google.internal/computeMetadata/v1/instance/machine-type', '-H', 'Metadata-Flavor: Google'] - stdout: stdout - - diff --git a/centaur/src/main/resources/standardTestCases/cwl_resources_papiv2.test b/centaur/src/main/resources/standardTestCases/cwl_resources_papiv2.test deleted file mode 100644 index ccb4f0ad534..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_resources_papiv2.test +++ /dev/null @@ -1,17 +0,0 @@ -name: cwl_resources_papiv2 -testFormat: workflowsuccess -backends: [Papiv2] -skipDescribeEndpointValidation: true - -files { - workflow: cwl_resources/cwl_resources.cwl -} - -metadata { - status: Succeeded - "outputs.machineTypeTool.machine_type": "projects/1005074806481/machineTypes/custom-2-7168" -} - -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: machineTypeTool diff --git a/centaur/src/main/resources/standardTestCases/cwl_restart/cwl_restart.cwl b/centaur/src/main/resources/standardTestCases/cwl_restart/cwl_restart.cwl deleted file mode 100644 index 82664f739b1..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_restart/cwl_restart.cwl +++ /dev/null @@ -1,117 +0,0 @@ -cwlVersion: v1.0 -class: Workflow -id: cwl_restart -# Workflow-level DockerRequirement -requirements: - DockerRequirement: - dockerPull: "ubuntu:latest" -inputs: [] -outputs: -- id: flag - outputSource: "#third_step/flag" - type: boolean -steps: -- id: first_step - in: [] - out: - - id: firstfoofile - # A dash is not valid in a WDL identifier and would have broken older `IdentifierAndPathPattern` parsing. - - id: first-bar-file - run: - inputs: [] - outputs: - - id: firstfoofile - outputBinding: - glob: out.txt - type: File - - id: first-bar-file - outputBinding: - glob: out.txt - type: File - class: CommandLineTool - requirements: - - class: ShellCommandRequirement - arguments: - - valueFrom: echo - - valueFrom: "I have a bad feeling about this" - - valueFrom: ">" - shellQuote: false - - valueFrom: "out.txt" -- id: cromwell_killer - in: - - id: fooinput - source: "#first_step/firstfoofile" - - id: bar-input - source: "#first_step/first-bar-file" - out: - - id: "footxt" - - id: "bar-txt" - run: - inputs: - - id: fooinput - type: File - - id: bar-input - type: File - outputs: - - id: "footxt" - outputBinding: - glob: foo.txt - type: File - - id: "bar-txt" - outputBinding: - glob: bar.txt - type: File - class: CommandLineTool - requirements: - - class: ShellCommandRequirement - arguments: - - valueFrom: sleep - - valueFrom: "60" - - valueFrom: "&&" - shellQuote: false - - valueFrom: echo - - valueFrom: foo - - valueFrom: ">" - shellQuote: false - - valueFrom: "foo.txt" - - valueFrom: "&&" - shellQuote: false - - valueFrom: echo - - valueFrom: bar - - valueFrom: ">" - shellQuote: false - - valueFrom: "bar.txt" -- id: third_step - in: - - id: foo - source: "#cromwell_killer/footxt" - - id: bar - source: "#cromwell_killer/bar-txt" - out: - - id: flag - run: - inputs: - - id: foo - type: File - - id: bar - type: File - outputs: - - id: flag - outputBinding: - outputEval: $(true) - type: boolean - class: CommandLineTool - requirements: - - class: InlineJavascriptRequirement - - class: ShellCommandRequirement - arguments: - - valueFrom: echo - - valueFrom: "Are we alive???" - - valueFrom: "&&" - shellQuote: false - - valueFrom: echo - - valueFrom: $(inputs.foo) - - valueFrom: "&&" - shellQuote: false - - valueFrom: echo - - valueFrom: $(inputs.bar) diff --git a/centaur/src/main/resources/standardTestCases/cwl_restart_local_with_recover.test b/centaur/src/main/resources/standardTestCases/cwl_restart_local_with_recover.test deleted file mode 100644 index a45e97769cd..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_restart_local_with_recover.test +++ /dev/null @@ -1,21 +0,0 @@ -name: cwl_restart_local_with_recover -testFormat: CromwellRestartWithRecover -callMark: cromwell_killer -backendsMode: "only" -backends: [Local, LocalNoDocker] -tags: [localdockertest] -skipDescribeEndpointValidation: true - -files { - workflow: cwl_restart/cwl_restart.cwl -} - -metadata { - status: Succeeded - "calls.first_step.executionStatus": Done - "calls.cromwell_killer.executionStatus": Done - "calls.third_step.executionStatus": Done -} - -workflowType: CWL -workflowTypeVersion: v1.0 diff --git a/centaur/src/main/resources/standardTestCases/cwl_run/1st-tool.cwl b/centaur/src/main/resources/standardTestCases/cwl_run/1st-tool.cwl deleted file mode 100644 index af0c4de297d..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_run/1st-tool.cwl +++ /dev/null @@ -1,9 +0,0 @@ -cwlVersion: v1.0 -class: CommandLineTool -baseCommand: echo -inputs: - message: - type: string - inputBinding: - position: 1 -outputs: [] diff --git a/centaur/src/main/resources/standardTestCases/cwl_run/echo-job.yml b/centaur/src/main/resources/standardTestCases/cwl_run/echo-job.yml deleted file mode 100644 index 913ff438103..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_run/echo-job.yml +++ /dev/null @@ -1 +0,0 @@ -message: Hello CWL! diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files.test b/centaur/src/main/resources/standardTestCases/cwl_secondary_files.test deleted file mode 100644 index 2c8f861e1e9..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_secondary_files.test +++ /dev/null @@ -1,21 +0,0 @@ -name: cwl_secondary_files -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: main -backendsMode: "any" -backends: [Local, LocalNoDocker] -tags: [localdockertest] -skipDescribeEndpointValidation: true - -files { - workflow: cwl_secondary_files/cwl_secondary_files.cwl - inputs: cwl_secondary_files/cwl_secondary_files.yaml - options: cwl_secondary_files/cwl_secondary_files.options -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "outputs.main.the_answer": "$(42)" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/bam.txt b/centaur/src/main/resources/standardTestCases/cwl_secondary_files/bam.txt deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/bam.txt.also b/centaur/src/main/resources/standardTestCases/cwl_secondary_files/bam.txt.also deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/bar.txt b/centaur/src/main/resources/standardTestCases/cwl_secondary_files/bar.txt deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/bar.txt.also b/centaur/src/main/resources/standardTestCases/cwl_secondary_files/bar.txt.also deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/baz.txt b/centaur/src/main/resources/standardTestCases/cwl_secondary_files/baz.txt deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/baz.txt.also b/centaur/src/main/resources/standardTestCases/cwl_secondary_files/baz.txt.also deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/bim.txt b/centaur/src/main/resources/standardTestCases/cwl_secondary_files/bim.txt deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/bim.txt.also b/centaur/src/main/resources/standardTestCases/cwl_secondary_files/bim.txt.also deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/cwl_secondary_files.cwl b/centaur/src/main/resources/standardTestCases/cwl_secondary_files/cwl_secondary_files.cwl deleted file mode 100644 index 123a7b574a1..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/cwl_secondary_files.cwl +++ /dev/null @@ -1,44 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: main - class: CommandLineTool - hints: - DockerRequirement: - dockerPull: "debian:stretch-slim" - inputs: - - id: command - type: string - - id: f - type: File - inputBinding: - position: 2 - secondaryFiles: [.also] - - id: of - type: File? - inputBinding: - position: 3 - secondaryFiles: [.also] - - id: fs - type: - type: array - items: File - inputBinding: - position: 4 - secondaryFiles: [.also] - - id: fr - secondaryFiles: [.also] - type: - type: record - name: foo - fields: - - name: a - type: File - inputBinding: - position: 5 - outputs: - the_answer: - type: string - outputBinding: - outputEval: ${ return "$(" + 42 + ")"; } - baseCommand: [] - arguments: ["bash", "-c", $(inputs.command)] diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/cwl_secondary_files.options b/centaur/src/main/resources/standardTestCases/cwl_secondary_files/cwl_secondary_files.options deleted file mode 100644 index 71d245d2bf6..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/cwl_secondary_files.options +++ /dev/null @@ -1,3 +0,0 @@ -{ - "backend": "Local" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/cwl_secondary_files.yaml b/centaur/src/main/resources/standardTestCases/cwl_secondary_files/cwl_secondary_files.yaml deleted file mode 100644 index 928cfb8cbb3..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/cwl_secondary_files.yaml +++ /dev/null @@ -1,17 +0,0 @@ -# This command will stat the secondaryFiles and if they're missing bad times -command: stat $(echo $* | sed 's/.txt/.txt.also/g') > /dev/null -f: - class: File - path: "centaur/src/main/resources/standardTestCases/cwl_secondary_files/foo.txt" -of: - class: File - path: "centaur/src/main/resources/standardTestCases/cwl_secondary_files/bim.txt" -fs: - - class: File - path: "centaur/src/main/resources/standardTestCases/cwl_secondary_files/bar.txt" - - class: File - path: "centaur/src/main/resources/standardTestCases/cwl_secondary_files/baz.txt" -fr: - a: - class: File - path: "centaur/src/main/resources/standardTestCases/cwl_secondary_files/bam.txt" \ No newline at end of file diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/foo.txt b/centaur/src/main/resources/standardTestCases/cwl_secondary_files/foo.txt deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files/foo.txt.also b/centaur/src/main/resources/standardTestCases/cwl_secondary_files/foo.txt.also deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow.test b/centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow.test deleted file mode 100644 index f6dff3fb727..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow.test +++ /dev/null @@ -1,21 +0,0 @@ -name: cwl_secondary_files_workflow -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: cwl_secondary_files_workflow -backendsMode: "any" -backends: [Local, LocalNoDocker] -tags: [localdockertest] -skipDescribeEndpointValidation: true - -files { - workflow: cwl_secondary_files_workflow/cwl_secondary_files_workflow.cwl - inputs: cwl_secondary_files_workflow/cwl_secondary_files_workflow.yaml - options: cwl_secondary_files_workflow/cwl_secondary_files_workflow.options -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "outputs.cwl_secondary_files_workflow.the_answer": "$(42)" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow/cwl_secondary_files_workflow.cwl b/centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow/cwl_secondary_files_workflow.cwl deleted file mode 100644 index d5f9dc3f8c9..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow/cwl_secondary_files_workflow.cwl +++ /dev/null @@ -1,55 +0,0 @@ -cwlVersion: v1.0 -$graph: -# The inputs of the workflow specify secondary files but NOT of the tool -# Verify that they are still propagated through -- id: cwl_secondary_files_workflow - class: Workflow - inputs: - - id: command - type: string - - id: wf_file_input - type: File - secondaryFiles: [.also] - - id: wf_file_input_array - type: - type: array - items: File - secondaryFiles: [.also] - outputs: - - id: the_answer - type: string - outputSource: run_tool/the_answer - steps: - - id: run_tool - run: "#cwl_secondary_files_workflow_tool" - in: - command: command - f: wf_file_input - fs: wf_file_input_array - out: - - id: the_answer -- id: cwl_secondary_files_workflow_tool - class: CommandLineTool - hints: - DockerRequirement: - dockerPull: "debian:stretch-slim" - inputs: - - id: command - type: string - - id: f - type: File - inputBinding: - position: 2 - - id: fs - type: - type: array - items: File - inputBinding: - position: 3 - outputs: - the_answer: - type: string - outputBinding: - outputEval: ${ return "$(" + 42 + ")"; } - baseCommand: [] - arguments: ["bash", "-c", $(inputs.command)] diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow/cwl_secondary_files_workflow.options b/centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow/cwl_secondary_files_workflow.options deleted file mode 100644 index 71d245d2bf6..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow/cwl_secondary_files_workflow.options +++ /dev/null @@ -1,3 +0,0 @@ -{ - "backend": "Local" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow/cwl_secondary_files_workflow.yaml b/centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow/cwl_secondary_files_workflow.yaml deleted file mode 100644 index e0cf43d4b1a..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_secondary_files_workflow/cwl_secondary_files_workflow.yaml +++ /dev/null @@ -1,10 +0,0 @@ -# This command will stat the secondaryFiles and if they're missing bad times -command: stat $(echo $* | sed 's/.txt/.txt.also/g') > /dev/null -wf_file_input: - class: File - path: "centaur/src/main/resources/standardTestCases/cwl_secondary_files/foo.txt" -wf_file_input_array: - - class: File - path: "centaur/src/main/resources/standardTestCases/cwl_secondary_files/bar.txt" - - class: File - path: "centaur/src/main/resources/standardTestCases/cwl_secondary_files/baz.txt" diff --git a/centaur/src/main/resources/standardTestCases/cwl_stdout_expression/cwl_stdout_expression.cwl b/centaur/src/main/resources/standardTestCases/cwl_stdout_expression/cwl_stdout_expression.cwl deleted file mode 100644 index b49ec9b1134..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_stdout_expression/cwl_stdout_expression.cwl +++ /dev/null @@ -1,17 +0,0 @@ -cwlVersion: v1.0 -$graph: -- id: cwl_stdout_expression - class: CommandLineTool - hints: - DockerRequirement: - dockerPull: "debian:stretch-slim" - inputs: - - id: foo - type: string - - id: bar - type: string - outputs: - b: stdout - stdout: "stdout-$(inputs.foo)-$(inputs.bar).txt" - baseCommand: [] - arguments: [echo, $(inputs.foo), $(inputs.bar)] diff --git a/centaur/src/main/resources/standardTestCases/cwl_stdout_expression/cwl_stdout_expression.json b/centaur/src/main/resources/standardTestCases/cwl_stdout_expression/cwl_stdout_expression.json deleted file mode 100644 index 82e0295d318..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_stdout_expression/cwl_stdout_expression.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "foo": "baz", - "bar": "qux" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_stdout_expression_local.test b/centaur/src/main/resources/standardTestCases/cwl_stdout_expression_local.test deleted file mode 100644 index 13bfbb68e24..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_stdout_expression_local.test +++ /dev/null @@ -1,21 +0,0 @@ -name: cwl_stdout_expression_local -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: cwl_stdout_expression -backendsMode: "any" -backends: [Local, LocalNoDocker] -tags: [localdockertest] - -skipDescribeEndpointValidation: true - -files { - workflow: cwl_stdout_expression/cwl_stdout_expression.cwl - inputs: cwl_stdout_expression/cwl_stdout_expression.json -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "calls.cwl_stdout_expression.stdout": "<>/call-cwl_stdout_expression/execution/stdout-baz-qux.txt" -} diff --git a/centaur/src/main/resources/standardTestCases/cwl_stdout_expression_papi.test b/centaur/src/main/resources/standardTestCases/cwl_stdout_expression_papi.test deleted file mode 100644 index b48a0effa79..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwl_stdout_expression_papi.test +++ /dev/null @@ -1,18 +0,0 @@ -name: cwl_stdout_expression_papi -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: cwl_stdout_expression -backends: [Papi2] -skipDescribeEndpointValidation: true - -files { - workflow: cwl_stdout_expression/cwl_stdout_expression.cwl - inputs: cwl_stdout_expression/cwl_stdout_expression.json -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "calls.cwl_stdout_expression.stdout": "<>call-cwl_stdout_expression/stdout-baz-qux.txt" -} diff --git a/centaur/src/main/resources/standardTestCases/cwls/scatter-job1.json b/centaur/src/main/resources/standardTestCases/cwls/scatter-job1.json deleted file mode 100644 index b3ab2751243..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwls/scatter-job1.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "inp": ["one", "two", "three", "four"], - "inp2": "yeahhh" -} diff --git a/centaur/src/main/resources/standardTestCases/cwls/scatter-wf1.cwl b/centaur/src/main/resources/standardTestCases/cwls/scatter-wf1.cwl deleted file mode 100644 index dc46aa7da4a..00000000000 --- a/centaur/src/main/resources/standardTestCases/cwls/scatter-wf1.cwl +++ /dev/null @@ -1,44 +0,0 @@ -#!/usr/bin/env cwl-runner -cwlVersion: v1.0 -class: Workflow -inputs: - inp: string[] - inp2: string -outputs: - out: - type: string[] - outputSource: step1/echo_out - -requirements: - - class: ScatterFeatureRequirement - - class: DockerRequirement - dockerPull: ubuntu:latest - -steps: - step1: - in: - echo_in: inp - echo_in2: inp2 - out: [echo_out] - scatter: echo_in - run: - class: CommandLineTool - inputs: - echo_in: - type: string - inputBinding: {} - echo_in2: - type: string - inputBinding: {} - outputs: - echo_out: - type: string - outputBinding: - glob: "step1_out" - loadContents: true - outputEval: $(self[0].contents) - baseCommand: "echo" - arguments: - - "-n" - - "foo" - stdout: "step1_out" diff --git a/centaur/src/main/resources/standardTestCases/drs_tests/drs_usa_jdr.inputs b/centaur/src/main/resources/standardTestCases/drs_tests/drs_usa_jdr.inputs index d5738ac4d17..d1712641dd2 100644 --- a/centaur/src/main/resources/standardTestCases/drs_tests/drs_usa_jdr.inputs +++ b/centaur/src/main/resources/standardTestCases/drs_tests/drs_usa_jdr.inputs @@ -1,5 +1,5 @@ { - # Martha does not return service accounts for JDR paths. Therefore they shouldn't need to be localized using the + # DRSHub does not return service accounts for JDR paths. Therefore they shouldn't need to be localized using the # Cromwell custom DOS/DRS localizer. # # However, the first file1 was generated before JDR stared saving file names at the end of the gsUri. diff --git a/centaur/src/main/resources/standardTestCases/drs_usa_jdr.test b/centaur/src/main/resources/standardTestCases/drs_usa_jdr.test index 4025860ed8d..510c9215c1f 100644 --- a/centaur/src/main/resources/standardTestCases/drs_usa_jdr.test +++ b/centaur/src/main/resources/standardTestCases/drs_usa_jdr.test @@ -1,6 +1,7 @@ name: drs_usa_jdr testFormat: WorkflowSuccess backends: ["papi-v2-usa"] +tags: [ drs ] skipDescribeEndpointValidation: true files { diff --git a/centaur/src/main/resources/standardTestCases/drs_usa_jdr_preresolve.test b/centaur/src/main/resources/standardTestCases/drs_usa_jdr_preresolve.test index ab050aea69f..15e41d95529 100644 --- a/centaur/src/main/resources/standardTestCases/drs_usa_jdr_preresolve.test +++ b/centaur/src/main/resources/standardTestCases/drs_usa_jdr_preresolve.test @@ -1,6 +1,7 @@ name: drs_usa_jdr_preresolve testFormat: WorkflowSuccess backends: ["papi-v2-usa"] +tags: [ drs ] skipDescribeEndpointValidation: true files { diff --git a/centaur/src/main/resources/standardTestCases/expressionLib.test b/centaur/src/main/resources/standardTestCases/expressionLib.test deleted file mode 100644 index 74299af9bc6..00000000000 --- a/centaur/src/main/resources/standardTestCases/expressionLib.test +++ /dev/null @@ -1,15 +0,0 @@ -name: expression_lib_cwl -cwlVersion: 1.0 -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -skipDescribeEndpointValidation: true - -files { - workflow: expressionLib/expressionLib.cwl -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 -} diff --git a/centaur/src/main/resources/standardTestCases/expressionLib/expressionLib.cwl b/centaur/src/main/resources/standardTestCases/expressionLib/expressionLib.cwl deleted file mode 100644 index fe8df12b4b0..00000000000 --- a/centaur/src/main/resources/standardTestCases/expressionLib/expressionLib.cwl +++ /dev/null @@ -1,14 +0,0 @@ -class: CommandLineTool -cwlVersion: v1.0 -hints: - DockerRequirement: - dockerPull: "ubuntu:latest" -requirements: - InlineJavascriptRequirement: - expressionLib: - - "function foo() { return 2; }" -inputs: [] -outputs: - out: stdout -arguments: [echo, $(foo())] -stdout: whatever.txt diff --git a/centaur/src/main/resources/standardTestCases/hello_cwl.test b/centaur/src/main/resources/standardTestCases/hello_cwl.test deleted file mode 100644 index 18712fd5404..00000000000 --- a/centaur/src/main/resources/standardTestCases/hello_cwl.test +++ /dev/null @@ -1,12 +0,0 @@ -name: hello_cwl -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -workflowRoot: hello -skipDescribeEndpointValidation: true - -files { - workflow: hello_cwl/hello.cwl - inputs: hello_cwl/hello.inputs - "outputs.hello.salutation": "Hello m'Lord!" -} diff --git a/centaur/src/main/resources/standardTestCases/hello_cwl/hello.cwl b/centaur/src/main/resources/standardTestCases/hello_cwl/hello.cwl deleted file mode 100644 index deb1e4d1f4a..00000000000 --- a/centaur/src/main/resources/standardTestCases/hello_cwl/hello.cwl +++ /dev/null @@ -1,21 +0,0 @@ -cwlVersion: v1.0 -$graph: - id: hello - class: CommandLineTool - requirements: - - class: DockerRequirement - dockerPull: ubuntu:latest - baseCommand: echo - inputs: - message: - type: string - inputBinding: - position: 1 - stdout: hello-stdout.txt - outputs: - - id: salutation - type: string - outputBinding: - glob: hello-stdout.txt - loadContents: true - outputEval: $(self[0].contents.trim()) diff --git a/centaur/src/main/resources/standardTestCases/hello_cwl/hello.inputs b/centaur/src/main/resources/standardTestCases/hello_cwl/hello.inputs deleted file mode 100644 index 11888b548ad..00000000000 --- a/centaur/src/main/resources/standardTestCases/hello_cwl/hello.inputs +++ /dev/null @@ -1,3 +0,0 @@ -# Seemingly innocuous but this particular message actually tests proper shell quoting of -# spaces, single quotes, and exclamation points. -message: Hello m'Lord! diff --git a/centaur/src/main/resources/standardTestCases/http_inputs/http_inputs.cwl b/centaur/src/main/resources/standardTestCases/http_inputs/http_inputs.cwl deleted file mode 100644 index e476be06d12..00000000000 --- a/centaur/src/main/resources/standardTestCases/http_inputs/http_inputs.cwl +++ /dev/null @@ -1,49 +0,0 @@ -cwlVersion: v1.0 -$graph: - - id: http_inputs - class: Workflow - inputs: - - id: jamie - type: File - outputs: - - id: md5 - outputSource: "#http_inputs/sum/md5" - type: int - steps: - - id: sum - in: - - id: jamie - source: "#http_inputs/jamie" - out: - - id: md5 - # Workflow step-level DockerRequirement - requirements: - DockerRequirement: - dockerPull: "ubuntu:latest" - run: - inputs: - - id: jamie - type: File - outputs: - - id: md5 - outputBinding: - glob: md5-stdOut.txt - loadContents: true - outputEval: $(self[0].contents) - type: string - class: CommandLineTool - requirements: - - class: ShellCommandRequirement - - class: InlineJavascriptRequirement - arguments: - - valueFrom: "/usr/bin/md5sum" - shellQuote: false - - valueFrom: $(inputs.jamie) - shellQuote: true - - valueFrom: '|' - shellQuote: false - - valueFrom: cut - shellQuote: false - - valueFrom: -c1-32 - shellQuote: false - stdout: md5-stdOut.txt diff --git a/centaur/src/main/resources/standardTestCases/http_inputs/http_inputs_cwl.inputs b/centaur/src/main/resources/standardTestCases/http_inputs/http_inputs_cwl.inputs deleted file mode 100644 index 9a7137d1bf9..00000000000 --- a/centaur/src/main/resources/standardTestCases/http_inputs/http_inputs_cwl.inputs +++ /dev/null @@ -1,3 +0,0 @@ -{ - "jamie": "https://raw.githubusercontent.com/broadinstitute/cromwell/develop/docs/jamie_the_cromwell_pig.png", -} diff --git a/centaur/src/main/resources/standardTestCases/http_inputs_cwl.test b/centaur/src/main/resources/standardTestCases/http_inputs_cwl.test deleted file mode 100644 index 7ed111e3ec7..00000000000 --- a/centaur/src/main/resources/standardTestCases/http_inputs_cwl.test +++ /dev/null @@ -1,15 +0,0 @@ -name: http_inputs_cwl -backends: [Local, Papiv2] -testFormat: workflowsuccess -workflowRoot: http_inputs -skipDescribeEndpointValidation: true - -files { - workflow: http_inputs/http_inputs.cwl - inputs: http_inputs/http_inputs_cwl.inputs -} - -metadata { - status: Succeeded - "outputs.http_inputs.md5": "602d0a4c1d54b7302e2d245a66a1b609\n" -} diff --git a/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_mounted_only_if_requested.wdl b/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_mounted_only_if_requested.wdl new file mode 100644 index 00000000000..33153c1157c --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_mounted_only_if_requested.wdl @@ -0,0 +1,49 @@ +version 1.0 + + +workflow ReferenceDiskMountedOnlyIfRequested { + call MentionsNirvanaReference {} + output { + Boolean disk_mounted = MentionsNirvanaReference.disk_mounted + } +} + + +task MentionsNirvanaReference { + input { + # Tiny 116 byte reference file, certainly not worth attaching a 55 GiB Nirvana reference disk for this. + File mention = + "gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/MITOMAP_20200819.nsa.idx" + } + command <<< + PS4='\D{+%F %T} \w $ ' + set -o nounset -o xtrace + + # Debug output + lsblk + + CANDIDATE_MOUNT_POINT=$(lsblk | sed -E -n 's!.*(/mnt/[a-f0-9]+).*!\1!p') + + if [[ ! -z ${CANDIDATE_MOUNT_POINT} ]]; then + echo "Found unexpected mounted disk, investigating further." + find ${CANDIDATE_MOUNT_POINT} -print | tee find.out + + if grep -i nirvana find.out; then + echo "Found what appears to be a Nirvana reference disk." + else + echo "Found unknown volume mounted, see 'find.out' for manifest." + fi + echo true > disk_mounted.out + else + echo false > disk_mounted.out + fi + >>> + runtime { + docker: "ubuntu:latest" + backend: "Papiv2-Reference-Disk-Localization" + } + output { + Boolean disk_mounted = read_boolean("disk_mounted.out") + File? find_out = "find.out" + } +} diff --git a/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test.inputs b/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test.inputs index 8e28a8ef541..4caf94c0f77 100644 --- a/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test.inputs +++ b/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test.inputs @@ -1,3 +1,5 @@ { - "wf_reference_disk_test.check_if_localized_as_symlink.reference_file_input": "gs://gcp-public-data--broad-references/hg19/v0/Homo_sapiens_assembly19.tile_db_header.vcf" + "wf_reference_disk_test.broad_reference_file_input": "gs://gcp-public-data--broad-references/hg19/v0/Homo_sapiens_assembly19.fasta.fai", + "wf_reference_disk_test.nirvana_reference_file_input": "gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/phyloP_hg38.npd.idx", + "wf_reference_disk_test.nirvana_reference_file_metachar_input": "gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/1000_Genomes_Project_(SV)_Phase_3_v5a.nsi" } diff --git a/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test.wdl b/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test.wdl index 43a298f18ab..51fb467598b 100644 --- a/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test.wdl +++ b/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test.wdl @@ -1,11 +1,38 @@ -task check_if_localized_as_symlink { - File reference_file_input - command { - # print true if file is a symlink, otherwise print false - if test -h ${reference_file_input}; then echo "true"; else echo "false"; fi; +version 1.0 + +task check_if_localized_with_valid_symlink { + input { + File broad_reference_file_input + File nirvana_reference_file_input + File nirvana_reference_file_metachar_input } + String broad_input_valid_symlink = "broad_input_valid_symlink.txt" + String nirvana_input_valid_symlink = "nirvana_input_valid_symlink.txt" + String nirvana_metachar_input_valid_symlink = "nirvana_metachar_input_valid_symlink.txt" + command <<< + PS4='\D{+%F %T} \w $ ' + set -o nounset -o pipefail -o xtrace + + # Echo true to stdout if the argument is a symlink pointing to an extant file, otherwise echo false. + check_if_valid_symlink() { + local reference_input="$1" + + if [[ -h "${reference_input}" && -f $(readlink "${reference_input}") ]]; then + echo true + else + echo false + fi + } + + check_if_valid_symlink "~{broad_reference_file_input}" > ~{broad_input_valid_symlink} + check_if_valid_symlink "~{nirvana_reference_file_input}" > ~{nirvana_input_valid_symlink} + check_if_valid_symlink "~{nirvana_reference_file_metachar_input}" > ~{nirvana_metachar_input_valid_symlink} + + >>> output { - Boolean is_symlink = read_boolean(stdout()) + Boolean is_broad_input_valid_symlink = read_boolean("~{broad_input_valid_symlink}") + Boolean is_nirvana_input_valid_symlink = read_boolean("~{nirvana_input_valid_symlink}") + Boolean is_nirvana_metachar_input_valid_symlink = read_boolean("~{nirvana_metachar_input_valid_symlink}") } runtime { docker: "ubuntu:latest" @@ -14,8 +41,20 @@ task check_if_localized_as_symlink { } workflow wf_reference_disk_test { - call check_if_localized_as_symlink + input { + File broad_reference_file_input + File nirvana_reference_file_input + File nirvana_reference_file_metachar_input + } + call check_if_localized_with_valid_symlink { + input: + broad_reference_file_input = broad_reference_file_input, + nirvana_reference_file_input = nirvana_reference_file_input, + nirvana_reference_file_metachar_input = nirvana_reference_file_metachar_input + } output { - Boolean is_input_file_a_symlink = check_if_localized_as_symlink.is_symlink + Boolean is_broad_input_file_a_valid_symlink = check_if_localized_with_valid_symlink.is_broad_input_valid_symlink + Boolean is_nirvana_input_file_a_valid_symlink = check_if_localized_with_valid_symlink.is_nirvana_input_valid_symlink + Boolean is_nirvana_metachar_input_file_a_valid_symlink = check_if_localized_with_valid_symlink.is_nirvana_metachar_input_valid_symlink } } diff --git a/centaur/src/main/resources/standardTestCases/reference_disk_false_options.test b/centaur/src/main/resources/standardTestCases/reference_disk_false_options.test index b75ef58dabc..bb8be222691 100644 --- a/centaur/src/main/resources/standardTestCases/reference_disk_false_options.test +++ b/centaur/src/main/resources/standardTestCases/reference_disk_false_options.test @@ -11,5 +11,7 @@ files { metadata { workflowName: wf_reference_disk_test status: Succeeded - "outputs.wf_reference_disk_test.is_input_file_a_symlink": false + "outputs.wf_reference_disk_test.is_broad_input_file_a_valid_symlink": false + "outputs.wf_reference_disk_test.is_nirvana_input_file_a_valid_symlink": false + "outputs.wf_reference_disk_test.is_nirvana_metachar_input_file_a_valid_symlink": false } diff --git a/centaur/src/main/resources/standardTestCases/reference_disk_mounted_only_if_requested_false.test b/centaur/src/main/resources/standardTestCases/reference_disk_mounted_only_if_requested_false.test new file mode 100644 index 00000000000..32c6d83e5c3 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/reference_disk_mounted_only_if_requested_false.test @@ -0,0 +1,14 @@ +name: reference_disk_mounted_only_if_requested_false +testFormat: workflowsuccess +backends: [Papiv2-Reference-Disk-Localization] + +files { + workflow: reference_disk/reference_disk_mounted_only_if_requested.wdl + options: reference_disk/reference_disk_test_false.options.json +} + +metadata { + workflowName: ReferenceDiskMountedOnlyIfRequested + status: Succeeded + "outputs.ReferenceDiskMountedOnlyIfRequested.disk_mounted": false +} diff --git a/centaur/src/main/resources/standardTestCases/reference_disk_mounted_only_if_requested_true.test b/centaur/src/main/resources/standardTestCases/reference_disk_mounted_only_if_requested_true.test new file mode 100644 index 00000000000..165e1c6d1f4 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/reference_disk_mounted_only_if_requested_true.test @@ -0,0 +1,14 @@ +name: reference_disk_mounted_only_if_requested_true +testFormat: workflowsuccess +backends: [Papiv2-Reference-Disk-Localization] + +files { + workflow: reference_disk/reference_disk_mounted_only_if_requested.wdl + options: reference_disk/reference_disk_test_true.options.json +} + +metadata { + workflowName: ReferenceDiskMountedOnlyIfRequested + status: Succeeded + "outputs.ReferenceDiskMountedOnlyIfRequested.disk_mounted": true +} diff --git a/centaur/src/main/resources/standardTestCases/reference_disk_mounted_only_if_requested_unspecified.test b/centaur/src/main/resources/standardTestCases/reference_disk_mounted_only_if_requested_unspecified.test new file mode 100644 index 00000000000..860159b594e --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/reference_disk_mounted_only_if_requested_unspecified.test @@ -0,0 +1,14 @@ +name: reference_disk_mounted_only_if_requested_unspecified +testFormat: workflowsuccess +backends: [Papiv2-Reference-Disk-Localization] + +files { + workflow: reference_disk/reference_disk_mounted_only_if_requested.wdl + options: reference_disk/reference_disk_test_unspecified.options.json +} + +metadata { + workflowName: ReferenceDiskMountedOnlyIfRequested + status: Succeeded + "outputs.ReferenceDiskMountedOnlyIfRequested.disk_mounted": false +} diff --git a/centaur/src/main/resources/standardTestCases/reference_disk_true_options.test b/centaur/src/main/resources/standardTestCases/reference_disk_true_options.test index dab9c516164..6d957bf55b7 100644 --- a/centaur/src/main/resources/standardTestCases/reference_disk_true_options.test +++ b/centaur/src/main/resources/standardTestCases/reference_disk_true_options.test @@ -11,5 +11,7 @@ files { metadata { workflowName: wf_reference_disk_test status: Succeeded - "outputs.wf_reference_disk_test.is_input_file_a_symlink": true + "outputs.wf_reference_disk_test.is_broad_input_file_a_valid_symlink": true + "outputs.wf_reference_disk_test.is_nirvana_input_file_a_valid_symlink": true + "outputs.wf_reference_disk_test.is_nirvana_metachar_input_file_a_valid_symlink": true } diff --git a/centaur/src/main/resources/standardTestCases/reference_disk_unspecified_options.test b/centaur/src/main/resources/standardTestCases/reference_disk_unspecified_options.test index 441db8748f2..7625bcd7806 100644 --- a/centaur/src/main/resources/standardTestCases/reference_disk_unspecified_options.test +++ b/centaur/src/main/resources/standardTestCases/reference_disk_unspecified_options.test @@ -11,5 +11,7 @@ files { metadata { workflowName: wf_reference_disk_test status: Succeeded - "outputs.wf_reference_disk_test.is_input_file_a_symlink": false + "outputs.wf_reference_disk_test.is_broad_input_file_a_valid_symlink": false + "outputs.wf_reference_disk_test.is_nirvana_input_file_a_valid_symlink": false + "outputs.wf_reference_disk_test.is_nirvana_metachar_input_file_a_valid_symlink": false } diff --git a/centaur/src/main/resources/standardTestCases/retry_with_more_memory/retry_with_more_memory_after_137.wdl b/centaur/src/main/resources/standardTestCases/retry_with_more_memory/retry_with_more_memory_after_137.wdl new file mode 100644 index 00000000000..2fe434475c6 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/retry_with_more_memory/retry_with_more_memory_after_137.wdl @@ -0,0 +1,22 @@ +version 1.0 + +task imitate_oom_error { + command { + printf "Exception in thread "main" java.lang.OutOfMemoryError: testing\n\tat Test.main(Test.java:1)\n" >&2 + touch foo + exit 137 + } + output { + File foo = "foo" + } + runtime { + docker: "python:latest" + memory: "1 GB" + maxRetries: 2 + backend: "Papiv2" + } +} + +workflow retry_with_more_memory_after_137 { + call imitate_oom_error +} diff --git a/centaur/src/main/resources/standardTestCases/retry_with_more_memory_after_137.test b/centaur/src/main/resources/standardTestCases/retry_with_more_memory_after_137.test new file mode 100644 index 00000000000..a69290ca511 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/retry_with_more_memory_after_137.test @@ -0,0 +1,21 @@ +name: retry_with_more_memory_after_137 +testFormat: workflowfailure +backends: [Papiv2] + +files { + workflow: retry_with_more_memory/retry_with_more_memory_after_137.wdl + options: retry_with_more_memory/retry_with_more_memory.options +} + +metadata { + workflowName: retry_with_more_memory_after_137 + status: Failed + "failures.0.message": "Workflow failed" + "failures.0.causedBy.0.message": "stderr for job `retry_with_more_memory_after_137.imitate_oom_error:NA:3` contained one of the `memory-retry-error-keys: [OutOfMemory,Killed]` specified in the Cromwell config. Job might have run out of memory." + "retry_with_more_memory_after_137.imitate_oom_error.-1.1.executionStatus": "RetryableFailure" + "retry_with_more_memory_after_137.imitate_oom_error.-1.1.runtimeAttributes.memory": "1 GB" + "retry_with_more_memory_after_137.imitate_oom_error.-1.2.executionStatus": "RetryableFailure" + "retry_with_more_memory_after_137.imitate_oom_error.-1.2.runtimeAttributes.memory": "1.1 GB" + "retry_with_more_memory_after_137.imitate_oom_error.-1.3.executionStatus": "Failed" + "retry_with_more_memory_after_137.imitate_oom_error.-1.3.runtimeAttributes.memory": "1.2100000000000002 GB" +} diff --git a/centaur/src/main/resources/standardTestCases/scatter-wf1.test b/centaur/src/main/resources/standardTestCases/scatter-wf1.test deleted file mode 100644 index 397b94710a3..00000000000 --- a/centaur/src/main/resources/standardTestCases/scatter-wf1.test +++ /dev/null @@ -1,11 +0,0 @@ -name: cwl_scatter_wf1 -testFormat: workflowsuccess -skipDescribeEndpointValidation: true - -files { - workflow: cwls/scatter-wf1.cwl - inputs: cwls/scatter-job1.json -} - -workflowType: CWL -workflowTypeVersion: v1.0 diff --git a/centaur/src/main/resources/standardTestCases/three_step.test b/centaur/src/main/resources/standardTestCases/three_step.test deleted file mode 100644 index b24e649ecce..00000000000 --- a/centaur/src/main/resources/standardTestCases/three_step.test +++ /dev/null @@ -1,21 +0,0 @@ -name: three_step_cwl -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 -skipDescribeEndpointValidation: true - -files { - workflow: three_step/three_step.cwl - inputs: three_step/inputs.json -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 - "calls.wc.executionStatus": Done - "calls.ps.executionStatus": Done - "calls.cgrep.executionStatus": Done - "calls.ps.runtimeAttributes.docker": "ubuntu:bionic" - "calls.wc.runtimeAttributes.docker": "ubuntu:latest" - "calls.cgrep.runtimeAttributes.docker": "debian:jessie" -} diff --git a/centaur/src/main/resources/standardTestCases/three_step/inputs.json b/centaur/src/main/resources/standardTestCases/three_step/inputs.json deleted file mode 100644 index 8dac4768fff..00000000000 --- a/centaur/src/main/resources/standardTestCases/three_step/inputs.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "pattern": "java" -} diff --git a/centaur/src/main/resources/standardTestCases/three_step/three_step.cwl b/centaur/src/main/resources/standardTestCases/three_step/three_step.cwl deleted file mode 100644 index e328003eab1..00000000000 --- a/centaur/src/main/resources/standardTestCases/three_step/three_step.cwl +++ /dev/null @@ -1,114 +0,0 @@ -cwlVersion: v1.0 -class: Workflow -id: three_step -# Workflow-level DockerRequirement -hints: - DockerRequirement: - dockerPull: "ubuntu:latest" -inputs: -- id: pattern - type: string -outputs: -- id: cgrep-count - outputSource: "#cgrep/cgrep-count" - type: int -- id: wc-count - outputSource: "#wc/wc-count" - type: int -steps: -- id: ps - in: [] - out: - - id: ps-stdOut - run: - inputs: [] - outputs: - - id: ps-stdOut - outputBinding: - glob: ps-stdOut.txt - type: File - class: CommandLineTool - requirements: - # Command line tool-level DockerRequirement - # Check it: this DockerRequirement does not use the `class` formatting but should still parse - # thanks to the magic of SALAD. - DockerRequirement: - dockerPull: "ubuntu:bionic" - baseCommand: ps - stdout: ps-stdOut.txt -- id: cgrep - in: - - id: pattern - source: "#pattern" - - id: file - source: "#ps/ps-stdOut" - out: - - id: cgrep-count - # Workflow step-level DockerRequirement - requirements: - DockerRequirement: - dockerPull: "debian:jessie" - run: - inputs: - - id: pattern - type: string - - id: file - type: File - outputs: - - id: cgrep-count - outputBinding: - glob: cgrep-stdOut.txt - loadContents: true - outputEval: $(parseInt(self[0].contents)) - type: int - class: CommandLineTool - requirements: - - class: ShellCommandRequirement - - class: InlineJavascriptRequirement - arguments: - - valueFrom: grep - shellQuote: false - - valueFrom: $(inputs.pattern) - shellQuote: false - - valueFrom: ${return inputs.file} - shellQuote: false - - valueFrom: '|' - shellQuote: false - - valueFrom: wc - shellQuote: false - - valueFrom: -l - shellQuote: false - stdout: cgrep-stdOut.txt -- id: wc - in: - - id: file - source: "#ps/ps-stdOut" - out: - - id: wc-count - run: - inputs: - - id: file - type: File - outputs: - - id: wc-count - outputBinding: - glob: wc-stdOut.txt - loadContents: true - outputEval: $(parseInt(self[0].contents)) - type: int - class: CommandLineTool - requirements: - - class: ShellCommandRequirement - - class: InlineJavascriptRequirement - arguments: - - valueFrom: cat - shellQuote: false - - valueFrom: $(inputs.file) - shellQuote: false - - valueFrom: '|' - shellQuote: false - - valueFrom: wc - shellQuote: false - - valueFrom: -l - shellQuote: false - stdout: wc-stdOut.txt diff --git a/centaur/src/main/resources/standardTestCases/three_step/three_step_caller_wf.cwl b/centaur/src/main/resources/standardTestCases/three_step/three_step_caller_wf.cwl deleted file mode 100644 index ab5d8ddc458..00000000000 --- a/centaur/src/main/resources/standardTestCases/three_step/three_step_caller_wf.cwl +++ /dev/null @@ -1,24 +0,0 @@ -#!/usr/bin/env cwl-runner -class: Workflow -cwlVersion: v1.0 - -inputs: -- id: pin - type: string - default: "v" - -outputs: - count_output: - type: int - outputSource: threestep/wc-count - -requirements: - - class: SubworkflowFeatureRequirement - -steps: - threestep: - run: three_step.cwl - in: - - id: pattern - source: pin - out: [wc-count] diff --git a/centaur/src/main/resources/standardTestCases/three_step_subworkflow.test b/centaur/src/main/resources/standardTestCases/three_step_subworkflow.test deleted file mode 100644 index 6ae478035b7..00000000000 --- a/centaur/src/main/resources/standardTestCases/three_step_subworkflow.test +++ /dev/null @@ -1,16 +0,0 @@ -name: three_step__subwf_cwl -testFormat: workflowsuccess -workflowType: CWL -workflowTypeVersion: v1.0 - -files { - workflow: three_step/three_step_caller_wf.cwl, - imports: [ - three_step/three_step.cwl - ] -} - -metadata { - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 -} diff --git a/centaur/src/main/resources/standardTestCases/workflow_type_and_version_cwl.test b/centaur/src/main/resources/standardTestCases/workflow_type_and_version_cwl.test deleted file mode 100644 index 14ba92c588f..00000000000 --- a/centaur/src/main/resources/standardTestCases/workflow_type_and_version_cwl.test +++ /dev/null @@ -1,16 +0,0 @@ -ignore: true -name: workflow_type_and_version_cwl -testFormat: workflowsuccess - -files { - workflow: workflow_type_and_version_cwl/workflow_type_and_version_cwl.wdl -} - -workflowType: CWL -workflowTypeVersion: v1.0 - -metadata { - workflowName: workflow_type_and_version_cwl - "submittedFiles.workflowType": CWL - "submittedFiles.workflowTypeVersion": v1.0 -} diff --git a/centaurCwlRunner/src/bin/centaur-cwl-runner.bash b/centaurCwlRunner/src/bin/centaur-cwl-runner.bash deleted file mode 100755 index 8f39c3ec130..00000000000 --- a/centaurCwlRunner/src/bin/centaur-cwl-runner.bash +++ /dev/null @@ -1,26 +0,0 @@ -#!/usr/bin/env bash - -# `sbt assembly` must have already been run. -build_root="$( dirname "${BASH_SOURCE[0]}" )/../../.." -centaur_cwl_jar="${CENTAUR_CWL_JAR:-"$( \ - find "${build_root}/centaurCwlRunner/target/scala-2.13" -name 'centaur-cwl-runner-*.jar' -print0 \ - | xargs -0 ls -1 -t \ - | head -n 1 \ - )"}" -centaur_cwl_skip_file="${build_root}/centaurCwlRunner/src/main/resources/skipped_tests.csv" - -centaur_cwl_java_args=("-Xmx1g") -if [[ -n "${CENTAUR_CWL_JAVA_ARGS-}" ]]; then - # Allow splitting on space to simulate an exported array - # https://stackoverflow.com/questions/5564418/exporting-an-array-in-bash-script#answer-5564589 - # shellcheck disable=SC2206 - centaur_cwl_java_args+=(${CENTAUR_CWL_JAVA_ARGS}) -fi - -# Handle empty arrays in older versions of bash -# https://stackoverflow.com/questions/7577052/bash-empty-array-expansion-with-set-u#answer-7577209 -java \ - ${centaur_cwl_java_args[@]+"${centaur_cwl_java_args[@]}"} \ - -jar "${centaur_cwl_jar}" \ - --skip-file "${centaur_cwl_skip_file}" \ - "$@" diff --git a/centaurCwlRunner/src/main/resources/application.conf b/centaurCwlRunner/src/main/resources/application.conf deleted file mode 100644 index fcdd3da9c57..00000000000 --- a/centaurCwlRunner/src/main/resources/application.conf +++ /dev/null @@ -1,23 +0,0 @@ -akka { - loggers = ["akka.event.slf4j.Slf4jLogger"] - logging-filter = "akka.event.slf4j.Slf4jLoggingFilter" - - # Silence Akka-Http warning logging of: - # [WARN] [01/28/2019 06:50:04.293] [centaur-acting-like-a-system-akka.actor.default-dispatcher-3] [centaur-acting-like-a-system/Pool(shared->http://localhost:8000)] Connection attempt failed. Backing off new connection attempts for at least 100 milliseconds. - # - # via: - # https://github.com/akka/akka/blob/v2.5.19/akka-actor/src/main/resources/reference.conf#L41-L44 - # https://github.com/akka/akka/blob/v2.5.19/akka-actor/src/main/scala/akka/event/Logging.scala#L377 - # https://github.com/akka/akka-http/blob/v10.1.7/akka-http-core/src/main/scala/akka/http/impl/engine/client/pool/NewHostConnectionPool.scala#L134 - stdout-loglevel = "ERROR" -} - -centaur { - # When running the tests in parallel on a Travis instance we need a bit more time for a reply to come back. - sendReceiveTimeout: 1 minute -} - -cwltool-runner { - # When running the tests in parallel it takes a while for each JVM to initialize Heterodon. Instead use the process. - class = "cwl.CwltoolProcess" -} diff --git a/centaurCwlRunner/src/main/resources/logback.xml b/centaurCwlRunner/src/main/resources/logback.xml deleted file mode 100644 index d92bac61949..00000000000 --- a/centaurCwlRunner/src/main/resources/logback.xml +++ /dev/null @@ -1,48 +0,0 @@ - - - - %d{HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n - - - - - - - - - - - - - - - - - - - - diff --git a/centaurCwlRunner/src/main/resources/reference.conf b/centaurCwlRunner/src/main/resources/reference.conf deleted file mode 100644 index a906e824372..00000000000 --- a/centaurCwlRunner/src/main/resources/reference.conf +++ /dev/null @@ -1,29 +0,0 @@ -centaur { - cwl-runner { - mode = local - - # End of actual references and begin BA-6546 exceptions... - # Not sure if this reference conf is used by testers outside of DSP/broad? - # If you know for sure then do some combination of: - # - update this comment with the location of those tests - # - update the external tests to explicitly pass in their own config - # - if the external tests are 100% confirmed not to exist state how you know that - # Not really reference, but leaving just in case someone is relying on these values. - papi.default-input-gcs-prefix = "gs://centaur-cwl-conformance-1f501e3/cwl-inputs/" - tesk.default-input-ftp-prefix = "ftp://ftp.hexdump.org/centaur-cwl-conformance/cwl-inputs/" - google { - application-name = "centaur-cwl-runner" - genomics.endpoint-url = "https://lifesciences.googleapis.com/" - genomics.location = "us-central1" - max-attempts = 3 - auth = "application-default" - auths = [ - { - name = "application-default" - scheme = "application_default" - } - ] - } - # End BA-6546 exceptions - } -} diff --git a/centaurCwlRunner/src/main/resources/skipped_tests.csv b/centaurCwlRunner/src/main/resources/skipped_tests.csv deleted file mode 100644 index 08b6b117906..00000000000 --- a/centaurCwlRunner/src/main/resources/skipped_tests.csv +++ /dev/null @@ -1,2 +0,0 @@ -vf-concat.cwl,empty.json -dir5.cwl,dir-job.yml diff --git a/centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunner.scala b/centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunner.scala deleted file mode 100644 index ec5dcad537e..00000000000 --- a/centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunner.scala +++ /dev/null @@ -1,255 +0,0 @@ -package centaur.cwl - -import better.files._ -import cats.effect.IO -import centaur.api.CentaurCromwellClient -import centaur.cwl.Outputs._ -import centaur.test.TestOptions -import centaur.test.standard.{CentaurTestCase, CentaurTestFormat} -import centaur.test.submit.{SubmitHttpResponse, SubmitWorkflowResponse} -import centaur.test.workflow.{AllBackendsRequired, SubmittedWorkflowTracker, Workflow, WorkflowData} -import com.typesafe.scalalogging.StrictLogging -import common.util.VersionUtil -import cromwell.api.model.{Aborted, Failed, NonTerminalStatus, Succeeded} -import cromwell.core.WorkflowOptions -import cromwell.core.path.PathBuilderFactory -import cromwell.core.path.BetterFileMethods.Cmds -import cwl.preprocessor.{CwlFileReference, CwlPreProcessor} -import spray.json._ - -import scala.concurrent.Await -import scala.concurrent.duration.Duration - -/** - * Runs workflows in a "cwl-runner" friendly way. - * - * https://github.com/broadinstitute/cromwell/issues/2590 - * https://github.com/common-workflow-language/common-workflow-language/blob/v1.0.1/CONFORMANCE_TESTS.md - * https://github.com/common-workflow-language/common-workflow-language/blob/v1.0.1/draft-3/cwl-runner.cwl#L5-L68 - * https://github.com/common-workflow-language/common-workflow-language/pull/278/files#diff-ee814a9c027fc9750beb075c283a973cR49 - */ -object CentaurCwlRunner extends StrictLogging { - - case class SkippedTest(sourceName: String, inputsName: String) { - def shouldSkip(args: CommandLineArguments) = { - args.workflowSource.exists(_.name.equalsIgnoreCase(sourceName)) && - args.workflowInputs.exists(_.name.equalsIgnoreCase(inputsName)) - } - } - - case class CommandLineArguments(workflowSource: Option[File] = None, - workflowInputs: Option[File] = None, - quiet: Boolean = false, - outdir: Option[File] = None, - skipFile: Option[File] = None) - - // TODO: This would be cleaner with Enumeratum - object ExitCode extends Enumeration { - - protected case class ExitVal(status: Int) extends super.Val - - implicit class ValueToVal(val exitCodeValue: Value) extends AnyVal { - def status: Int = exitCodeValue.asInstanceOf[ExitVal].status - } - - val Success = ExitVal(0) - val Failure = ExitVal(1) - val NotImplemented = ExitVal(33) - } - - private val cwlPreProcessor = new CwlPreProcessor() - private val centaurCwlRunnerRunMode = CentaurCwlRunnerRunMode.fromConfig(CentaurCwlRunnerConfig.conf) - private val parser = buildParser() - private lazy val centaurCwlRunnerVersion = VersionUtil.getVersion("centaur-cwl-runner") - private lazy val versionString = s"$centaurCwlRunnerVersion ${centaurCwlRunnerRunMode.description}" - - private def showUsage(): ExitCode.Value = { - System.err.println(parser.usage) - ExitCode.Failure - } - - private def buildParser(): scopt.OptionParser[CommandLineArguments] = { - new scopt.OptionParser[CommandLineArguments]("java -jar /path/to/centaurCwlRunner.jar") { - head("centaur-cwl-runner", versionString) - - help("help").text("Centaur CWL Runner - Cromwell integration testing environment") - - version("version").text("Print version and exit") - - arg[String]("workflow-source").text("Workflow source file.").required(). - action((s, c) => c.copy(workflowSource = Option(File(s)))) - - arg[String]("inputs").text("Workflow inputs file.").optional(). - action((s, c) => c.copy(workflowInputs = Option(File(s)))) - - opt[Unit]("quiet").text("Only print warnings and errors.").optional(). - action((_, c) => c.copy(quiet = true)) - - opt[String]("outdir").text("Output directory, default current directory.").optional(). - action((s, c) => - c.copy(outdir = Option(File(s)))) - - opt[String]("skip-file").text("A csv file describing workflows to be skipped based on their name.").optional(). - action((s, c) => - c.copy(skipFile = Option(File(s)))) - } - } - - private def runCentaur(args: CommandLineArguments): ExitCode.Value = { - - def zipSiblings(file: File): File = { - val zipFile = File.newTemporaryFile("cwl_imports.", ".zip") - val dir = file.parent - if (!args.quiet) { - logger.info(s"Zipping files under 1mb in $dir to $zipFile") - } - // TODO: Only include files under 1mb for now. When cwl runners run in parallel this can use a lot of space. - val files = dir - .children - .filter(_.isRegularFile) - .filter(_.size < 1 * 1024 * 1024) - Cmds.zip(files.toSeq: _*)(zipFile) - zipFile - } - - val workflowPath = args.workflowSource.get - val (parsedWorkflowPath, workflowRoot) = workflowPath.path.toAbsolutePath.toString.split("#") match { - case Array(file) => File(file) -> None - case Array(file, root) => File(file) -> Option(root) - } - val outdirOption = args.outdir.map(_.pathAsString) - val testName = workflowPath.name - val preProcessedWorkflow = cwlPreProcessor - .preProcessCwlToString(CwlFileReference(parsedWorkflowPath, workflowRoot)) - .value.unsafeRunSync() match { - case Left(errors) => - logger.error(s"Failed to pre process cwl workflow: ${errors.toList.mkString(", ")}") - return ExitCode.Failure - case Right(v) => v - } - - val workflowContents = centaurCwlRunnerRunMode.preProcessWorkflow(preProcessedWorkflow) - val inputContents = args.workflowInputs - .map(centaurCwlRunnerRunMode.preProcessInput) - .map(preProcessed => { - preProcessed.value.unsafeRunSync() match { - case Left(errors) => - logger.error(s"Failed to pre process cwl workflow: ${errors.toList.mkString(", ")}") - return ExitCode.Failure - case Right(value) => value - } - }) - - val workflowType = Option("cwl") - val workflowTypeVersion = None - val optionsContents = outdirOption map { outdir => - JsObject("cwl_outdir" -> JsString(outdir)).compactPrint - } - val labels = List.empty - val zippedImports = Option(zipSiblings(workflowPath)) // TODO: Zipping all the things! Be more selective. - val backends = AllBackendsRequired(List.empty) - val workflowMetadata = None - val notInMetadata = List.empty - val directoryContentCounts = None - val testFormat = CentaurTestFormat.WorkflowSuccessTest - val testOptions = TestOptions(List.empty, ignore = false) - val submitResponseOption = None - - val submittedWorkflowTracker = new SubmittedWorkflowTracker() - - val workflowData = WorkflowData( - Option(workflowContents), - None, - None, - workflowType, - workflowTypeVersion, - inputContents.map(IO.pure), - optionsContents.map(IO.pure), - labels, - zippedImports - ) - val workflow = Workflow( - testName, - workflowData, - workflowMetadata, - notInMetadata, - directoryContentCounts, - backends, - retryTestFailures = false, - allowOtherOutputs = true, - skipDescribeEndpointValidation = true, - submittedWorkflowTracker = submittedWorkflowTracker, - maximumAllowedTime = None - ) - - val testCase = CentaurTestCase(workflow, testFormat, testOptions, submittedWorkflowTracker, submitResponseOption)(cromwellTracker = None) - - if (!args.quiet) { - logger.info(s"Starting test for $workflowPath") - } - - val pathBuilderFactory: PathBuilderFactory = centaurCwlRunnerRunMode.pathBuilderFactory - - try { - import CentaurCromwellClient.{blockingEc, system} - lazy val pathBuilder = Await.result(pathBuilderFactory.withOptions(WorkflowOptions.empty), Duration.Inf) - - testCase.testFunction.run.unsafeRunSync() match { - case unexpected: SubmitHttpResponse => - logger.error(s"Unexpected response: $unexpected") - ExitCode.Failure - case SubmitWorkflowResponse(submittedWorkflow) => - val status = CentaurCromwellClient.status(submittedWorkflow).unsafeRunSync() - status match { - case unexpected: NonTerminalStatus => - logger.error(s"Unexpected status: $unexpected") - ExitCode.Failure - case Aborted => - logger.error(s"Unexpected abort.") - ExitCode.Failure - case Failed => - logger.error(s"Unexpected failure.") - ExitCode.Failure - case Succeeded => - val result = handleOutput(submittedWorkflow, pathBuilder) - if (!args.quiet) { - logger.info(s"Result: $result") - } else { - // Print directly to stdout during --quiet - println(result) - } - ExitCode.Success - } - } - } finally { - zippedImports map { zipFile => - if (!args.quiet) { - logger.info(s"Deleting $zipFile") - } - zipFile.delete(swallowIOExceptions = true) - } - Await.result(CentaurCromwellClient.system.terminate(), Duration.Inf) - () - } - } - - private def skip(args: CommandLineArguments): Boolean = - args.skipFile - .map(_.lines).getOrElse(List.empty) - .map(_.split(',')) - .map({ - case Array(source, inputs) => SkippedTest(source, inputs) - case invalid => throw new RuntimeException(s"Invalid skipped_tests file: $invalid") - }) - .exists(_.shouldSkip(args)) - - def main(args: Array[String]): Unit = { - val parsedArgsOption = parser.parse(args, CommandLineArguments()) - val exitCode = parsedArgsOption match { - case Some(parsedArgs) if skip(parsedArgs) => ExitCode.NotImplemented - case Some(parsedArgs) => runCentaur(parsedArgs) - case None => showUsage() - } - System.exit(exitCode.status) - } -} diff --git a/centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunnerConfig.scala b/centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunnerConfig.scala deleted file mode 100644 index 0123efd7b8d..00000000000 --- a/centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunnerConfig.scala +++ /dev/null @@ -1,7 +0,0 @@ -package centaur.cwl - -import centaur.CentaurConfig - -object CentaurCwlRunnerConfig { - lazy val conf = CentaurConfig.conf.getConfig("cwl-runner") -} diff --git a/centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunnerRunMode.scala b/centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunnerRunMode.scala deleted file mode 100644 index 83b6626c983..00000000000 --- a/centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunnerRunMode.scala +++ /dev/null @@ -1,94 +0,0 @@ -package centaur.cwl - -import better.files.File -import cloud.nio.impl.ftp.FtpFileSystems -import com.typesafe.config.Config -import common.validation.IOChecked._ -import cromwell.core.path.Obsolete.Paths -import cromwell.core.path.{DefaultPathBuilderFactory, PathBuilderFactory} -import cromwell.filesystems.ftp.{CromwellFtpFileSystems, FtpPathBuilderFactory} -import cromwell.filesystems.gcs.GcsPathBuilderFactory -import cwl.preprocessor.CwlPreProcessor - -sealed trait CentaurCwlRunnerRunMode { - /** - * Returns a user friendly of this run mode. - */ - def description: String - - /** - * Returns a factory that can create a path builder for this run mode. - */ - def pathBuilderFactory: PathBuilderFactory - - /** - * Runs any preprocessing as needed on a workflow. - * - * For example, may prefix relative paths so that absolute URLs are used. - */ - def preProcessWorkflow(workflow: String): String = workflow - - /** - * Runs any preprocessing as needed on inputs. - * - * For example, may prefix relative paths so that absolute URLs are used. - */ - def preProcessInput(path: File): IOChecked[String] = path.contentAsString.validIOChecked -} - -object CentaurCwlRunnerRunMode { - def fromConfig(conf: Config): CentaurCwlRunnerRunMode = { - conf.getString("mode") match { - case "local" => LocalRunMode - case "papi_v2alpha1" | "papi_v2beta" => PapiRunMode(conf) - case "tesk" => TeskRunMode(conf) - case unknown => throw new UnsupportedOperationException(s"mode not recognized: $unknown") - } - } -} - -case object LocalRunMode extends CentaurCwlRunnerRunMode { - val cwlPreProcessor = new CwlPreProcessor() - override lazy val description: String = "local" - override lazy val pathBuilderFactory: PathBuilderFactory = DefaultPathBuilderFactory - /* - * If the file is a relative local path, resolve it against the path of the input json. - */ - private def inputFilesMapper(inputJsonPath: File)(file: String) = { - if (!Paths.get(file).isAbsolute) inputJsonPath.sibling(file).toString - else file - } - - override def preProcessInput(input: File): IOChecked[String] = cwlPreProcessor.preProcessInputFiles(input.contentAsString, inputFilesMapper(input)) -} - -case class PapiRunMode(conf: Config) extends CentaurCwlRunnerRunMode { - private lazy val googleConfig = conf.getConfig("google") - private lazy val preprocessor = new CloudPreprocessor(conf, "papi.default-input-gcs-prefix") - - override lazy val description: String = s"papi ${googleConfig.getString("auth")}" - - override lazy val pathBuilderFactory: PathBuilderFactory = { - GcsPathBuilderFactory(conf, googleConfig) - } - - override def preProcessWorkflow(workflow: String): String = preprocessor.preProcessWorkflow(workflow) - - override def preProcessInput(input: File): IOChecked[String] = preprocessor.preProcessInput(input.contentAsString) -} - -case class TeskRunMode(conf: Config) extends CentaurCwlRunnerRunMode { - private lazy val ftpConfig = conf.getConfig("ftp") - private lazy val preprocessor = new CloudPreprocessor(conf, "tesk.default-input-ftp-prefix") - - override lazy val description: String = "tesk" - - override lazy val pathBuilderFactory: PathBuilderFactory = { - val fileSystemsConfiguration = FtpFileSystems.DefaultConfig.copy(capacity = 2) - new FtpPathBuilderFactory(conf, ftpConfig, new CromwellFtpFileSystems(new FtpFileSystems(fileSystemsConfiguration))) - } - - override def preProcessWorkflow(workflow: String): String = preprocessor.preProcessWorkflow(workflow) - - override def preProcessInput(input: File): IOChecked[String] = preprocessor.preProcessInput(input.contentAsString) -} diff --git a/centaurCwlRunner/src/main/scala/centaur/cwl/CloudPreprocessor.scala b/centaurCwlRunner/src/main/scala/centaur/cwl/CloudPreprocessor.scala deleted file mode 100644 index 68e79b9b701..00000000000 --- a/centaurCwlRunner/src/main/scala/centaur/cwl/CloudPreprocessor.scala +++ /dev/null @@ -1,102 +0,0 @@ -package centaur.cwl -import better.files.File -import com.typesafe.config.Config -import common.util.StringUtil._ -import common.validation.IOChecked.IOChecked -import cwl.preprocessor.CwlPreProcessor -import io.circe.optics.JsonPath -import io.circe.optics.JsonPath._ -import io.circe.yaml.Printer.StringStyle -import io.circe.{Json, yaml} -import net.ceedubs.ficus.Ficus._ -import wom.util.YamlUtils - -/** - * Tools to pre-process the CWL workflows and inputs before feeding them to Cromwell so they can be executed on PAPI. - */ -class CloudPreprocessor(config: Config, prefixConfigPath: String) { - val cwlPreProcessor = new CwlPreProcessor() - - // Cloud directory where inputs for conformance tests are stored - private val cloudPrefix = config.as[String](prefixConfigPath).ensureSlashed - - // Default docker pull image - val DefaultDockerPull = "dockerPull" -> Json.fromString("ubuntu:latest") - - // Default docker image to be injected in a pre-existing requirements array - private val DefaultDockerHint: Json = { - Json.obj( - "class" -> Json.fromString("DockerRequirement"), - DefaultDockerPull - ) - } - - // hints array with default docker requirement - private val DefaultDockerHintList: Json = { - Json.obj( - "hints" -> Json.arr(DefaultDockerHint) - ) - } - - // Parse value, apply f to it, and print it back to String using the printer - private def process(value: String, f: Json => Json, printer: Json => String) = { - YamlUtils.parse(value) match { - case Left(error) => throw new Exception(error.getMessage) - case Right(json) => printer(f(json)) - } - } - - // Process and print back as YAML - private def processYaml(value: String)(f: Json => Json) = - process(value, f, yaml.Printer.spaces2.copy(stringStyle = StringStyle.DoubleQuoted).pretty) - - // Prefix the string at "key" with the cloud prefix - private def prefixLocation(value: String): String = { - cloudPrefix + File(value.stripPrefix("file://")).name - } - - // Function to check if the given json has the provided key / value pair - private def hasKeyValue(key: String, value: String): Json => Boolean = { - root.selectDynamic(key).string.exist(_.equalsIgnoreCase(value)) - } - - /** - * Pre-process input file by prefixing all files and directories with the cloud prefix - */ - def preProcessInput(input: String): IOChecked[String] = cwlPreProcessor.preProcessInputFiles(input, prefixLocation) - - // Check if the given path (as an array or object) has a DockerRequirement element - def hasDocker(jsonPath: JsonPath)(json: Json): Boolean = { - val hasDockerInArray: Json => Boolean = jsonPath.arr.exist(_.exists(hasKeyValue("class", "DockerRequirement"))) - val hasDockerInObject: Json => Boolean = jsonPath.obj.exist(_.kleisli("DockerRequirement").nonEmpty) - - hasDockerInArray(json) || hasDockerInObject(json) - } - - // Check if the given Json has a docker image in hints or requirements - def hasDocker(json: Json): Boolean = hasDocker(root.hints)(json) || hasDocker(root.requirements)(json) - - // Add a default docker hint to the workflow if it doesn't have one - private val addDefaultDocker: Json => Json = workflow => if (!hasDocker(workflow)) { - /* - * deepMerge does not combine objects together but replaces keys which would overwrite existing hints - * so first check if there are hints already and if so add our docker one. - * Also turns out that the hints section can be either an array or an object. - * When it gets saladed the object is transformed to an array but because we deal with unsaladed cwl here - * we have to handle both cases. - */ - val hintsAsArray = root.hints.arr.modifyOption(_ :+ DefaultDockerHint)(workflow) - val hintsAsObject = root.hints.obj.modifyOption(_.add("DockerRequirement", Json.obj(DefaultDockerPull)))(workflow) - - hintsAsArray - .orElse(hintsAsObject) - .getOrElse(workflow.deepMerge(DefaultDockerHintList)) - } else workflow - - private val prefixDefaultFilesInCwl = CwlPreProcessor.mapFilesAndDirectories(prefixLocation) _ - - /** - * Pre-process the workflow by adding a default docker hint iff it doesn't have one - */ - def preProcessWorkflow(workflow: String): String = processYaml(workflow)(addDefaultDocker.andThen(prefixDefaultFilesInCwl)) -} diff --git a/centaurCwlRunner/src/main/scala/centaur/cwl/OutputManipulator.scala b/centaurCwlRunner/src/main/scala/centaur/cwl/OutputManipulator.scala deleted file mode 100644 index dcbe3a70ec8..00000000000 --- a/centaurCwlRunner/src/main/scala/centaur/cwl/OutputManipulator.scala +++ /dev/null @@ -1,237 +0,0 @@ -package centaur.cwl - -import common.util.StringUtil._ -import cromwell.core.path.{Path, PathBuilder} -import cwl.CwlCodecs._ -import cwl.command.ParentName -import cwl.ontology.Schema -import cwl.{File => CwlFile, _} -import io.circe.literal._ -import io.circe.syntax._ -import io.circe.{Json, JsonObject} -import mouse.all._ -import org.apache.commons.codec.digest.DigestUtils -import shapeless.{Inl, Poly1} -import spray.json.{JsArray, JsBoolean, JsNull, JsNumber, JsObject, JsString, JsValue} - -//Take cromwell's outputs and format them as expected by the spec -object OutputManipulator extends Poly1 { - - //In an Ideal world I'd return a Coproduct of these types and leave the asJson-ing to the handleOutput - def resolveOutput(jsValue: JsValue, - pathBuilder: PathBuilder, - mot: MyriadOutputType, - schemaOption: Option[Schema]): Json = { - mot.fold(this).apply(jsValue, pathBuilder, schemaOption) - } - - private def hashFile(path: Path): Option[String] = path.exists.option("sha1$" + path.sha1.toLowerCase) - private def sizeFile(path: Path): Option[Long] = path.exists.option(path.size) - - private def stringToFile(pathAsString: String, pathBuilder: PathBuilder): Json = { - val path = pathBuilder.build(pathAsString).get - - if (path.exists()) { - CwlFile( - location = Option(path.name), - checksum = hashFile(path), - size = sizeFile(path) - ).asJson - } else { - Json.Null - } - } - - private def populateFileFields(pathBuilder: PathBuilder, - schemaOption: Option[Schema] - )(obj: JsonObject): JsonObject = { - val path = pathBuilder.build(obj.kleisli("location").get.asString.get).get - val isFile = obj.kleisli("class").exists(_.asString.contains("File")) - val isDirectory = obj.kleisli("class").exists(_.asString.contains("Directory")) - - // Return the default value if the value at key is either absent or null, otherwise return the value - def valueOrNull(key: String, default: Json) = obj.kleisli(key) match { - case Some(Json.Null) => default - case Some(other) => other - case None => default - } - - def populateInnerFiles(json: Json): Option[Json] = { - - def populateInnerFile(file: Json): Json = { - file.asObject - .map(populateFileFields(pathBuilder, schemaOption)) - .map(Json.fromJsonObject) - .orElse(file.asString.map(stringToFile(_, pathBuilder))) - .getOrElse(file) - } - - // Assume the json is an array ("secondaryFiles" and "listing" are both arrays). - val innerFiles = json.asArray.get - // Remove any files that don't actually exist. - val filteredFiles = innerFiles.map(populateInnerFile).filterNot(_.isNull) - // The cwl test runner doesn't expect a "secondaryFiles" or "listing" field at all if it's empty. - if (filteredFiles.nonEmpty) Option(Json.arr(filteredFiles: _*)) else None - } - - def updateFileOrDirectoryWithNestedFiles(obj: JsonObject, fieldName: String) = { - // Cromwell metadata has a field for all values even if their content is empty - // remove it as the cwl test runner expects nothing instead - val withoutField = obj.remove(fieldName) - - // If the field was not empty, add it back with each inner file / directory properly updated as well - populateInnerFiles(obj.kleisli(fieldName).get) - .map(withoutField.add(fieldName, _)) - .getOrElse(withoutField) - } - - // Get sha1 from the content field - // We need this because some task return a "File" object that actually does not exist on disk but has its content provided directly instead - def hashContent: Option[String] = obj.kleisli("contents").flatMap(_.asString.map(DigestUtils.sha1Hex).map("sha1$" + _)) - - // Get size from the content field - // We need this because some task return a "File" object that actually does not exist on disk but has its content provided directly instead - def sizeContent: Option[Long] = obj.kleisli("contents").flatMap(_.asString.map(_.length.toLong)) - - // "as a special case, cwltest only matches the trailing part of location in the output sections so that it - // can do something reasonable regardless of URL scheme or prefix" - Peter Amstutz, 2019-04-16, CWL gitter - val updatedLocation = obj.add("location", Json.fromString(path.pathAsString)) - - // Get the format - def formatOption: Option[String] = for { - klesliFormat <- obj.kleisli("format") - format <- klesliFormat.asString - } yield format - - // Lookup the full format using the namespaces - def fullFormatOption: Option[String] = for { - format <- formatOption - schema <- schemaOption - } yield schema.fullIri(format) - - if (isFile) { - /* - * In order of priority use: - * 1) the checksum value in the metadata - * 2) the checksum calculated from the value of the "contents" field of the metadata - * 3) the checksum calculated from the file itself - */ - val defaultChecksum = hashContent.orElse(hashFile(path)).map(Json.fromString).getOrElse(Json.Null) - val checksum = valueOrNull("checksum", defaultChecksum) - - /* - * In order of priority use: - * 1) the size value in the metadata - * 2) the size calculated from the value of the "contents" field of the metadata - * 3) the size calculated from the file itself - */ - val defaultSize = sizeContent.orElse(sizeFile(path)).map(Json.fromLong).getOrElse(Json.Null) - val size = valueOrNull("size", defaultSize) - - val basename: Option[Json] = - Option(valueOrNull("basename", path.exists.option(path.name).map(Json.fromString).getOrElse(Json.Null))) - - /* - In order of priority use: - 1) the full format if $namespaces are available - 2) the original format - */ - val defaultFormatOption: Option[Json] = fullFormatOption.orElse(formatOption).map(Json.fromString) - - val withChecksumAndSize = updatedLocation - .add("checksum", checksum) - .add("size", size) - // conformance test does not expect "contents" in the output - .remove("contents") - - val withBasename = basename - .map(withChecksumAndSize.add("basename", _)) - .getOrElse(withChecksumAndSize) - - val withFormat = defaultFormatOption - .map(withBasename.add("format", _)) - .getOrElse(withBasename) - - updateFileOrDirectoryWithNestedFiles(withFormat, "secondaryFiles") - } else if (isDirectory) { - updateFileOrDirectoryWithNestedFiles(updatedLocation, "listing"). - add("basename", path.nameWithoutExtension |> Json.fromString) - } else throw new RuntimeException(s"${path.pathAsString} is neither a valid file or a directory") - } - - private def resolveOutputViaInnerType(moits: Array[MyriadOutputInnerType]) - (jsValue: JsValue, - pathBuilder: PathBuilder, - schemaOption: Option[Schema]): Json = { - def sprayJsonToCirce(jsValue: JsValue): Json = { - import io.circe.parser._ - val jsonString = jsValue.compactPrint - parse(jsonString).getOrElse( - sys.error(s"Failed to parse Json output as Json... something is very wrong: '$jsonString'") - ) - } - - (jsValue, moits) match { - //CWL expects quite a few enhancements to the File structure, hence... - case (JsString(metadata), Array(Inl(CwlType.File))) => stringToFile(metadata, pathBuilder) - // If it's a JsObject it means it's already in the right format, we just want to fill in some values that might not - // have been populated like "checksum" and "size" - case (obj: JsObject, a) if a.contains(Inl(CwlType.File)) || a.contains(Inl(CwlType.Directory)) => - val json: Json = sprayJsonToCirce(obj) - val fileExists = (for { - o <- json.asObject - l <- o.kleisli("location") - s <- l.asString - c = if (a.contains(Inl(CwlType.Directory))) s.ensureSlashed else s - p <- pathBuilder.build(c).toOption - } yield p.exists).getOrElse(false) - if (fileExists) json.mapObject(populateFileFields(pathBuilder, schemaOption)) else Json.Null - case (JsNumber(metadata), Array(Inl(CwlType.Long))) => metadata.longValue.asJson - case (JsNumber(metadata), Array(Inl(CwlType.Float))) => metadata.floatValue.asJson - case (JsNumber(metadata), Array(Inl(CwlType.Double))) => metadata.doubleValue.asJson - case (JsNumber(metadata), Array(Inl(CwlType.Int))) => metadata.intValue.asJson - case (JsString(metadata), Array(Inl(CwlType.String))) => metadata.asJson - - //The Anys. They have to be done for each type so that the asJson can use this type information when going back to Json representation - case (JsString(metadata), Array(Inl(CwlType.Any))) => metadata.asJson - case (JsNumber(metadata), Array(Inl(CwlType.Any))) => metadata.asJson - case (obj: JsObject, Array(Inl(CwlType.Any))) => sprayJsonToCirce(obj) - case (JsBoolean(metadata), Array(Inl(CwlType.Any))) => metadata.asJson - case (array: JsArray, Array(Inl(CwlType.Any))) => sprayJsonToCirce(array) - case (JsNull, a) if a.contains(Inl(CwlType.Any)) || a.contains(Inl(CwlType.Null)) => Json.Null - - case (JsArray(metadata), Array(tpe)) if tpe.select[OutputArraySchema].isDefined => - (for { - schema <- tpe.select[OutputArraySchema] - items = schema.items - innerType <- items.select[MyriadOutputInnerType] - outputJson = metadata.map(m => - resolveOutputViaInnerType(Array(innerType))(m, pathBuilder, schemaOption)).asJson - } yield outputJson).getOrElse(throw new RuntimeException(s"We currently do not support output arrays with ${tpe.select[OutputArraySchema].get.items} inner type")) - case (JsObject(metadata), Array(tpe)) if tpe.select[OutputRecordSchema].isDefined => - def processField(field: OutputRecordField) = { - val parsedName = FullyQualifiedName(field.name)(ParentName.empty).id - field.`type`.select[MyriadOutputInnerType] map { parsedName -> _ } - } - - (for { - schema <- tpe.select[OutputRecordSchema] - fields <- schema.fields - typeMap = fields.flatMap(processField).toMap - outputJson = metadata.map({ - case (k, v) => k -> resolveOutputViaInnerType(Array(typeMap(k)))(v, pathBuilder, schemaOption) - }).asJson - } yield outputJson).getOrElse(throw new RuntimeException(s"We currently do not support output record schemas with ${tpe.select[OutputArraySchema].get.items} inner type")) - case (json, tpe) => throw new RuntimeException(s"We currently do not support outputs (${json.getClass.getSimpleName}) of $json and type $tpe") - } - } - - implicit val moit: Case.Aux[MyriadOutputInnerType, (JsValue, PathBuilder, Option[Schema]) => Json] = at { t => - resolveOutputViaInnerType(Array(t)) - } - - implicit val amoit: Case.Aux[Array[MyriadOutputInnerType], (JsValue, PathBuilder, Option[Schema]) => Json] = - at { amoit => - resolveOutputViaInnerType(amoit) - } -} diff --git a/centaurCwlRunner/src/main/scala/centaur/cwl/Outputs.scala b/centaurCwlRunner/src/main/scala/centaur/cwl/Outputs.scala deleted file mode 100644 index c63d95399d7..00000000000 --- a/centaurCwlRunner/src/main/scala/centaur/cwl/Outputs.scala +++ /dev/null @@ -1,93 +0,0 @@ -package centaur.cwl - -import centaur.api.CentaurCromwellClient -import centaur.test.metadata.WorkflowFlatMetadata._ -import common.collections.EnhancedCollections._ -import common.validation.IOChecked._ -import cromwell.api.model.SubmittedWorkflow -import cromwell.core.path.PathBuilder -import cwl.ontology.Schema -import cwl.{Cwl, CwlDecoder, MyriadOutputType} -import io.circe.Json -import io.circe.syntax._ -import mouse.map._ -import shapeless.Poly1 -import spray.json.{JsObject, JsString, JsValue} - -object Outputs { - - //When the string returned is not valid JSON, it is effectively an exception as CWL runner expects JSON to be returned - def handleOutput(submittedWorkflow: SubmittedWorkflow, pathBuilder: PathBuilder): String = { - val metadata: Map[String, JsValue] = CentaurCromwellClient.metadata(submittedWorkflow).unsafeRunSync().asFlat.value - - // Wrapper function to provide the right signature for `intersectWith` below. - def outputResolver(schemaOption: Option[Schema])(jsValue: JsValue, mot: MyriadOutputType): Json = { - OutputManipulator.resolveOutput(jsValue, pathBuilder, mot, schemaOption) - } - - //Sorry for all the nesting, but spray json JsValue doesn't have optional access methods like Argonaut/circe, - //thus no for comprehensions for us :( - metadata.get("submittedFiles.workflow") match { - case Some(JsString(workflow)) => - - val parseCwl: IOChecked[Cwl] = CwlDecoder.decodeCwlString( - workflow, - submittedWorkflow.workflow.zippedImports, - submittedWorkflow.workflow.workflowRoot - ) - - parseCwl.value.attempt.unsafeRunSync() match { - case Right(Right(cwl)) => - - CentaurCromwellClient.outputs(submittedWorkflow).unsafeRunSync().outputs match { - case JsObject(map) => - val typeMap: Map[String, MyriadOutputType] = cwl.fold(CwlOutputsFold) - val mungeTypeMap = typeMap.mapKeys(stripTypeMapKey) - - val mungeOutputMap = map.mapKeys(stripOutputKey) - - mungeOutputMap. - //This lets us operate on the values of the output values and types for a particular output key - intersectWith(mungeTypeMap)(outputResolver(cwl.schemaOption)). - //converting the whole response to Json using Circe's auto-encoder derivation - asJson. - //drop null values so that we don't print when Option == None - printWith(io.circe.Printer.spaces2.copy(dropNullValues = true)) - case other => s"it seems cromwell is not returning outputs as a Jsobject but is instead a $other" - } - case Right(Left(error)) => s"couldn't parse workflow: $workflow failed with error: $error" - case Left(error) => s"Exception when trying to read workflow: $workflow failed with error: $error" - } - case Some(other) => s"received the value $other when the workflow string was expected" - case None => "the workflow is no longer in the metadata payload, it's a problem" - } - } - - //Ids come out of SALAD pre-processing with a filename prepended. This gets rid of it - // Also gets rid of the parent workflow name if present - def stripTypeMapKey(key: String): String = key - .substring(key.lastIndexOf("#") + 1, key.length) - .split("/").last - - //Ids come out of Cromwell with a prefix, separated by a ".". This takes everything to the right, - //as CWL wants it - def stripOutputKey(key: String): String = key.substring(key.lastIndexOf(".") + 1, key.length) - -} - -object CwlOutputsFold extends Poly1 { - import cwl._ - - implicit def wf: Case.Aux[cwl.Workflow, Map[String, MyriadOutputType]] = at[cwl.Workflow] { - _.outputs.map(output => output.id -> output.`type`.get).toMap - } - - implicit def clt: Case.Aux[cwl.CommandLineTool, Map[String, MyriadOutputType]] = at[cwl.CommandLineTool] { - _.outputs.map(output => output.id -> output.`type`.get).toMap - } - - implicit def et: Case.Aux[cwl.ExpressionTool, Map[String, MyriadOutputType]] = at[cwl.ExpressionTool] { - _.outputs.map(output => output.id -> output.`type`.get).toMap - } -} - diff --git a/centaurCwlRunner/src/test/resources/application.conf b/centaurCwlRunner/src/test/resources/application.conf deleted file mode 100644 index 27c24fa07bf..00000000000 --- a/centaurCwlRunner/src/test/resources/application.conf +++ /dev/null @@ -1 +0,0 @@ -papi.default-input-gcs-prefix="gs://centaur-cwl-conformance-1f501e3/cwl-inputs/" diff --git a/centaurCwlRunner/src/test/scala/CloudPreprocessorSpec.scala b/centaurCwlRunner/src/test/scala/CloudPreprocessorSpec.scala deleted file mode 100644 index df13e49b666..00000000000 --- a/centaurCwlRunner/src/test/scala/CloudPreprocessorSpec.scala +++ /dev/null @@ -1,342 +0,0 @@ -import centaur.cwl.CloudPreprocessor -import com.typesafe.config.ConfigFactory -import common.assertion.CromwellTimeoutSpec -import org.scalatest.flatspec.AnyFlatSpec -import org.scalatest.matchers.should.Matchers -import wom.util.YamlUtils - -class CloudPreprocessorSpec extends AnyFlatSpec with CromwellTimeoutSpec with Matchers { - behavior of "PAPIPreProcessor" - - val pAPIPreprocessor = new CloudPreprocessor(ConfigFactory.load(), "papi.default-input-gcs-prefix") - - def validate(result: String, expectation: String) = { - val parsedResult = YamlUtils.parse(result).toOption.get - val parsedExpectation = YamlUtils.parse(expectation).toOption.get - - // This is an actual Json comparison from circe - parsedResult shouldBe parsedExpectation - } - - it should "prefix files and directories in inputs" in { - validate( - pAPIPreprocessor.preProcessInput( - """|{ - | "input": { - | "null": null, - | "file": { - | "location": "whale.txt", - | "class": "File", - | "secondaryFiles": [ - | { - | "class": File, - | "location": "hello.txt" - | } - | ], - | "default": { - | "location": "default_whale.txt", - | "class": "File", - | } - | }, - | "directory": { - | "location": "ref.fasta", - | "class": "Directory", - | "listing": [ - | { - | "class": File, - | "location": "hello.txt" - | } - | ] - | } - | } - |} - |""".stripMargin).value.unsafeRunSync().toOption.get, - """|{ - | "input": { - | "null": null, - | "file": { - | "location": "gs://centaur-cwl-conformance-1f501e3/cwl-inputs/whale.txt", - | "class": "File", - | "secondaryFiles": [ - | { - | "class": File, - | "location": "gs://centaur-cwl-conformance-1f501e3/cwl-inputs/hello.txt" - | } - | ], - | "default": { - | "location": "gs://centaur-cwl-conformance-1f501e3/cwl-inputs/default_whale.txt", - | "class": "File", - | } - | }, - | "directory": { - | "location": "gs://centaur-cwl-conformance-1f501e3/cwl-inputs/ref.fasta", - | "class": "Directory", - | "listing": [ - | { - | "class": File, - | "location": "gs://centaur-cwl-conformance-1f501e3/cwl-inputs/hello.txt" - | } - | ] - | } - | } - |} - |""".stripMargin - ) - } - - it should "prefix files and directories in workflow" in { - validate( - pAPIPreprocessor.preProcessWorkflow( - """|class: CommandLineTool - |cwlVersion: v1.0 - |requirements: - | - class: DockerRequirement - | dockerPull: ubuntu:latest - |inputs: - | - id: reference - | type: File - | inputBinding: { position: 2 } - | default: - | class: File - | location: args.py - | - |outputs: - | args: string[] - | - |baseCommand: python - |arguments: ["bwa", "mem"] - |""".stripMargin), - """|class: CommandLineTool - |cwlVersion: v1.0 - |requirements: - | - class: DockerRequirement - | dockerPull: ubuntu:latest - |inputs: - | - id: reference - | type: File - | inputBinding: { position: 2 } - | default: - | class: File - | location: gs://centaur-cwl-conformance-1f501e3/cwl-inputs/args.py - | - |outputs: - | args: string[] - | - |baseCommand: python - |arguments: ["bwa", "mem"] - |""".stripMargin - ) - } - - it should "add default docker image if there's no hint" in { - validate( - pAPIPreprocessor.preProcessWorkflow( - """|class: CommandLineTool - |cwlVersion: v1.0 - |inputs: - | - id: reference - | type: File - | inputBinding: { position: 2 } - | - |outputs: - | args: string[] - | - |baseCommand: python - |arguments: ["bwa", "mem"] - |""".stripMargin), - """|class: CommandLineTool - |hints: - | - class: DockerRequirement - | dockerPull: ubuntu:latest - |cwlVersion: v1.0 - |inputs: - | - id: reference - | type: File - | inputBinding: { position: 2 } - | - |outputs: - | args: string[] - | - |baseCommand: python - |arguments: ["bwa", "mem"] - |""".stripMargin - ) - } - - it should "append default docker image to existing hint as an array" in { - validate( - pAPIPreprocessor.preProcessWorkflow( - """|class: CommandLineTool - |hints: - | - class: EnvVarRequirement - | envDef: - | TEST_ENV: $(inputs.in) - |cwlVersion: v1.0 - |inputs: - | - id: reference - | type: File - | inputBinding: { position: 2 } - | - |outputs: - | args: string[] - | - |baseCommand: python - |arguments: ["bwa", "mem"] - |""".stripMargin), - """|class: CommandLineTool - |hints: - | - class: EnvVarRequirement - | envDef: - | TEST_ENV: $(inputs.in) - | - class: DockerRequirement - | dockerPull: ubuntu:latest - |cwlVersion: v1.0 - |inputs: - | - id: reference - | type: File - | inputBinding: { position: 2 } - | - |outputs: - | args: string[] - | - |baseCommand: python - |arguments: ["bwa", "mem"] - |""".stripMargin - ) - } - - it should "append default docker image to existing hints as an object" in { - validate( - pAPIPreprocessor.preProcessWorkflow( - """|class: CommandLineTool - |cwlVersion: v1.0 - |inputs: - | in: string - |outputs: - | out: - | type: File - | outputBinding: - | glob: out - | - |hints: - | EnvVarRequirement: - | envDef: - | TEST_ENV: $(inputs.in) - | - |baseCommand: ["/bin/bash", "-c", "echo $TEST_ENV"] - | - |stdout: out - |""".stripMargin), - """|class: CommandLineTool - |cwlVersion: v1.0 - |inputs: - | in: string - |outputs: - | out: - | type: File - | outputBinding: - | glob: out - | - |hints: - | EnvVarRequirement: - | envDef: - | TEST_ENV: $(inputs.in) - | DockerRequirement: - | dockerPull: ubuntu:latest - | - |baseCommand: ["/bin/bash", "-c", "echo $TEST_ENV"] - | - |stdout: out - |""".stripMargin - ) - } - - it should "not replace existing docker requirement in an object" in { - val workflow = """|class: CommandLineTool - |cwlVersion: v1.0 - |requirements: - | DockerRequirement: - | dockerPull: python27:slim - |inputs: - | - id: reference - | type: File - | inputBinding: { position: 2 } - | - |outputs: - | args: string[] - | - |baseCommand: python - |arguments: ["bwa", "mem"] - |""".stripMargin - validate(pAPIPreprocessor.preProcessWorkflow(workflow), workflow) - } - - it should "not replace existing docker hint in an object" in { - val workflow = """|class: CommandLineTool - |cwlVersion: v1.0 - |hints: - | DockerRequirement: - | dockerPull: python27:slim - |inputs: - | - id: reference - | type: File - | inputBinding: { position: 2 } - | - |outputs: - | args: string[] - | - |baseCommand: python - |arguments: ["bwa", "mem"] - |""".stripMargin - validate(pAPIPreprocessor.preProcessWorkflow(workflow), workflow) - } - - it should "not replace existing docker hint in an array" in { - val workflow = """|class: CommandLineTool - |cwlVersion: v1.0 - |hints: - |- class: DockerRequirement - | dockerPull: python27:slim - |inputs: - | - id: reference - | type: File - | inputBinding: { position: 2 } - | - |outputs: - | args: string[] - | - |baseCommand: python - |arguments: ["bwa", "mem"] - |""".stripMargin - validate(pAPIPreprocessor.preProcessWorkflow(workflow), workflow) - } - - it should "not replace existing docker requirement in an array" in { - val workflow = """|class: CommandLineTool - |cwlVersion: v1.0 - |requirements: - |- class: DockerRequirement - | dockerPull: python27:slim - |inputs: - | - id: reference - | type: File - | inputBinding: { position: 2 } - | - |outputs: - | args: string[] - | - |baseCommand: python - |arguments: ["bwa", "mem"] - |""".stripMargin - validate(pAPIPreprocessor.preProcessWorkflow(workflow), workflow) - } - - it should "throw an exception if yaml / json can't be parse" in { - val invalid = - """ - |{ [invalid]: } - """.stripMargin - - an[Exception] shouldBe thrownBy(pAPIPreprocessor.preProcessWorkflow(invalid)) - } -} diff --git a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCloudNioFileProvider.scala b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCloudNioFileProvider.scala index a1156fc33e9..93f4cf77d34 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCloudNioFileProvider.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCloudNioFileProvider.scala @@ -15,7 +15,7 @@ import org.apache.http.HttpStatus class DrsCloudNioFileProvider(drsPathResolver: EngineDrsPathResolver, drsReadInterpreter: DrsReadInterpreter) extends CloudNioFileProvider { - private def checkIfPathExistsThroughMartha(drsPath: String): IO[Boolean] = { + private def checkIfPathExistsThroughDrsResolver(drsPath: String): IO[Boolean] = { /* * Unlike other cloud providers where directories are identified with a trailing slash at the end like `gs://bucket/dir/`, * DRS has a concept of bundles for directories (not supported yet). Hence for method `checkDirectoryExists` which appends a trailing '/' @@ -23,15 +23,15 @@ class DrsCloudNioFileProvider(drsPathResolver: EngineDrsPathResolver, */ if (drsPath.endsWith("/")) IO(false) else { - drsPathResolver.rawMarthaResponse(drsPath, NonEmptyList.one(MarthaField.GsUri)).use { marthaResponse => - val errorMsg = s"Status line was null for martha response $marthaResponse." - toIO(Option(marthaResponse.getStatusLine), errorMsg) + drsPathResolver.rawDrsResolverResponse(drsPath, NonEmptyList.one(DrsResolverField.GsUri)).use { drsResolverResponse => + val errorMsg = s"Status line was null for DRS Resolver response $drsResolverResponse." + toIO(Option(drsResolverResponse.getStatusLine), errorMsg) }.map(_.getStatusCode == HttpStatus.SC_OK) } } override def existsPath(drsPath: String, unused: String): Boolean = - checkIfPathExistsThroughMartha(drsPath).unsafeRunSync() + checkIfPathExistsThroughDrsResolver(drsPath).unsafeRunSync() override def existsPaths(cloudHost: String, cloudPathPrefix: String): Boolean = existsPath(cloudHost, cloudPathPrefix) @@ -47,11 +47,11 @@ class DrsCloudNioFileProvider(drsPathResolver: EngineDrsPathResolver, throw new UnsupportedOperationException("DRS currently doesn't support delete.") override def read(drsPath: String, unused: String, offset: Long): ReadableByteChannel = { - val fields = NonEmptyList.of(MarthaField.GsUri, MarthaField.GoogleServiceAccount, MarthaField.AccessUrl) + val fields = NonEmptyList.of(DrsResolverField.GsUri, DrsResolverField.GoogleServiceAccount, DrsResolverField.AccessUrl) val byteChannelIO = for { - marthaResponse <- drsPathResolver.resolveDrsThroughMartha(drsPath, fields) - byteChannel <- drsReadInterpreter(drsPathResolver, marthaResponse) + drsResolverResponse <- drsPathResolver.resolveDrs(drsPath, fields) + byteChannel <- drsReadInterpreter(drsPathResolver, drsResolverResponse) } yield byteChannel byteChannelIO.handleErrorWith { @@ -63,14 +63,14 @@ class DrsCloudNioFileProvider(drsPathResolver: EngineDrsPathResolver, throw new UnsupportedOperationException("DRS currently doesn't support write.") override def fileAttributes(drsPath: String, unused: String): Option[CloudNioRegularFileAttributes] = { - val fields = NonEmptyList.of(MarthaField.Size, MarthaField.TimeCreated, MarthaField.TimeUpdated, MarthaField.Hashes) + val fields = NonEmptyList.of(DrsResolverField.Size, DrsResolverField.TimeCreated, DrsResolverField.TimeUpdated, DrsResolverField.Hashes) val fileAttributesIO = for { - marthaResponse <- drsPathResolver.resolveDrsThroughMartha(drsPath, fields) - sizeOption = marthaResponse.size - hashOption = getPreferredHash(marthaResponse.hashes) - timeCreatedOption <- convertToFileTime(drsPath, MarthaField.TimeCreated, marthaResponse.timeCreated) - timeUpdatedOption <- convertToFileTime(drsPath, MarthaField.TimeUpdated, marthaResponse.timeUpdated) + drsResolverResponse <- drsPathResolver.resolveDrs(drsPath, fields) + sizeOption = drsResolverResponse.size + hashOption = getPreferredHash(drsResolverResponse.hashes) + timeCreatedOption <- convertToFileTime(drsPath, DrsResolverField.TimeCreated, drsResolverResponse.timeCreated) + timeUpdatedOption <- convertToFileTime(drsPath, DrsResolverField.TimeUpdated, drsResolverResponse.timeUpdated) } yield new DrsCloudNioRegularFileAttributes(drsPath, sizeOption, hashOption, timeCreatedOption, timeUpdatedOption) Option(fileAttributesIO.unsafeRunSync()) @@ -78,5 +78,5 @@ class DrsCloudNioFileProvider(drsPathResolver: EngineDrsPathResolver, } object DrsCloudNioFileProvider { - type DrsReadInterpreter = (DrsPathResolver, MarthaResponse) => IO[ReadableByteChannel] + type DrsReadInterpreter = (DrsPathResolver, DrsResolverResponse) => IO[ReadableByteChannel] } diff --git a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCloudNioFileSystemProvider.scala b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCloudNioFileSystemProvider.scala index 8721ab9b43e..884072e4a31 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCloudNioFileSystemProvider.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCloudNioFileSystemProvider.scala @@ -11,7 +11,8 @@ class DrsCloudNioFileSystemProvider(rootConfig: Config, drsReadInterpreter: DrsReadInterpreter, ) extends CloudNioFileSystemProvider { - lazy val drsConfig: DrsConfig = DrsConfig.fromConfig(rootConfig.getConfig("martha")) + lazy val drsResolverConfig = if (rootConfig.hasPath("resolver")) rootConfig.getConfig("resolver") else rootConfig.getConfig("martha") + lazy val drsConfig: DrsConfig = DrsConfig.fromConfig(drsResolverConfig) lazy val drsPathResolver: EngineDrsPathResolver = EngineDrsPathResolver(drsConfig, drsCredentials) diff --git a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCloudNioRegularFileAttributes.scala b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCloudNioRegularFileAttributes.scala index 3c9293f2bd7..778d9b8380e 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCloudNioRegularFileAttributes.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCloudNioRegularFileAttributes.scala @@ -61,7 +61,7 @@ object DrsCloudNioRegularFileAttributes { .map(FileTime.from) } - def convertToFileTime(drsPath: String, key: MarthaField.Value, timeInStringOption: Option[String]): IO[Option[FileTime]] = { + def convertToFileTime(drsPath: String, key: DrsResolverField.Value, timeInStringOption: Option[String]): IO[Option[FileTime]] = { timeInStringOption match { case None => IO.pure(None) case Some(timeInString) => @@ -71,7 +71,7 @@ object DrsCloudNioRegularFileAttributes { throwable => IO.raiseError( new RuntimeException( - s"Error while parsing '$key' value from Martha to FileTime for DRS path $drsPath. " + + s"Error while parsing '$key' value from DRS Resolver to FileTime for DRS path $drsPath. " + s"Reason: ${ExceptionUtils.getMessage(throwable)}.", throwable, ) diff --git a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsConfig.scala b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsConfig.scala index e9dc760162e..c8333a57a66 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsConfig.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsConfig.scala @@ -6,7 +6,7 @@ import net.ceedubs.ficus.Ficus._ import scala.concurrent.duration._ import scala.language.postfixOps -final case class DrsConfig(marthaUrl: String, +final case class DrsConfig(drsResolverUrl: String, numRetries: Int, waitInitial: FiniteDuration, waitMaximum: FiniteDuration, @@ -22,45 +22,46 @@ object DrsConfig { private val DefaultWaitMultiplier = 1.5d private val DefaultWaitRandomizationFactor = 0.1 - private val EnvMarthaUrl = "MARTHA_URL" - private val EnvMarthaNumRetries = "MARTHA_NUM_RETRIES" - private val EnvMarthaWaitInitialSeconds = "MARTHA_WAIT_INITIAL_SECONDS" - private val EnvMarthaWaitMaximumSeconds = "MARTHA_WAIT_MAXIMUM_SECONDS" - private val EnvMarthaWaitMultiplier = "MARTHA_WAIT_MULTIPLIER" - private val EnvMarthaWaitRandomizationFactor = "MARTHA_WAIT_RANDOMIZATION_FACTOR" + private val EnvDrsResolverUrl = "DRS_RESOLVER_URL" + private val EnvDrsResolverNumRetries = "DRS_RESOLVER_NUM_RETRIES" + private val EnvDrsResolverWaitInitialSeconds = "DRS_RESOLVER_WAIT_INITIAL_SECONDS" + private val EnvDrsResolverWaitMaximumSeconds = "DRS_RESOLVER_WAIT_MAXIMUM_SECONDS" + private val EnvDrsResolverWaitMultiplier = "DRS_RESOLVER_WAIT_MULTIPLIER" + private val EnvDrsResolverWaitRandomizationFactor = "DRS_RESOLVER_WAIT_RANDOMIZATION_FACTOR" - def fromConfig(marthaConfig: Config): DrsConfig = { + + def fromConfig(drsResolverConfig: Config): DrsConfig = { DrsConfig( - marthaUrl = marthaConfig.getString("url"), - numRetries = marthaConfig.getOrElse("num-retries", DefaultNumRetries), - waitInitial = marthaConfig.getOrElse("wait-initial", DefaultWaitInitial), - waitMaximum = marthaConfig.getOrElse("wait-maximum", DefaultWaitMaximum), - waitMultiplier = marthaConfig.getOrElse("wait-multiplier", DefaultWaitMultiplier), + drsResolverUrl = drsResolverConfig.getString("url"), + numRetries = drsResolverConfig.getOrElse("num-retries", DefaultNumRetries), + waitInitial = drsResolverConfig.getOrElse("wait-initial", DefaultWaitInitial), + waitMaximum = drsResolverConfig.getOrElse("wait-maximum", DefaultWaitMaximum), + waitMultiplier = drsResolverConfig.getOrElse("wait-multiplier", DefaultWaitMultiplier), waitRandomizationFactor = - marthaConfig.getOrElse("wait-randomization-factor", DefaultWaitRandomizationFactor), + drsResolverConfig.getOrElse("wait-randomization-factor", DefaultWaitRandomizationFactor), ) } def fromEnv(env: Map[String, String]): DrsConfig = { DrsConfig( - marthaUrl = env(EnvMarthaUrl), - numRetries = env.get(EnvMarthaNumRetries).map(_.toInt).getOrElse(DefaultNumRetries), - waitInitial = env.get(EnvMarthaWaitInitialSeconds).map(_.toLong.seconds).getOrElse(DefaultWaitInitial), - waitMaximum = env.get(EnvMarthaWaitMaximumSeconds).map(_.toLong.seconds).getOrElse(DefaultWaitMaximum), - waitMultiplier = env.get(EnvMarthaWaitMultiplier).map(_.toDouble).getOrElse(DefaultWaitMultiplier), + drsResolverUrl = env(EnvDrsResolverUrl), + numRetries = env.get(EnvDrsResolverNumRetries).map(_.toInt).getOrElse(DefaultNumRetries), + waitInitial = env.get(EnvDrsResolverWaitInitialSeconds).map(_.toLong.seconds).getOrElse(DefaultWaitInitial), + waitMaximum = env.get(EnvDrsResolverWaitMaximumSeconds).map(_.toLong.seconds).getOrElse(DefaultWaitMaximum), + waitMultiplier = env.get(EnvDrsResolverWaitMultiplier).map(_.toDouble).getOrElse(DefaultWaitMultiplier), waitRandomizationFactor = - env.get(EnvMarthaWaitRandomizationFactor).map(_.toDouble).getOrElse(DefaultWaitRandomizationFactor), + env.get(EnvDrsResolverWaitRandomizationFactor).map(_.toDouble).getOrElse(DefaultWaitRandomizationFactor), ) } def toEnv(drsConfig: DrsConfig): Map[String, String] = { Map( - EnvMarthaUrl -> drsConfig.marthaUrl, - EnvMarthaNumRetries -> s"${drsConfig.numRetries}", - EnvMarthaWaitInitialSeconds -> s"${drsConfig.waitInitial.toSeconds}", - EnvMarthaWaitMaximumSeconds -> s"${drsConfig.waitMaximum.toSeconds}", - EnvMarthaWaitMultiplier -> s"${drsConfig.waitMultiplier}", - EnvMarthaWaitRandomizationFactor -> s"${drsConfig.waitRandomizationFactor}", + EnvDrsResolverUrl -> drsConfig.drsResolverUrl, + EnvDrsResolverNumRetries -> s"${drsConfig.numRetries}", + EnvDrsResolverWaitInitialSeconds -> s"${drsConfig.waitInitial.toSeconds}", + EnvDrsResolverWaitMaximumSeconds -> s"${drsConfig.waitMaximum.toSeconds}", + EnvDrsResolverWaitMultiplier -> s"${drsConfig.waitMultiplier}", + EnvDrsResolverWaitRandomizationFactor -> s"${drsConfig.waitRandomizationFactor}", ) } } diff --git a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCredentials.scala b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCredentials.scala index 66a89fe4014..2d3e972508a 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCredentials.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsCredentials.scala @@ -1,25 +1,32 @@ package cloud.nio.impl.drs import cats.syntax.validated._ +import com.azure.core.credential.TokenRequestContext +import com.azure.core.management.AzureEnvironment +import com.azure.core.management.profile.AzureProfile import com.azure.identity.DefaultAzureCredentialBuilder -import com.azure.security.keyvault.secrets.{SecretClient, SecretClientBuilder} -import com.google.auth.oauth2.{AccessToken, OAuth2Credentials} +import com.google.auth.oauth2.{AccessToken, GoogleCredentials, OAuth2Credentials} import com.typesafe.config.Config -import common.validation.ErrorOr import common.validation.ErrorOr.ErrorOr import net.ceedubs.ficus.Ficus._ import scala.concurrent.duration._ +import scala.jdk.DurationConverters._ +import scala.util.{Failure, Success, Try} /** * This trait allows us to abstract away different token attainment strategies * for different cloud environments. **/ -sealed trait DrsCredentials { +trait DrsCredentials { def getAccessToken: ErrorOr[String] } -case class GoogleDrsCredentials(credentials: OAuth2Credentials, acceptableTTL: Duration) extends DrsCredentials { +/** + * Strategy for obtaining an access token from an existing OAuth credential. This class + * is designed for use within the Cromwell engine. + */ +case class GoogleOauthDrsCredentials(credentials: OAuth2Credentials, acceptableTTL: Duration) extends DrsCredentials { //Based on method from GoogleRegistry def getAccessToken: ErrorOr[String] = { def accessTokenTTLIsAcceptable(accessToken: AccessToken): Boolean = { @@ -39,25 +46,68 @@ case class GoogleDrsCredentials(credentials: OAuth2Credentials, acceptableTTL: D } } -object GoogleDrsCredentials { - def apply(credentials: OAuth2Credentials, config: Config): GoogleDrsCredentials = - GoogleDrsCredentials(credentials, config.as[FiniteDuration]("access-token-acceptable-ttl")) +object GoogleOauthDrsCredentials { + def apply(credentials: OAuth2Credentials, config: Config): GoogleOauthDrsCredentials = + GoogleOauthDrsCredentials(credentials, config.as[FiniteDuration]("access-token-acceptable-ttl")) +} + + +/** + * Strategy for obtaining an access token from Google Application Default credentials that are assumed to already exist + * in the environment. This class is designed for use by standalone executables running in environments + * that have direct access to a Google identity (ex. CromwellDrsLocalizer). + */ +case object GoogleAppDefaultTokenStrategy extends DrsCredentials { + private final val UserInfoEmailScope = "https://www.googleapis.com/auth/userinfo.email" + private final val UserInfoProfileScope = "https://www.googleapis.com/auth/userinfo.profile" + + def getAccessToken: ErrorOr[String] = { + Try { + val scopedCredentials = GoogleCredentials.getApplicationDefault().createScoped(UserInfoEmailScope, UserInfoProfileScope) + scopedCredentials.refreshAccessToken().getTokenValue + } match { + case Success(null) => "null token value attempting to refresh access token".invalidNel + case Success(value) => value.validNel + case Failure(e) => s"Failed to refresh access token: ${e.getMessage}".invalidNel + } + } } -case class AzureDrsCredentials(identityClientId: Option[String], vaultName: String, secretName: String) extends DrsCredentials { +/** + * Strategy for obtaining an access token in an environment with available Azure identity. + * If you need to disambiguate among multiple active user-assigned managed identities, pass + * in the client id of the identity that should be used. + */ +case class AzureDrsCredentials(identityClientId: Option[String]) extends DrsCredentials { + + final val tokenAcquisitionTimeout = 5.seconds - lazy val secretClient: ErrorOr[SecretClient] = ErrorOr { - val defaultCreds = identityClientId.map(identityId => - new DefaultAzureCredentialBuilder().managedIdentityClientId(identityId) - ).getOrElse( - new DefaultAzureCredentialBuilder() - ).build() + val azureProfile = new AzureProfile(AzureEnvironment.AZURE) + val tokenScope = "https://management.azure.com/.default" - new SecretClientBuilder() - .vaultUrl(s"https://${vaultName}.vault.azure.net") - .credential(defaultCreds) - .buildClient() + def tokenRequestContext: TokenRequestContext = { + val trc = new TokenRequestContext() + trc.addScopes(tokenScope) + trc } - def getAccessToken: ErrorOr[String] = secretClient.map(_.getSecret(secretName).getValue) + def defaultCredentialBuilder: DefaultAzureCredentialBuilder = + new DefaultAzureCredentialBuilder() + .authorityHost(azureProfile.getEnvironment.getActiveDirectoryEndpoint) + + def getAccessToken: ErrorOr[String] = { + val credentials = identityClientId.foldLeft(defaultCredentialBuilder) { + (builder, clientId) => builder.managedIdentityClientId(clientId) + }.build() + + Try( + credentials + .getToken(tokenRequestContext) + .block(tokenAcquisitionTimeout.toJava) + ) match { + case Success(null) => "null token value attempting to obtain access token".invalidNel + case Success(token) => token.getToken.validNel + case Failure(error) => s"Failed to refresh access token: ${error.getMessage}".invalidNel + } + } } diff --git a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsPathResolver.scala b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsPathResolver.scala index 9e0b45e259e..f9ae5b62e03 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsPathResolver.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsPathResolver.scala @@ -5,7 +5,7 @@ import cats.data.Validated.{Invalid, Valid} import cats.effect.{IO, Resource} import cats.implicits._ import cloud.nio.impl.drs.DrsPathResolver.{FatalRetryDisposition, RegularRetryDisposition} -import cloud.nio.impl.drs.MarthaResponseSupport._ +import cloud.nio.impl.drs.DrsResolverResponseSupport._ import common.exception.{AggregatedMessageException, toIO} import common.validation.ErrorOr.ErrorOr import io.circe._ @@ -29,7 +29,7 @@ abstract class DrsPathResolver(drsConfig: DrsConfig, retryInternally: Boolean = protected lazy val httpClientBuilder: HttpClientBuilder = { val clientBuilder = HttpClientBuilder.create() if (retryInternally) { - val retryHandler = new MarthaHttpRequestRetryStrategy(drsConfig) + val retryHandler = new DrsResolverHttpRequestRetryStrategy(drsConfig) clientBuilder .setRetryHandler(retryHandler) .setServiceUnavailableRetryStrategy(retryHandler) @@ -39,11 +39,11 @@ abstract class DrsPathResolver(drsConfig: DrsConfig, retryInternally: Boolean = def getAccessToken: ErrorOr[String] - private def makeHttpRequestToMartha(drsPath: String, fields: NonEmptyList[MarthaField.Value]): Resource[IO, HttpPost] = { + private def makeHttpRequestToDrsResolver(drsPath: String, fields: NonEmptyList[DrsResolverField.Value]): Resource[IO, HttpPost] = { val io = getAccessToken match { case Valid(token) => IO { - val postRequest = new HttpPost(drsConfig.marthaUrl) - val requestJson = MarthaRequest(drsPath, fields).asJson.noSpaces + val postRequest = new HttpPost(drsConfig.drsResolverUrl) + val requestJson = DrsResolverRequest(drsPath, fields).asJson.noSpaces postRequest.setEntity(new StringEntity(requestJson, ContentType.APPLICATION_JSON)) postRequest.setHeader("Authorization", s"Bearer $token") postRequest @@ -54,7 +54,7 @@ abstract class DrsPathResolver(drsConfig: DrsConfig, retryInternally: Boolean = Resource.eval(io) } - private def httpResponseToMarthaResponse(drsPathForDebugging: String)(httpResponse: HttpResponse): IO[MarthaResponse] = { + private def httpResponseToDrsResolverResponse(drsPathForDebugging: String)(httpResponse: HttpResponse): IO[DrsResolverResponse] = { val responseStatusLine = httpResponse.getStatusLine val status = responseStatusLine.getStatusCode @@ -71,39 +71,39 @@ abstract class DrsPathResolver(drsConfig: DrsConfig, retryInternally: Boolean = case s if s / 100 == 5 => IO.raiseError(new RuntimeException(retryMessage) with RegularRetryDisposition) case _ => - val marthaResponseEntityOption = Option(httpResponse.getEntity).map(EntityUtils.toString) - val exceptionMsg = errorMessageFromResponse(drsPathForDebugging, marthaResponseEntityOption, responseStatusLine, drsConfig.marthaUrl) - val responseEntityOption = (responseStatusLine.getStatusCode == HttpStatus.SC_OK).valueOrZero(marthaResponseEntityOption) + val drsResolverResponseEntityOption = Option(httpResponse.getEntity).map(EntityUtils.toString) + val exceptionMsg = errorMessageFromResponse(drsPathForDebugging, drsResolverResponseEntityOption, responseStatusLine, drsConfig.drsResolverUrl) + val responseEntityOption = (responseStatusLine.getStatusCode == HttpStatus.SC_OK).valueOrZero(drsResolverResponseEntityOption) val responseContentIO = toIO(responseEntityOption, exceptionMsg) responseContentIO.flatMap { responseContent => - IO.fromEither(decode[MarthaResponse](responseContent)) + IO.fromEither(decode[DrsResolverResponse](responseContent)) }.handleErrorWith { e => IO.raiseError(new RuntimeException(s"Unexpected response during DRS resolution: ${ExceptionUtils.getMessage(e)}")) } } } - private def executeMarthaRequest(httpPost: HttpPost): Resource[IO, HttpResponse]= { + private def executeDrsResolverRequest(httpPost: HttpPost): Resource[IO, HttpResponse]= { for { httpClient <- Resource.fromAutoCloseable(IO(httpClientBuilder.build())) httpResponse <- Resource.fromAutoCloseable(IO(httpClient.execute(httpPost))) } yield httpResponse } - def rawMarthaResponse(drsPath: String, fields: NonEmptyList[MarthaField.Value]): Resource[IO, HttpResponse] = { + def rawDrsResolverResponse(drsPath: String, fields: NonEmptyList[DrsResolverField.Value]): Resource[IO, HttpResponse] = { for { - httpPost <- makeHttpRequestToMartha(drsPath, fields) - response <- executeMarthaRequest(httpPost) + httpPost <- makeHttpRequestToDrsResolver(drsPath, fields) + response <- executeDrsResolverRequest(httpPost) } yield response } /** * - * Resolves the DRS path through Martha url provided in the config. - * Please note, this method returns an IO that would make a synchronous HTTP request to Martha when run. + * Resolves the DRS path through DRS Resolver url provided in the config. + * Please note, this method returns an IO that would make a synchronous HTTP request to DRS Resolver when run. */ - def resolveDrsThroughMartha(drsPath: String, fields: NonEmptyList[MarthaField.Value]): IO[MarthaResponse] = { - rawMarthaResponse(drsPath, fields).use(httpResponseToMarthaResponse(drsPathForDebugging = drsPath)) + def resolveDrs(drsPath: String, fields: NonEmptyList[DrsResolverField.Value]): IO[DrsResolverResponse] = { + rawDrsResolverResponse(drsPath, fields).use(httpResponseToDrsResolverResponse(drsPathForDebugging = drsPath)) } def openChannel(accessUrl: AccessUrl): IO[ReadableByteChannel] = { @@ -144,7 +144,7 @@ abstract class DrsPathResolver(drsConfig: DrsConfig, retryInternally: Boolean = } object DrsPathResolver { - final val ExtractUriErrorMsg = "No access URL nor GCS URI starting with 'gs://' found in Martha response!" + final val ExtractUriErrorMsg = "No access URL nor GCS URI starting with 'gs://' found in the DRS Resolver response!" sealed trait RetryDisposition // Should immediately fail the download attempt. trait FatalRetryDisposition extends RetryDisposition @@ -152,27 +152,27 @@ object DrsPathResolver { trait RegularRetryDisposition extends RetryDisposition } -object MarthaField extends Enumeration { - val GsUri: MarthaField.Value = Value("gsUri") - val Size: MarthaField.Value = Value("size") - val TimeCreated: MarthaField.Value = Value("timeCreated") - val TimeUpdated: MarthaField.Value = Value("timeUpdated") - val BondProvider: MarthaField.Value = Value("bondProvider") - val GoogleServiceAccount: MarthaField.Value = Value("googleServiceAccount") - val Hashes: MarthaField.Value = Value("hashes") - val FileName: MarthaField.Value = Value("fileName") - val AccessUrl: MarthaField.Value = Value("accessUrl") - val LocalizationPath: MarthaField.Value = Value("localizationPath") +object DrsResolverField extends Enumeration { + val GsUri: DrsResolverField.Value = Value("gsUri") + val Size: DrsResolverField.Value = Value("size") + val TimeCreated: DrsResolverField.Value = Value("timeCreated") + val TimeUpdated: DrsResolverField.Value = Value("timeUpdated") + val BondProvider: DrsResolverField.Value = Value("bondProvider") + val GoogleServiceAccount: DrsResolverField.Value = Value("googleServiceAccount") + val Hashes: DrsResolverField.Value = Value("hashes") + val FileName: DrsResolverField.Value = Value("fileName") + val AccessUrl: DrsResolverField.Value = Value("accessUrl") + val LocalizationPath: DrsResolverField.Value = Value("localizationPath") } -final case class MarthaRequest(url: String, fields: NonEmptyList[MarthaField.Value]) +final case class DrsResolverRequest(url: String, fields: NonEmptyList[DrsResolverField.Value]) final case class SADataObject(data: Json) final case class AccessUrl(url: String, headers: Option[Map[String, String]]) /** - * A response from `martha_v3`. + * A response from `drshub_v4`. * * @param size Size of the object stored at gsUri * @param timeCreated The creation time of the object at gsUri @@ -187,34 +187,33 @@ final case class AccessUrl(url: String, headers: Option[Map[String, String]]) * DRS metadata, via the `aliases` field. As this is a distinct field from `fileName` in DRS * metadata it is also made a distinct field in this response object. */ -final case class MarthaResponse(size: Option[Long] = None, - timeCreated: Option[String] = None, - timeUpdated: Option[String] = None, - gsUri: Option[String] = None, - bondProvider: Option[String] = None, - googleServiceAccount: Option[SADataObject] = None, - fileName: Option[String] = None, - hashes: Option[Map[String, String]] = None, - accessUrl: Option[AccessUrl] = None, - localizationPath: Option[String] = None +final case class DrsResolverResponse(size: Option[Long] = None, + timeCreated: Option[String] = None, + timeUpdated: Option[String] = None, + gsUri: Option[String] = None, + bondProvider: Option[String] = None, + googleServiceAccount: Option[SADataObject] = None, + fileName: Option[String] = None, + hashes: Option[Map[String, String]] = None, + accessUrl: Option[AccessUrl] = None, + localizationPath: Option[String] = None ) -// Adapted from https://github.com/broadinstitute/martha/blob/f31933a3a11e20d30698ec4b4dc1e0abbb31a8bc/common/helpers.js#L210-L218 -final case class MarthaFailureResponse(response: MarthaFailureResponsePayload) -final case class MarthaFailureResponsePayload(text: String) +final case class DrsResolverFailureResponse(response: DrsResolverFailureResponsePayload) +final case class DrsResolverFailureResponsePayload(text: String) -object MarthaResponseSupport { +object DrsResolverResponseSupport { - implicit lazy val marthaFieldEncoder: Encoder[MarthaField.Value] = Encoder.encodeEnumeration(MarthaField) - implicit lazy val marthaRequestEncoder: Encoder[MarthaRequest] = deriveEncoder + implicit lazy val drsResolverFieldEncoder: Encoder[DrsResolverField.Value] = Encoder.encodeEnumeration(DrsResolverField) + implicit lazy val drsResolverRequestEncoder: Encoder[DrsResolverRequest] = deriveEncoder implicit lazy val saDataObjectDecoder: Decoder[SADataObject] = deriveDecoder - implicit lazy val marthaResponseDecoder: Decoder[MarthaResponse] = deriveDecoder + implicit lazy val drsResolverResponseDecoder: Decoder[DrsResolverResponse] = deriveDecoder - implicit lazy val marthaFailureResponseDecoder: Decoder[MarthaFailureResponse] = deriveDecoder - implicit lazy val marthaFailureResponsePayloadDecoder: Decoder[MarthaFailureResponsePayload] = deriveDecoder + implicit lazy val drsResolverFailureResponseDecoder: Decoder[DrsResolverFailureResponse] = deriveDecoder + implicit lazy val drsResolverFailureResponsePayloadDecoder: Decoder[DrsResolverFailureResponsePayload] = deriveDecoder - implicit lazy val marthaAccessUrlDecoder: Decoder[AccessUrl] = deriveDecoder + implicit lazy val drsResolverAccessUrlDecoder: Decoder[AccessUrl] = deriveDecoder private val GcsScheme = "gs://" @@ -223,18 +222,18 @@ object MarthaResponseSupport { (array(0), array(1)) } - def errorMessageFromResponse(drsPathForDebugging: String, marthaResponseEntityOption: Option[String], responseStatusLine: StatusLine, marthaUri: String): String = { - val baseMessage = s"Could not access object \'$drsPathForDebugging\'. Status: ${responseStatusLine.getStatusCode}, reason: \'${responseStatusLine.getReasonPhrase}\', Martha location: \'$marthaUri\', message: " + def errorMessageFromResponse(drsPathForDebugging: String, drsResolverResponseEntityOption: Option[String], responseStatusLine: StatusLine, drsResolverUri: String): String = { + val baseMessage = s"Could not access object \'$drsPathForDebugging\'. Status: ${responseStatusLine.getStatusCode}, reason: \'${responseStatusLine.getReasonPhrase}\', DRS Resolver location: \'$drsResolverUri\', message: " - marthaResponseEntityOption match { + drsResolverResponseEntityOption match { case Some(entity) => - val maybeErrorResponse: Either[Error, MarthaFailureResponse] = decode[MarthaFailureResponse](entity) + val maybeErrorResponse: Either[Error, DrsResolverFailureResponse] = decode[DrsResolverFailureResponse](entity) maybeErrorResponse match { case Left(_) => - // Not parsable as a `MarthaFailureResponse` + // Not parsable as a `DrsResolverFailureResponse` baseMessage + s"\'$entity\'" case Right(decoded) => - // Is a `MarthaFailureResponse` + // Is a `DrsResolverFailureResponse` baseMessage + s"\'${decoded.response.text}\'" } case None => diff --git a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/MarthaHttpRequestRetryStrategy.scala b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsResolverHttpRequestRetryStrategy.scala similarity index 96% rename from cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/MarthaHttpRequestRetryStrategy.scala rename to cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsResolverHttpRequestRetryStrategy.scala index 2881fc60b05..39020f5a684 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/MarthaHttpRequestRetryStrategy.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsResolverHttpRequestRetryStrategy.scala @@ -7,7 +7,7 @@ import org.apache.http.HttpResponse import org.apache.http.client.{HttpRequestRetryHandler, ServiceUnavailableRetryStrategy} import org.apache.http.protocol.HttpContext -class MarthaHttpRequestRetryStrategy(drsConfig: DrsConfig) +class DrsResolverHttpRequestRetryStrategy(drsConfig: DrsConfig) extends ServiceUnavailableRetryStrategy with HttpRequestRetryHandler { // We can execute a total of one time, plus the number of retries diff --git a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/DrsCloudNioFileProviderSpec.scala b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/DrsCloudNioFileProviderSpec.scala index fd7b71ef4ee..f61c608bd71 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/DrsCloudNioFileProviderSpec.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/DrsCloudNioFileProviderSpec.scala @@ -25,15 +25,15 @@ class DrsCloudNioFileProviderSpec extends AnyFlatSpecLike with CromwellTimeoutSp it should "parse a config and create a working file system provider" in { val config = ConfigFactory.parseString( - """martha.url = "https://from.config" + """resolver.url = "https://from.config" |access-token-acceptable-ttl = 1 minute |""".stripMargin ) val fileSystemProvider = new MockDrsCloudNioFileSystemProvider(config = config) - fileSystemProvider.drsConfig.marthaUrl should be("https://from.config") + fileSystemProvider.drsConfig.drsResolverUrl should be("https://from.config") fileSystemProvider.drsCredentials match { - case GoogleDrsCredentials(_, ttl) => ttl should be(1.minute) + case GoogleOauthDrsCredentials(_, ttl) => ttl should be(1.minute) case error => fail(s"Expected GoogleDrsCredentials, found $error") } fileSystemProvider.fileProvider should be(a[DrsCloudNioFileProvider]) @@ -79,19 +79,19 @@ class DrsCloudNioFileProviderSpec extends AnyFlatSpecLike with CromwellTimeoutSp it should "return a file provider that can read bytes from gcs" in { val drsPathResolver = new MockEngineDrsPathResolver() { - override def resolveDrsThroughMartha(drsPath: String, - fields: NonEmptyList[MarthaField.Value], - ): IO[MarthaResponse] = { - IO(MarthaResponse(gsUri = Option("gs://bucket/object/path"))) + override def resolveDrs(drsPath: String, + fields: NonEmptyList[DrsResolverField.Value], + ): IO[DrsResolverResponse] = { + IO(DrsResolverResponse(gsUri = Option("gs://bucket/object/path"))) } } val readChannel = mock[ReadableByteChannel] - val drsReadInterpreter: DrsReadInterpreter = (_, marthaResponse) => { + val drsReadInterpreter: DrsReadInterpreter = (_, drsResolverResponse) => { IO( - (marthaResponse.gsUri, marthaResponse.googleServiceAccount) match { + (drsResolverResponse.gsUri, drsResolverResponse.googleServiceAccount) match { case (Some("gs://bucket/object/path"), None) => readChannel - case _ => fail(s"Unexpected parameters passed: $marthaResponse") + case _ => fail(s"Unexpected parameters passed: $drsResolverResponse") } ) } @@ -105,19 +105,19 @@ class DrsCloudNioFileProviderSpec extends AnyFlatSpecLike with CromwellTimeoutSp it should "return a file provider that can read bytes from an access url" in { val drsPathResolver = new MockEngineDrsPathResolver() { - override def resolveDrsThroughMartha(drsPath: String, - fields: NonEmptyList[MarthaField.Value], - ): IO[MarthaResponse] = { - IO(MarthaResponse(accessUrl = Option(AccessUrl("https://host/object/path", None)))) + override def resolveDrs(drsPath: String, + fields: NonEmptyList[DrsResolverField.Value], + ): IO[DrsResolverResponse] = { + IO(DrsResolverResponse(accessUrl = Option(AccessUrl("https://host/object/path", None)))) } } val readChannel = mock[ReadableByteChannel] - val drsReadInterpreter: DrsReadInterpreter = (_, marthaResponse) => { + val drsReadInterpreter: DrsReadInterpreter = (_, drsResolverResponse) => { IO( - marthaResponse.accessUrl match { + drsResolverResponse.accessUrl match { case Some(AccessUrl("https://host/object/path", None)) => readChannel - case _ => fail(s"Unexpected parameters passed: $marthaResponse") + case _ => fail(s"Unexpected parameters passed: $drsResolverResponse") } ) } @@ -131,13 +131,13 @@ class DrsCloudNioFileProviderSpec extends AnyFlatSpecLike with CromwellTimeoutSp it should "return a file provider that can return file attributes" in { val drsPathResolver = new MockEngineDrsPathResolver() { - override def resolveDrsThroughMartha(drsPath: String, - fields: NonEmptyList[MarthaField.Value], - ): IO[MarthaResponse] = { + override def resolveDrs(drsPath: String, + fields: NonEmptyList[DrsResolverField.Value], + ): IO[DrsResolverResponse] = { val instantCreated = Instant.ofEpochMilli(123L) val instantUpdated = Instant.ofEpochMilli(456L) IO( - MarthaResponse( + DrsResolverResponse( size = Option(789L), timeCreated = Option(OffsetDateTime.ofInstant(instantCreated, ZoneOffset.UTC).toString), timeUpdated = Option(OffsetDateTime.ofInstant(instantUpdated, ZoneOffset.UTC).toString), diff --git a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/DrsPathResolverSpec.scala b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/DrsPathResolverSpec.scala index 78585c96ed9..b4e5968e048 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/DrsPathResolverSpec.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/DrsPathResolverSpec.scala @@ -17,7 +17,7 @@ class DrsPathResolverSpec extends AnyFlatSpecLike with CromwellTimeoutSpec with private val md5HashValue = "336ea55913bc261b72875bd259753046" private val shaHashValue = "f76877f8e86ec3932fd2ae04239fbabb8c90199dab0019ae55fa42b31c314c44" - private val fullMarthaResponse = MarthaResponse( + private val fullDrsResolverResponse = DrsResolverResponse( size = Option(34905345), timeCreated = Option(OffsetDateTime.parse("2020-04-27T15:56:09.696Z").toString), timeUpdated = Option(OffsetDateTime.parse("2020-04-27T15:56:09.696Z").toString), @@ -27,17 +27,17 @@ class DrsPathResolverSpec extends AnyFlatSpecLike with CromwellTimeoutSpec with hashes = Option(Map("md5" -> md5HashValue, "crc32c" -> crcHashValue)) ) - private val fullMarthaResponseNoTz = - fullMarthaResponse - .copy(timeUpdated = fullMarthaResponse.timeUpdated.map(_.stripSuffix("Z"))) + private val fullDrsResolverResponseNoTz = + fullDrsResolverResponse + .copy(timeUpdated = fullDrsResolverResponse.timeUpdated.map(_.stripSuffix("Z"))) - private val fullMarthaResponseNoTime = - fullMarthaResponse + private val fullDrsResolverResponseNoTime = + fullDrsResolverResponse .copy(timeUpdated = None) - private val fullMarthaResponseBadTz = - fullMarthaResponse - .copy(timeUpdated = fullMarthaResponse.timeUpdated.map(_.stripSuffix("Z") + "BADTZ")) + private val fullDrsResolverResponseBadTz = + fullDrsResolverResponse + .copy(timeUpdated = fullDrsResolverResponse.timeUpdated.map(_.stripSuffix("Z") + "BADTZ")) private val etagHashValue = "something" private val completeHashesMap = Option(Map( @@ -74,19 +74,19 @@ class DrsPathResolverSpec extends AnyFlatSpecLike with CromwellTimeoutSpec with behavior of "fileHash()" - it should "return crc32c hash from `hashes` in Martha response when there is a crc32c" in { + it should "return crc32c hash from `hashes` in DRS Resolver response when there is a crc32c" in { DrsCloudNioRegularFileAttributes.getPreferredHash(completeHashesMap) shouldBe Option(FileHash(HashType.Crc32c, crcHashValue)) } - it should "return md5 hash from `hashes` in Martha response when there is no crc32c" in { + it should "return md5 hash from `hashes` in DRS Resolver response when there is no crc32c" in { DrsCloudNioRegularFileAttributes.getPreferredHash(missingCRCHashesMap) shouldBe Option(FileHash(HashType.Md5, md5HashValue)) } - it should "return sha256 hash from `hashes` in Martha response when there is only a sha256" in { + it should "return sha256 hash from `hashes` in DRS Resolver response when there is only a sha256" in { DrsCloudNioRegularFileAttributes.getPreferredHash(onlySHAHashesMap) shouldBe Option(FileHash(HashType.Sha256, shaHashValue)) } - it should "return etag hash from `hashes` in Martha response when there is only an etag" in { + it should "return etag hash from `hashes` in DRS Resolver response when there is only an etag" in { DrsCloudNioRegularFileAttributes.getPreferredHash(onlyEtagHashesMap) shouldBe Option(FileHash(HashType.S3Etag, etagHashValue)) } @@ -127,10 +127,10 @@ class DrsPathResolverSpec extends AnyFlatSpecLike with CromwellTimeoutSpec with it should "successfully parse a failure" in { import io.circe.parser.decode - import cloud.nio.impl.drs.MarthaResponseSupport.marthaFailureResponseDecoder + import cloud.nio.impl.drs.DrsResolverResponseSupport.drsResolverFailureResponseDecoder - val maybeDecoded = decode[MarthaFailureResponse](failureResponseJson) - maybeDecoded map { decoded: MarthaFailureResponse => + val maybeDecoded = decode[DrsResolverFailureResponse](failureResponseJson) + maybeDecoded map { decoded: DrsResolverFailureResponse => decoded.response.text shouldBe "{\"msg\":\"User 'null' does not have required action: read_data\",\"status_code\":500}" } } @@ -139,37 +139,37 @@ class DrsPathResolverSpec extends AnyFlatSpecLike with CromwellTimeoutSpec with val drsPathForDebugging = "drs://my_awesome_drs" val responseStatusLine = new BasicStatusLine(new ProtocolVersion("http", 1, 2) , 345, "test-reason") - val testMarthaUri = "www.martha_v3.com" + val testDrsResolverUri = "www.drshub_v4.com" it should "construct an error message from a populated, well-formed failure response" in { val failureResponse = Option(failureResponseJson) - MarthaResponseSupport.errorMessageFromResponse(drsPathForDebugging, failureResponse, responseStatusLine, testMarthaUri) shouldBe { - "Could not access object 'drs://my_awesome_drs'. Status: 345, reason: 'test-reason', Martha location: 'www.martha_v3.com', message: '{\"msg\":\"User 'null' does not have required action: read_data\",\"status_code\":500}'" + DrsResolverResponseSupport.errorMessageFromResponse(drsPathForDebugging, failureResponse, responseStatusLine, testDrsResolverUri) shouldBe { + "Could not access object 'drs://my_awesome_drs'. Status: 345, reason: 'test-reason', DRS Resolver location: 'www.drshub_v4.com', message: '{\"msg\":\"User 'null' does not have required action: read_data\",\"status_code\":500}'" } } it should "construct an error message from an empty failure response" in { - MarthaResponseSupport.errorMessageFromResponse(drsPathForDebugging, None, responseStatusLine, testMarthaUri) shouldBe { - "Could not access object 'drs://my_awesome_drs'. Status: 345, reason: 'test-reason', Martha location: 'www.martha_v3.com', message: (empty response)" + DrsResolverResponseSupport.errorMessageFromResponse(drsPathForDebugging, None, responseStatusLine, testDrsResolverUri) shouldBe { + "Could not access object 'drs://my_awesome_drs'. Status: 345, reason: 'test-reason', DRS Resolver location: 'www.drshub_v4.com', message: (empty response)" } } - // Technically we enter this case when preparing the "error message" for a successful response, because naturally `MarthaResponse` does not deserialize to `MarthaFailureResponse` + // Technically we enter this case when preparing the "error message" for a successful response, because naturally `DrsResolverResponse` does not deserialize to `DrsResolverFailureResponse` // But then there's no error so we throw it away :shrug: it should "construct an error message from a malformed failure response" in { val unparsableFailureResponse = Option("something went horribly wrong") - MarthaResponseSupport.errorMessageFromResponse(drsPathForDebugging, unparsableFailureResponse, responseStatusLine, testMarthaUri) shouldBe { - "Could not access object 'drs://my_awesome_drs'. Status: 345, reason: 'test-reason', Martha location: 'www.martha_v3.com', message: 'something went horribly wrong'" + DrsResolverResponseSupport.errorMessageFromResponse(drsPathForDebugging, unparsableFailureResponse, responseStatusLine, testDrsResolverUri) shouldBe { + "Could not access object 'drs://my_awesome_drs'. Status: 345, reason: 'test-reason', DRS Resolver location: 'www.drshub_v4.com', message: 'something went horribly wrong'" } } it should "resolve an ISO-8601 date with timezone" in { val lastModifiedTimeIO = convertToFileTime( "drs://my_awesome_drs", - MarthaField.TimeUpdated, - fullMarthaResponse.timeUpdated, + DrsResolverField.TimeUpdated, + fullDrsResolverResponse.timeUpdated, ) lastModifiedTimeIO.unsafeRunSync() should be(Option(FileTime.from(OffsetDateTime.parse("2020-04-27T15:56:09.696Z").toInstant))) @@ -178,8 +178,8 @@ class DrsPathResolverSpec extends AnyFlatSpecLike with CromwellTimeoutSpec with it should "resolve an ISO-8601 date without timezone" in { val lastModifiedTimeIO = convertToFileTime( "drs://my_awesome_drs", - MarthaField.TimeUpdated, - fullMarthaResponseNoTz.timeUpdated, + DrsResolverField.TimeUpdated, + fullDrsResolverResponseNoTz.timeUpdated, ) lastModifiedTimeIO.unsafeRunSync() should be(Option(FileTime.from(OffsetDateTime.parse("2020-04-27T15:56:09.696Z").toInstant))) @@ -188,8 +188,8 @@ class DrsPathResolverSpec extends AnyFlatSpecLike with CromwellTimeoutSpec with it should "not resolve an date that does not contain a timeUpdated" in { val lastModifiedTimeIO = convertToFileTime( "drs://my_awesome_drs", - MarthaField.TimeUpdated, - fullMarthaResponseNoTime.timeUpdated, + DrsResolverField.TimeUpdated, + fullDrsResolverResponseNoTime.timeUpdated, ) lastModifiedTimeIO.unsafeRunSync() should be(None) } @@ -197,11 +197,11 @@ class DrsPathResolverSpec extends AnyFlatSpecLike with CromwellTimeoutSpec with it should "not resolve an date that is not ISO-8601" in { val lastModifiedTimeIO = convertToFileTime( "drs://my_awesome_drs", - MarthaField.TimeUpdated, - fullMarthaResponseBadTz.timeUpdated, + DrsResolverField.TimeUpdated, + fullDrsResolverResponseBadTz.timeUpdated, ) the[RuntimeException] thrownBy lastModifiedTimeIO.unsafeRunSync() should have message - "Error while parsing 'timeUpdated' value from Martha to FileTime for DRS path drs://my_awesome_drs. " + + "Error while parsing 'timeUpdated' value from DRS Resolver to FileTime for DRS path drs://my_awesome_drs. " + "Reason: DateTimeParseException: Text '2020-04-27T15:56:09.696BADTZ' could not be parsed at index 23." } } diff --git a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MarthaHttpRequestRetryStrategySpec.scala b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/DrsResolverHttpRequestRetryStrategySpec.scala similarity index 89% rename from cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MarthaHttpRequestRetryStrategySpec.scala rename to cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/DrsResolverHttpRequestRetryStrategySpec.scala index 8fd1e948607..b221038bedb 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MarthaHttpRequestRetryStrategySpec.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/DrsResolverHttpRequestRetryStrategySpec.scala @@ -12,13 +12,13 @@ import common.mock.MockSugar import scala.concurrent.duration._ -class MarthaHttpRequestRetryStrategySpec extends AnyFlatSpec with Matchers with MockSugar { +class DrsResolverHttpRequestRetryStrategySpec extends AnyFlatSpec with Matchers with MockSugar { - behavior of "MarthaHttpRequestRetryStrategy" + behavior of "DrsResolverHttpRequestRetryStrategy" it should "retry 500 errors a configured number of times" in { val drsConfig = MockDrsPaths.mockDrsConfig.copy(numRetries = 3) - val retryStrategy = new MarthaHttpRequestRetryStrategy(drsConfig) + val retryStrategy = new DrsResolverHttpRequestRetryStrategy(drsConfig) val http500Response = mock[CloseableHttpResponse] http500Response.getStatusLine returns new BasicStatusLine(HttpVersion.HTTP_1_1, 500, "Testing 500") val httpContext = mock[HttpContext] @@ -35,7 +35,7 @@ class MarthaHttpRequestRetryStrategySpec extends AnyFlatSpec with Matchers with it should "retry 500 errors even after a number of 408/429 errors" in { val drsConfig = MockDrsPaths.mockDrsConfig.copy(numRetries = 3) - val retryStrategy = new MarthaHttpRequestRetryStrategy(drsConfig) + val retryStrategy = new DrsResolverHttpRequestRetryStrategy(drsConfig) val http500Response = mock[CloseableHttpResponse] http500Response.getStatusLine returns new BasicStatusLine(HttpVersion.HTTP_1_1, 500, "Testing 500") val http408Response = mock[CloseableHttpResponse] @@ -63,7 +63,7 @@ class MarthaHttpRequestRetryStrategySpec extends AnyFlatSpec with Matchers with it should "not retry an HTTP 401" in { val drsConfig = MockDrsPaths.mockDrsConfig.copy(numRetries = 3) - val retryStrategy = new MarthaHttpRequestRetryStrategy(drsConfig) + val retryStrategy = new DrsResolverHttpRequestRetryStrategy(drsConfig) val http400Response = mock[CloseableHttpResponse] http400Response.getStatusLine returns new BasicStatusLine(HttpVersion.HTTP_1_1, 401, "Testing 401") val httpContext = mock[HttpContext] @@ -73,7 +73,7 @@ class MarthaHttpRequestRetryStrategySpec extends AnyFlatSpec with Matchers with it should "retry IO exceptions a configured number of times" in { val drsConfig = MockDrsPaths.mockDrsConfig.copy(numRetries = 3) - val retryStrategy = new MarthaHttpRequestRetryStrategy(drsConfig) + val retryStrategy = new DrsResolverHttpRequestRetryStrategy(drsConfig) val exception = mock[IOException] val httpContext = mock[HttpContext] @@ -95,7 +95,7 @@ class MarthaHttpRequestRetryStrategySpec extends AnyFlatSpec with Matchers with waitMaximum = 1.minute, waitRandomizationFactor = 0d, ) - val retryStrategy = new MarthaHttpRequestRetryStrategy(drsConfig) + val retryStrategy = new DrsResolverHttpRequestRetryStrategy(drsConfig) retryStrategy.getRetryInterval should be(10000L) retryStrategy.getRetryInterval should be(20000L) diff --git a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockDrsCloudNioFileSystemProvider.scala b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockDrsCloudNioFileSystemProvider.scala index 025d7c6d543..07a02e096dc 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockDrsCloudNioFileSystemProvider.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockDrsCloudNioFileSystemProvider.scala @@ -17,7 +17,7 @@ class MockDrsCloudNioFileSystemProvider(config: Config = mockConfig, ), mockResolver: Option[EngineDrsPathResolver] = None, ) - extends DrsCloudNioFileSystemProvider(config, GoogleDrsCredentials(NoCredentials.getInstance, config), drsReadInterpreter) { + extends DrsCloudNioFileSystemProvider(config, GoogleOauthDrsCredentials(NoCredentials.getInstance, config), drsReadInterpreter) { override lazy val drsPathResolver: EngineDrsPathResolver = { mockResolver getOrElse @@ -31,7 +31,7 @@ class MockDrsCloudNioFileSystemProvider(config: Config = mockConfig, object MockDrsCloudNioFileSystemProvider { private lazy val mockConfig = ConfigFactory.parseString( - """martha.url = "https://mock.martha" + """resolver.url = "https://mock.drshub" |access-token-acceptable-ttl = 1 hour |""".stripMargin ) diff --git a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockDrsPaths.scala b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockDrsPaths.scala index b6825dcbbd2..a24385bc15f 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockDrsPaths.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockDrsPaths.scala @@ -3,9 +3,9 @@ package cloud.nio.impl.drs import scala.concurrent.duration._ object MockDrsPaths { - val marthaUrl = "http://mock.martha" + val drsResolverUrl = "http://mock.drshub" - val mockDrsConfig: DrsConfig = DrsConfig(MockDrsPaths.marthaUrl, 1, 1.seconds, 1.seconds, 1d, 0d) + val mockDrsConfig: DrsConfig = DrsConfig(MockDrsPaths.drsResolverUrl, 1, 1.seconds, 1.seconds, 1d, 0d) val mockToken = "mock.token" @@ -35,5 +35,5 @@ object MockDrsPaths { val drsPathResolvingToNoGcsPath = s"$drsPathPrefix/226686cf-22c9-4472-9f79-7a0b0044f253" - val drsPathNotExistingInMartha = s"$drsPathPrefix/5e21b8c3-8eda-48d5-9a04-2b32e1571765" + val drsPathNotExistingInDrsResolver = s"$drsPathPrefix/5e21b8c3-8eda-48d5-9a04-2b32e1571765" } diff --git a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockEngineDrsPathResolver.scala b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockEngineDrsPathResolver.scala index 116155753f1..9e22544c38b 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockEngineDrsPathResolver.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockEngineDrsPathResolver.scala @@ -14,12 +14,12 @@ class MockEngineDrsPathResolver(drsConfig: DrsConfig = MockDrsPaths.mockDrsConfi httpClientBuilderOverride: Option[HttpClientBuilder] = None, accessTokenAcceptableTTL: Duration = Duration.Inf, ) - extends EngineDrsPathResolver(drsConfig, GoogleDrsCredentials(NoCredentials.getInstance, accessTokenAcceptableTTL)) { + extends EngineDrsPathResolver(drsConfig, GoogleOauthDrsCredentials(NoCredentials.getInstance, accessTokenAcceptableTTL)) { override protected lazy val httpClientBuilder: HttpClientBuilder = httpClientBuilderOverride getOrElse MockSugar.mock[HttpClientBuilder] - private lazy val mockMarthaUri = drsConfig.marthaUrl + private lazy val mockDrsResolverUri = drsConfig.drsResolverUrl private val hashesObj = Map( "md5" -> "336ea55913bc261b72875bd259753046", @@ -27,8 +27,8 @@ class MockEngineDrsPathResolver(drsConfig: DrsConfig = MockDrsPaths.mockDrsConfi "crc32c" -> "8a366443" ) - private val marthaObjWithGcsPath = - MarthaResponse( + private val drsResolverObjWithGcsPath = + DrsResolverResponse( size = Option(156018255), timeCreated = Option("2020-04-27T15:56:09.696Z"), timeUpdated = Option("2020-04-27T15:56:09.696Z"), @@ -36,27 +36,27 @@ class MockEngineDrsPathResolver(drsConfig: DrsConfig = MockDrsPaths.mockDrsConfi hashes = Option(hashesObj) ) - private val marthaObjWithFileName = marthaObjWithGcsPath.copy(fileName = Option("file.txt")) + private val drsResolverObjWithFileName = drsResolverObjWithGcsPath.copy(fileName = Option("file.txt")) - private val marthaObjWithLocalizationPath = marthaObjWithGcsPath.copy(localizationPath = Option("/dir/subdir/file.txt")) + private val drsResolverObjWithLocalizationPath = drsResolverObjWithGcsPath.copy(localizationPath = Option("/dir/subdir/file.txt")) - private val marthaObjWithAllThePaths = marthaObjWithLocalizationPath.copy(fileName = marthaObjWithFileName.fileName) + private val drsResolverObjWithAllThePaths = drsResolverObjWithLocalizationPath.copy(fileName = drsResolverObjWithFileName.fileName) - private val marthaObjWithNoGcsPath = marthaObjWithGcsPath.copy(gsUri = None) + private val drsResolverObjWithNoGcsPath = drsResolverObjWithGcsPath.copy(gsUri = None) - override def resolveDrsThroughMartha(drsPath: String, fields: NonEmptyList[MarthaField.Value]): IO[MarthaResponse] = { + override def resolveDrs(drsPath: String, fields: NonEmptyList[DrsResolverField.Value]): IO[DrsResolverResponse] = { drsPath match { - case MockDrsPaths.drsPathResolvingGcsPath => IO(marthaObjWithGcsPath) - case MockDrsPaths.drsPathWithNonPathChars => IO(marthaObjWithGcsPath) - case MockDrsPaths.drsPathResolvingWithFileName => IO(marthaObjWithFileName) - case MockDrsPaths.drsPathResolvingWithLocalizationPath => IO.pure(marthaObjWithLocalizationPath) - case MockDrsPaths.drsPathResolvingWithAllThePaths => IO.pure(marthaObjWithAllThePaths) - case MockDrsPaths.drsPathResolvingToNoGcsPath => IO(marthaObjWithNoGcsPath) - case MockDrsPaths.drsPathNotExistingInMartha => + case MockDrsPaths.drsPathResolvingGcsPath => IO(drsResolverObjWithGcsPath) + case MockDrsPaths.drsPathWithNonPathChars => IO(drsResolverObjWithGcsPath) + case MockDrsPaths.drsPathResolvingWithFileName => IO(drsResolverObjWithFileName) + case MockDrsPaths.drsPathResolvingWithLocalizationPath => IO.pure(drsResolverObjWithLocalizationPath) + case MockDrsPaths.drsPathResolvingWithAllThePaths => IO.pure(drsResolverObjWithAllThePaths) + case MockDrsPaths.drsPathResolvingToNoGcsPath => IO(drsResolverObjWithNoGcsPath) + case MockDrsPaths.`drsPathNotExistingInDrsResolver` => IO.raiseError( new RuntimeException( - s"Unexpected response resolving ${MockDrsPaths.drsPathNotExistingInMartha} " + - s"through Martha url $mockMarthaUri. Error: 404 Not Found." + s"Unexpected response resolving ${MockDrsPaths.drsPathNotExistingInDrsResolver} " + + s"through DRS Resolver url $mockDrsResolverUri. Error: 404 Not Found." ) ) } diff --git a/codegen_java/build.sbt b/codegen_java/build.sbt index 2ba35a693c3..0ad461a66b8 100644 --- a/codegen_java/build.sbt +++ b/codegen_java/build.sbt @@ -6,7 +6,7 @@ lazy val root = (project in file(".")). Seq(organization := "org.broadinstitute.cromwell", name := "cromwell-client", version := createVersion("0.1"), - scalaVersion := "2.13.8", + scalaVersion := "2.13.9", scalacOptions ++= Seq("-feature"), compile / javacOptions ++= Seq("-Xlint:deprecation"), Compile / packageDoc / publishArtifact := false, diff --git a/common/src/main/scala/common/util/TerminalUtil.scala b/common/src/main/scala/common/util/TerminalUtil.scala index dab7fb9ce17..cb7c980aecb 100644 --- a/common/src/main/scala/common/util/TerminalUtil.scala +++ b/common/src/main/scala/common/util/TerminalUtil.scala @@ -6,10 +6,10 @@ object TerminalUtil { def maxWidth(lengths: Seq[Seq[Int]], column: Int) = lengths.map { length => length(column) }.max val widths = (rows :+ header).map { row => row.map { s => s.length } } val maxWidths = widths.head.indices.map { column => maxWidth(widths, column) } - val tableHeader = header.indices.map { i => header(i).padTo(maxWidths(i), " ").mkString("") }.mkString("|") + val tableHeader = header.indices.map { i => header(i).padTo(maxWidths(i), ' ').mkString("") }.mkString("|") val tableDivider = header.indices.map { i => "-" * maxWidths(i) }.mkString("|") val tableRows = rows.map { row => - val mdRow = row.indices.map { i => row(i).padTo(maxWidths(i), " ").mkString("") }.mkString("|") + val mdRow = row.indices.map { i => row(i).padTo(maxWidths(i), ' ').mkString("") }.mkString("|") s"|$mdRow|" } s"|$tableHeader|\n|$tableDivider|\n${tableRows.mkString("\n")}\n" diff --git a/common/src/test/scala/common/collections/EnhancedCollectionsSpec.scala b/common/src/test/scala/common/collections/EnhancedCollectionsSpec.scala index 12dba2e15d9..d2e0a38878c 100644 --- a/common/src/test/scala/common/collections/EnhancedCollectionsSpec.scala +++ b/common/src/test/scala/common/collections/EnhancedCollectionsSpec.scala @@ -11,7 +11,7 @@ class EnhancedCollectionsSpec extends AsyncFlatSpec with Matchers { behavior of "EnhancedCollections" it should "filter a List by type and return a List" in { - val objectList = List("hello", 3, None, "world") + val objectList = List[Any]("hello", 3, None, "world") val stringList = objectList.filterByType[String] stringList should be(List("hello", "world")) @@ -28,14 +28,14 @@ class EnhancedCollectionsSpec extends AsyncFlatSpec with Matchers { } it should "filter a Set by type and return a Set" in { - val objectSet = Set("hello", 3, None, "world") + val objectSet = Set[Any]("hello", 3, None, "world") val intSet: Set[Int] = objectSet.filterByType[Int] intSet should be(Set(3)) } it should "find the first Int in a List" in { - val objectSet = List("hello", 3, None, 4, "world") + val objectSet = List[Any]("hello", 3, None, 4, "world") objectSet.firstByType[Int] should be(Some(3)) } diff --git a/core/src/main/resources/reference.conf b/core/src/main/resources/reference.conf index 41d2b2c14bd..dd2f14c06c8 100644 --- a/core/src/main/resources/reference.conf +++ b/core/src/main/resources/reference.conf @@ -328,8 +328,8 @@ filesystems { # Class to instantiate and propagate to all factories. Takes a single typesafe config argument class = "cromwell.filesystems.drs.DrsFileSystemConfig" config { - martha { - url = "https://martha-url-here" + resolver { + url = "https://martha-url-here or https://drshub-url-here" # When true (and when possible) any DrsPath will be internally converted to another compatible Path such as a # GcsPath. This enables other optimizations such as using batch requests for GCS for hashes, using the # existing GCS downloader in Papi, etc. @@ -441,6 +441,10 @@ engine { languages { default: WDL WDL { + http-allow-list { + enabled: false + allowed-http-hosts: [] + } versions { default: "draft-2" "draft-2" { @@ -468,29 +472,6 @@ languages { } } } - CWL { - versions { - default: "v1.0" - "v1.0" { - language-factory = "languages.cwl.CwlV1_0LanguageFactory" - config { - strict-validation: false - enabled: true - - # Optional section to define a command returning paths to output files that - # can only be known after the user's command has been run - output-runtime-extractor { - # Optional docker image on which the command should be run - Must have bash if provided - docker-image = "stedolan/jq@sha256:a61ed0bca213081b64be94c5e1b402ea58bc549f457c2682a86704dd55231e09" - # Single line command executed after the tool has run. - # It should write to stdout the file paths that are referenced in the cwl.output.json (or any other file path - # not already defined in the outputs of the tool and therefore unknown when the tool is run) - command = "cat cwl.output.json 2>/dev/null | jq -r '.. | .path? // .location? // empty'" - } - } - } - } - } } ontology { diff --git a/core/src/main/scala/cromwell/core/io/IoCommand.scala b/core/src/main/scala/cromwell/core/io/IoCommand.scala index a21b6f8cebd..7d71f9bd307 100644 --- a/core/src/main/scala/cromwell/core/io/IoCommand.scala +++ b/core/src/main/scala/cromwell/core/io/IoCommand.scala @@ -38,6 +38,12 @@ trait IoCommand[+T] { def commandDescription: String + /** + * IO commands side-effect and/or have exceptions. We don't want that when evaluating identity, e.g. to check presence in cache. + * @return Hash code based on the description, which captures file path & action + */ + override def hashCode(): Int = commandDescription.hashCode + def logIOMsgOverLimit(message: => String): Unit = { val millis: Long = java.time.Duration.between(creation, OffsetDateTime.now).toMillis if (millis > IoCommand.IOCommandWarnLimit.toMillis) { diff --git a/core/src/main/scala/cromwell/core/io/IoPromiseProxyActor.scala b/core/src/main/scala/cromwell/core/io/IoPromiseProxyActor.scala index 15097a92a99..dd8f44464f3 100644 --- a/core/src/main/scala/cromwell/core/io/IoPromiseProxyActor.scala +++ b/core/src/main/scala/cromwell/core/io/IoPromiseProxyActor.scala @@ -10,6 +10,8 @@ import scala.concurrent.duration.FiniteDuration object IoPromiseProxyActor { case class IoCommandWithPromise[A](ioCommand: IoCommand[A], timeout: FiniteDuration = defaultTimeout) { val promise = Promise[A]() + + override def hashCode(): Int = ioCommand.hashCode() } def props(ioActor: ActorRef) = Props(new IoPromiseProxyActor(ioActor)) } diff --git a/core/src/main/scala/cromwell/core/logging/EnhancedDateConverter.scala b/core/src/main/scala/cromwell/core/logging/EnhancedDateConverter.scala new file mode 100644 index 00000000000..dcf62bc0b28 --- /dev/null +++ b/core/src/main/scala/cromwell/core/logging/EnhancedDateConverter.scala @@ -0,0 +1,70 @@ +package cromwell.core.logging + +import ch.qos.logback.classic.pattern.DateConverter +import ch.qos.logback.classic.spi.ILoggingEvent +import ch.qos.logback.core.CoreConstants +import ch.qos.logback.core.util.CachingDateFormatter + +import java.util.TimeZone +import scala.jdk.CollectionConverters._ + +/** + * Log the Akka akkaTimestamp if found in the MDC, otherwise log the original event timestamp. + * + * - https://doc.akka.io/docs/akka/current/logging.html#more-accurate-timestamps-for-log-output-in-mdc + * - https://logback.qos.ch/manual/layouts.html#customConversionSpecifier + * + * NOTE: For proper configuration both this EnhancedDateConverter should be configured into the logback.xml AND the + * configuration file should set akka.loggers = ["cromwell.core.logging.EnhancedSlf4jLogger"]. + */ +class EnhancedDateConverter extends DateConverter { + protected var cachingDateFormatterProtected: CachingDateFormatter = _ + + /* Duplicated from ch.qos.logback.classic.pattern.DateConverter as cachingDateFormatter is package private. */ + override def start(): Unit = { + cachingDateFormatterProtected = Option(getFirstOption) match { + case Some(CoreConstants.ISO8601_STR) | None => new CachingDateFormatter(CoreConstants.ISO8601_PATTERN) + case Some(datePattern) => + try { + new CachingDateFormatter(datePattern) + } catch { + case e: IllegalArgumentException => + addWarn("Could not instantiate SimpleDateFormat with pattern " + datePattern, e) + // default to the ISO8601 format + new CachingDateFormatter(CoreConstants.ISO8601_PATTERN) + } + } + // if the option list contains a TZ option, then set it. + Option(getOptionList) + .toList + .flatMap(_.asScala) + .drop(1) + .headOption + .map(TimeZone.getTimeZone) + .foreach(cachingDateFormatterProtected.setTimeZone) + + // Allow the parent class to start/initialize its private members. + super.start() + } + + /** + * Look for the Akka timestamp and use that to format the date. + * + * Until this (currently 6+ year) issue is resolved, formatting the date as a Long requires using the + * [[EnhancedSlf4jLogger]] versus Akka's basic Slf4jLogger. + * + * - https://github.com/akka/akka/issues/18079#issuecomment-125175884 + */ + override def convert(event: ILoggingEvent): String = { + val mdc = event.getMDCPropertyMap + if (mdc.containsKey("akkaTimestamp")) { + val timestamp = mdc.get("akkaTimestamp") + timestamp.toLongOption match { + case Some(value) => cachingDateFormatterProtected.format(value) + case None => timestamp // Return the original timestamp string. + } + } else { + super.convert(event) + } + } +} diff --git a/core/src/main/scala/cromwell/core/logging/EnhancedSlf4jLogger.scala b/core/src/main/scala/cromwell/core/logging/EnhancedSlf4jLogger.scala new file mode 100644 index 00000000000..0999ec18055 --- /dev/null +++ b/core/src/main/scala/cromwell/core/logging/EnhancedSlf4jLogger.scala @@ -0,0 +1,16 @@ +package cromwell.core.logging + +import akka.event.slf4j.Slf4jLogger + +class EnhancedSlf4jLogger extends Slf4jLogger { + /** + * Format the timestamp as a simple long. Allows the akkaTimestamp to be retrieved later from the MDC by custom + * converters. + * + * NOTE: Should not be necessary once this issue is resolved: + * - https://github.com/akka/akka/issues/18079#issuecomment-125175884 + * + * @see [[EnhancedDateConverter.convert()]] + */ + override protected def formatTimestamp(timestamp: Long): String = String.valueOf(timestamp) +} diff --git a/core/src/main/scala/cromwell/core/logging/EnhancedThreadConverter.scala b/core/src/main/scala/cromwell/core/logging/EnhancedThreadConverter.scala new file mode 100644 index 00000000000..73f5876597b --- /dev/null +++ b/core/src/main/scala/cromwell/core/logging/EnhancedThreadConverter.scala @@ -0,0 +1,21 @@ +package cromwell.core.logging + +import ch.qos.logback.classic.pattern.ThreadConverter +import ch.qos.logback.classic.spi.ILoggingEvent + +/** + * Log the Akka sourceThread if found, otherwise log the event thread. + * + * - https://doc.akka.io/docs/akka/current/logging.html#logging-thread-akka-source-and-actor-system-in-mdc + * - https://logback.qos.ch/manual/layouts.html#customConversionSpecifier + */ +class EnhancedThreadConverter extends ThreadConverter { + override def convert(event: ILoggingEvent): String = { + val mdc = event.getMDCPropertyMap + if (mdc.containsKey("sourceThread")) { + mdc.get("sourceThread") + } else { + super.convert(event) + } + } +} diff --git a/core/src/main/scala/cromwell/core/logging/JavaLoggingBridge.scala b/core/src/main/scala/cromwell/core/logging/JavaLoggingBridge.scala new file mode 100644 index 00000000000..1dc10fab46d --- /dev/null +++ b/core/src/main/scala/cromwell/core/logging/JavaLoggingBridge.scala @@ -0,0 +1,39 @@ +package cromwell.core.logging + +import ch.qos.logback.classic.LoggerContext +import ch.qos.logback.classic.jul.LevelChangePropagator +import org.slf4j.LoggerFactory +import org.slf4j.bridge.SLF4JBridgeHandler + +import scala.jdk.CollectionConverters._ + +object JavaLoggingBridge { + /** + * Replace java.util.logging with SLF4J while ensuring Logback is configured with a LevelChangePropogator. + * + * One likely won't need to do this but just in case: note that any libraries using JUL running BEFORE this + * initialization which require increasing or decreasing verbosity must be configured via JUL not Logback. + * + * See also: + * - https://www.slf4j.org/api/org/slf4j/bridge/SLF4JBridgeHandler.html + * - https://docs.oracle.com/en/java/javase/11/docs/api/java.logging/java/util/logging/LogManager.html + */ + def init(): Unit = { + // Retrieve the Logback context, and as a side effect initialize Logback. + val ctx = LoggerFactory.getILoggerFactory.asInstanceOf[LoggerContext] + + // Ensure that Logback has a LevelChangePropagator, either here or via a logback.xml. + val listeners = ctx.getCopyOfListenerList.asScala + if (!listeners.exists(_.isInstanceOf[LevelChangePropagator])) { + val propagator = new LevelChangePropagator() + propagator.setContext(ctx) + propagator.start() + } + + // Remove all the JUL logging handlers. + SLF4JBridgeHandler.removeHandlersForRootLogger() + + // Send all JUL logging to SLF4J. + SLF4JBridgeHandler.install() + } +} diff --git a/core/src/main/scala/cromwell/core/path/DefaultPathBuilderFactory.scala b/core/src/main/scala/cromwell/core/path/DefaultPathBuilderFactory.scala index 91f05b10ef7..20a2afdd5ce 100644 --- a/core/src/main/scala/cromwell/core/path/DefaultPathBuilderFactory.scala +++ b/core/src/main/scala/cromwell/core/path/DefaultPathBuilderFactory.scala @@ -2,6 +2,7 @@ package cromwell.core.path import akka.actor.ActorSystem import cromwell.core.WorkflowOptions +import cromwell.core.path.PathBuilderFactory.PriorityDefault import scala.concurrent.{ExecutionContext, Future} @@ -9,4 +10,6 @@ case object DefaultPathBuilderFactory extends PathBuilderFactory { override def withOptions(options: WorkflowOptions)(implicit actorSystem: ActorSystem, ec: ExecutionContext) = Future.successful(DefaultPathBuilder) val name = "local" val tuple = name -> this + + override def priority: Int = PriorityDefault } diff --git a/core/src/main/scala/cromwell/core/path/PathBuilderFactory.scala b/core/src/main/scala/cromwell/core/path/PathBuilderFactory.scala index 1f215ba2933..79adaae4d6b 100644 --- a/core/src/main/scala/cromwell/core/path/PathBuilderFactory.scala +++ b/core/src/main/scala/cromwell/core/path/PathBuilderFactory.scala @@ -5,6 +5,7 @@ import cromwell.core.{Dispatcher, WorkflowOptions} import cats.syntax.traverse._ import cats.instances.list._ import cats.instances.future._ +import cromwell.core.path.PathBuilderFactory.PriorityStandard import scala.concurrent.{ExecutionContext, Future} @@ -12,16 +13,13 @@ object PathBuilderFactory { // Given a list of factories, instantiates the corresponding path builders def instantiatePathBuilders(factories: List[PathBuilderFactory], workflowOptions: WorkflowOptions)(implicit as: ActorSystem): Future[List[PathBuilder]] = { implicit val ec: ExecutionContext = as.dispatchers.lookup(Dispatcher.IoDispatcher) - // The DefaultPathBuilderFactory always needs to be last. - // The reason is path builders are tried in order, and the default one is very generous in terms of paths it "thinks" it supports - // For instance, it will return a Path for a gcs url even though it doesn't really support it - val sortedFactories = factories.sortWith({ - case (_, DefaultPathBuilderFactory) => true - case (DefaultPathBuilderFactory, _) => false - case (a, b) => factories.indexOf(a) < factories.indexOf(b) - }) + val sortedFactories = factories.sortBy(_.priority) sortedFactories.traverse(_.withOptions(workflowOptions)) } + + val PriorityBlob = 100 // High priority to evaluate first, because blob files may inadvertently match other filesystems + val PriorityStandard = 1000 + val PriorityDefault = 10000 // "Default" is a fallback, evaluate last } /** @@ -29,4 +27,11 @@ object PathBuilderFactory { */ trait PathBuilderFactory { def withOptions(options: WorkflowOptions)(implicit as: ActorSystem, ec: ExecutionContext): Future[PathBuilder] + + /** + * Candidate filesystems are considered in a stable order, as some requests may match multiple filesystems. + * To customize this order, the priority of a filesystem may be adjusted. Lower number == higher priority. + * @return This filesystem's priority + */ + def priority: Int = PriorityStandard } diff --git a/core/src/test/scala/cromwell/core/io/IoClientHelperSpec.scala b/core/src/test/scala/cromwell/core/io/IoClientHelperSpec.scala index f77df6fa011..06132ba152b 100644 --- a/core/src/test/scala/cromwell/core/io/IoClientHelperSpec.scala +++ b/core/src/test/scala/cromwell/core/io/IoClientHelperSpec.scala @@ -27,7 +27,7 @@ class IoClientHelperSpec extends TestKitSuite with AnyFlatSpecLike with Matchers val testActor = TestActorRef(new IoClientHelperTestActor(ioActorProbe.ref, delegateProbe.ref, backoff, noResponseTimeout)) val command = DefaultIoSizeCommand(mock[Path]) - val response = IoSuccess(command, 5) + val response = IoSuccess(command, 5L) // Send the command testActor.underlyingActor.sendMessage(command) @@ -58,7 +58,7 @@ class IoClientHelperSpec extends TestKitSuite with AnyFlatSpecLike with Matchers val commandContext = "context" val command = DefaultIoSizeCommand(mock[Path]) - val response = IoSuccess(command, 5) + val response = IoSuccess(command, 5L) // Send the command testActor.underlyingActor.sendMessageWithContext(commandContext, command) diff --git a/core/src/test/scala/cromwell/core/retry/BackoffSpec.scala b/core/src/test/scala/cromwell/core/retry/BackoffSpec.scala index 58fa597dc03..c6a008feb02 100644 --- a/core/src/test/scala/cromwell/core/retry/BackoffSpec.scala +++ b/core/src/test/scala/cromwell/core/retry/BackoffSpec.scala @@ -54,7 +54,7 @@ class BackoffSpec extends AnyFlatSpec with CromwellTimeoutSpec with Matchers { it should "parse config" in { val config = ConfigFactory.parseMap( - Map( + Map[String, Any]( "min" -> "5 seconds", "max" -> "30 seconds", "multiplier" -> 6D, diff --git a/cromwell-drs-localizer/src/main/scala/drs/localizer/CommandLineParser.scala b/cromwell-drs-localizer/src/main/scala/drs/localizer/CommandLineParser.scala index df0f7c2eba6..3b7be2b38bd 100644 --- a/cromwell-drs-localizer/src/main/scala/drs/localizer/CommandLineParser.scala +++ b/cromwell-drs-localizer/src/main/scala/drs/localizer/CommandLineParser.scala @@ -14,40 +14,55 @@ class CommandLineParser extends scopt.OptionParser[CommandLineArguments](Usage) head("cromwell-drs-localizer", localizerVersion) - arg[String]("drs-object-id").text("DRS object ID").required(). + arg[String]("drs-object-id").text("DRS object ID").optional(). action((s, c) => c.copy(drsObject = Option(s))) - arg[String]("container-path").text("Container path").required(). + arg[String]("container-path").text("Container path").optional(). action((s, c) => c.copy(containerPath = Option(s))) - arg[String]("requester-pays-project").text("Requester pays project").optional(). + arg[String]("requester-pays-project").text(s"Requester pays project (only valid with '$Google' auth strategy)").optional(). action((s, c) => c.copy(googleRequesterPaysProject = Option(s))) + opt[String]('m', "manifest-path").text("File path of manifest containing multiple files to localize"). + action((s, c) => + c.copy(manifestPath = Option(s))) + opt[String]('r', "requester-pays-project").text(s"Requester pays project (only valid with '$Google' auth strategy)").optional(). + action((s, c) => { + c.copy( + googleRequesterPaysProject = Option(s), + googleRequesterPaysProjectConflict = c.googleRequesterPaysProject.exists(_ != s) + ) + }) opt[String]('t', "access-token-strategy").text(s"Access token strategy, must be one of '$Azure' or '$Google' (default '$Google')"). action((s, c) => c.copy(accessTokenStrategy = Option(s.toLowerCase()))) - opt[String]('v', "vault-name").text("Azure vault name"). - action((s, c) => - c.copy(azureVaultName = Option(s))) - opt[String]('s', "secret-name").text("Azure secret name"). - action((s, c) => - c.copy(azureSecretName = Option(s))) opt[String]('i', "identity-client-id").text("Azure identity client id"). action((s, c) => c.copy(azureIdentityClientId = Option(s))) + checkConfig(c => + if (c.googleRequesterPaysProjectConflict) + failure("Requester pays project differs between positional argument and option flag") + else success + ) checkConfig(c => c.accessTokenStrategy match { case Some(Azure) if c.googleRequesterPaysProject.nonEmpty => Left(s"Requester pays project is only valid with access token strategy '$Google'") - case Some(Azure) if List(c.azureVaultName, c.azureSecretName).exists(_.isEmpty) => - Left(s"Both vault name and secret name must be specified for access token strategy $Azure") + case Some(Google) if c.azureIdentityClientId.nonEmpty => + Left(s"Identity client id is only valid with access token strategy '$Azure'") case Some(Azure) => Right(()) - case Some(Google) if List(c.azureSecretName, c.azureVaultName, c.azureIdentityClientId).forall(_.isEmpty) => Right(()) - case Some(Google) => Left(s"One or more specified options are only valid with access token strategy '$Azure'") + case Some(Google) => Right(()) case Some(huh) => Left(s"Unrecognized access token strategy '$huh'") case None => Left("Programmer error, access token strategy should not be None") } ) + checkConfig(c => + (c.drsObject, c.containerPath, c.manifestPath) match { + case (Some(_), Some(_), None) => Right(()) + case (None, None, Some(_)) => Right(()) + case _ => Left("Must provide either DRS path and container path, OR manifest file (-m).") + } + ) } object CommandLineParser { @@ -64,9 +79,12 @@ object CommandLineParser { val Usage = s""" Usage: - java -jar /path/to/localizer.jar [options] drs://provider/object /local/path/to/file.txt [requester pays project] - Note that the optional argument is only valid with access token strategy 'Google'. + Can be run to localize a single file with DRS id and local container path provided in args: + java -jar /path/to/localizer.jar [options] drs://provider/object /local/path/to/file.txt + + Can also be used to localize multiple files in one invocation with manifest file provided in args: + java -jar /path/to/localizer.jar [options] -m /local/path/to/manifest/file """ } @@ -75,6 +93,6 @@ case class CommandLineArguments(accessTokenStrategy: Option[String] = Option(Goo drsObject: Option[String] = None, containerPath: Option[String] = None, googleRequesterPaysProject: Option[String] = None, - azureVaultName: Option[String] = None, - azureSecretName: Option[String] = None, - azureIdentityClientId: Option[String] = None) + azureIdentityClientId: Option[String] = None, + manifestPath: Option[String] = None, + googleRequesterPaysProjectConflict: Boolean = false) diff --git a/cromwell-drs-localizer/src/main/scala/drs/localizer/DrsLocalizerDrsPathResolver.scala b/cromwell-drs-localizer/src/main/scala/drs/localizer/DrsLocalizerDrsPathResolver.scala index ac27367b932..944d418acbb 100644 --- a/cromwell-drs-localizer/src/main/scala/drs/localizer/DrsLocalizerDrsPathResolver.scala +++ b/cromwell-drs-localizer/src/main/scala/drs/localizer/DrsLocalizerDrsPathResolver.scala @@ -1,10 +1,9 @@ package drs.localizer -import cloud.nio.impl.drs.{DrsConfig, DrsPathResolver} +import cloud.nio.impl.drs.{DrsConfig, DrsCredentials, DrsPathResolver} import common.validation.ErrorOr.ErrorOr -import drs.localizer.accesstokens.AccessTokenStrategy -class DrsLocalizerDrsPathResolver(drsConfig: DrsConfig, accessTokenStrategy: AccessTokenStrategy) extends DrsPathResolver(drsConfig) { - override def getAccessToken: ErrorOr[String] = accessTokenStrategy.getAccessToken() +class DrsLocalizerDrsPathResolver(drsConfig: DrsConfig, drsCredentials: DrsCredentials) extends DrsPathResolver(drsConfig) { + override def getAccessToken: ErrorOr[String] = drsCredentials.getAccessToken } diff --git a/cromwell-drs-localizer/src/main/scala/drs/localizer/DrsLocalizerMain.scala b/cromwell-drs-localizer/src/main/scala/drs/localizer/DrsLocalizerMain.scala index c7ecdd3f3e4..1858f395024 100644 --- a/cromwell-drs-localizer/src/main/scala/drs/localizer/DrsLocalizerMain.scala +++ b/cromwell-drs-localizer/src/main/scala/drs/localizer/DrsLocalizerMain.scala @@ -2,16 +2,20 @@ package drs.localizer import cats.data.NonEmptyList import cats.effect.{ExitCode, IO, IOApp} +import cats.implicits._ import cloud.nio.impl.drs.DrsPathResolver.{FatalRetryDisposition, RegularRetryDisposition} -import cloud.nio.impl.drs.{AccessUrl, DrsConfig, DrsPathResolver, MarthaField} +import cloud.nio.impl.drs._ import cloud.nio.spi.{CloudNioBackoff, CloudNioSimpleExponentialBackoff} import com.typesafe.scalalogging.StrictLogging import drs.localizer.CommandLineParser.AccessTokenStrategy.{Azure, Google} -import drs.localizer.accesstokens.{AccessTokenStrategy, AzureB2CAccessTokenStrategy, GoogleAccessTokenStrategy} import drs.localizer.downloaders.AccessUrlDownloader.Hashes import drs.localizer.downloaders._ +import org.apache.commons.csv.{CSVFormat, CSVParser} +import java.io.File +import java.nio.charset.Charset import scala.concurrent.duration._ +import scala.jdk.CollectionConverters._ import scala.language.postfixOps object DrsLocalizerMain extends IOApp with StrictLogging { @@ -24,8 +28,8 @@ object DrsLocalizerMain extends IOApp with StrictLogging { val localize: Option[IO[ExitCode]] = for { pa <- parsedArgs run <- pa.accessTokenStrategy.collect { - case Azure => runLocalizer(pa, AzureB2CAccessTokenStrategy(pa)) - case Google => runLocalizer(pa, GoogleAccessTokenStrategy) + case Azure => runLocalizer(pa, AzureDrsCredentials(pa.azureIdentityClientId)) + case Google => runLocalizer(pa, GoogleAppDefaultTokenStrategy) } } yield run @@ -50,23 +54,40 @@ object DrsLocalizerMain extends IOApp with StrictLogging { IO.pure(ExitCode.Error) } - def runLocalizer(commandLineArguments: CommandLineArguments, accessTokenStrategy: AccessTokenStrategy): IO[ExitCode] = { - val drsObject = commandLineArguments.drsObject.get - val containerPath = commandLineArguments.containerPath.get - new DrsLocalizerMain(drsObject, containerPath, accessTokenStrategy, commandLineArguments.googleRequesterPaysProject). + def runLocalizer(commandLineArguments: CommandLineArguments, drsCredentials: DrsCredentials): IO[ExitCode] = { + commandLineArguments.manifestPath match { + case Some(manifestPath) => + val manifestFile = new File(manifestPath) + val csvParser = CSVParser.parse(manifestFile, Charset.defaultCharset(), CSVFormat.DEFAULT) + val exitCodes: IO[List[ExitCode]] = csvParser.asScala.map(record => { + val drsObject = record.get(0) + val containerPath = record.get(1) + localizeFile(commandLineArguments, drsCredentials, drsObject, containerPath) + }).toList.sequence + exitCodes.map(_.find(_ != ExitCode.Success).getOrElse(ExitCode.Success)) + case None => + val drsObject = commandLineArguments.drsObject.get + val containerPath = commandLineArguments.containerPath.get + localizeFile(commandLineArguments, drsCredentials, drsObject, containerPath) + } + } + + private def localizeFile(commandLineArguments: CommandLineArguments, drsCredentials: DrsCredentials, drsObject: String, containerPath: String) = { + new DrsLocalizerMain(drsObject, containerPath, drsCredentials, commandLineArguments.googleRequesterPaysProject). resolveAndDownloadWithRetries(downloadRetries = 3, checksumRetries = 1, defaultDownloaderFactory, Option(defaultBackoff)).map(_.exitCode) } } class DrsLocalizerMain(drsUrl: String, downloadLoc: String, - accessTokenStrategy: AccessTokenStrategy, + drsCredentials: DrsCredentials, requesterPaysProjectIdOption: Option[String]) extends StrictLogging { def getDrsPathResolver: IO[DrsLocalizerDrsPathResolver] = { IO { val drsConfig = DrsConfig.fromEnv(sys.env) - new DrsLocalizerDrsPathResolver(drsConfig, accessTokenStrategy) + logger.info(s"Using ${drsConfig.drsResolverUrl} to resolve DRS Objects") + new DrsLocalizerDrsPathResolver(drsConfig, drsCredentials) } } @@ -121,17 +142,17 @@ class DrsLocalizerMain(drsUrl: String, } private [localizer] def resolve(downloaderFactory: DownloaderFactory): IO[Downloader] = { - val fields = NonEmptyList.of(MarthaField.GsUri, MarthaField.GoogleServiceAccount, MarthaField.AccessUrl, MarthaField.Hashes) + val fields = NonEmptyList.of(DrsResolverField.GsUri, DrsResolverField.GoogleServiceAccount, DrsResolverField.AccessUrl, DrsResolverField.Hashes) for { resolver <- getDrsPathResolver - marthaResponse <- resolver.resolveDrsThroughMartha(drsUrl, fields) + drsResolverResponse <- resolver.resolveDrs(drsUrl, fields) - // Currently Martha only supports resolving DRS paths to access URLs or GCS paths. - downloader <- (marthaResponse.accessUrl, marthaResponse.gsUri) match { + // Currently DRS Resolver only supports resolving DRS paths to access URLs or GCS paths. + downloader <- (drsResolverResponse.accessUrl, drsResolverResponse.gsUri) match { case (Some(accessUrl), _) => - downloaderFactory.buildAccessUrlDownloader(accessUrl, downloadLoc, marthaResponse.hashes) + downloaderFactory.buildAccessUrlDownloader(accessUrl, downloadLoc, drsResolverResponse.hashes) case (_, Some(gcsPath)) => - val serviceAccountJsonOption = marthaResponse.googleServiceAccount.map(_.data.spaces2) + val serviceAccountJsonOption = drsResolverResponse.googleServiceAccount.map(_.data.spaces2) downloaderFactory.buildGcsUriDownloader( gcsPath = gcsPath, serviceAccountJsonOption = serviceAccountJsonOption, diff --git a/cromwell-drs-localizer/src/main/scala/drs/localizer/accesstokens/AccessTokenStrategy.scala b/cromwell-drs-localizer/src/main/scala/drs/localizer/accesstokens/AccessTokenStrategy.scala deleted file mode 100644 index 756f5371c0f..00000000000 --- a/cromwell-drs-localizer/src/main/scala/drs/localizer/accesstokens/AccessTokenStrategy.scala +++ /dev/null @@ -1,7 +0,0 @@ -package drs.localizer.accesstokens - -import common.validation.ErrorOr.ErrorOr - -trait AccessTokenStrategy { - def getAccessToken(): ErrorOr[String] -} diff --git a/cromwell-drs-localizer/src/main/scala/drs/localizer/accesstokens/AzureB2CAccessTokenStrategy.scala b/cromwell-drs-localizer/src/main/scala/drs/localizer/accesstokens/AzureB2CAccessTokenStrategy.scala deleted file mode 100644 index cf83171adf0..00000000000 --- a/cromwell-drs-localizer/src/main/scala/drs/localizer/accesstokens/AzureB2CAccessTokenStrategy.scala +++ /dev/null @@ -1,40 +0,0 @@ -package drs.localizer.accesstokens - -import cats.syntax.validated._ -import com.azure.identity.DefaultAzureCredentialBuilder -import com.azure.security.keyvault.secrets.{SecretClient, SecretClientBuilder} -import common.validation.ErrorOr -import common.validation.ErrorOr.{ErrorOr,ShortCircuitingFlatMap} -import drs.localizer.CommandLineArguments - -case class AzureB2CAccessTokenStrategy(commandLineArguments: CommandLineArguments) extends AccessTokenStrategy { - override def getAccessToken(): ErrorOr[String] = { - commandLineArguments match { - case CommandLineArguments(_, _, _, _, Some(vault), Some(secret), clientId) => - AzureKeyVaultClient(vault, clientId) flatMap { _.getSecret(secret) } - case invalid => s"Invalid command line arguments: $invalid".invalidNel - } - } -} - -// Note: The AzureKeyVaultClient code here is basically a copy/paste of the code temporarily living in the TES backend. -// All the current KeyVault interaction in Cromwell is temporary, but the code in the TES backend might be even more -// temporary than this. -class AzureKeyVaultClient(client: SecretClient) { - def getSecret(secretName: String): ErrorOr[String] = ErrorOr(client.getSecret(secretName).getValue) -} - -object AzureKeyVaultClient { - def apply(vaultName: String, identityClientId: Option[String]): ErrorOr[AzureKeyVaultClient] = ErrorOr { - val credentialBuilder = identityClientId.foldLeft(new DefaultAzureCredentialBuilder()) { - (builder, clientId) => builder.managedIdentityClientId(clientId) - } - - val client = new SecretClientBuilder() - .vaultUrl(s"https://${vaultName}.vault.azure.net") - .credential(credentialBuilder.build()) - .buildClient() - - new AzureKeyVaultClient(client) - } -} diff --git a/cromwell-drs-localizer/src/main/scala/drs/localizer/accesstokens/GoogleAccessTokenStrategy.scala b/cromwell-drs-localizer/src/main/scala/drs/localizer/accesstokens/GoogleAccessTokenStrategy.scala deleted file mode 100644 index 5af2b8af441..00000000000 --- a/cromwell-drs-localizer/src/main/scala/drs/localizer/accesstokens/GoogleAccessTokenStrategy.scala +++ /dev/null @@ -1,28 +0,0 @@ -package drs.localizer.accesstokens - -import cats.syntax.validated._ -import com.google.auth.oauth2.GoogleCredentials -import common.validation.ErrorOr.ErrorOr - -import scala.jdk.CollectionConverters._ -import scala.util.{Failure, Success, Try} - -/** - * Strategy for obtaining an access token from Google Application Default credentials that are assumed to already exist. - */ -case object GoogleAccessTokenStrategy extends AccessTokenStrategy { - private final val UserInfoEmailScope = "https://www.googleapis.com/auth/userinfo.email" - private final val UserInfoProfileScope = "https://www.googleapis.com/auth/userinfo.profile" - private final val UserInfoScopes = List(UserInfoEmailScope, UserInfoProfileScope) - - override def getAccessToken(): ErrorOr[String] = { - Try { - val scopedCredentials = GoogleCredentials.getApplicationDefault().createScoped(UserInfoScopes.asJava) - scopedCredentials.refreshAccessToken().getTokenValue - } match { - case Success(null) => "null token value attempting to refresh access token".invalidNel - case Success(value) => value.validNel - case Failure(e) => s"Failed to refresh access token: ${e.getMessage}".invalidNel - } - } -} diff --git a/cromwell-drs-localizer/src/main/scala/drs/localizer/downloaders/GcsUriDownloader.scala b/cromwell-drs-localizer/src/main/scala/drs/localizer/downloaders/GcsUriDownloader.scala index 345a14f63e4..d4c81af6300 100644 --- a/cromwell-drs-localizer/src/main/scala/drs/localizer/downloaders/GcsUriDownloader.scala +++ b/cromwell-drs-localizer/src/main/scala/drs/localizer/downloaders/GcsUriDownloader.scala @@ -19,7 +19,7 @@ case class GcsUriDownloader(gcsUrl: String, val copyProcess = serviceAccountJson match { case Some(sa) => - // if Martha returned a SA, use that SA for gsutil instead of default credentials + // if DRS Resolver returned a SA, use that SA for gsutil instead of default credentials val tempCredentialDir: Path = Files.createTempDirectory("gcloudTemp_").toAbsolutePath val saJsonPath: Path = tempCredentialDir.resolve("sa.json") Files.write(saJsonPath, sa.getBytes(StandardCharsets.UTF_8)) @@ -27,7 +27,7 @@ case class GcsUriDownloader(gcsUrl: String, val copyCommand = Seq("bash", "-c", generateDownloadScript(gcsUrl, Option(saJsonPath))) Process(copyCommand, None, extraEnv.toSeq: _*) case None => - // No SA returned from Martha. gsutil will use the application default credentials. + // No SA returned from DRS Resolver. gsutil will use the application default credentials. val copyCommand = Seq("bash", "-c", generateDownloadScript(gcsUrl, None)) Process(copyCommand) } @@ -50,11 +50,11 @@ case class GcsUriDownloader(gcsUrl: String, def setServiceAccount(): String = { saJsonPathOption match { case Some(saJsonPath) => - s"""# Set gsutil to use the service account returned from Martha + s"""# Set gsutil to use the service account returned from the DRS Resolver |gcloud auth activate-service-account --key-file=$saJsonPath > gcloud_output.txt 2>&1 |RC_GCLOUD=$$? |if [ "$$RC_GCLOUD" != "0" ]; then - | echo "Failed to activate service account returned from Martha. File won't be downloaded. Error: $$(cat gcloud_output.txt)" >&2 + | echo "Failed to activate service account returned from the DRS Resolver. File won't be downloaded. Error: $$(cat gcloud_output.txt)" >&2 | exit "$$RC_GCLOUD" |else | echo "Successfully activated service account; Will continue with download. $$(cat gcloud_output.txt)" diff --git a/cromwell-drs-localizer/src/main/scala/drs/localizer/downloaders/GetmChecksum.scala b/cromwell-drs-localizer/src/main/scala/drs/localizer/downloaders/GetmChecksum.scala index 252971c0b5e..2a39a6543a3 100644 --- a/cromwell-drs-localizer/src/main/scala/drs/localizer/downloaders/GetmChecksum.scala +++ b/cromwell-drs-localizer/src/main/scala/drs/localizer/downloaders/GetmChecksum.scala @@ -4,9 +4,8 @@ import cats.syntax.validated._ import cloud.nio.impl.drs.AccessUrl import common.validation.ErrorOr.ErrorOr import drs.localizer.downloaders.AccessUrlDownloader.Hashes -import mouse.all.anySyntaxMouse -import org.apache.commons.codec.binary.Base64.{decodeBase64, isBase64} -import org.apache.commons.codec.binary.Hex.encodeHexString +import org.apache.commons.codec.binary.Base64.encodeBase64String +import org.apache.commons.codec.binary.Hex.decodeHex import org.apache.commons.text.StringEscapeUtils @@ -26,23 +25,18 @@ sealed trait GetmChecksum { } case class Md5(override val rawValue: String) extends GetmChecksum { - override def value: ErrorOr[String] = { - val trimmed = rawValue.trim - if (trimmed.matches("[A-Fa-f0-9]+")) - trimmed.validNel - // TDR currently returns a base64-encoded MD5 because that's what Azure seems to do. However, - // the DRS spec does not specify that any checksums should be base64-encoded, and `getm` also - // does not expect base64. This case handles the current behavior in the short term until - // https://broadworkbench.atlassian.net/browse/DR-2259 is done. - else if (isBase64(trimmed)) - (trimmed |> decodeBase64 |> encodeHexString).validNel - else - s"Invalid md5 checksum value is neither hex nor base64: $rawValue".invalidNel - } + override def value: ErrorOr[String] = GetmChecksum.validateHex(rawValue) override def getmAlgorithm: String = "md5" } case class Crc32c(override val rawValue: String) extends GetmChecksum { + // The DRS spec says that all hash values should be hex strings, + // but getm expects crc32c values to be base64. + override def value: ErrorOr[String] = + GetmChecksum.validateHex(rawValue) + .map(decodeHex) + .map(encodeBase64String) + override def getmAlgorithm: String = "gs_crc32c" } case class AwsEtag(override val rawValue: String) extends GetmChecksum { @@ -61,7 +55,7 @@ object GetmChecksum { def apply(hashes: Hashes, accessUrl: AccessUrl): GetmChecksum = { hashes match { case Some(hashes) if hashes.nonEmpty => - // `hashes` is keyed by the Martha names for these hash algorithms, which in turn are the forwarded DRS + // `hashes` is keyed by the DRS Resolver names for these hash algorithms, which in turn are the forwarded DRS // providers' names for the algorithms. `getm` has its own notions of what these algorithms are called. // For the specific case of `md5` the algorithm names are the same between DRS providers and `getm`, // but all of the other algorithm names currently differ between DRS providers and `getm`. @@ -88,4 +82,12 @@ object GetmChecksum { case _ => Null // None or an empty hashes map. } } + + def validateHex(s: String): ErrorOr[String] = { + val trimmed = s.trim + if (trimmed.matches("[A-Fa-f0-9]+")) + trimmed.validNel + else + s"Invalid checksum value, expected hex but got: $trimmed".invalidNel + } } diff --git a/cromwell-drs-localizer/src/test/scala/drs/localizer/CommandLineParserSpec.scala b/cromwell-drs-localizer/src/test/scala/drs/localizer/CommandLineParserSpec.scala index 3f7b96f5edc..7be30be8ac0 100644 --- a/cromwell-drs-localizer/src/test/scala/drs/localizer/CommandLineParserSpec.scala +++ b/cromwell-drs-localizer/src/test/scala/drs/localizer/CommandLineParserSpec.scala @@ -13,9 +13,8 @@ class CommandLineParserSpec extends AnyFlatSpec with CromwellTimeoutSpec with Ma private val drsObject = "drs://provider/object" private val containerPath = "/cromwell_root/my.bam" private val requesterPaysProject = "i_heart_egress" - private val azureVaultName = "Kwikset" - private val azureSecretName = "shhh" private val azureIdentityClientId = "itme@azure.com" + private val manifestPath = "/my/manifest.txt" behavior of "DRS Localizer command line parser" @@ -38,67 +37,89 @@ class CommandLineParserSpec extends AnyFlatSpec with CromwellTimeoutSpec with Ma args.containerPath.get shouldBe containerPath args.accessTokenStrategy.get shouldBe AccessTokenStrategy.Google args.googleRequesterPaysProject shouldBe empty - args.azureVaultName shouldBe empty - args.azureSecretName shouldBe empty args.azureIdentityClientId shouldBe empty + args.manifestPath shouldBe empty } it should "successfully parse with three arguments" in { val args = parser.parse(Array(drsObject, containerPath, requesterPaysProject), CommandLineArguments()).get + args.drsObject.get shouldBe drsObject + args.containerPath.get shouldBe containerPath + args.accessTokenStrategy.get shouldBe AccessTokenStrategy.Google + args.googleRequesterPaysProject.get shouldBe requesterPaysProject + args.azureIdentityClientId shouldBe empty + args.manifestPath shouldBe empty + } + + it should "successfully parse requester pays project" in { + val args = parser.parse(Array(drsObject, containerPath, "-r", requesterPaysProject), CommandLineArguments()).get args.drsObject.get shouldBe drsObject args.containerPath.get shouldBe containerPath args.accessTokenStrategy.get shouldBe AccessTokenStrategy.Google args.googleRequesterPaysProject.get shouldBe requesterPaysProject - args.azureVaultName shouldBe empty - args.azureSecretName shouldBe empty args.azureIdentityClientId shouldBe empty + args.manifestPath shouldBe empty } - it should "successfully parse an explicit Google access token strategy invocation" in { - val args = parser.parse(Array("--access-token-strategy", "google", drsObject, containerPath, requesterPaysProject), CommandLineArguments()).get + it should "successfully parse with three arguments and requester pays project" in { + val args = parser.parse(Array(drsObject, containerPath, requesterPaysProject, "-r", requesterPaysProject), CommandLineArguments()).get args.drsObject.get shouldBe drsObject args.containerPath.get shouldBe containerPath args.accessTokenStrategy.get shouldBe AccessTokenStrategy.Google args.googleRequesterPaysProject.get shouldBe requesterPaysProject - args.azureVaultName shouldBe empty - args.azureSecretName shouldBe empty args.azureIdentityClientId shouldBe empty + args.manifestPath shouldBe empty } - it should "fail to parse an Azure invocation missing vault name and secret name" in { - val args = parser.parse(Array( - "--access-token-strategy", AccessTokenStrategy.Azure, - drsObject, containerPath), CommandLineArguments()) + it should "fail if requester pays argument and flag specify different projects" in { + parser.parse(Array(drsObject, containerPath, requesterPaysProject, "-r", "boom!"), CommandLineArguments()) shouldBe None + } - args shouldBe None + it should "successfully parse args with a manifest file" in { + val args = parser.parse(Array("-m", manifestPath), CommandLineArguments()).get + + args.drsObject shouldBe empty + args.containerPath shouldBe empty + args.accessTokenStrategy.get shouldBe AccessTokenStrategy.Google + args.googleRequesterPaysProject shouldBe empty + args.azureIdentityClientId shouldBe empty + args.manifestPath.get shouldBe manifestPath } - it should "fail to parse an Azure invocation missing vault name" in { - val args = parser.parse(Array( - "--access-token-strategy", AccessTokenStrategy.Azure, - "--secret-name", azureSecretName, - drsObject, containerPath), CommandLineArguments()) + it should "fail to parse with a manifest file and one single-file arg" in { + val args = parser.parse(Array(drsObject, "--manifest-path", manifestPath), CommandLineArguments()) + args shouldBe None + } + it should "fail to parse with a manifest file and two single-file args" in { + val args = parser.parse(Array(drsObject, containerPath, "--manifest-path", manifestPath), CommandLineArguments()) args shouldBe None } - it should "fail to parse an Azure invocation missing secret name" in { + it should "successfully parse an explicit Google access token strategy invocation" in { val args = parser.parse(Array( - "--access-token-strategy", AccessTokenStrategy.Azure, - "--vault-name", azureVaultName, - drsObject, containerPath), CommandLineArguments()) + "--access-token-strategy", "google", + drsObject, + containerPath, + "--requester-pays-project", requesterPaysProject + ), CommandLineArguments()).get - args shouldBe None + args.drsObject.get shouldBe drsObject + args.containerPath.get shouldBe containerPath + args.accessTokenStrategy.get shouldBe AccessTokenStrategy.Google + args.googleRequesterPaysProject.get shouldBe requesterPaysProject + args.azureIdentityClientId shouldBe empty + args.manifestPath shouldBe empty } it should "fail to parse an Azure invocation that specifies requester pays" in { val args = parser.parse(Array( "--access-token-strategy", AccessTokenStrategy.Azure, - "--secret-name", azureSecretName, - "--vault-name", azureVaultName, - drsObject, containerPath, requesterPaysProject), CommandLineArguments()) + drsObject, + containerPath, + "--requester-pays-project", requesterPaysProject), CommandLineArguments()) args shouldBe None } @@ -106,24 +127,19 @@ class CommandLineParserSpec extends AnyFlatSpec with CromwellTimeoutSpec with Ma it should "successfully parse an Azure invocation" in { val args = parser.parse(Array( "--access-token-strategy", AccessTokenStrategy.Azure, - "--secret-name", azureSecretName, - "--vault-name", azureVaultName, drsObject, containerPath), CommandLineArguments()).get args.drsObject.get shouldBe drsObject args.containerPath.get shouldBe containerPath args.accessTokenStrategy.get shouldBe AccessTokenStrategy.Azure args.googleRequesterPaysProject shouldBe empty - args.azureVaultName.get shouldBe azureVaultName - args.azureSecretName.get shouldBe azureSecretName args.azureIdentityClientId shouldBe empty + args.manifestPath shouldBe empty } - it should "successfully parse an Azure invocation with all the trimmings" in { + it should "successfully parse an Azure invocation with identity" in { val args = parser.parse(Array( "--access-token-strategy", AccessTokenStrategy.Azure, - "--vault-name", azureVaultName, - "--secret-name", azureSecretName, "--identity-client-id", azureIdentityClientId, drsObject, containerPath), CommandLineArguments()).get @@ -131,9 +147,8 @@ class CommandLineParserSpec extends AnyFlatSpec with CromwellTimeoutSpec with Ma args.containerPath.get shouldBe containerPath args.accessTokenStrategy.get shouldBe AccessTokenStrategy.Azure args.googleRequesterPaysProject shouldBe empty - args.azureVaultName.get shouldBe azureVaultName - args.azureSecretName.get shouldBe azureSecretName args.azureIdentityClientId.get shouldBe azureIdentityClientId + args.manifestPath shouldBe empty } it should "fail to parse with an unrecognized access token strategy" in { diff --git a/cromwell-drs-localizer/src/test/scala/drs/localizer/DrsLocalizerMainSpec.scala b/cromwell-drs-localizer/src/test/scala/drs/localizer/DrsLocalizerMainSpec.scala index 84b407cec41..66799fcc099 100644 --- a/cromwell-drs-localizer/src/test/scala/drs/localizer/DrsLocalizerMainSpec.scala +++ b/cromwell-drs-localizer/src/test/scala/drs/localizer/DrsLocalizerMainSpec.scala @@ -4,10 +4,10 @@ import cats.data.NonEmptyList import cats.effect.{ExitCode, IO} import cats.syntax.validated._ import cloud.nio.impl.drs.DrsPathResolver.FatalRetryDisposition -import cloud.nio.impl.drs.{AccessUrl, DrsConfig, MarthaField, MarthaResponse} +import cloud.nio.impl.drs.{AccessUrl, DrsConfig, DrsCredentials, DrsResolverField, DrsResolverResponse} import common.assertion.CromwellTimeoutSpec +import common.validation.ErrorOr.ErrorOr import drs.localizer.MockDrsLocalizerDrsPathResolver.{FakeAccessTokenStrategy, FakeHashes} -import drs.localizer.accesstokens.AccessTokenStrategy import drs.localizer.downloaders.AccessUrlDownloader.Hashes import drs.localizer.downloaders._ import org.scalatest.flatspec.AnyFlatSpec @@ -49,12 +49,12 @@ class DrsLocalizerMainSpec extends AnyFlatSpec with CromwellTimeoutSpec with Mat mockDrsLocalizer.resolve(DrsLocalizerMain.defaultDownloaderFactory).unsafeRunSync() shouldBe expected } - it should "fail and throw error if Martha response does not have gs:// url" in { + it should "fail and throw error if the DRS Resolver response does not have gs:// url" in { val mockDrsLocalizer = new MockDrsLocalizerMain(MockDrsPaths.fakeDrsUrlWithoutAnyResolution, fakeDownloadLocation, None) the[RuntimeException] thrownBy { mockDrsLocalizer.resolve(DrsLocalizerMain.defaultDownloaderFactory).unsafeRunSync() - } should have message "No access URL nor GCS URI starting with 'gs://' found in Martha response!" + } should have message "No access URL nor GCS URI starting with 'gs://' found in the DRS Resolver response!" } it should "resolve to use the correct downloader for an access url" in { @@ -67,7 +67,7 @@ class DrsLocalizerMainSpec extends AnyFlatSpec with CromwellTimeoutSpec with Mat mockDrsLocalizer.resolve(DrsLocalizerMain.defaultDownloaderFactory).unsafeRunSync() shouldBe expected } - it should "resolve to use the correct downloader for an access url when the Martha response also contains a gs url" in { + it should "resolve to use the correct downloader for an access url when the DRS Resolver response also contains a gs url" in { val mockDrsLocalizer = new MockDrsLocalizerMain(MockDrsPaths.fakeDrsUrlWithAccessUrlAndGcsResolution, fakeDownloadLocation, None) val expected = AccessUrlDownloader( accessUrl = AccessUrl(url = "http://abc/def/ghi.bam", headers = None), downloadLoc = fakeDownloadLocation, @@ -315,23 +315,23 @@ class MockDrsLocalizerMain(drsUrl: String, class MockDrsLocalizerDrsPathResolver(drsConfig: DrsConfig) extends DrsLocalizerDrsPathResolver(drsConfig, FakeAccessTokenStrategy) { - override def resolveDrsThroughMartha(drsPath: String, fields: NonEmptyList[MarthaField.Value]): IO[MarthaResponse] = { - val marthaResponse = MarthaResponse( + override def resolveDrs(drsPath: String, fields: NonEmptyList[DrsResolverField.Value]): IO[DrsResolverResponse] = { + val drsResolverResponse = DrsResolverResponse( size = Option(1234), hashes = FakeHashes ) IO.pure(drsPath) map { case MockDrsPaths.fakeDrsUrlWithGcsResolutionOnly => - marthaResponse.copy( + drsResolverResponse.copy( gsUri = Option("gs://abc/foo-123/abc123")) case MockDrsPaths.fakeDrsUrlWithoutAnyResolution => - marthaResponse + drsResolverResponse case MockDrsPaths.fakeDrsUrlWithAccessUrlResolutionOnly => - marthaResponse.copy( + drsResolverResponse.copy( accessUrl = Option(AccessUrl(url = "http://abc/def/ghi.bam", headers = None))) case MockDrsPaths.fakeDrsUrlWithAccessUrlAndGcsResolution => - marthaResponse.copy( + drsResolverResponse.copy( accessUrl = Option(AccessUrl(url = "http://abc/def/ghi.bam", headers = None)), gsUri = Option("gs://some/uri")) case e => throw new RuntimeException(s"Unexpected exception in DRS localization test code: $e") @@ -341,5 +341,7 @@ class MockDrsLocalizerDrsPathResolver(drsConfig: DrsConfig) extends object MockDrsLocalizerDrsPathResolver { val FakeHashes: Option[Map[String, String]] = Option(Map("md5" -> "abc123", "crc32c" -> "34fd67")) - val FakeAccessTokenStrategy: AccessTokenStrategy = () => "testing code: do not call me".invalidNel + val FakeAccessTokenStrategy: DrsCredentials = new DrsCredentials { + override def getAccessToken: ErrorOr[String] = "testing code: do not call me".invalidNel + } } diff --git a/cromwell-drs-localizer/src/test/scala/drs/localizer/downloaders/GcsUriDownloaderSpec.scala b/cromwell-drs-localizer/src/test/scala/drs/localizer/downloaders/GcsUriDownloaderSpec.scala index 66c4ff6fabd..07eb5ede181 100644 --- a/cromwell-drs-localizer/src/test/scala/drs/localizer/downloaders/GcsUriDownloaderSpec.scala +++ b/cromwell-drs-localizer/src/test/scala/drs/localizer/downloaders/GcsUriDownloaderSpec.scala @@ -11,7 +11,7 @@ class GcsUriDownloaderSpec extends AnyFlatSpec with CromwellTimeoutSpec with Mat val fakeDownloadLocation = "/root/foo/foo-123.bam" val fakeRequesterPaysId = "fake-billing-project" - it should "return correct download script for a drs url without Requester Pays ID and Google SA returned from Martha" in { + it should "return correct download script for a drs url without Requester Pays ID and Google SA returned from the DRS Resolver" in { val gcsUrl = "gs://foo/bar.bam" val downloader = new GcsUriDownloader( gcsUrl = gcsUrl, @@ -44,7 +44,7 @@ class GcsUriDownloaderSpec extends AnyFlatSpec with CromwellTimeoutSpec with Mat downloader.generateDownloadScript(gcsUrl = gcsUrl, saJsonPathOption = None) shouldBe expectedDownloadScript } - it should "inject Requester Pays flag & gcloud auth using SA returned from Martha" in { + it should "inject Requester Pays flag & gcloud auth using SA returned from the DRS Resolver" in { val gcsUrl = "gs://foo/bar.bam" val downloader = new GcsUriDownloader( gcsUrl = gcsUrl, @@ -60,11 +60,11 @@ class GcsUriDownloaderSpec extends AnyFlatSpec with CromwellTimeoutSpec with Mat s"""set -euo pipefail |set +e | - |# Set gsutil to use the service account returned from Martha + |# Set gsutil to use the service account returned from the DRS Resolver |gcloud auth activate-service-account --key-file=${fakeSAJsonPath.toString} > gcloud_output.txt 2>&1 |RC_GCLOUD=$$? |if [ "$$RC_GCLOUD" != "0" ]; then - | echo "Failed to activate service account returned from Martha. File won't be downloaded. Error: $$(cat gcloud_output.txt)" >&2 + | echo "Failed to activate service account returned from the DRS Resolver. File won't be downloaded. Error: $$(cat gcloud_output.txt)" >&2 | exit "$$RC_GCLOUD" |else | echo "Successfully activated service account; Will continue with download. $$(cat gcloud_output.txt)" diff --git a/cromwell-drs-localizer/src/test/scala/drs/localizer/downloaders/GetmChecksumSpec.scala b/cromwell-drs-localizer/src/test/scala/drs/localizer/downloaders/GetmChecksumSpec.scala index 24ea087cb8f..69c063ff616 100644 --- a/cromwell-drs-localizer/src/test/scala/drs/localizer/downloaders/GetmChecksumSpec.scala +++ b/cromwell-drs-localizer/src/test/scala/drs/localizer/downloaders/GetmChecksumSpec.scala @@ -31,9 +31,10 @@ class GetmChecksumSpec extends AnyFlatSpec with CromwellTimeoutSpec with Matcher val results = Table( ("description", "algorithm", "expected"), ("md5 hex", Md5("abcdef"), "--checksum-algorithm 'md5' --checksum abcdef".validNel), - ("md5 base64", Md5("cR84lXY1y17c3q7/7riLEA=="), "--checksum-algorithm 'md5' --checksum 711f38957635cb5edcdeaeffeeb88b10".validNel), - ("md5 gibberish", Md5("what is this???"), "Invalid md5 checksum value is neither hex nor base64: what is this???".invalidNel), - ("crc32c", Crc32c("012345"), "--checksum-algorithm 'gs_crc32c' --checksum 012345".validNel), + ("md5 base64", Md5("cR84lXY1y17c3q7/7riLEA=="), "Invalid checksum value, expected hex but got: cR84lXY1y17c3q7/7riLEA==".invalidNel), + ("md5 gibberish", Md5("what is this???"), "Invalid checksum value, expected hex but got: what is this???".invalidNel), + ("crc32c", Crc32c("012345"), "--checksum-algorithm 'gs_crc32c' --checksum ASNF".validNel), + ("crc32c gibberish", Crc32c("????"), "Invalid checksum value, expected hex but got: ????".invalidNel), ("AWS ETag", AwsEtag("012345"), "--checksum-algorithm 's3_etag' --checksum 012345".validNel), // Escape checksum values constructed from unvalidated data returned by DRS servers. ("Unsupported", Unsupported("Robert'); DROP TABLE Students;\n --\\"), raw"--checksum-algorithm 'null' --checksum Robert\'\)\;\ DROP\ TABLE\ Students\;\ --\\".validNel), @@ -46,4 +47,19 @@ class GetmChecksumSpec extends AnyFlatSpec with CromwellTimeoutSpec with Matcher } } } + + it should "correctly validate hex strings" in { + val results = Table( + ("test string", "expected"), + ("", "Invalid checksum value, expected hex but got: ".invalidNel), + (" ", "Invalid checksum value, expected hex but got: ".invalidNel), + ("myfavoritestring", "Invalid checksum value, expected hex but got: myfavoritestring".invalidNel), + (" AbC123 ", "AbC123".validNel), + ("456", "456".validNel), + ) + + forAll(results) { (testString, expected) => + GetmChecksum.validateHex(testString) shouldBe expected + } + } } diff --git a/cromwell.example.backends/cromwell.examples.conf b/cromwell.example.backends/cromwell.examples.conf index 49b676a2c7c..4ca250d1202 100644 --- a/cromwell.example.backends/cromwell.examples.conf +++ b/cromwell.example.backends/cromwell.examples.conf @@ -308,17 +308,6 @@ languages { } } } - CWL { - versions { - "v1.0" { - # language-factory = "languages.cwl.CwlV1_0LanguageFactory" - # config { - # strict-validation: false - # enabled: true - # } - } - } - } } # Here is where you can define the backend providers that Cromwell understands. diff --git a/cwl/src/main/resources/reference.conf b/cwl/src/main/resources/reference.conf deleted file mode 100644 index 0d0d8a980f9..00000000000 --- a/cwl/src/main/resources/reference.conf +++ /dev/null @@ -1,3 +0,0 @@ -cwltool-runner { - class = "cwl.CwltoolHeterodon" -} diff --git a/cwl/src/main/scala/cwl/ArgumentToCommandPart.scala b/cwl/src/main/scala/cwl/ArgumentToCommandPart.scala deleted file mode 100644 index ca67eaa7ab3..00000000000 --- a/cwl/src/main/scala/cwl/ArgumentToCommandPart.scala +++ /dev/null @@ -1,26 +0,0 @@ -package cwl - -import cwl.command.StringCommandPart -import shapeless._ -import wom.CommandPart - -object ArgumentToCommandPart extends Poly1 { - type MakeCommandPart = (Boolean, ExpressionLib) => CommandPart - implicit def caseStringOrExpression: Case.Aux[StringOrExpression, MakeCommandPart] = at { - _.fold(this) - } - - implicit def caseExpression: Case.Aux[Expression, MakeCommandPart] = at { - CwlExpressionCommandPart.apply - } - - implicit def caseString: Case.Aux[String, MakeCommandPart] = at { - string => - (_, _) => - StringCommandPart(string) - } - - implicit def caseCommandLineBinding: Case.Aux[ArgumentCommandLineBinding, MakeCommandPart] = at { - ArgumentCommandLineBindingCommandPart.apply - } -} diff --git a/cwl/src/main/scala/cwl/BaseCommandToCommandParts.scala b/cwl/src/main/scala/cwl/BaseCommandToCommandParts.scala deleted file mode 100644 index 2f597918082..00000000000 --- a/cwl/src/main/scala/cwl/BaseCommandToCommandParts.scala +++ /dev/null @@ -1,11 +0,0 @@ -package cwl - -import cwl.command.StringCommandPart -import shapeless.Poly1 -import wom.CommandPart - -object BaseCommandToCommandParts extends Poly1 { - implicit def one = at[String] { Seq(_).map(StringCommandPart.apply): Seq[CommandPart] } - - implicit def many = at[Array[String]] { _.toSeq.map(StringCommandPart.apply): Seq[CommandPart] } -} diff --git a/cwl/src/main/scala/cwl/CommandLineTool.scala b/cwl/src/main/scala/cwl/CommandLineTool.scala deleted file mode 100644 index 49a2de8c861..00000000000 --- a/cwl/src/main/scala/cwl/CommandLineTool.scala +++ /dev/null @@ -1,439 +0,0 @@ -package cwl - -import cats.data.{NonEmptyList, OptionT} -import cats.syntax.apply._ -import cats.syntax.traverse._ -import cats.syntax.validated._ -import cats.instances.list._ -import cats.instances.future._ -import common.Checked -import common.validation.ErrorOr._ -import cwl.CommandLineTool._ -import cwl.CwlVersion._ -import cwl.internal.CommandPartSortingAlgorithm -import io.circe.Json -import shapeless.syntax.singleton._ -import shapeless.{:+:, CNil, Coproduct, Poly1, Witness} -import wom.callable.CommandTaskDefinition.{EvaluatedOutputs, OutputFunctionResponse} -import wom.callable.{Callable, CallableTaskDefinition, ContainerizedInputExpression} -import wom.expression.{IoFunctionSet, ValueAsAnExpression, WomExpression} -import wom.graph.GraphNodePort.OutputPort -import wom.types.{WomArrayType, WomIntegerType, WomOptionalType} -import wom.util.YamlUtils -import wom.values.{WomArray, WomEvaluatedCallInputs, WomGlobFile, WomInteger, WomString, WomValue} -import wom.{CommandPart, RuntimeAttributes, RuntimeAttributesKeys} - -import scala.concurrent.{ExecutionContext, Future} -import scala.math.Ordering - -/** - * @param `class` This _should_ always be "CommandLineTool," however the spec does not -er- specify this. - */ -case class CommandLineTool private( - inputs: Array[CommandInputParameter], - outputs: Array[CommandOutputParameter], - `class`: Witness.`"CommandLineTool"`.T, - id: String, - requirements: Option[Array[Requirement]], - hints: Option[Array[Hint]], - label: Option[String], - doc: Option[String], - cwlVersion: Option[CwlVersion], - baseCommand: Option[BaseCommand], - arguments: Option[Array[CommandLineTool.Argument]], - stdin: Option[StringOrExpression], - stderr: Option[StringOrExpression], - stdout: Option[StringOrExpression], - successCodes: Option[Array[Int]], - temporaryFailCodes: Option[Array[Int]], - permanentFailCodes: Option[Array[Int]], - `$namespaces`: Option[Map[String, String]], - `$schemas`: Option[Array[String]] - ) extends Tool { - - def asCwl: Cwl = Coproduct[Cwl](this) - - lazy val baseCommandPart: List[CommandPart] = baseCommand.toList.flatMap(_.fold(BaseCommandToCommandParts)) - - /* - * The command template is built following the rules described here: http://www.commonwl.org/v1.0/CommandLineTool.html#Input_binding - * - The baseCommand goes first - * - Then the arguments are assigned a sorting key and transformed into a CommandPart - * - Finally the inputs are folded one by one into a CommandPartsList - * - arguments and inputs CommandParts are sorted according to their sort key - */ - private [cwl] def buildCommandTemplate: CommandPartExpression[List[CommandPart]] = - ( - CommandPartSortingAlgorithm.inputBindingsCommandParts(inputs), - CommandPartSortingAlgorithm.argumentCommandParts(arguments) - ).mapN ( _ ++ _ ). - //sort the highest level, then recursively pull in the nested (and sorted) command parts within each SortKeyAndCommandPart - map{ _.sorted.flatMap(_.sortedCommandParts) }. - map(baseCommandPart ++ _) - - // This seems like it makes sense only for CommandLineTool and hence is not abstracted in Tool. If this assumption is wrong it could be moved up. - private def environmentDefs(requirementsAndHints: List[Requirement], expressionLib: ExpressionLib): ErrorOr[Map[String, WomExpression]] = { - // For environment variables we need to make sure that we aren't being asked to evaluate expressions from a containing - // workflow step or its containing workflow or anything containing the workflow. The current structure of this code - // is not prepared to evaluate those expressions. Actually this is true for attributes too and we're totally not - // checking for this condition there. Blurgh. - // TODO CWL: for runtime attributes, detect unevaluatable expressions in the containment hierarchy. - - // This traverses all `EnvironmentDef`s within all `EnvVarRequirement`s. The spec doesn't appear to say how to handle - // duplicate `envName` keys in a single array of `EnvironmentDef`s; this code gives precedence to the last occurrence. - val allEnvVarDefs = for { - req <- requirementsAndHints - envVarReq <- req.select[EnvVarRequirement].toList - // Reverse the defs within an env var requirement so that when we fold from the right below the later defs - // will take precedence over the earlier defs. - envDef <- envVarReq.envDef.toList.reverse - } yield envDef - - // Compact the `EnvironmentDef`s. Don't convert to `WomExpression`s yet, the `StringOrExpression`s need to be - // compared to the `EnvVarRequirement`s that were defined on this tool. - val effectiveEnvironmentDefs = allEnvVarDefs.foldRight(Map.empty[String, StringOrExpression]) { - case (envVarReq, envVarMap) => envVarMap + (envVarReq.envName -> envVarReq.envValue) - } - - // These are the effective environment defs irrespective of where they were found in the - // Run / WorkflowStep / Workflow containment hierarchy. - val effectiveExpressionEnvironmentDefs = effectiveEnvironmentDefs filter { case (_, expr) => expr.select[Expression].isDefined } - - // These are only the environment defs defined on this tool. - val cltRequirements = requirements.toList.flatten ++ hints.toList.flatten.flatMap(_.select[Requirement]) - val cltEnvironmentDefExpressions = (for { - cltEnvVarRequirement <- cltRequirements flatMap { - _.select[EnvVarRequirement] - } - cltEnvironmentDef <- cltEnvVarRequirement.envDef.toList - expr <- cltEnvironmentDef.envValue.select[Expression].toList - } yield expr).toSet - - // If there is an expression in an effective environment def that wasn't defined on this tool then error out since - // there isn't currently a way of evaluating it. - val unevaluatableEnvironmentDefs = for { - (name, stringOrExpression) <- effectiveExpressionEnvironmentDefs.toList - expression <- stringOrExpression.select[Expression].toList - if !cltEnvironmentDefExpressions.contains(expression) - } yield name - - unevaluatableEnvironmentDefs match { - case Nil => - // No unevaluatable environment defs => keep on truckin' - effectiveEnvironmentDefs.foldRight(Map.empty[String, WomExpression]) { case ((envName, envValue), acc) => - acc + (envName -> envValue.fold(StringOrExpressionToWomExpression).apply(inputNames, expressionLib)) - }.validNel - case xs => - s"Could not evaluate environment variable expressions defined in the call hierarchy of tool $id: ${xs.mkString(", ")}.".invalidNel - } - } - - /* - * Custom evaluation of the outputs of the command line tool (custom as in bypasses the engine provided default implementation). - * This is needed because the output of a CWL tool might be defined by the presence of a cwl.output.json file in the output directory. - * In that case, the content of the file should be used as output. This means that otherwise provided output expressions should not be evaluated. - * This method checks for the existence of this file, and optionally return a Map[OutputPort, WomValue] if the file was found. - * It ensures all the output ports of the corresponding WomCallNode get a WomValue which is needed for the engine to keep running the workflow properly. - * If the json file happens to be missing values for one or more output ports, it's a failure. - * If the file is not present, an empty value is returned and the engine will execute its own output evaluation logic. - * - * TODO: use IO instead of Future ? - */ - final private def outputEvaluationJsonFunction(outputPorts: Set[OutputPort], - inputs: Map[String, WomValue], - ioFunctionSet: IoFunctionSet, - executionContext: ExecutionContext): OutputFunctionResponse = { - implicit val ec = executionContext - import cats.syntax.either._ - - // Convert the parsed json to Wom-usable output values - def jsonToOutputs(json: Map[String, Json]): Checked[List[(OutputPort, WomValue)]] = { - - outputPorts.toList.traverse[ErrorOr, (OutputPort, WomValue)]({ outputPort => - // If the type is optional, then we can set the value to none if there's nothing in the json - def emptyValue = outputPort.womType match { - case optional: WomOptionalType => Option((outputPort -> optional.none).validNel) - case _ => None - } - - json.get(outputPort.internalName) - .map( - _.foldWith(CwlJsonToDelayedCoercionFunction).apply(outputPort.womType) - .map(outputPort -> _) - ) - .orElse(emptyValue) - .getOrElse(s"Cannot find a value for output ${outputPort.internalName} in output json $json".invalidNel) - }).toEither - } - - // Parse content as json and return output values for each output port - def parseContent(content: String): EvaluatedOutputs = { - val yaml = YamlUtils.parse(content) - for { - parsed <- yaml.flatMap(_.as[Map[String, Json]]).leftMap(error => NonEmptyList.one(error.getMessage)) - jobOutputsMap <- jsonToOutputs(parsed) - } yield jobOutputsMap.toMap - } - - for { - // Glob for "cwl.output.json" - outputJsonGlobs <- OptionT.liftF { ioFunctionSet.glob(CwlOutputJson) } - // There can only be 0 or 1, so try to take the head of the list - outputJsonFile <- OptionT.fromOption[Future] { outputJsonGlobs.headOption } - // Read the content using the ioFunctionSet.readFile function - content <- OptionT.liftF { ioFunctionSet.readFile(outputJsonFile, None, failOnOverflow = false) } - // parse the content and validate it - outputs = parseContent(content) - } yield outputs - } - - override protected def toolAttributes: Map[String, WomExpression] = { - val codes: List[Int] = successCodes match { - case Some(c) => c.toList // Use the provided list of success codes. - case None => List(0) // Default to allowing only 0 for a success code. - } - - val arr = WomArray(WomArrayType(WomIntegerType), codes map WomInteger.apply) - Map(RuntimeAttributesKeys.ContinueOnReturnCodeKey -> ValueAsAnExpression(arr)) - } - - override def buildTaskDefinition(taskName: String, - inputDefinitions: List[_ <: Callable.InputDefinition], - outputDefinitions: List[Callable.OutputDefinition], - runtimeAttributes: RuntimeAttributes, - requirementsAndHints: List[cwl.Requirement], - expressionLib: ExpressionLib): ErrorOr[CallableTaskDefinition] = { - - def redirect(soe: Option[StringOrExpression]): Option[WomExpression] = soe map { - _.fold(StringOrExpressionToWomExpression).apply(inputNames, expressionLib) - } - - // This seems like it makes sense only for CommandLineTool and hence is not abstracted in Tool. If this assumption is wrong it could be moved up. - val adHocFileCreations: Set[ContainerizedInputExpression] = (for { - requirements <- requirements.getOrElse(Array.empty[Requirement]) - initialWorkDirRequirement <- requirements.select[InitialWorkDirRequirement].toArray - listing <- initialWorkDirRequirement.listings - } yield InitialWorkDirFileGeneratorExpression(listing, expressionLib)).toSet[ContainerizedInputExpression] - - val dockerOutputDirectory = requirementsAndHints - .flatMap(_.select[DockerRequirement]) - .flatMap(_.dockerOutputDirectory) - .headOption - - def inputsToCommandParts(inputs: WomEvaluatedCallInputs) : ErrorOr[Seq[CommandPart]] = - buildCommandTemplate.run((RequirementsAndHints(requirementsAndHints), expressionLib, inputs)) - - environmentDefs(requirementsAndHints, expressionLib) map { environmentExpressions => - CallableTaskDefinition( - taskName, - inputsToCommandParts, - runtimeAttributes, - Map.empty, - Map.empty, - outputDefinitions, - inputDefinitions, - adHocFileCreation = adHocFileCreations, - environmentExpressions = environmentExpressions, - // TODO: This doesn't work in all cases and it feels clunky anyway - find a way to sort that out - prefixSeparator = "#", - commandPartSeparator = " ", - stdinRedirection = redirect(stdin), - stdoutOverride = redirect(stdout), - stderrOverride = redirect(stderr), - additionalGlob = Option(WomGlobFile(CwlOutputJson)), - customizedOutputEvaluation = outputEvaluationJsonFunction, - homeOverride = Option(_.outputPath), - dockerOutputDirectory = dockerOutputDirectory, - sourceLocation = None - ) - } - } -} - -object CommandLineTool { - val CwlOutputJson = "cwl.output.json" - - val DefaultPosition = Coproduct[StringOrInt](0) - - // Elements of the sorting key can be either Strings or Ints - type StringOrInt = String :+: Int :+: CNil - object StringOrInt { - object String { def unapply(stringOrInt: StringOrInt): Option[String] = stringOrInt.select[String] } - object Int { def unapply(stringOrInt: StringOrInt): Option[Int] = stringOrInt.select[Int] } - } - - /* - * The algorithm described here http://www.commonwl.org/v1.0/CommandLineTool.html#Input_binding to sort the command line parts - * uses a sorting key assigned to each binding. This class represents such a key - */ - object CommandBindingSortingKey { - def empty = CommandBindingSortingKey(List.empty, List.empty) - } - case class CommandBindingSortingKey(head: List[StringOrInt], - tail: List[StringOrInt] = List.empty) { - val value = head ++ tail - - def append(binding: Option[CommandLineBinding], name: StringOrInt): CommandBindingSortingKey = binding match { - // If there's an input binding, add the position to the key (or 0) - case Some(b) => - // The spec is inconsistent about this as it says "If position is not specified, it is not added to the sorting key" - // but also that the position defaults to 0. cwltool uses 0 when there's no position so we'll do that too. - val position = b.position.map(Coproduct[StringOrInt](_)) getOrElse DefaultPosition - copy(head = head :+ position, tail = tail :+ name) - // Otherwise do nothing - case None => this - } - - /** - * Creates a new key with head and tail combined into the new head. - */ - def asNewKey = CommandBindingSortingKey(value) - } - - // Maps a sorting key to its (sorted) bindings - case class SortKeyAndCommandPart( - sortingKey: CommandBindingSortingKey, - commandPart: CommandPart, - nestedCommandParts: List[SortKeyAndCommandPart] = List.empty) { - def sortedCommandParts:List[CommandPart] = - List(commandPart) ++ nestedCommandParts.sorted.flatMap(_.sortedCommandParts) - } - - type CommandPartsList = List[SortKeyAndCommandPart] - - // Ordering for CommandBindingSortingKeyElement - implicit val SortingKeyTypeOrdering: Ordering[StringOrInt] = Ordering.fromLessThan[StringOrInt]({ - // String comparison - case (StringOrInt.String(s1), StringOrInt.String(s2)) => s1 < s2 - // Int comparison - case (StringOrInt.Int(i1), StringOrInt.Int(i2)) => i1 < i2 - // Int < String (from the spec: "Numeric entries sort before strings.") - case (StringOrInt.Int(_), StringOrInt.String(_)) => true - // String > Int - case (StringOrInt.String(_), StringOrInt.Int(_)) => false - case oh => throw new Exception(s"Programmer Error! Unexpected case match: $oh") - }) - - // https://github.com/scala/bug/issues/4097#issuecomment-292388627 - implicit def IterableSubclass[CC[X] <: Iterable[X], T: Ordering] : Ordering[CC[T]] = { - new Ordering[CC[T]] { - val ord = implicitly[Ordering[T]] - def compare(x: CC[T], y: CC[T]): Int = { - val xe = x.iterator - val ye = y.iterator - - while (xe.hasNext && ye.hasNext) { - val res = ord.compare(xe.next(), ye.next()) - if (res != 0) return res - } - - Ordering.Boolean.compare(xe.hasNext, ye.hasNext) - } - } - } - - // Ordering for a CommandBindingSortingKey - implicit val SortingKeyOrdering: Ordering[CommandBindingSortingKey] = Ordering.by(_.value) - - // Ordering for a CommandPartSortMapping: order by sorting key - implicit val SortKeyAndCommandPartOrdering: Ordering[SortKeyAndCommandPart] = Ordering.by(_.sortingKey) - - def apply(inputs: Array[CommandInputParameter] = Array.empty, - outputs: Array[CommandOutputParameter] = Array.empty, - id: String, - requirements: Option[Array[Requirement]] = None, - hints: Option[Array[Hint]] = None, - label: Option[String] = None, - doc: Option[String] = None, - cwlVersion: Option[CwlVersion] = Option(CwlVersion.Version1), - baseCommand: Option[BaseCommand] = None, - arguments: Option[Array[CommandLineTool.Argument]] = None, - stdin: Option[StringOrExpression] = None, - stderr: Option[StringOrExpression] = None, - stdout: Option[StringOrExpression] = None, - successCodes: Option[Array[Int]] = None, - temporaryFailCodes: Option[Array[Int]] = None, - permanentFailCodes: Option[Array[Int]] = None, - namespaces: Option[Map[String, String]] = None, - schemas: Option[Array[String]] = None - ): CommandLineTool = { - CommandLineTool( - inputs, - outputs, - "CommandLineTool".narrow, - id, - requirements, - hints, - label, - doc, - cwlVersion, - baseCommand, - arguments, - stdin, - stderr, - stdout, - successCodes, - temporaryFailCodes, - permanentFailCodes, - namespaces, - schemas - ) - } - - type BaseCommand = SingleOrArrayOfStrings - - type Argument = ArgumentCommandLineBinding :+: StringOrExpression :+: CNil - - case class CommandInputParameter( - id: String, - label: Option[String] = None, - secondaryFiles: Option[SecondaryFiles] = None, - format: Option[InputParameterFormat] = None, //only valid when type: File - streamable: Option[Boolean] = None, //only valid when type: File - doc: Option[Doc] = None, - inputBinding: Option[InputCommandLineBinding] = None, - default: Option[CwlAny] = None, - `type`: Option[MyriadInputType] = None) extends InputParameter - - case class CommandOutputParameter( - id: String, - label: Option[String] = None, - secondaryFiles: Option[SecondaryFiles] = None, - format: Option[OutputParameterFormat] = None, //only valid when type: File - streamable: Option[Boolean] = None, //only valid when type: File - doc: Option[Doc] = None, - outputBinding: Option[CommandOutputBinding] = None, - `type`: Option[MyriadOutputType] = None) extends OutputParameter { - - /** Overridden to strip off the intentionally uniquified bits and leave only the stuff that we want to look - * at for the purposes of determining cache hits. - * - * before: - * - * CommandOutputParameter(file:///var/folders/qh/vvrlr2q92mvb9bb45sttxll4y3gg2g/T/e17762d1-9921-46c5-833e-cb47e3c3bdfd.temp.1197611185902619026/e17762d1-9921-46c5-833e-cb47e3c3bdfd.cwl#ps/ea5165fc-8948-43ae-8dec-3c0468b56bcb/ps-stdOut,None,None,None,None,None,Some(CommandOutputBinding(Some(Inl(Inr(Inl(ps-stdOut.txt)))),None,None)),Some(Inl(Inl(File)))) - * - * after: - * - * CommandOutputParameter(ps-stdOut,None,None,None,None,None,Some(CommandOutputBinding(Some(Inl(Inr(Inl(ps-stdOut.txt)))),None,None)),Some(Inl(Inl(File)))) - * - * This is possibly too strict (i.e. some of these fields may be irrelevant for cache hit determination), but preferable - * to having false positives. - * Also two coproduct types that can be either single or Arrays have custom stringifying folds for arrays. */ - override def cacheString: String = { - val cacheableId: String = id.substring(id.lastIndexOf('/') + 1) - val cacheableSecondaryFiles = secondaryFiles map { _.fold(SecondaryFilesCacheableString)} - val cacheableType: Option[String] = `type`.map(_.fold(MyriadOutputTypeCacheableString)) - s"CommandOutputParameter($cacheableId,$label,$cacheableSecondaryFiles,$format,$streamable,$doc,$outputBinding,$cacheableType)" - } - } -} - -object StringOrExpressionToWomExpression extends Poly1 { - implicit def string: Case.Aux[String, (Set[String], ExpressionLib) => WomExpression] = at[String] { s => (inputNames, expressionLib) => - ValueAsAnExpression(WomString(s)) - } - - implicit def expression: Case.Aux[Expression, (Set[String], ExpressionLib) => WomExpression] = at[Expression] { e => (inputNames, expressionLib) => - cwl.ECMAScriptWomExpression(e, inputNames, expressionLib) - } -} diff --git a/cwl/src/main/scala/cwl/CommandOutputBinding.scala b/cwl/src/main/scala/cwl/CommandOutputBinding.scala deleted file mode 100644 index bab616f2995..00000000000 --- a/cwl/src/main/scala/cwl/CommandOutputBinding.scala +++ /dev/null @@ -1,295 +0,0 @@ -package cwl - -import cats.effect.IO -import cats.effect.IO._ -import cats.syntax.functor._ -import cats.syntax.traverse._ -import cats.syntax.validated._ -import cats.syntax.parallel._ -import cats.instances.list._ -import common.validation.ErrorOr._ -import common.validation.IOChecked._ -import common.validation.Validation._ -import wom.expression.IoFunctionSet.{IoDirectory, IoFile} -import wom.expression.{FileEvaluation, IoFunctionSet} -import wom.types._ -import wom.values._ - -/** @see CommandOutputBinding */ -case class CommandOutputBinding(glob: Option[Glob] = None, - loadContents: Option[Boolean] = None, - outputEval: Option[StringOrExpression] = None) - -object CommandOutputBinding { - /** - * TODO: WOM: WOMFILE: Need to support returning globs for primary and secondary. - * - * Right now, when numerous glob files are created, the backends place each glob result into a different subdirectory. - * Later, when the secondary files are being generated each of their globs are in different sub-directories. - * Unfortunately, the secondary file resolution assumes that the secondaries are _next to_ the primaries, not in - * sibling directories. - * - * While this is being worked on, instead of looking up globs return regular files until this can be fixed. - */ - def isRegularFile(path: String): Boolean = { - path match { - case _ if path.contains("*") => false - case _ if path.contains("?") => false - case _ => true - } - } - - /** - * Returns all the primary and secondary files that _will be_ created by this command output binding. - */ - def getOutputWomFiles(inputValues: Map[String, WomValue], - outputWomType: WomType, - commandOutputBinding: CommandOutputBinding, - secondaryFilesOption: Option[SecondaryFiles], - ioFunctionSet: IoFunctionSet, - expressionLib: ExpressionLib): IOChecked[Set[FileEvaluation]] = { - val parameterContext = ParameterContext(ioFunctionSet, expressionLib, inputs = inputValues) - - /* - CWL can output two types of "glob" path types: - - File will be a glob path - - Directory is a path that will be searched for files. It's assumed that it will not be a glob, but the spec is - unclear and doesn't specifically say how Directories are listed. If the backend uses a POSIX - `find -type f`, then a globbed path should still work. - - Either way create one of the two types as a return type that the backend should be looking for when the command - completes. - - UPDATE: - It turns out that secondary files can be directories. TODO: WOM: WOMFILE: Not sure what to do with that yet. - https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/v1.0/search.cwl#L42 - https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/v1.0/index.py#L36-L37 - */ - def primaryPathsToWomFiles(primaryPaths: List[String]): ErrorOr[List[WomFile]] = { - validate { - primaryPaths map { primaryPath => - val outputWomFlatType = outputWomType match { - case WomMaybeListedDirectoryType => WomUnlistedDirectoryType - case WomArrayType(WomMaybeListedDirectoryType) => WomUnlistedDirectoryType - case _ if isRegularFile(primaryPath) => WomSingleFileType - case _ => WomGlobFileType - } - - WomFile(outputWomFlatType, primaryPath) - } - } - } - - def secondaryFilesToWomFiles(primaryWomFiles: List[WomFile], ioFunctionSet: IoFunctionSet): IOChecked[List[WomFile]] = { - primaryWomFiles.flatTraverse[IOChecked, WomFile] { primaryWomFile => - FileParameter.secondaryFiles(primaryWomFile, - primaryWomFile.womFileType, secondaryFilesOption, parameterContext, expressionLib, ioFunctionSet) - } - } - - for { - primaryPaths <- GlobEvaluator.globs(commandOutputBinding.glob, parameterContext, expressionLib).toIOChecked - primaryWomFiles <- primaryPathsToWomFiles(primaryPaths).toIOChecked - // This sets optional = false arbitrarily for now as this code doesn't have the context to make that determination, - // the caller can change this if necessary. - primaryEvaluations = primaryWomFiles map { FileEvaluation(_, optional = false, secondary = false) } - secondaryWomFiles <- secondaryFilesToWomFiles(primaryWomFiles, ioFunctionSet) - secondaryEvaluations = secondaryWomFiles map { FileEvaluation(_, optional = false, secondary = true) } - } yield (primaryEvaluations ++ secondaryEvaluations).toSet - } - - /** - * Generates an output wom value based on the specification in command output binding. - * - * Depending on the outputWomType, the following steps will be applied as specified in the CWL spec: - * 1. glob: get a list the globbed files as our primary files - * 2. loadContents: load the contents of the primary files - * 3. outputEval: pass in the primary files to an expression to generate our return value - * 4. secondaryFiles: just before returning the value, fill in the secondary files on the return value - * - * The result type will be coerced to the output type. - */ - def generateOutputWomValue(inputValues: Map[String, WomValue], - ioFunctionSet: IoFunctionSet, - outputWomType: WomType, - commandOutputBinding: CommandOutputBinding, - secondaryFilesCoproduct: Option[SecondaryFiles], - formatCoproduct: Option[StringOrExpression], - expressionLib: ExpressionLib): IOChecked[WomValue] = { - import ioFunctionSet.cs - - val parameterContext = ParameterContext(ioFunctionSet, expressionLib, inputs = inputValues) - - // 3. outputEval: pass in the primary files to an expression to generate our return value - def evaluateWomValue(womFilesArray: WomArray): ErrorOr[WomValue] = { - commandOutputBinding.outputEval match { - case Some(StringOrExpression.String(string)) => WomString(string).valid - case Some(StringOrExpression.Expression(expression)) => - val outputEvalParameterContext = parameterContext.copy(self = womFilesArray) - ExpressionEvaluator.eval(expression, outputEvalParameterContext) - case None => - womFilesArray.valid - case oh => throw new Exception(s"Programmer Error! Unexpected case match: $oh") - } - } - - // Used to retrieve the file format to be injected into a file result. - def formatOptionErrorOr = OutputParameter.format(formatCoproduct, parameterContext, expressionLib) - - // 4. secondaryFiles: just before returning the value, fill in the secondary files on the return value - def populateSecondaryFiles(evaluatedWomValue: WomValue): IOChecked[WomValue] = { - for { - formatOption <- formatOptionErrorOr.toIOChecked - womValue <- FileParameter.populateSecondaryFiles( - evaluatedWomValue, - secondaryFilesCoproduct, - formatOption, - parameterContext, - expressionLib, - ioFunctionSet - ) - } yield womValue - } - - // CWL tells us the type this output is expected to be. Attempt to coerce the actual output into this type. - def coerceWomValue(populatedWomValue: WomValue): ErrorOr[WomValue] = { - (outputWomType, populatedWomValue) match { - - case (womType: WomArrayType, womValue: WomArray) => - // Array -from- Array then coerce normally - // NOTE: this evaluates to the same as the last case, but guards against the next cases accidentally matching - womType.coerceRawValue(womValue).toErrorOr - - case (womType: WomArrayType, womValue) => - // Array -from- Single then coerce a single value to an array - womType.coerceRawValue(womValue).toErrorOr.map(value => WomArray(womType, List(value))) - - case (womType, womValue: WomArray) if womValue.value.lengthCompare(1) == 0 => - // Single -from- Array then coerce the head (if there's only a head) - womType.coerceRawValue(womValue.value.head).toErrorOr - - case (womType, womValue) => - // to then coerce normally - womType.coerceRawValue(womValue).toErrorOr - - } - } - - for { - // 1. glob: get a list the globbed files as our primary files - primaryPaths <- GlobEvaluator.globs(commandOutputBinding.glob, parameterContext, expressionLib).toIOChecked - - // 2. loadContents: load the contents of the primary files - primaryAsDirectoryOrFiles <- primaryPaths.parTraverse[IOChecked, List[WomFile]] { - loadPrimaryWithContents(ioFunctionSet, outputWomType, commandOutputBinding) - } map (_.flatten) - - // Make globbed files absolute paths by prefixing them with the output dir if necessary - absolutePaths = primaryAsDirectoryOrFiles.map(_.mapFile(ioFunctionSet.pathFunctions.relativeToHostCallRoot)) - - // Load file size - withFileSizes <- absolutePaths.parTraverse[IOChecked, WomFile](_.withSize(ioFunctionSet).to[IOChecked]) - - womFilesArray = WomArray(withFileSizes) - - // 3. outputEval: pass in the primary files to an expression to generate our return value - evaluatedWomValue <- evaluateWomValue(womFilesArray).toIOChecked - - // 4. secondaryFiles: just before returning the value, fill in the secondary files on the return value - populatedWomValue <- populateSecondaryFiles(evaluatedWomValue) - - // CWL tells us the type this output is expected to be. Attempt to coerce the actual output into this type. - coercedWomValue <- coerceWomValue(populatedWomValue).toIOChecked - } yield coercedWomValue - } - - /** - * Given a cwl glob path and an output type, gets the listing using the ioFunctionSet, and optionally loads the - * contents of the file(s). - */ - private def loadPrimaryWithContents(ioFunctionSet: IoFunctionSet, - outputWomType: WomType, - commandOutputBinding: CommandOutputBinding) - (cwlPath: String): IOChecked[List[WomFile]] = { - import ioFunctionSet.cs - - /* - For each file matched in glob, read up to the first 64 KiB of text from the file and place it in the contents field - of the file object for manipulation by outputEval. - */ - val womMaybeListedDirectoryOrFileType = outputWomType match { - case WomMaybeListedDirectoryType => WomMaybeListedDirectoryType - case WomArrayType(WomMaybeListedDirectoryType) => WomMaybeListedDirectoryType - case _ => WomMaybePopulatedFileType - } - womMaybeListedDirectoryOrFileType match { - case WomMaybeListedDirectoryType => - // Even if multiple directories are _somehow_ requested, a single flattened directory is returned. - loadDirectoryWithListing(ioFunctionSet, commandOutputBinding)(cwlPath).map(List(_)) - case WomMaybePopulatedFileType if isRegularFile(cwlPath) => - loadFileWithContents(ioFunctionSet, commandOutputBinding)(cwlPath).to[IOChecked].map(List(_)) - case WomMaybePopulatedFileType => - //TODO: HACK ALERT - DB: I am starting on ticket https://github.com/broadinstitute/cromwell/issues/3092 which will redeem me of this mortal sin. - val detritusFiles = List( - "docker_cid", - "gcs_delocalization.sh", - "gcs_localization.sh", - "gcs_transfer.sh", - "rc.tmp", - "script", - "script.background", - "script.submit", - "stderr", - "stderr.background", - "stdout", - "stdout.background", - ) - val globs: IOChecked[Seq[String]] = - ioFunctionSet.glob(cwlPath).toIOChecked - .map({ - _ - .filterNot{ s => - detritusFiles exists s.endsWith - } - }) - - globs.flatMap({ files => - files.toList.parTraverse[IOChecked, WomFile](v => loadFileWithContents(ioFunctionSet, commandOutputBinding)(v).to[IOChecked]) - }) - case other => s"Program error: $other type was not expected".invalidIOChecked - } - } - - /** - * Loads a directory with the files listed, each file with contents populated. - */ - private def loadDirectoryWithListing(ioFunctionSet: IoFunctionSet, - commandOutputBinding: CommandOutputBinding)(path: String, visited: Vector[String] = Vector.empty): IOChecked[WomMaybeListedDirectory] = { - import ioFunctionSet.cs - - for { - listing <- IO.fromFuture(IO { ioFunctionSet.listDirectory(path)(visited) }).to[IOChecked] - loadedListing <- listing.toList.parTraverse[IOChecked, WomFile]({ - case IoFile(p) => loadFileWithContents(ioFunctionSet, commandOutputBinding)(p).to[IOChecked] - case IoDirectory(p) => loadDirectoryWithListing(ioFunctionSet, commandOutputBinding)(p, visited :+ path).widen - }) - } yield WomMaybeListedDirectory(valueOption = Option(path), listingOption = Option(loadedListing)) - } - - /** - * Loads a file at path reading 64KiB of data into the contents. - */ - private def loadFileWithContents(ioFunctionSet: IoFunctionSet, - commandOutputBinding: CommandOutputBinding)(path: String): IO[WomFile] = { - val contentsOptionErrorOr = { - if (commandOutputBinding.loadContents getOrElse false) { - FileParameter.load64KiB(path, ioFunctionSet) map Option.apply - } else { - IO.pure(None) - } - } - contentsOptionErrorOr map { contentsOption => - WomMaybePopulatedFile(valueOption = Option(path), contentsOption = contentsOption) - } - } -} diff --git a/cwl/src/main/scala/cwl/CwlCodecs.scala b/cwl/src/main/scala/cwl/CwlCodecs.scala deleted file mode 100644 index d7010494bca..00000000000 --- a/cwl/src/main/scala/cwl/CwlCodecs.scala +++ /dev/null @@ -1,153 +0,0 @@ -package cwl - -import cats.data.NonEmptyList -import cats.syntax.either._ -import cats.syntax.show._ -import common.Checked -import common.validation.Checked._ -import cwl.CommandLineTool.{CommandInputParameter, CommandOutputParameter} -import cwl.CwlType.CwlType -import cwl.CwlVersion.CwlVersion -import cwl.ExpressionTool.{ExpressionToolInputParameter, ExpressionToolOutputParameter} -import cwl.InitialWorkDirRequirement.IwdrListingArrayEntry -import cwl.LinkMergeMethod.LinkMergeMethod -import cwl.ScatterMethod.ScatterMethod -import cwl.SchemaDefRequirement.SchemaDefTypes -import cwl.Workflow.{WorkflowInputParameter, WorkflowOutputParameter} -import io.circe.DecodingFailure._ -import io.circe.Json._ -import io.circe._ -import io.circe.generic.semiauto._ -import io.circe.literal._ -import io.circe.refined._ -import io.circe.shapes._ -import shapeless.Coproduct - -object CwlCodecs { - /* - Semi-auto codecs for the Cwl types - https://circe.github.io/circe/codecs/semiauto-derivation.html - - Some types, such as BaseCommand are not listed, as they are actually one-for-one type aliases. Having two exact - implicits for the same type will confuse the compiler, leading to cryptic error messages elsewhere. - - For example, a duplicated decoder such as: - - ``` - implicit lazy val decodeSingleOrArrayOfStrings: Decoder[SingleOrArrayOfStrings] = decodeCCons - implicit lazy val decodeBaseCommand: Decoder[BaseCommand] = decodeCCons - ``` - - causes cryptic errors such as: - - ``` - [error] cwl/src/main/scala/cwl/CwlCodecs.scala:123:456: could not find Lazy implicit value of type io.circe.generic.decoding.DerivedDecoder[A] - [error] implicit lazy val decodeCommandInputParameter: Decoder[CommandInputParameter] = deriveDecoder - - (etc. etc. etc.) - ``` - */ - - // Semi-Automatically derived codecs - implicit lazy val codecArgumentCommandLineBinding: Codec[ArgumentCommandLineBinding] = deriveCodec - implicit lazy val codecCommandInputParameter: Codec[CommandInputParameter] = deriveCodec - implicit lazy val codecCommandLineTool: Codec[CommandLineTool] = deriveCodec - implicit lazy val codecCommandOutputBinding: Codec[CommandOutputBinding] = deriveCodec - implicit lazy val codecCommandOutputParameter: Codec[CommandOutputParameter] = deriveCodec - implicit lazy val codecCwlType: Codec[CwlType] = Codec.codecForEnumeration(CwlType) - implicit lazy val codecCwlVersion: Codec[CwlVersion] = Codec.codecForEnumeration(CwlVersion) - implicit lazy val codecDirectory: Codec[Directory] = deriveCodec - implicit lazy val codecDockerRequirement: Codec[DockerRequirement] = deriveCodec - implicit lazy val codecEnvVarRequirement: Codec[EnvVarRequirement] = deriveCodec - implicit lazy val codecEnvironmentDef: Codec[EnvironmentDef] = deriveCodec - implicit lazy val codecExpressionDirent: Codec[ExpressionDirent] = deriveCodec - implicit lazy val codecExpressionTool: Codec[ExpressionTool] = deriveCodec - implicit lazy val codecExpressionToolInputParameter: Codec[ExpressionToolInputParameter] = deriveCodec - implicit lazy val codecExpressionToolOutputParameter: Codec[ExpressionToolOutputParameter] = deriveCodec - implicit lazy val codecFile: Codec[File] = deriveCodec - implicit lazy val codecInitialWorkDirRequirement: Codec[InitialWorkDirRequirement] = deriveCodec - implicit lazy val codecInlineJavascriptRequirement: Codec[InlineJavascriptRequirement] = deriveCodec - implicit lazy val codecInputArraySchema: Codec[InputArraySchema] = deriveCodec - implicit lazy val codecInputBinding: Codec[InputBinding] = deriveCodec - implicit lazy val codecInputCommandLineBinding: Codec[InputCommandLineBinding] = deriveCodec - implicit lazy val codecInputEnumSchema: Codec[InputEnumSchema] = deriveCodec - implicit lazy val codecInputRecordField: Codec[InputRecordField] = deriveCodec - implicit lazy val codecInputRecordSchema: Codec[InputRecordSchema] = deriveCodec - implicit lazy val codecInputResourceRequirement: Codec[DnaNexusInputResourceRequirement] = deriveCodec - implicit lazy val codecLinkMergeMethod: Codec[LinkMergeMethod] = Codec.codecForEnumeration(LinkMergeMethod) - implicit lazy val codecMultipleInputFeatureRequirement: Codec[MultipleInputFeatureRequirement] = deriveCodec - implicit lazy val codecOutputArraySchema: Codec[OutputArraySchema] = deriveCodec - implicit lazy val codecOutputEnumSchema: Codec[OutputEnumSchema] = deriveCodec - implicit lazy val codecOutputRecordField: Codec[OutputRecordField] = deriveCodec - implicit lazy val codecOutputRecordSchema: Codec[OutputRecordSchema] = deriveCodec - implicit lazy val codecResourceRequirement: Codec[ResourceRequirement] = deriveCodec - implicit lazy val codecScatterFeatureRequirement: Codec[ScatterFeatureRequirement] = deriveCodec - implicit lazy val codecScatterMethod: Codec[ScatterMethod] = Codec.codecForEnumeration(ScatterMethod) - implicit lazy val codecSchemaDefRequirement: Codec[SchemaDefRequirement] = deriveCodec - implicit lazy val codecShellCommandRequirement: Codec[ShellCommandRequirement] = deriveCodec - implicit lazy val codecSoftwarePackage: Codec[SoftwarePackage] = deriveCodec - implicit lazy val codecSoftwareRequirement: Codec[SoftwareRequirement] = deriveCodec - implicit lazy val codecStepInputExpressionRequirement: Codec[StepInputExpressionRequirement] = deriveCodec - implicit lazy val codecStringDirent: Codec[StringDirent] = deriveCodec - implicit lazy val codecSubworkflowFeatureRequirement: Codec[SubworkflowFeatureRequirement] = deriveCodec - implicit lazy val codecWorkflow: Codec[Workflow] = deriveCodec - implicit lazy val codecWorkflowInputParameter: Codec[WorkflowInputParameter] = deriveCodec - implicit lazy val codecWorkflowOutputParameter: Codec[WorkflowOutputParameter] = deriveCodec - implicit lazy val codecWorkflowStep: Codec[WorkflowStep] = deriveCodec - implicit lazy val codecWorkflowStepInput: Codec[WorkflowStepInput] = deriveCodec - implicit lazy val codecWorkflowStepOutput: Codec[WorkflowStepOutput] = deriveCodec - - // Encoders and decoders for Coproduct-based types must be explicitly derived - implicit lazy val decodeCwlAny: Decoder[CwlAny] = decodeCCons - implicit lazy val decodeExpression: Decoder[Expression] = decodeCCons - implicit lazy val decodeFileOrDirectory: Decoder[FileOrDirectory] = decodeCCons - implicit lazy val decodeGlob: Decoder[Glob] = decodeCCons - implicit lazy val decodeHint: Decoder[Hint] = decodeCCons - implicit lazy val decodeIwdrListingArrayEntry: Decoder[IwdrListingArrayEntry] = decodeCCons - implicit lazy val decodeMyriadInputInnerType: Decoder[MyriadInputInnerType] = decodeCCons - implicit lazy val decodeMyriadInputType: Decoder[MyriadInputType] = decodeCCons - implicit lazy val decodeMyriadOutputType: Decoder[MyriadOutputType] = decodeCCons - implicit lazy val decodeRequirement: Decoder[Requirement] = decodeCCons - implicit lazy val decodeResourceRequirementType: Decoder[ResourceRequirementType] = decodeCCons - implicit lazy val decodeSchemaDefTypes: Decoder[SchemaDefTypes] = decodeCCons - implicit lazy val decodeSecondaryFiles: Decoder[SecondaryFiles] = decodeCCons - implicit lazy val decodeSingleOrArrayOfStrings: Decoder[SingleOrArrayOfStrings] = decodeCCons - implicit lazy val decodeStringOrExpression: Decoder[StringOrExpression] = decodeCCons - - implicit lazy val encodeCwlAny: Encoder[CwlAny] = encodeCCons - implicit lazy val encodeExpression: Encoder[Expression] = encodeCCons - implicit lazy val encodeFileOrDirectory: Encoder[FileOrDirectory] = encodeCCons - implicit lazy val encodeGlob: Encoder[Glob] = encodeCCons - implicit lazy val encodeHint: Encoder[Hint] = encodeCCons - implicit lazy val encodeIwdrListingArrayEntry: Encoder[IwdrListingArrayEntry] = encodeCCons - implicit lazy val encodeMyriadInputInnerType: Encoder[MyriadInputInnerType] = encodeCCons - implicit lazy val encodeMyriadInputType: Encoder[MyriadInputType] = encodeCCons - implicit lazy val encodeMyriadOutputType: Encoder[MyriadOutputType] = encodeCCons - implicit lazy val encodeRequirement: Encoder[Requirement] = encodeCCons - implicit lazy val encodeResourceRequirementType: Encoder[ResourceRequirementType] = encodeCCons - implicit lazy val encodeSchemaDefTypes: Encoder[SchemaDefTypes] = encodeCCons - implicit lazy val encodeSecondaryFiles: Encoder[SecondaryFiles] = encodeCCons - implicit lazy val encodeSingleOrArrayOfStrings: Encoder[SingleOrArrayOfStrings] = encodeCCons - implicit lazy val encodeStringOrExpression: Encoder[StringOrExpression] = encodeCCons - - def decodeCwl(json: Json): Checked[Cwl] = { - findClass(json) match { - case Some("Workflow") => decodeWithErrorStringsJson[Workflow](json).map(Coproduct[Cwl].apply(_)) - case Some("CommandLineTool") => decodeWithErrorStringsJson[CommandLineTool](json).map(Coproduct[Cwl].apply(_)) - case Some("ExpressionTool") => decodeWithErrorStringsJson[ExpressionTool](json).map(Coproduct[Cwl].apply(_)) - case Some(other) => s"Class field was declared incorrectly: $other is not one of Workflow, CommandLineTool, or ExpressionTool! as seen in ${json.show}".invalidNelCheck - case None => s"Class field was omitted in ${json.show}".invalidNelCheck - } - } - - private def decodeWithErrorStringsJson[A](in: Json)(implicit d: Codec[A]): Checked[A] = - in.as[A].leftMap(_.show).leftMap(NonEmptyList.one) - - private def findClass(json: Json): Option[String] = - for { - obj <- json.asObject - map = obj.toMap - classObj <- map.get("class") - classString <- classObj.asString - } yield classString -} diff --git a/cwl/src/main/scala/cwl/CwlDecoder.scala b/cwl/src/main/scala/cwl/CwlDecoder.scala deleted file mode 100644 index b7a4125e5a3..00000000000 --- a/cwl/src/main/scala/cwl/CwlDecoder.scala +++ /dev/null @@ -1,84 +0,0 @@ -package cwl - -import better.files.{File => BFile} -import cats.data.EitherT._ -import cats.effect.IO -import cats.syntax.either._ -import cats.{Applicative, Monad} -import common.validation.ErrorOr._ -import common.validation.IOChecked._ -import common.validation.Validation._ -import cwl.preprocessor.{CwlFileReference, CwlPreProcessor, CwlReference} -import io.circe.Json - -import scala.util.Try - -object CwlDecoder { - - implicit val composedApplicative = Applicative[IO] compose Applicative[ErrorOr] - - def saladCwlFile(reference: CwlReference): IOChecked[String] = { - val cwlToolResult = - Try(CwltoolRunner.instance.salad(reference)) - .toCheckedWithContext(s"run cwltool on file ${reference.pathAsString}", throwableToStringFunction = t => t.toString) - - fromEither[IO](cwlToolResult) - } - - private lazy val cwlPreProcessor = new CwlPreProcessor() - - // TODO: WOM: During conformance testing the saladed-CWLs are referring to files in the temp directory. - // Thus we can't delete the temp directory until after the workflow is complete, like the workflow logs. - // All callers to this method should be fixed around the same time. - // https://github.com/broadinstitute/cromwell/issues/3186 - def todoDeleteCwlFileParentDirectory(cwlFile: BFile): IOChecked[Unit] = { - goIOChecked { - //cwlFile.parent.delete(swallowIOExceptions = true) - } - } - - def parseJson(json: Json, from: String): IOChecked[Cwl] = fromEither[IO](CwlCodecs.decodeCwl(json).contextualizeErrors(s"parse '$from'")) - - /** - * Notice it gives you one instance of Cwl. This has transformed all embedded files into scala object state - */ - def decodeCwlReference(reference: CwlReference)(implicit processor: CwlPreProcessor = cwlPreProcessor): IOChecked[Cwl] = { - def makeStandaloneWorkflow(): IOChecked[Json] = processor.preProcessCwl(reference) - - for { - standaloneWorkflow <- makeStandaloneWorkflow() - parsedCwl <- parseJson(standaloneWorkflow, reference.pathAsString) - } yield parsedCwl - } - - def decodeCwlFile(file: BFile, workflowRoot: Option[String] = None) = { - decodeCwlReference(CwlFileReference(file, workflowRoot)) - } - - def decodeCwlString(cwl: String, - zipOption: Option[BFile] = None, - rootName: Option[String] = None, - cwlFilename: String = "cwl_temp_file"): IOChecked[Cwl] = { - for { - parentDir <- goIOChecked(BFile.newTemporaryDirectory("cwl_temp_dir_")) // has a random long appended like `cwl_temp_dir_100000000000` - file <- fromEither[IO](parentDir./(cwlFilename + ".cwl").write(cwl).asRight) // serves as the basis for the output directory name; must remain stable across restarts - _ <- zipOption match { - case Some(zip) => goIOChecked(zip.unzipTo(parentDir)) - case None => Monad[IOChecked].unit - } - out <- decodeCwlFile(file, rootName) - _ <- todoDeleteCwlFileParentDirectory(file) - } yield out - } - - //This is used when traversing over Cwl and replacing links w/ embedded data - private[cwl] def decodeCwlAsValidated(fileName: String): IOCheckedValidated[(String, Cwl)] = { - //The SALAD preprocess step puts "file://" as a prefix to all filenames. Better files doesn't like this. - val bFileName = fileName.stripPrefix("file://") - - decodeCwlReference(CwlFileReference(BFile(bFileName), None)). - map(fileName.toString -> _). - value. - map(_.toValidated) - } -} diff --git a/cwl/src/main/scala/cwl/CwlExecutableValidation.scala b/cwl/src/main/scala/cwl/CwlExecutableValidation.scala deleted file mode 100644 index f6f8237c5ff..00000000000 --- a/cwl/src/main/scala/cwl/CwlExecutableValidation.scala +++ /dev/null @@ -1,38 +0,0 @@ -package cwl - -import common.Checked -import common.validation.Checked._ -import io.circe.Json -import wom.callable.{ExecutableCallable, TaskDefinition} -import wom.executable.Executable -import wom.executable.Executable.{InputParsingFunction, ParsedInputMap} -import wom.expression.IoFunctionSet -import wom.util.YamlUtils - -object CwlExecutableValidation { - - // Decodes the input file, and build the ParsedInputMap - private val inputCoercionFunction: InputParsingFunction = - inputFile => { - val yaml = YamlUtils.parse(inputFile) - yaml.flatMap(_.as[Map[String, Json]]) match { - case Left(error) => error.getMessage.invalidNelCheck[ParsedInputMap] - case Right(inputValue) => inputValue.map({ case (key, value) => key -> value.foldWith(CwlJsonToDelayedCoercionFunction) }).validNelCheck - } - } - - def buildWomExecutableCallable(callable: Checked[ExecutableCallable], inputFile: Option[String], ioFunctions: IoFunctionSet, strictValidation: Boolean): Checked[Executable] = { - for { - womDefinition <- callable - executable <- Executable.withInputs(womDefinition, inputCoercionFunction, inputFile, ioFunctions, strictValidation) - } yield executable - } - - def buildWomExecutable(callableTaskDefinition: Checked[TaskDefinition], inputFile: Option[String], ioFunctions: IoFunctionSet, strictValidation: Boolean): Checked[Executable] = { - for { - taskDefinition <- callableTaskDefinition - executableTaskDefinition = taskDefinition.toExecutable.toEither - executable <- CwlExecutableValidation.buildWomExecutableCallable(executableTaskDefinition, inputFile, ioFunctions, strictValidation) - } yield executable - } -} diff --git a/cwl/src/main/scala/cwl/CwlExpressionCommandPart.scala b/cwl/src/main/scala/cwl/CwlExpressionCommandPart.scala deleted file mode 100644 index f323c74dbd2..00000000000 --- a/cwl/src/main/scala/cwl/CwlExpressionCommandPart.scala +++ /dev/null @@ -1,146 +0,0 @@ -package cwl - -import cats.syntax.either._ -import cats.syntax.validated._ -import common.Checked -import common.validation.Checked._ -import common.validation.ErrorOr.ErrorOr -import wom.callable.RuntimeEnvironment -import wom.expression.IoFunctionSet -import wom.graph.LocalName -import wom.values._ -import wom.{CommandPart, InstantiatedCommand} - -import scala.language.postfixOps - -case class CwlExpressionCommandPart(expr: Expression)(hasShellCommandRequirement: Boolean, expressionLib: ExpressionLib) extends CommandPart { - override def instantiate(inputsMap: Map[LocalName, WomValue], - functions: IoFunctionSet, - valueMapper: (WomValue) => WomValue, - runtimeEnvironment: RuntimeEnvironment): ErrorOr[List[InstantiatedCommand]] = { - val stringKeyMap = inputsMap map { - case (LocalName(localName), value) => localName -> valueMapper(value) - } - val parameterContext = ParameterContext( - functions, - expressionLib, - inputs = stringKeyMap, runtimeOption = Option(runtimeEnvironment) - ) - ExpressionEvaluator.eval(expr, parameterContext) map { womValue => - List(InstantiatedCommand(valueMapper(womValue).valueString.shellQuote)) - } - } -} - -/** - * Generates command parts from a CWL Binding - */ -abstract class CommandLineBindingCommandPart(commandLineBinding: CommandLineBinding)(hasShellCommandRequirement: Boolean, expressionLib: ExpressionLib) extends CommandPart { - - - private lazy val prefixAsString = commandLineBinding.prefix.getOrElse("") - private lazy val prefixAsList = commandLineBinding.prefix.toList - private lazy val separate = commandLineBinding.effectiveSeparate - - def handlePrefix(value: String) = { - if (separate) prefixAsList :+ value else List(s"$prefixAsString$value") - } - - /** - * Value bound to this command part. - * InputCommandLineBindingCommandPart has one - * ArgumentCommandLineBindingCommandPart does not - */ - def boundValue: Option[WomValue] - - // If the bound value is defined but contains an empty optional value, we should not evaluate the valueFrom - // Conformance test "stage-unprovided-file" tests this behavior - private lazy val evaluateValueFrom = boundValue.forall { - case WomOptionalValue(_, None) => false - case _ => true - } - - override def instantiate(inputsMap: Map[LocalName, WomValue], - functions: IoFunctionSet, - valueMapper: (WomValue) => WomValue, - runtimeEnvironment: RuntimeEnvironment): ErrorOr[List[InstantiatedCommand]] = { - val stringInputsMap = inputsMap map { - case (LocalName(localName), value) => localName -> valueMapper(value) - } - val parameterContext = ParameterContext( - functions, - expressionLib, - inputs = stringInputsMap, - runtimeOption = Option(runtimeEnvironment), - self = boundValue.getOrElse(ParameterContext.EmptySelf) - ) - - val evaluatedValueFrom = commandLineBinding.optionalValueFrom flatMap { - case StringOrExpression.Expression(expression) if evaluateValueFrom => Option(ExpressionEvaluator.eval(expression, parameterContext) map valueMapper) - case StringOrExpression.String(string) if evaluateValueFrom => Option(WomString(string).validNel) - case _ => None - } - - val evaluatedWomValue: Checked[WomValue] = evaluatedValueFrom.orElse(boundValue.map(_.validNel)) match { - case Some(womValue) => womValue.map(valueMapper).toEither - case None => "Command line binding has no valueFrom field and no bound value".invalidNelCheck - } - - def applyShellQuote(value: String): String = commandLineBinding.shellQuote match { - // Only honor shellQuote = false if ShellCommandRequirement is enabled. - // Conformance test "Test that shell directives are not interpreted." - case Some(false) if hasShellCommandRequirement => value - case _ => value.shellQuote - } - - def processValue(womValue: WomValue): List[String] = womValue match { - case WomOptionalValue(_, Some(value)) => processValue(valueMapper(value)) - case WomOptionalValue(_, None) => List.empty - case _: WomString | _: WomInteger | _: WomFile | _: WomLong | _: WomFloat => handlePrefix(valueMapper(womValue).valueString) - // For boolean values, use the value of the boolean to choose whether to print the prefix or not - case WomBoolean(false) => List.empty - case WomBoolean(true) => prefixAsList - case WomArray(_, values) => commandLineBinding.itemSeparator match { - case Some(_) if values.isEmpty => List.empty - case Some(itemSeparator) => handlePrefix(values.map(valueMapper(_).valueString).mkString(itemSeparator)) - - /* - via: http://www.commonwl.org/v1.0/CommandLineTool.html#CommandLineBinding - - > If itemSeparator is specified, add prefix and the join the array into a single string with itemSeparator - > separating the items. Otherwise first add prefix, then recursively process individual elements. - - Not 100% sure if this is conformant, as we are only recurse the head into `processValue` here... However the - conformance test "Test InlineJavascriptRequirement with multiple expressions in the same tool" is happy with the - behavior. - */ - // InstantiatedCommand elements are generated here when there exists an optionalValueFrom - case None if commandLineBinding.optionalValueFrom.isDefined => values.toList match { - case head :: tail => processValue(head) ++ tail.map(valueMapper(_).valueString) - case Nil => prefixAsList - } - - // When there is no optionalValueFrom the InstantiatedCommand elements for the womValue are appended elsewhere - case _ => prefixAsList - } - case _: WomObjectLike => prefixAsList - case WomEnumerationValue(_, value) => handlePrefix(value) - case WomCoproductValue(_, value) => processValue(value) - case w => throw new RuntimeException(s"Unhandled CwlExpressionCommandPart value '$w' of type ${w.womType.stableName}") - } - - evaluatedWomValue map { v => processValue(v) map applyShellQuote map (InstantiatedCommand(_)) } toValidated - } -} - -case class InputCommandLineBindingCommandPart(commandLineBinding: InputCommandLineBinding, associatedValue: WomValue) - (hasShellCommandRequirement: Boolean, expressionLib: ExpressionLib) - extends CommandLineBindingCommandPart(commandLineBinding)(hasShellCommandRequirement, expressionLib) { - override lazy val boundValue = Option(associatedValue) -} - -case class ArgumentCommandLineBindingCommandPart(commandLineBinding: ArgumentCommandLineBinding) - (hasShellCommandRequirement: Boolean, expressionLib: ExpressionLib) - extends CommandLineBindingCommandPart(commandLineBinding)(hasShellCommandRequirement, expressionLib) { - override lazy val boundValue = None -} diff --git a/cwl/src/main/scala/cwl/CwlJsonToDelayedCoercionFunction.scala b/cwl/src/main/scala/cwl/CwlJsonToDelayedCoercionFunction.scala deleted file mode 100644 index 55dc7f9e3ba..00000000000 --- a/cwl/src/main/scala/cwl/CwlJsonToDelayedCoercionFunction.scala +++ /dev/null @@ -1,90 +0,0 @@ -package cwl - -import cats.syntax.option._ -import cats.syntax.traverse._ -import cats.syntax.validated._ -import cats.instances.list._ -import common.validation.ErrorOr.ErrorOr -import common.validation.Validation._ -import cwl.CwlCodecs._ -import io.circe.{Json, JsonNumber, JsonObject} -import wom.executable.Executable.DelayedCoercionFunction -import wom.types._ -import wom.values._ - -// With recursive types we could let circe parse it for us, but until we figure that out just parse it as Json and -// manually check for File / Directory structures -private [cwl] object CwlJsonToDelayedCoercionFunction extends Json.Folder[DelayedCoercionFunction] { - private def simpleCoercion(value: Any)(womType: WomType): ErrorOr[WomValue] = womType.coerceRawValue(value).toErrorOr - - override def onNull: DelayedCoercionFunction = WomOptionalValue.none(_).validNel - - override def onBoolean(value: Boolean): DelayedCoercionFunction = simpleCoercion(value) - - override def onNumber(value: JsonNumber): DelayedCoercionFunction = { - case WomFloatType => WomFloat(value.toDouble).validNel - case WomIntegerType => value.toInt.map(WomInteger.apply).toValidNel(s"$value is not a valid Int") - case WomLongType => value.toLong.map(WomLong.apply).toValidNel(s"$value is not a valid Long") - case other => other.coerceRawValue(value.toString).toErrorOr - } - - override def onString(value: String): DelayedCoercionFunction = simpleCoercion(value) - - override def onArray(value: Vector[Json]): DelayedCoercionFunction = { - case womArrayType: WomArrayType => - value.toList - .traverse(_.foldWith(this).apply(womArrayType.memberType)) - .map { - WomArray(womArrayType, _) - } - - case WomOptionalType(otherType) => - onArray(value).apply(otherType) - case WomCoproductType(types) => - val attempts: List[ErrorOr[WomValue]] = types.toList.map(onArray(value)(_)) - attempts.find(_.isValid).getOrElse(attempts.sequence[ErrorOr, WomValue].map(_.head)) - case WomAnyType => - // Make an array of WomAny - value.toList - .traverse(_.foldWith(this).apply(WomAnyType)) - .map { - WomArray(WomArrayType(WomAnyType), _) - } - case other => s"Cannot convert an array input value into a non array type: $other".invalidNel - } - - override def onObject(value: JsonObject): DelayedCoercionFunction = { - // CWL files are represented as Json objects, so this could be a file - case WomSingleFileType | WomMaybePopulatedFileType if value.toMap.get("class").flatMap(_.asString).contains("File") => - Json.fromJsonObject(value).as[File] match { - case Left(errors) => errors.message.invalidNel - case Right(file) => file.asWomValue - } - case WomMaybeListedDirectoryType | WomUnlistedDirectoryType if value.toMap.get("class").flatMap(_.asString).contains("Directory") => - Json.fromJsonObject(value).as[Directory] match { - case Left(errors) => errors.message.invalidNel - case Right(directory) => directory.asWomValue - } - case composite: WomCompositeType => - val foldedMap = value.toList.traverse[ErrorOr, (String, WomValue)]({ - case (k, v) => - composite.typeMap.get(k).map({ valueType => - v.foldWith(this).apply(valueType).map(k -> _) - }).getOrElse(s"Input field $k is not defined in the composite input type $composite".invalidNel) - }).map(_.toMap[String, WomValue]) - - foldedMap.map(WomObject.withTypeUnsafe(_, composite)) - case WomObjectType => - val foldedMap = value.toList.traverse[ErrorOr, (String, WomValue)]({ - case (k, v) => v.foldWith(this).apply(WomAnyType).map(k -> _) - }).map(_.toMap[String, WomValue]) - - foldedMap.map(WomObject.apply) - case WomOptionalType(otherType) => onObject(value).apply(otherType) - case WomCoproductType(types) => - val attempts: List[ErrorOr[WomValue]] = types.toList.map(onObject(value)(_)) - //these are all Invalid, just taking head to satisfy required type of WomValue instead of List[WomValue] - attempts.find(_.isValid).getOrElse(attempts.sequence[ErrorOr, WomValue].map(_.head)) - case other => s"Cannot convert an object value $value into a non array type: $other".invalidNel - } -} diff --git a/cwl/src/main/scala/cwl/CwlToRun.scala b/cwl/src/main/scala/cwl/CwlToRun.scala deleted file mode 100644 index 1571958abb5..00000000000 --- a/cwl/src/main/scala/cwl/CwlToRun.scala +++ /dev/null @@ -1,11 +0,0 @@ -package cwl - -import shapeless._ - -object CwlToRun extends Poly1 { - implicit def commandLineTool = at[CommandLineTool] { Coproduct[WorkflowStep.Run](_) } - implicit def workflow = at[Workflow] { Coproduct[WorkflowStep.Run](_) } - implicit def expressionTool = at[ExpressionTool] { Coproduct[WorkflowStep.Run](_) } -} - - diff --git a/cwl/src/main/scala/cwl/CwlType.scala b/cwl/src/main/scala/cwl/CwlType.scala deleted file mode 100644 index 7ec171d97fb..00000000000 --- a/cwl/src/main/scala/cwl/CwlType.scala +++ /dev/null @@ -1,284 +0,0 @@ -package cwl - -import cats.instances.list._ -import cats.instances.option._ -import cats.syntax.functor._ -import cats.syntax.traverse._ -import cats.syntax.validated._ -import common.validation.ErrorOr._ -import common.validation.IOChecked._ -import eu.timepit.refined._ -import mouse.all._ -import shapeless.Poly1 -import shapeless.syntax.singleton._ -import wom.expression.IoFunctionSet.{IoDirectory, IoFile} -import wom.expression.{IoFunctionSet, PathFunctionSet} -import wom.types.WomFileType -import wom.values._ - -import scala.annotation.tailrec - -object CwlType extends Enumeration { - type CwlType = Value - - val Any = Value("Any") - val Null = Value("null") - val Boolean = Value("boolean") - val Int = Value("int") - val Long = Value("long") - val Float = Value("float") - val Double = Value("double") - val String = Value("string") - val File = Value("File") - val Directory = Value("Directory") -} - -case class File private -( - `class`: W.`"File"`.T, - location: Option[String], //TODO refine w/ regex of IRI - path: Option[String], - basename: Option[String], - checksum: Option[String], - size: Option[Long], - secondaryFiles: Option[Array[FileOrDirectory]], - format: Option[String], - contents: Option[String] -) { - lazy val effectivePath = path.orElse(location) - - lazy val errorOrSecondaryFiles: ErrorOr[List[WomFile]] = { - val dirsOrFiles: List[FileOrDirectory] = secondaryFiles.getOrElse(Array.empty).toList - dirsOrFiles.traverse{ - _.fold(CwlDirectoryOrFileAsWomSingleDirectoryOrFile) - } - } - - lazy val asWomValue: ErrorOr[WomMaybePopulatedFile] = { - errorOrSecondaryFiles flatMap { secondaryFiles => - val valueOption = location.orElse(path) - (valueOption, contents) match { - case (None, None) => - "Cannot convert CWL File to WomValue without either a location, a path, or contents".invalidNel - case (None, Some(content)) => - val initializeFunction: WomMaybePopulatedFile => IoFunctionSet => IOChecked[WomValue] = { file =>ioFunctionSet => - val name = basename.getOrElse(content.hashCode.toString) - ioFunctionSet.writeFile(name, content).toIOChecked(ioFunctionSet.cs) map { writtenFile => - file.copy(valueOption = Option(writtenFile.value)) - } - } - - new WomMaybePopulatedFile(None, checksum, size, format, contents, initializeFunction = initializeFunction).valid - case (_, _) => - WomMaybePopulatedFile(valueOption, checksum, size, format, contents, secondaryFiles).valid - } - } - } -} - -object File { - def apply( - location: Option[String] = None, //TODO refine w/ regex of IRI - path: Option[String] = None, - basename: Option[String] = None, - checksum: Option[String] = None, - size: Option[Long] = None, - secondaryFiles: Option[Array[FileOrDirectory]] = None, - format: Option[String] = None, - contents: Option[String] = None): File = { - new cwl.File( - "File".narrow, - location, - path, - basename, - checksum, - size, - secondaryFiles, - format, - contents - ) - } - - def dirname(value: String): String = { - val index = value.lastIndexOf('/') - if (index >= 0) { - value.substring(0, index) - } else { - "" - } - } - - def basename(value: String): String = value.substring(value.lastIndexOf('/') + 1) - - def nameroot(value: String): String = basename(value).stripSuffix(nameext(value)) - - def nameext(value: String): String = { - val base = basename(value) - val index = base.lastIndexOf('.') - if (index >= 0) { - base.substring(index) - } else { - "" - } - } - - def recursivelyBuildDirectory(directory: String, ioFunctions: IoFunctionSet)(visited: Vector[String] = Vector.empty): IOChecked[WomMaybeListedDirectory] = { - for { - listing <- ioFunctions.listDirectory(directory)(visited).toIOChecked(ioFunctions.cs) - fileListing <- listing.toList.traverse[IOChecked, WomFile]({ - case IoDirectory(e) => recursivelyBuildDirectory(e, ioFunctions)(visited :+ directory).widen - case IoFile(e) => WomMaybePopulatedFile(e).validIOChecked.widen - }) - } yield WomMaybeListedDirectory(Option(directory), Option(fileListing)) - } - - private def asAbsoluteSiblingOfPrimary(primary: WomFile, pathFunctions: PathFunctionSet)(path: String) = { - pathFunctions.absoluteSibling(primary.value, path) - } - - def secondaryStringFile(primaryWomFile: WomFile, - stringWomFileType: WomFileType, - secondaryValue: String, - ioFunctions: IoFunctionSet): IOChecked[WomFile] = { - val secondaryRelativeFileName = File.relativeFileName(primaryWomFile.value, secondaryValue) - - // If the primary file is an absolute path, and the secondary is not, make the secondary file an absolute path and a sibling of the primary - // http://www.commonwl.org/v1.0/CommandLineTool.html#CommandInputParameter - val filePath = asAbsoluteSiblingOfPrimary(primaryWomFile, ioFunctions.pathFunctions)(secondaryRelativeFileName) - - // If the secondary file is in fact a directory, look into it and build its listing - for { - isDirectory <- ioFunctions.isDirectory(filePath).toIOChecked(ioFunctions.cs) - file <- if (isDirectory) recursivelyBuildDirectory(filePath, ioFunctions)() else WomFile(stringWomFileType, filePath).validIOChecked - } yield file - } - - def secondaryExpressionFiles(primaryWomFile: WomFile, - stringWomFileType: WomFileType, - expression: Expression, - parameterContext: ParameterContext, - expressionLib: ExpressionLib, - ioFunctions: IoFunctionSet): ErrorOr[List[WomFile]] = { - - /* - If the value is an expression, the value of self in the expression must be the primary input or output File object - to which this binding applies. - */ - val secondaryParameterContext = parameterContext.copy(self = primaryWomFile) - - /* - The expression must return a filename string relative to the path to the primary File, a File or Directory object - with either path or location and basename fields set, or an array consisting of strings or File or Directory - objects. - */ - def parseResult(nestedLevel: Int)(womValue: WomValue): ErrorOr[List[WomFile]] = { - womValue match { - case womString: WomString => - List(WomFile(stringWomFileType, womString.value |> asAbsoluteSiblingOfPrimary(primaryWomFile, ioFunctions.pathFunctions))).valid - case womMaybeListedDirectory: WomMaybeListedDirectory => - List(womMaybeListedDirectory.mapFile(asAbsoluteSiblingOfPrimary(primaryWomFile, ioFunctions.pathFunctions))).valid - case womMaybePopulatedFile: WomMaybePopulatedFile => - List(womMaybePopulatedFile.mapFile(asAbsoluteSiblingOfPrimary(primaryWomFile, ioFunctions.pathFunctions))).valid - case womArray: WomArray if nestedLevel == 0 => - womArray.value.toList flatTraverse parseResult(nestedLevel + 1) - case other => s"Not a valid secondary file: $other".invalidNel - } - } - - val possibleArrayErrorOr: ErrorOr[WomValue] = ExpressionEvaluator.eval(expression, secondaryParameterContext) - possibleArrayErrorOr.flatMap(parseResult(nestedLevel = 0)) - } - - def relativeFileName(primary: String, secondary: String): String = { - /* - If a value in secondaryFiles is a string that is not an expression, it specifies that the following pattern should - be applied to the path of the primary file to yield a filename relative to the primary File: - - 1. If string begins with one or more caret ^ characters, for each caret, remove the last file extension from the - path (the last period . and all following characters). If there are no file extensions, the path is unchanged. - - 2. Append the remainder of the string to the end of the file path. - */ - if (secondary.startsWith("^")) { - @tailrec - def stripCaret(primaryAcc: String, secondaryAcc: String): (String, String) = { - if (secondaryAcc.startsWith("^")) { - val idx = primaryAcc.lastIndexOf('.') - if (idx < 0) { - (primaryAcc, secondaryAcc.dropWhile(_ == '^')) - } else { - val primaryNext = primaryAcc.substring(0, idx) - val secondaryNext = secondaryAcc.drop(1) - stripCaret(primaryNext, secondaryNext) - } - } else { - (primaryAcc, secondaryAcc) - } - } - - val (prefix, suffix) = stripCaret(primary, secondary) - prefix + suffix - } else { - primary + secondary - } - - } -} - -case class Directory private -( - `class`: W.`"Directory"`.T, - location: Option[String], - path: Option[String], - basename: Option[String], - listing: Option[Array[FileOrDirectory]] -) { - lazy val errorOrListingOption: ErrorOr[Option[List[WomFile]]] = { - val maybeErrorOrList: Option[ErrorOr[List[WomFile]]] = - listing map { - _.toList.traverse { - _.fold(CwlDirectoryOrFileAsWomSingleDirectoryOrFile) - } - } - maybeErrorOrList.sequence[ErrorOr, List[WomFile]] - } - - lazy val asWomValue: ErrorOr[WomFile] = { - errorOrListingOption flatMap { listingOption => - path.orElse(location) map { value => - WomMaybeListedDirectory(Option(value), listingOption, basename).valid - } getOrElse { - val initializeFunction: WomMaybeListedDirectory => IoFunctionSet => IOChecked[WomValue] = { dir =>ioFunctionSet => - ioFunctionSet.createTemporaryDirectory(basename).toIOChecked(ioFunctionSet.cs) map { tempDir => - dir.copy(valueOption = Option(tempDir)) - } - } - new WomMaybeListedDirectory(None, listingOption, basename, initializeFunction = initializeFunction).valid - } - } - } -} - -object Directory { - def apply(location: Option[String] = None, - path: Option[String] = None, - basename: Option[String] = None, - listing: Option[Array[FileOrDirectory]] = None - ): Directory = - new cwl.Directory("Directory".narrow, location, path, basename, listing) - - def basename(value: String): String = { - val stripped = value.stripSuffix("/") - stripped.substring(stripped.lastIndexOf('/') + 1) - } -} - -private[cwl] object CwlDirectoryOrFileAsWomSingleDirectoryOrFile extends Poly1 { - implicit def caseFile: Case.Aux[File, ErrorOr[WomFile]] = at { - _.asWomValue - } - - implicit def caseDirectory: Case.Aux[Directory, ErrorOr[WomFile]] = at { - _.asWomValue - } -} diff --git a/cwl/src/main/scala/cwl/CwlVersion.scala b/cwl/src/main/scala/cwl/CwlVersion.scala deleted file mode 100644 index 12336b1cceb..00000000000 --- a/cwl/src/main/scala/cwl/CwlVersion.scala +++ /dev/null @@ -1,7 +0,0 @@ -package cwl - -object CwlVersion extends Enumeration { - type CwlVersion = Value - - val Version1 = Value("v1.0") -} diff --git a/cwl/src/main/scala/cwl/CwlWomExpression.scala b/cwl/src/main/scala/cwl/CwlWomExpression.scala deleted file mode 100644 index 5d77370ba49..00000000000 --- a/cwl/src/main/scala/cwl/CwlWomExpression.scala +++ /dev/null @@ -1,260 +0,0 @@ -package cwl - -import cats.instances.list._ -import cats.syntax.functor._ -import cats.syntax.traverse._ -import cats.syntax.validated._ -import common.validation.ErrorOr.{ErrorOr, ShortCircuitingFlatMap} -import common.validation.IOChecked._ -import common.validation.Validation._ -import cwl.ExpressionEvaluator.{ECMAScriptExpression, ECMAScriptFunction} -import cwl.InitialWorkDirFileGeneratorExpression._ -import cwl.InitialWorkDirRequirement.IwdrListingArrayEntry -import shapeless.Poly1 -import wom.callable.{AdHocValue, ContainerizedInputExpression} -import wom.expression.IoFunctionSet.{IoDirectory, IoFile} -import wom.expression.{FileEvaluation, IoFunctionSet, WomExpression} -import wom.types._ -import wom.values._ - -import scala.concurrent.Await -import scala.concurrent.duration.Duration - -trait CwlWomExpression extends WomExpression { - - def cwlExpressionType: WomType - - override def evaluateType(inputTypes: Map[String, WomType]): ErrorOr[WomType] = cwlExpressionType.validNel - - def expressionLib: ExpressionLib - - def evaluate(inputs: Map[String, WomValue], parameterContext: ParameterContext, expression: Expression): ErrorOr[WomValue] = - expression. - fold(EvaluateExpression). - apply(parameterContext) -} - -case class ECMAScriptWomExpression(expression: Expression, - override val inputs: Set[String], - override val expressionLib: ExpressionLib) extends CwlWomExpression { - val cwlExpressionType = WomAnyType - - override def sourceString = expression match { - case Expression.ECMAScriptExpression(s) => s.value - case Expression.ECMAScriptFunction(s) => s.value - case oh => throw new Exception(s"Programmer Error! Unexpected case match: $oh") - } - - override def evaluateValue(inputValues: Map[String, WomValue], ioFunctionSet: IoFunctionSet) = { - val pc = ParameterContext(ioFunctionSet, expressionLib, inputValues) - evaluate(inputValues, pc, expression) - } - - override def evaluateFiles(inputTypes: Map[String, WomValue], ioFunctionSet: IoFunctionSet, coerceTo: WomType) = Set.empty[FileEvaluation].validNel -} - -final case class InitialWorkDirFileGeneratorExpression(entry: IwdrListingArrayEntry, expressionLib: ExpressionLib) extends ContainerizedInputExpression { - - def evaluate(inputValues: Map[String, WomValue], mappedInputValues: Map[String, WomValue], ioFunctionSet: IoFunctionSet): IOChecked[List[AdHocValue]] = { - def recursivelyBuildDirectory(directory: String): IOChecked[WomMaybeListedDirectory] = { - import cats.syntax.traverse._ - for { - listing <- ioFunctionSet.listDirectory(directory)().toIOChecked(ioFunctionSet.cs) - fileListing <- listing.toList.traverse[IOChecked, WomFile] { - case IoDirectory(e) => recursivelyBuildDirectory(e).widen - case IoFile(e) => WomSingleFile(e).validIOChecked.widen - } - } yield WomMaybeListedDirectory(Option(directory), Option(fileListing)) - } - - inputValues.toList.traverse[IOChecked, (String, WomValue)]({ - case (k, v: WomMaybeListedDirectory) => - val absolutePathString = ioFunctionSet.pathFunctions.relativeToHostCallRoot(v.value) - recursivelyBuildDirectory(absolutePathString).contextualizeErrors(s"Error building directory $absolutePathString") map { k -> _ } - case kv => kv.validIOChecked - }).map(_.toMap) - .flatMap({ updatedValues => - val unmappedParameterContext = ParameterContext(ioFunctionSet, expressionLib, updatedValues) - entry.fold(InitialWorkDirFilePoly).apply(unmappedParameterContext, mappedInputValues).toIOChecked - }) - } -} - -object InitialWorkDirFileGeneratorExpression { - type InitialWorkDirFileEvaluator = (ParameterContext, Map[String, WomValue]) => ErrorOr[List[AdHocValue]] - - /** - * Converts an InitialWorkDir. - * - * TODO: Review against the spec. Especially for Dirent values. For example: - * - * "If the value is an expression that evaluates to a Dirent object, this indicates that the File or Directory in - * entry should be added to the designated output directory with the name in entryname." - * - * - http://www.commonwl.org/v1.0/CommandLineTool.html#InitialWorkDirRequirement - * - http://www.commonwl.org/v1.0/CommandLineTool.html#Dirent - */ - object InitialWorkDirFilePoly extends Poly1 { - implicit val caseExpressionDirent: Case.Aux[ExpressionDirent, InitialWorkDirFileEvaluator] = { - at { expressionDirent => - (unmappedParameterContext, mappedInputValues) => { - - val mutableInputOption = expressionDirent.entry.fold(ExpressionToMutableInputOptionPoly) - - val entryEvaluation = - expressionDirent.entry match { - //we need to catch this special case to feed in "value-mapped" input values - case expr@Expression.ECMAScriptExpression(exprString) if exprString.value.trim() == "$(JSON.stringify(inputs))" => - val specialParameterContext = ParameterContext( - unmappedParameterContext.ioFunctionSet, - unmappedParameterContext.expressionLib, - mappedInputValues - ) - ExpressionEvaluator.eval(expr, specialParameterContext) - case _ => ExpressionEvaluator.eval(expressionDirent.entry, unmappedParameterContext) - } - - val womValueErrorOr: ErrorOr[AdHocValue] = entryEvaluation flatMap { - case womFile: WomFile => - val errorOrEntryName: ErrorOr[Option[String]] = expressionDirent.entryname match { - case Some(actualEntryName) => actualEntryName.fold(EntryNamePoly).apply(unmappedParameterContext).map(Option.apply) - case None => None.valid - } - errorOrEntryName map { entryName => - AdHocValue(womFile, entryName, inputName = mutableInputOption) - } - case other => for { - coerced <- WomStringType.coerceRawValue(other).toErrorOr - contentString = coerced.asInstanceOf[WomString].value - // We force the entryname to be specified, and then evaluate it: - entryNameStringOrExpression <- expressionDirent.entryname.toErrorOr( - "Invalid dirent: Entry was a string but no file name was supplied") - entryName <- entryNameStringOrExpression.fold(EntryNamePoly).apply(unmappedParameterContext) - writeFile = unmappedParameterContext.ioFunctionSet.writeFile(entryName, contentString) - writtenFile <- validate(Await.result(writeFile, Duration.Inf)) - } yield AdHocValue(writtenFile, alternativeName = None, inputName = mutableInputOption) - } - - womValueErrorOr.map(List(_)) - } - } - } - - implicit val caseStringDirent: Case.Aux[StringDirent, InitialWorkDirFileEvaluator] = { - at { - stringDirent => { - (unmappedParameterContext, _) => - val womValueErrorOr = for { - entryName <- stringDirent.entryname.fold(EntryNamePoly).apply(unmappedParameterContext) - contentString = stringDirent.entry - writeFile = unmappedParameterContext.ioFunctionSet.writeFile(entryName, contentString) - writtenFile <- validate(Await.result(writeFile, Duration.Inf)) - } yield writtenFile - - womValueErrorOr.map(AdHocValue(_, alternativeName = None, inputName = None)).map(List(_)) - } - } - } - - implicit val caseExpression: Case.Aux[Expression, InitialWorkDirFileEvaluator] = { - at { expression => - (unmappedParameterContext, _) => { - // A single expression which must evaluate to an array of Files - val expressionEvaluation = ExpressionEvaluator.eval(expression, unmappedParameterContext) - - expressionEvaluation flatMap { - case array: WomArray if array.value.forall(_.isInstanceOf[WomFile]) => - array.value.toList.map(_.asInstanceOf[WomFile]).map(AdHocValue(_, alternativeName = None, inputName = None)).validNel - case file: WomFile => - List(AdHocValue(file, alternativeName = None, inputName = None)).validNel - case other => - val error = "InitialWorkDirRequirement listing expression must be File or Array[File] but got %s: %s" - .format(other, other.womType.stableName) - error.invalidNel - } - } - } - } - - implicit val caseString: Case.Aux[String, InitialWorkDirFileEvaluator] = { - at { string => - (_, _) => { - List(AdHocValue(WomSingleFile(string), alternativeName = None, inputName = None)).valid - } - } - } - - implicit val caseStringOrExpression: Case.Aux[StringOrExpression, InitialWorkDirFileEvaluator] = { - at { - _.fold(this) - } - } - - implicit val caseFile: Case.Aux[File, InitialWorkDirFileEvaluator] = { - at { file => - (_, _) => { - file.asWomValue.map(AdHocValue(_, alternativeName = None, inputName = None)).map(List(_)) - } - } - } - - implicit val caseDirectory: Case.Aux[Directory, InitialWorkDirFileEvaluator] = { - at { directory => - (_, _) => { - directory.asWomValue.map(AdHocValue(_, alternativeName = None, inputName = None)).map(List(_)) - } - } - } - - } - - type EntryNameEvaluator = ParameterContext => ErrorOr[String] - - object EntryNamePoly extends Poly1 { - implicit val caseString: Case.Aux[String, EntryNameEvaluator] = { - at { - string => { - _ => - string.valid - } - } - } - - implicit val caseExpression: Case.Aux[Expression, EntryNameEvaluator] = { - at { - expression => { - parameterContext => - for { - entryNameExpressionEvaluated <- ExpressionEvaluator.eval(expression, parameterContext) - entryNameValidated <- mustBeString(entryNameExpressionEvaluated) - } yield entryNameValidated - } - } - } - - private def mustBeString(womValue: WomValue): ErrorOr[String] = { - womValue match { - case WomString(s) => s.valid - case other => WomStringType.coerceRawValue(other).map(_.asInstanceOf[WomString].value).toErrorOr - } - } - } - - /** - * Searches for ECMAScript expressions that mutate input paths. - * - * @see [[AdHocValue]] - */ - object ExpressionToMutableInputOptionPoly extends Poly1 { - private val MutableInputRegex = """(?s)\s*\$\(\s*inputs\.(\w+)\s*\)\s*""".r - implicit val caseECMAScriptExpression: Case.Aux[ECMAScriptExpression, Option[String]] = { - at { eCMAScriptExpression => - eCMAScriptExpression.value match { - case MutableInputRegex(mutableInput) => Option(mutableInput) - case _ => None - } - } - } - implicit val caseECMAScriptFunction: Case.Aux[ECMAScriptFunction, Option[String]] = at { _ => None } - } -} diff --git a/cwl/src/main/scala/cwl/CwltoolRunner.scala b/cwl/src/main/scala/cwl/CwltoolRunner.scala deleted file mode 100644 index 834c9d80b23..00000000000 --- a/cwl/src/main/scala/cwl/CwltoolRunner.scala +++ /dev/null @@ -1,71 +0,0 @@ -package cwl - -import ammonite.ops.ImplicitWd._ -import ammonite.ops._ -import com.typesafe.config.ConfigFactory -import cwl.preprocessor.CwlReference -import org.broadinstitute.heterodon.ExecAndEval - -/** - * Interface for running cwltool. - */ -sealed trait CwltoolRunner { - def salad(reference: CwlReference): String -} - -object CwltoolRunner { - private lazy val config = ConfigFactory.load - - lazy val instance: CwltoolRunner = { - val runnerClass = config.getString("cwltool-runner.class") - Class.forName(runnerClass).getDeclaredConstructor().newInstance().asInstanceOf[CwltoolRunner] - } -} - -/** - * Runs cwltool as an external process. - */ -final class CwltoolProcess extends CwltoolRunner { - override def salad(reference: CwlReference): String = { - val commandResult: CommandResult = %%("cwltool", "--quiet", "--print-pre", reference.pathAsString) - commandResult.exitCode match { - case 0 => commandResult.out.string - case error => - throw new RuntimeException( - s"running CwlTool on file ${reference.pathAsString} resulted in exit code $error and stderr ${commandResult.err.string}") - } - } -} - -/** - * Runs cwltool via heterodon. - * - * https://github.com/broadinstitute/heterodon - */ -final class CwltoolHeterodon extends CwltoolRunner { - // Trimmed down version of - // https://github.com/common-workflow-language/cwltool/blob/1a839255795882894b4bbfea6d909a74cacb1d6a/cwltool/main.py#L355 - private val cwltoolSaladExecScript = - """|import json - |import logging - | - |from cwltool.load_tool import fetch_document, resolve_tool_uri, validate_document - |from cwltool.loghandler import _logger - | - | - |def cwltool_salad(path): - | _logger.setLevel(logging.WARN) - | uri, tool_file_uri = resolve_tool_uri(path) - | document_loader, workflowobj, uri = fetch_document(uri) - | document_loader, avsc_names, processobj, metadata, uri \ - | = validate_document(document_loader, workflowobj, uri, preprocess_only=True) - | return json.dumps(processobj, indent=4) - |""".stripMargin - - private def cwltoolSaladEvalStatement(reference: CwlReference): String = s"cwltool_salad('${reference.pathAsString}')" - - def salad(reference: CwlReference): String = { - val execAndEval = new ExecAndEval() - execAndEval.apply(cwltoolSaladExecScript, cwltoolSaladEvalStatement(reference)).asInstanceOf[String] - } -} diff --git a/cwl/src/main/scala/cwl/EnumSchema.scala b/cwl/src/main/scala/cwl/EnumSchema.scala deleted file mode 100644 index 83ff1b05813..00000000000 --- a/cwl/src/main/scala/cwl/EnumSchema.scala +++ /dev/null @@ -1,35 +0,0 @@ -package cwl - -import cats.data.NonEmptyList -import shapeless.Witness -import wom.types.WomEnumerationType - -trait EnumSchema { - - val name: Option[String] - val symbols: Array[String] - val `type`: Witness.`"enum"`.T - val label: Option[String] - - def toWomEnumerationType: WomEnumerationType = { - val symbolIds = symbols.toList.map{ - s => s.substring(s.lastIndexOf("/") + 1) - } - WomEnumerationType(NonEmptyList.fromListUnsafe(symbolIds)) - } -} - -case class InputEnumSchema( - name: Option[String], - symbols: Array[String], - `type`: Witness.`"enum"`.T = Witness("enum").value, - label: Option[String] = None, - inputBinding: Option[InputCommandLineBinding] = None) extends EnumSchema - -case class OutputEnumSchema( - name: Option[String], - symbols: Array[String], - `type`: Witness.`"enum"`.T, - label: Option[String], - outputBinding: Option[CommandOutputBinding]) extends EnumSchema - diff --git a/cwl/src/main/scala/cwl/EvaluateExpression.scala b/cwl/src/main/scala/cwl/EvaluateExpression.scala deleted file mode 100644 index 00036d49de4..00000000000 --- a/cwl/src/main/scala/cwl/EvaluateExpression.scala +++ /dev/null @@ -1,20 +0,0 @@ -package cwl - -import common.validation.ErrorOr.ErrorOr -import cwl.ExpressionEvaluator.{ECMAScriptExpression, ECMAScriptFunction} -import shapeless.Poly1 -import wom.values.WomValue - -object EvaluateExpression extends Poly1 { - implicit val script: Case.Aux[ECMAScriptExpression, ParameterContext => ErrorOr[WomValue]] = { - at { - ExpressionEvaluator.evalExpression - } - } - - implicit val function: Case.Aux[ECMAScriptFunction, ParameterContext => ErrorOr[WomValue]] = { - at { - ExpressionEvaluator.evalFunction - } - } -} diff --git a/cwl/src/main/scala/cwl/ExpressionEvaluator.scala b/cwl/src/main/scala/cwl/ExpressionEvaluator.scala deleted file mode 100644 index a3ed2c7a7c4..00000000000 --- a/cwl/src/main/scala/cwl/ExpressionEvaluator.scala +++ /dev/null @@ -1,101 +0,0 @@ -package cwl - -import cats.syntax.validated._ -import common.validation.ErrorOr._ -import cwl.internal.{CwlEcmaScriptDecoder, EcmaScriptEncoder, EcmaScriptUtil} -import eu.timepit.refined.api.Refined -import eu.timepit.refined.string.MatchesRegex -import shapeless.Witness -import wom.callable.RuntimeEnvironment -import wom.values.{WomFloat, WomInteger, WomString, WomValue} - -// http://www.commonwl.org/v1.0/CommandLineTool.html#Expressions -object ExpressionEvaluator { - // A code fragment wrapped in the $(...) syntax must be evaluated as a ECMAScript expression. - /* - There are several places in cwltool where a simple contains check is used: - - https://github.com/common-workflow-language/cwltool/blob/353dbed/cwltool/builder.py#L157 - - https://github.com/common-workflow-language/cwltool/blob/353dbed/cwltool/command_line_tool.py#L669 - - https://github.com/common-workflow-language/cwltool/blob/353dbed/cwltool/expression.py#L219 - - https://github.com/common-workflow-language/cwltool/blob/353dbed/cwltool/process.py#L645 - */ - val ECMAScriptExpressionWitness = Witness("""(?s).*\$\(.*""") - val ECMAScriptExpressionRegex = ECMAScriptExpressionWitness.value.r - type MatchesECMAScriptExpression = MatchesRegex[ECMAScriptExpressionWitness.T] - type ECMAScriptExpression = String Refined MatchesECMAScriptExpression - - // A code fragment wrapped in the ${...} syntax must be evaluated as a ECMAScript function body for an anonymous, - // zero-argument function. - val ECMAScriptFunctionWitness = Witness("""(?s)\s*\$\{(.*)\}\s*""") - val ECMAScriptFunctionRegex = ECMAScriptFunctionWitness.value.r - type MatchesECMAScriptFunction = MatchesRegex[ECMAScriptFunctionWitness.T] - type ECMAScriptFunction = String Refined MatchesECMAScriptFunction - - def eval(expr: Expression, parameterContext: ParameterContext): ErrorOr[WomValue] = { - expr.fold(EvaluateExpression).apply(parameterContext) - } - - def evalExpression(expression: ECMAScriptExpression)(parameterContext: ParameterContext): ErrorOr[WomValue] = { - def evaluator(string: String): ErrorOr[WomValue] = { - eval(string, parameterContext) - } - - ExpressionInterpolator.interpolate(expression.value, evaluator, strip_whitespace = false) - } - - def evalFunction(function: ECMAScriptFunction)(parameterContext: ParameterContext): ErrorOr[WomValue] = { - function.value match { - case ECMAScriptFunctionRegex(script) => - - val functionExpression = - s"""|(function() { - |FUNCTION_BODY - |})(); - |""".stripMargin.replace("FUNCTION_BODY", script) - - eval(functionExpression, parameterContext) - case unmatched => - s"Expression '$unmatched' was unable to be matched to regex '${ECMAScriptFunctionWitness.value}'".invalidNel - } - } - - private lazy val cwlJsDecoder = new CwlEcmaScriptDecoder() - - def eval(expr: String, parameterContext: ParameterContext): ErrorOr[WomValue] = { - val script = if (parameterContext.expressionLib.isEmpty) { - expr - } else { - parameterContext.expressionLib.mkString("", ";", s";$expr") - } - val (rawVals, mapVals) = paramValues(parameterContext) - EcmaScriptUtil.evalStructish( - script, - rawVals, - mapVals, - new EcmaScriptEncoder(), - cwlJsDecoder - ) - } - - def paramValues(parameterContext: ParameterContext): ((String, WomValue), Map[String, Map[String, WomValue]]) = { - ( - "self" -> parameterContext.self - , - Map( - "inputs" -> parameterContext.inputs, - "runtime" -> parameterContext.runtimeOption.map(cwlMap).getOrElse(Map.empty) - ) - ) - } - - def cwlMap(runtime: RuntimeEnvironment): Map[String, WomValue] = { - Map( - "outdir" -> WomString(runtime.outputPath), - "tmpdir" -> WomString(runtime.tempPath), - "cores" -> WomInteger(runtime.cores.value), - "ram" -> WomFloat(runtime.ram), - "outdirSize" -> WomFloat(runtime.outputPathSize.toDouble), - "tmpdirSize" -> WomFloat(runtime.tempPathSize.toDouble) - ) - } -} diff --git a/cwl/src/main/scala/cwl/ExpressionInterpolator.scala b/cwl/src/main/scala/cwl/ExpressionInterpolator.scala deleted file mode 100644 index 9a40ee8e06f..00000000000 --- a/cwl/src/main/scala/cwl/ExpressionInterpolator.scala +++ /dev/null @@ -1,173 +0,0 @@ -package cwl - -import cats.syntax.traverse._ -import cats.syntax.validated._ -import cats.instances.list._ -import common.validation.ErrorOr._ -import wom.types.WomNothingType -import wom.values._ - -import scala.jdk.CollectionConverters._ - -/** - * Partial copy-port of cwltool's expression.py. - * - * Minimizes refactor so that as expression.py continues to update the updates may be manually copied here also. - * - * Current omissions in `def interpolate`: - * - `def evaluator` - * - Utility not copy-ported. - * - `def interpolate` instead ignores absence of InlineJavascriptRequirement and instead always uses fullJS=True - * - `leaf = json.dumps(e, sort_keys=True)` - * - When the interpolated string is not just-a-single-expression there is undefined behavior of rendering - * non-WomPrimitive values into json. - * - Even if it did, does not sort the keys. - * - Thus we do not currently return character-for-character equality for these expressions with cwltool, however - * these cases are not explicitly defined in the CWL spec as far as I know. - * - * @see https://github.com/common-workflow-language/cwltool/blob/353dbed/cwltool/expression.py - */ -//noinspection ZeroIndexToHead,RemoveRedundantReturn -object ExpressionInterpolator { - - class SubstitutionException(message: String) extends RuntimeException(message) - - /** - * Copy-port of expression.py's scanner. - */ - private def scanner(scan: String): List[Int] = { - val DEFAULT = 0 - val DOLLAR = 1 - val PAREN = 2 - val BRACE = 3 - val SINGLE_QUOTE = 4 - val DOUBLE_QUOTE = 5 - val BACKSLASH = 6 - - var i = 0 - val stack = new java.util.Stack[Int] - stack.push(DEFAULT) - var start = 0 - while (i < scan.length) { - val state = stack.peek - val c = scan(i) - - if (state == DEFAULT) { - if (c == '$') { - stack.push(DOLLAR) - } else if (c == '\\') { - stack.push(BACKSLASH) - } - } else if (state == BACKSLASH) { - stack.pop() - if (stack.peek == DEFAULT) { - return List(i - 1, i + 1) - } - } else if (state == DOLLAR) { - if (c == '(') { - start = i - 1 - stack.push(PAREN) - } else if (c == '{') { - start = i - 1 - stack.push(BRACE) - } else { - stack.pop() - } - } else if (state == PAREN) { - if (c == '(') { - stack.push(PAREN) - } else if (c == ')') { - stack.pop() - if (stack.peek == DOLLAR) { - return List(start, i + 1) - } - } else if (c == '\'') { - stack.push(SINGLE_QUOTE) - } else if (c == '"') { - stack.push(DOUBLE_QUOTE) - } - } else if (state == BRACE) { - if (c == '{') { - stack.push(BRACE) - } else if (c == '}') { - stack.pop() - if (stack.peek == DOLLAR) { - return List(start, i + 1) - } - } else if (c == '\'') { - stack.push(SINGLE_QUOTE) - } else if (c == '"') { - stack.push(DOUBLE_QUOTE) - } - } else if (state == SINGLE_QUOTE) { - if (c == '\'') { - stack.pop() - } else if (c == '\\') { - stack.push(BACKSLASH) - } - } else if (state == DOUBLE_QUOTE) { - if (c == '"') { - stack.pop() - } else if (c == '\\') { - stack.push(BACKSLASH) - } - } - i += 1 - } - if (stack.size > 1) { - throw new SubstitutionException( - "Substitution error, unfinished block starting at position %s: %s".format(start, scan.drop(start))) - } else { - return null - } - } - - /** - * Copy-port of expression.py's interpolate. - */ - def interpolate(scanParam: String, - evaluator: String => ErrorOr[WomValue], - strip_whitespace: Boolean = true): ErrorOr[WomValue] = { - var scan = scanParam - if (strip_whitespace) { - scan = scan.trim - } - val parts = new java.util.Stack[ErrorOr[WomValue]] - var w = scanner(scan) - while (w != null) { - parts.push(WomString(scan.slice(0, w(0))).valid) - - if (scan(w(0)) == '$') { - val e = evaluator(scan.slice(w(0) + 1, w(1))) - if (w(0) == 0 && w(1) == scan.length && parts.size <= 1) { - return e - } - /* - Original-ish: - var leaf = json.dumps(e, sort_keys=True) - if (leaf(0) == '"') { - leaf = leaf.drop(1).dropRight(1) - } - Instead, push the raw WomValue. WomString's won't have been quoted as python's json.dumps does. - Other WomPrimitive's should be fine. CWL using other structures (Maps, Arrays, etc.) may have problems. - */ - val leaf = e - parts.push(leaf) - } else if (scan(w(0)) == '\\') { - val e = scan(w(1) - 1) - parts.push(WomString(e.toString).valid) - } - - scan = scan.drop(w(1)) - w = scanner(scan) - } - parts.push(WomString(scan).valid) - return parts.asScala.toList.sequence[ErrorOr, WomValue] map { list => - WomString(list.map{ - //we represent nulls as this type because Wom doesn't have a "null" value, but it does have a nothing type - case WomOptionalValue(WomNothingType, None) => "null" - case other => other.valueString - }.mkString("")) - } - } -} diff --git a/cwl/src/main/scala/cwl/ExpressionTool.scala b/cwl/src/main/scala/cwl/ExpressionTool.scala deleted file mode 100644 index 53c8a675a4c..00000000000 --- a/cwl/src/main/scala/cwl/ExpressionTool.scala +++ /dev/null @@ -1,136 +0,0 @@ -package cwl - -import cats.syntax.either._ -import cats.syntax.validated._ -import cats.instances.list._ -import common.Checked -import common.validation.Checked._ -import common.validation.ErrorOr.ErrorOr -import common.validation.Validation._ -import cwl.CwlVersion._ -import cwl.ExpressionTool.{ExpressionToolInputParameter, ExpressionToolOutputParameter} -import shapeless.{Coproduct, Witness} -import wom.RuntimeAttributes -import wom.callable.{Callable, CallableExpressionTaskDefinition} -import wom.expression.{IoFunctionSet, ValueAsAnExpression} -import wom.graph.GraphNodePort.OutputPort -import wom.types.{WomMapType, WomStringType, WomType} -import wom.values.{WomMap, WomObject, WomString, WomValue} - -case class ExpressionTool( - inputs: Array[ExpressionToolInputParameter] = Array.empty, - outputs: Array[ExpressionToolOutputParameter] = Array.empty, - `class`: Witness.`"ExpressionTool"`.T, - expression: StringOrExpression, - id: String, - requirements: Option[Array[Requirement]] = None, - hints: Option[Array[Hint]] = None, - label: Option[String] = None, - doc: Option[String] = None, - cwlVersion: Option[CwlVersion] = None, - `$namespaces`: Option[Map[String, String]] = None, - `$schemas`: Option[Array[String]] = None - ) extends Tool { - - def asCwl: Cwl = Coproduct[Cwl](this) - - def castWomValueAsMap(evaluatedExpression: WomValue): Checked[Map[String, WomValue]] = { - evaluatedExpression match { - case WomMap(WomMapType(WomStringType, _), map) => map.map{ - case (WomString(key), value) => key -> value - case (key, _) => throw new RuntimeException(s"saw a non-string value $key in wom map $evaluatedExpression typed with string keys ") - }.asRight - case obj: WomObject => obj.values.asRight - case _ => s"Could not cast value $evaluatedExpression to a Map[String, WomValue]".invalidNelCheck - } - } - - def buildTaskDefinition(taskName: String, - inputDefinitions: List[_ <: Callable.InputDefinition], - outputDefinitions: List[Callable.OutputDefinition], - runtimeAttributes: RuntimeAttributes, - requirementsAndHints: List[cwl.Requirement], - expressionLib: ExpressionLib): ErrorOr[CallableExpressionTaskDefinition] = { - - def evaluate(inputs: Map[String, WomValue], ioFunctionSet: IoFunctionSet, outputPorts: List[OutputPort]): Checked[Map[OutputPort, WomValue]] = { - val womExpression = expression match { - case StringOrExpression.String(str) => ValueAsAnExpression(WomString(str)) - case StringOrExpression.Expression(expr) => ECMAScriptWomExpression(expr, inputNames, expressionLib) - case oh => throw new Exception(s"Programmer Error! Unexpected case match: $oh") - } - - // If we expect a certain type for - def coerce(womValue: WomValue, womType: WomType): Checked[WomValue] = womType.coerceRawValue(womValue).toChecked - - /* - * Look in the object (the result of the expression evaluation) for values to match output each port. - * Ideally we'd want to find a value for each output port declared by the tool. - * However since the spec is not clear on this and cwltool seems to ignore entirely the outputs declaration, - * we'll do the same for now, meaning we'll assign a value to the output port iff the result of the expression has one for them, - * otherwise we'll remove the port for the final map. - * If there are fields in the expression's result that do not have a matching declared output port, they won't be added either, - * because we'd have to create an output port for them now which would be useless anyway since nothing could point to it. - */ - def mapPortsToValues(values: Map[String, WomValue]): Checked[Map[OutputPort, WomValue]] = { - import cats.syntax.traverse._ - - val coercedValues: List[ErrorOr[(OutputPort, WomValue)]] = for { - outputPort <- outputPorts - value <- values.get(outputPort.internalName) - coerced = coerce(value, outputPort.womType).toValidated - } yield coerced.map(outputPort -> _) - - coercedValues.sequence[ErrorOr, (OutputPort, WomValue)].map(_.toMap).toEither - } - - for { - // Evaluate the expression - evaluatedExpression <- womExpression.evaluateValue(inputs, ioFunctionSet).toEither - /* - * We expect the result to be an object of the form: - * { - * "outputName": value, - * ... - * } - */ - womMap <- castWomValueAsMap(evaluatedExpression) - - mappedValues <- mapPortsToValues(womMap) - } yield mappedValues - } - - CallableExpressionTaskDefinition( - taskName, - evaluate, - runtimeAttributes, - Map.empty, - Map.empty, - outputDefinitions, - inputDefinitions, - // TODO: This doesn't work in all cases and it feels clunky anyway - find a way to sort that out - prefixSeparator = "#" - ).validNel - } -} - -object ExpressionTool { - - case class ExpressionToolInputParameter(id: String, - label: Option[String] = None, - secondaryFiles: Option[SecondaryFiles] = None, - format: Option[InputParameterFormat] = None, - streamable: Option[Boolean] = None, - doc: Option[Doc] = None, - inputBinding: Option[InputCommandLineBinding] = None, - default: Option[CwlAny] = None, - `type`: Option[MyriadInputType] = None) extends InputParameter - - case class ExpressionToolOutputParameter(id: String, - label: Option[String] = None, - secondaryFiles: Option[SecondaryFiles] = None, - format: Option[OutputParameterFormat] = None, - streamable: Option[Boolean] = None, - doc: Option[Doc] = None, - outputBinding: Option[CommandOutputBinding] = None, - `type`: Option[MyriadOutputType] = None) extends OutputParameter -} diff --git a/cwl/src/main/scala/cwl/FileParameter.scala b/cwl/src/main/scala/cwl/FileParameter.scala deleted file mode 100644 index 2413d9d7c02..00000000000 --- a/cwl/src/main/scala/cwl/FileParameter.scala +++ /dev/null @@ -1,150 +0,0 @@ -package cwl - -import cats.effect.IO -import cats.syntax.traverse._ -import cats.instances.list._ -import common.validation.ErrorOr._ -import common.validation.IOChecked._ -import common.validation.Validation._ -import cwl.ontology.Schema -import shapeless.Poly1 -import wom.expression.IoFunctionSet -import wom.types.{WomFileType, WomMaybePopulatedFileType} -import wom.values.{WomArray, WomFile, WomMaybePopulatedFile, WomValue} - -object FileParameter { - private val ReadLimit = Option(64 * 1024) - - def populateSecondaryFiles(womValue: WomValue, - secondaryFilesCoproduct: Option[SecondaryFiles], - formatOption: Option[String], - parameterContext: ParameterContext, - expressionLib: ExpressionLib, - ioFunctions: IoFunctionSet): IOChecked[WomValue] = { - - womValue match { - - case womMaybePopulatedFile: WomMaybePopulatedFile => - val secondaryFilesErrorOr = FileParameter.secondaryFiles( - womMaybePopulatedFile, - WomMaybePopulatedFileType, - secondaryFilesCoproduct, - parameterContext, - expressionLib, - ioFunctions - ) - - secondaryFilesErrorOr map { secondaryFiles => - womMaybePopulatedFile.copy(secondaryFiles = secondaryFiles, formatOption = formatOption) - } - - case womArray: WomArray => - womArray.value.toList.traverse( - populateSecondaryFiles(_, secondaryFilesCoproduct, formatOption, parameterContext, expressionLib, ioFunctions) - ).map(WomArray(_)) - - case womValue: WomValue => womValue.validIOChecked - } - } - - /** - * Checks if the file is compatible with a format. - */ - def checkFormat(womMaybePopulatedFile: WomMaybePopulatedFile, - formatsOption: Option[List[String]], - schemaOption: Option[Schema]): ErrorOr[Unit] = { - validate { - for { - schema <- schemaOption - fileFormat <- womMaybePopulatedFile.formatOption - formats <- formatsOption - } yield { - if (!formats.exists(schema.isSubClass(fileFormat, _))) - throw new RuntimeException(s"$fileFormat is not compatible with ${formats.mkString(", ")}") - } - () - } - } - - /** - * Populates the contents if they aren't loaded already. - */ - def maybeLoadContents(womMaybePopulatedFile: WomMaybePopulatedFile, - ioFunctionSet: IoFunctionSet, - loadContents: Boolean): IO[Option[String]] = { - womMaybePopulatedFile.contentsOption match { - case someContents@Some(_) => IO.pure(someContents) - case None if !loadContents => IO.pure(None) - case _ => FileParameter.load64KiB(womMaybePopulatedFile.value, ioFunctionSet).map(Option(_)) - } - } - - def load64KiB(path: String, ioFunctionSet: IoFunctionSet): IO[String] = { - implicit val ec = IO.contextShift(ioFunctionSet.ec) - IO.fromFuture(IO { ioFunctionSet.readFile(path, ReadLimit, failOnOverflow = false) }) - } - - /** - * Returns the list of secondary files for the primary file. - */ - def secondaryFiles(primaryWomFile: WomFile, - stringWomFileType: WomFileType, - secondaryFilesOption: Option[SecondaryFiles], - parameterContext: ParameterContext, - expressionLib: ExpressionLib, - ioFunctions: IoFunctionSet): IOChecked[List[WomFile]] = { - secondaryFilesOption - .map(secondaryFiles(primaryWomFile, stringWomFileType, _, parameterContext, expressionLib, ioFunctions)) - .getOrElse(List.empty[WomFile].validIOChecked) - } - - /** - * Returns the list of secondary files for the primary file. - */ - def secondaryFiles(primaryWomFile: WomFile, - stringWomFileType: WomFileType, - secondaryFiles: SecondaryFiles, - parameterContext: ParameterContext, - expressionLib: ExpressionLib, - ioFunctions: IoFunctionSet): IOChecked[List[WomFile]] = { - secondaryFiles - .fold(SecondaryFilesPoly) - .apply(primaryWomFile, stringWomFileType, parameterContext, expressionLib, ioFunctions) - } - - type SecondaryFilesFunction = (WomFile, WomFileType, ParameterContext, ExpressionLib, IoFunctionSet) => IOChecked[List[WomFile]] - - object SecondaryFilesPoly extends Poly1 { - implicit def caseStringOrExpression: Case.Aux[StringOrExpression, SecondaryFilesFunction] = { - at { - _.fold(this) - } - } - - implicit def caseExpression: Case.Aux[Expression, SecondaryFilesFunction] = { - at { - expression => - (primaryWomFile, stringWomFileType, parameterContext, expressionLib, ioFunctions) => - File.secondaryExpressionFiles(primaryWomFile, stringWomFileType, expression, parameterContext, expressionLib, ioFunctions).toIOChecked - } - } - - implicit def caseString: Case.Aux[String, SecondaryFilesFunction] = { - at { - string => - (primaryWomFile, stringWomFileType, _, _, ioFunctions) => - File.secondaryStringFile(primaryWomFile, stringWomFileType, string, ioFunctions).map(List(_)) - } - } - - implicit def caseArray: Case.Aux[Array[StringOrExpression], SecondaryFilesFunction] = { - at { - array => - (primaryWomFile, stringWomFileType, parameterContext, expressionLib, ioFunctions) => - val functions: List[SecondaryFilesFunction] = array.toList.map(_.fold(this)) - functions.flatTraverse(_ (primaryWomFile, stringWomFileType, parameterContext, expressionLib, ioFunctions)) - } - } - } - -} diff --git a/cwl/src/main/scala/cwl/FullyQualifiedName.scala b/cwl/src/main/scala/cwl/FullyQualifiedName.scala deleted file mode 100644 index faabfa44fc4..00000000000 --- a/cwl/src/main/scala/cwl/FullyQualifiedName.scala +++ /dev/null @@ -1,61 +0,0 @@ -package cwl - -import cwl.command.ParentName - -/** - * All of these classes decompose a "fully qualified" id into its constituent parts. They have unique types as - * they are seen in different parts of a CWL document and they differ in content. - * - * The fully qualified names are created by the Schema salad preprocessing step. - * - * @see Schema salad Identifier Resolution - */ -trait FullyQualifiedName { - def fileName: String - def id: String - def parent: Option[String] -} - -case class FileAndId private(fileName: String, parent: Option[String], id: String) extends FullyQualifiedName - -object FileAndId { - def apply(in: String)(implicit parent: ParentName): FileAndId = { - val Array(fileName, id) = in.split("#") - val cleanID = parent.stripParent(id) - FileAndId(fileName, parent.value, cleanID) - } -} - -case class FileStepAndId private(fileName: String, parent: Option[String], stepId: String, id: String) extends FullyQualifiedName - -object FileStepAndId { - def apply(in: String)(implicit parent: ParentName): FileStepAndId = { - val Array(fileName, id) = in.split("#") - val cleanID = parent.stripParent(id) - val Array(stepId, outputId) = cleanID.split("/") - FileStepAndId(fileName, parent.value, stepId, outputId) - } -} - -case class ArbitrarilyNested(fileName: String, parent: Option[String], id: String) extends FullyQualifiedName - -case class FileStepUUID(fileName: String, parent: Option[String], id: String, uuid: String, stepId: String) extends FullyQualifiedName - -object FullyQualifiedName { - def apply(in: String)(implicit parent: ParentName): FullyQualifiedName = maybeApply(in)(parent).getOrElse(throw new Exception(s"malformed FQN: $in")) - - def maybeApply(in: String)(implicit parent: ParentName): Option[FullyQualifiedName] = { - - in.split("#") match { - case Array(file, after) => - val cleanAfter = parent.stripParent(after) - cleanAfter.split("/").toList match { - case step :: uuid :: id :: Nil => Option(FileStepUUID(file, parent.value, id, uuid, step)) - case step :: id :: Nil => Option(FileStepAndId(file, parent.value, step, id)) - case id :: Nil => Option(FileAndId(file, parent.value, id)) - case many => Option(ArbitrarilyNested(file, parent.value, many.last)) - } - case _ => None - } - } -} diff --git a/cwl/src/main/scala/cwl/GlobEvaluator.scala b/cwl/src/main/scala/cwl/GlobEvaluator.scala deleted file mode 100644 index d50b154da2c..00000000000 --- a/cwl/src/main/scala/cwl/GlobEvaluator.scala +++ /dev/null @@ -1,67 +0,0 @@ -package cwl - -import cats.syntax.validated._ -import common.validation.ErrorOr._ -import shapeless._ -import wom.types.{WomArrayType, WomStringType} -import wom.values._ - -/* -CommandOutputBinding.glob: -Find files relative to the output directory, using POSIX glob(3) pathname matching. If an array is provided, find -files that match any pattern in the array. If an expression is provided, the expression must return a string or an -array of strings, which will then be evaluated as one or more glob patterns. Must only match and return files which -actually exist. - -http://www.commonwl.org/v1.0/CommandLineTool.html#CommandOutputBinding - */ -object GlobEvaluator { - - type GlobEvaluatorFunction = (ParameterContext, ExpressionLib) => ErrorOr[List[String]] - - def globs(globOption: Option[Glob], parameterContext: ParameterContext, expressionLib: ExpressionLib): ErrorOr[List[String]] = { - globOption map { - globs(_, parameterContext, expressionLib) - } getOrElse { - Nil.valid - } - } - - def globs(glob: Glob, parameterContext: ParameterContext, expressionLib: ExpressionLib): ErrorOr[List[String]] = { - glob.fold(GlobEvaluator.GlobEvaluatorPoly).apply(parameterContext, expressionLib) - } - - object GlobEvaluatorPoly extends Poly1 { - implicit def caseStringOrExpression: Case.Aux[StringOrExpression, GlobEvaluatorFunction] = { - at { - _.fold(this) - } - } - - implicit def caseExpression: Case.Aux[Expression, GlobEvaluatorFunction] = { - at { - expression => - (parameterContext, expressionLib) => { - ExpressionEvaluator.eval(expression, parameterContext) flatMap { - case WomArray(_, values) if values.isEmpty => Nil.valid - case WomString(value) => List(value).valid - case WomArray(WomArrayType(WomStringType), values) => values.toList.map(_.valueString).valid - case womValue => - val message = s"Unexpected expression result: $womValue while evaluating expression '$expression' " + - s"using inputs '$parameterContext'" - message.invalidNel - } - } - } - } - - implicit def caseArrayString: Case.Aux[Array[String], GlobEvaluatorFunction] = { - at { array => (_, _) => array.toList.valid } - } - - implicit def caseString: Case.Aux[String, GlobEvaluatorFunction] = { - at { string => (_, _) => List(string).valid } - } - } - -} diff --git a/cwl/src/main/scala/cwl/InitialWorkDirRequirement.scala b/cwl/src/main/scala/cwl/InitialWorkDirRequirement.scala deleted file mode 100644 index 28e49fe71d9..00000000000 --- a/cwl/src/main/scala/cwl/InitialWorkDirRequirement.scala +++ /dev/null @@ -1,63 +0,0 @@ -package cwl - -import cwl.InitialWorkDirRequirement._ -import eu.timepit.refined.W -import shapeless.{:+:, CNil, _} - -final case class InitialWorkDirRequirement( - `class`: W.`"InitialWorkDirRequirement"`.T, - listing: IwdrListing - ) { - val listings: Array[IwdrListingArrayEntry] = listing.fold(IwdrListingArrayPoly) - - override def toString: WorkflowStepInputId = - s"""InitialWorkDirRequirement( - | ${listings.mkString(System.lineSeparator + " ")} - |)""".stripMargin -} - -/** - * Short for "Directory Entry" - * @see Dirent Specification - * - * Split into two cases because entryName is only optional if entry is an Expression - */ -trait Dirent { - def writable: Option[Boolean] - def writableWithDefault = writable.getOrElse(false) -} - -final case class ExpressionDirent( - entry: Expression, - entryname: Option[StringOrExpression], - writable: Option[Boolean] - ) extends Dirent - -final case class StringDirent( - entry: String, - entryname: StringOrExpression, - writable: Option[Boolean] - ) extends Dirent - -object InitialWorkDirRequirement { - - // "ExpressionDirent" has to come before StringDirent because expressions are matched by "String" first if we do it the - // other way round. - final type IwdrListingArrayEntry = File :+: Directory :+: ExpressionDirent :+: StringDirent :+: StringOrExpression :+: CNil - final type IwdrListing = Array[IwdrListingArrayEntry] :+: StringOrExpression :+: CNil - - object IwdrListingArrayPoly extends Poly1 { - implicit val caseArrayIwdrListingArrayEntry: Case.Aux[Array[IwdrListingArrayEntry], Array[IwdrListingArrayEntry]] = { - at { - identity - } - } - - implicit val caseStringOrExpression: Case.Aux[StringOrExpression, Array[IwdrListingArrayEntry]] = { - at { - stringOrExpression => - Array(Coproduct[IwdrListingArrayEntry](stringOrExpression)) - } - } - } -} diff --git a/cwl/src/main/scala/cwl/InputParameter.scala b/cwl/src/main/scala/cwl/InputParameter.scala deleted file mode 100644 index d8b1579dcef..00000000000 --- a/cwl/src/main/scala/cwl/InputParameter.scala +++ /dev/null @@ -1,190 +0,0 @@ -package cwl - -import cats.syntax.functor._ -import cats.syntax.parallel._ -import cats.syntax.traverse._ -import cats.syntax.validated._ -import cats.instances.list._ -import cats.instances.option._ -import common.validation.ErrorOr._ -import common.validation.IOChecked._ -import common.validation.Validation._ -import cwl.FileParameter._ -import cwl.ontology.Schema -import shapeless.Poly1 -import wom.callable.Callable.InputDefinition.InputValueMapper -import wom.expression.IoFunctionSet -import wom.types.{WomSingleFileType, WomType} -import wom.values.{WomArray, WomMaybeListedDirectory, WomMaybePopulatedFile, WomObject, WomObjectLike, WomOptionalValue, WomValue} - -trait InputParameter { - def id: String - def label: Option[String] - def secondaryFiles: Option[SecondaryFiles] - def format: Option[InputParameterFormat] - def streamable: Option[Boolean] - def doc: Option[Doc] - def inputBinding: Option[InputCommandLineBinding] - def default: Option[CwlAny] - def `type`: Option[MyriadInputType] - def loadContents = inputBinding.flatMap(_.loadContents).getOrElse(false) -} - -object InputParameter { - object IdDefaultAndType { - def unapply(arg: InputParameter): Option[(String, CwlAny, MyriadInputType)] = (arg.default, arg.`type`) match { - case (Some(default), Some(tpe)) => Option((arg.id, default, tpe)) - case _ => None - } - } - - object IdAndType { - def unapply(arg: InputParameter): Option[(String, MyriadInputType)] = (arg.default, arg.`type`) match { - case (None, Some(tpe)) => Option((arg.id, tpe)) - case _ => None - } - } - - type DefaultToWomValueFunction = WomType => ErrorOr[WomValue] - - object DefaultToWomValuePoly extends Poly1 { - implicit def caseFileOrDirectory: Case.Aux[FileOrDirectory, DefaultToWomValueFunction] = { - at { - _.fold(this) - } - } - - implicit def caseFileOrDirectoryArray: Case.Aux[Array[FileOrDirectory], DefaultToWomValueFunction] = { - at { - fileOrDirectoryArray => - womType => - fileOrDirectoryArray - .toList - .traverse(_.fold(this).apply(womType)) - .map(WomArray(_)) - } - } - - implicit def caseFile: Case.Aux[File, DefaultToWomValueFunction] = { - at { - file => - womType => - file.asWomValue.flatMap(womType.coerceRawValue(_).toErrorOr) - } - } - - implicit def caseDirectory: Case.Aux[Directory, DefaultToWomValueFunction] = { - at { - directory => - womType => - directory.asWomValue.flatMap(womType.coerceRawValue(_).toErrorOr) - } - } - - implicit def caseJson: Case.Aux[io.circe.Json, DefaultToWomValueFunction] = { - at { - circeJson => - womType => - val stringJson = circeJson.noSpaces - import spray.json._ - val sprayJson = stringJson.parseJson - womType.coerceRawValue(sprayJson).toErrorOr - } - } - } - - object InputParameterFormatPoly extends Poly1 { - implicit val caseExpression: Case.Aux[Expression, ParameterContext => ErrorOr[List[String]]] = { - at { expression => - parameterContext => - ExpressionEvaluator.eval(expression, parameterContext) map { - case WomArray(_, values) => values.toList.map(_.valueString) - case womValue => List(womValue.valueString) - } - } - } - - implicit val caseString: Case.Aux[String, ParameterContext => ErrorOr[List[String]]] = { - at { string => - _ => - List(string).valid - } - } - - implicit val caseArrayString: Case.Aux[Array[String], ParameterContext => ErrorOr[List[String]]] = { - at { array => - _ => - array.toList.valid - } - } - } - - /** - * Yet another value mapper. This one is needed because in CWL we might need to "augment" inputs which we can only do - * once they have been linked to a WomValue. This input value mapper encapsulates logic to be applied once that is - * done. For now, if the inputParameter has an input binding with loadContents = true, load the content of the file. - * - * This is based on the spec in http://www.commonwl.org/v1.0/CommandLineTool.html#Input_binding - * - * NOTE: There may be _many_ cases not implemented here that need to be fixed. - */ - def inputValueMapper(inputParameter: InputParameter, - inputType: MyriadInputType, - expressionLib: ExpressionLib, - schemaOption: Option[Schema]): InputValueMapper = { - ioFunctionSet: IoFunctionSet => { - import ioFunctionSet.cs - - def populateFiles(womValue: WomValue): IOChecked[WomValue] = { - womValue match { - case womMaybePopulatedFile: WomMaybePopulatedFile => - // Don't include the secondary files in the self variables - val parameterContext = ParameterContext(ioFunctionSet, expressionLib, self = womMaybePopulatedFile.copy(secondaryFiles = List.empty)) - val secondaryFilesFromInputParameter = inputParameter.secondaryFiles - val secondaryFilesFromType = inputType.fold(MyriadInputTypeToSecondaryFiles) - val secondaryFiles = secondaryFilesFromInputParameter orElse secondaryFilesFromType - val inputFormatsErrorOr = inputParameter.format - .traverse(_.fold(InputParameterFormatPoly).apply(parameterContext)) - - for { - inputFormatsOption <- inputFormatsErrorOr.toIOChecked - _ <- checkFormat(womMaybePopulatedFile, inputFormatsOption, schemaOption).toIOChecked - contentsOption <- FileParameter.maybeLoadContents( - womMaybePopulatedFile, - ioFunctionSet, - inputParameter.loadContents - ).to[IOChecked] - withSize <- womMaybePopulatedFile.withSize(ioFunctionSet).to[IOChecked] - loaded = withSize.copy(contentsOption = contentsOption) - secondaries <- FileParameter.secondaryFiles( - loaded, - WomSingleFileType, - secondaryFiles, - parameterContext, - expressionLib, - ioFunctionSet - ) - updated = loaded.copy(secondaryFiles = loaded.secondaryFiles ++ secondaries) - } yield updated - case womMaybeListedDirectory: WomMaybeListedDirectory => womMaybeListedDirectory.withSize(ioFunctionSet).to[IOChecked].widen - case WomArray(_, values) => values.toList.parTraverse[IOChecked, WomValue](populateFiles).map(WomArray(_)) - case WomOptionalValue(_, Some(innerValue)) => populateFiles(innerValue).map(WomOptionalValue(_)) - case obj: WomObjectLike => - // Map the values - val populated: IOChecked[WomObject] = obj.values.toList.parTraverse[IOChecked, (String, WomValue)]({ - case (key, value) => populateFiles(value).map(key -> _) - }) - .map(_.toMap) - // Validate new types are still valid w.r.t the object - .flatMap(WomObject.withTypeChecked(_, obj.womObjectTypeLike).toIOChecked) - - populated.widen[WomValue] - case womValue: WomValue => - pure(womValue) - } - } - - womValue => populateFiles(womValue) - } - } -} diff --git a/cwl/src/main/scala/cwl/LinkMergeMethod.scala b/cwl/src/main/scala/cwl/LinkMergeMethod.scala deleted file mode 100644 index bd54ca6b56a..00000000000 --- a/cwl/src/main/scala/cwl/LinkMergeMethod.scala +++ /dev/null @@ -1,8 +0,0 @@ -package cwl - -object LinkMergeMethod extends Enumeration { - type LinkMergeMethod = Value - - val MergeNested = Value("merge_nested") - val MergeFlattened = Value("merge_flattened") -} diff --git a/cwl/src/main/scala/cwl/MyriadInputTypeToSecondaryFiles.scala b/cwl/src/main/scala/cwl/MyriadInputTypeToSecondaryFiles.scala deleted file mode 100644 index 4ddc09625d3..00000000000 --- a/cwl/src/main/scala/cwl/MyriadInputTypeToSecondaryFiles.scala +++ /dev/null @@ -1,25 +0,0 @@ -package cwl - -import cwl.CwlType.CwlType -import shapeless.Poly1 - -// IAS.secondaryFiles are NOT listed in 1.0 spec, but according to jgentry they will be, maybe -object MyriadInputTypeToSecondaryFiles extends Poly1 { - implicit val caseMyriadInputInnerType: Case.Aux[MyriadInputInnerType, Option[SecondaryFiles]] = at { - _.fold(MyriadInputInnerTypeToSecondaryFiles) - } - - implicit val caseArrayMyriadInputInnerType: Case.Aux[Array[MyriadInputInnerType], Option[SecondaryFiles]] = at { - _.to(LazyList).flatMap(_.fold(MyriadInputInnerTypeToSecondaryFiles)).headOption - } -} - -object MyriadInputInnerTypeToSecondaryFiles extends Poly1 { - implicit val caseCwlType: Case.Aux[CwlType, Option[SecondaryFiles]] = at { _ => None } - implicit val caseInputRecordSchema: Case.Aux[InputRecordSchema, Option[SecondaryFiles]] = at { _ => None } - implicit val caseInputEnumSchema: Case.Aux[InputEnumSchema, Option[SecondaryFiles]] = at { _ => None } - implicit val caseInputArraySchema: Case.Aux[InputArraySchema, Option[SecondaryFiles]] = at { - inputArraySchema => inputArraySchema.secondaryFiles - } - implicit val caseString: Case.Aux[String, Option[SecondaryFiles]] = at { _ => None } -} diff --git a/cwl/src/main/scala/cwl/MyriadInputTypeToSortedCommandParts.scala b/cwl/src/main/scala/cwl/MyriadInputTypeToSortedCommandParts.scala deleted file mode 100644 index 498e97c016f..00000000000 --- a/cwl/src/main/scala/cwl/MyriadInputTypeToSortedCommandParts.scala +++ /dev/null @@ -1,254 +0,0 @@ -package cwl - -import cwl.CommandLineTool.{CommandBindingSortingKey, CommandPartsList, SortKeyAndCommandPart, StringOrInt} -import cwl.CwlType.CwlType -import cwl.SchemaDefRequirement.SchemaDefTypes -import cwl.command.ParentName -import shapeless.{Coproduct, Poly1} -import wom.values.{WomArray, WomCoproductValue, WomObjectLike, WomOptionalValue, WomValue} -import cats.syntax.option._ -import cats.syntax.traverse._ -import cats.instances.list._ -import cats.instances.option._ - -/** - * Poly1 to fold over a MyriadInputType and create a List of SortingKeyAndCommandPart based on the rules described here: - * http://www.commonwl.org/v1.0/CommandLineTool.html#Input_binding - */ -object MyriadInputTypeToSortedCommandParts extends Poly1 { - import Case._ - - /** - * CommandLineBinding: binding of the element being processed. - * It's possible for an input not to have an "inputBinding", but to have a complex type which itself has inputBindings for its fields or items. - * e.g: - * - id: arrayInput - * type: - * type: array - * items: File - * inputBinding: { prefix: "-YYY" } - * - * Here the array type has an inputBinding but the input itself (arrayInput) does not. - * This would yield on the command line something like "-YYY arrayItem0 -YYY arrayItem1 ..." - * - * On the other hand: - * - id: arrayInput - * type: - * type: array - * items: File - * inputBinding: { prefix: "-YYY" } - * inputBinding: { prefix: "-XXX" } - * - * In this case the input itself has a binding as well, which would yield something like "-XXX -YYY arrayItem0 -YYY arrayItem1 ..." - * - * For arrays specifically, the presence of an "itemSeparator" and / or "valueFrom" in the input binding will change the way the array is processed, - * which is why we need to know about it in this Poly1 that folds over the input types. - * - * WomValue: value bound to the element (in the cases above it would be the WomArray) - * CommandBindingSortingKey: The current sorting key. Because we might need to recurse in nested types we need to propagate the key as we do. - */ - type CommandPartBuilder = (Option[InputCommandLineBinding], WomValue, CommandBindingSortingKey, Boolean, ExpressionLib, SchemaDefRequirement) => CommandPartsList - - implicit def m: Aux[MyriadInputInnerType, CommandPartBuilder] = { - at { - innerType => { - (binding, womValue, sortingKey, hasShellCommandRequirement, expressionLib, schemaDefRequirement) => - innerType - .fold(MyriadInputInnerTypeToSortedCommandParts) - .apply(binding, womValue, sortingKey, hasShellCommandRequirement, expressionLib, schemaDefRequirement).fold( - throw new RuntimeException(s"inner type to command part failed! on type $innerType w/ womValue $womValue"))(identity) - } - } - } - - implicit def am: Aux[Array[MyriadInputInnerType], CommandPartBuilder] = { - at { - types => { - (binding, womValue, sortingKey, hasShellCommandRequirement, expressionLib, schemaDefRequirement) => - def lookupTypes(innerTypes: Array[MyriadInputInnerType]) = { - innerTypes.toList.map{ - _.fold(MyriadInputInnerTypeToSortedCommandParts) - .apply(binding, womValue, sortingKey, hasShellCommandRequirement, expressionLib, schemaDefRequirement) - }.reduce(_ orElse _) - } - val result: Option[CommandPartsList] = - types.partition(_.select[CwlType].contains(CwlType.Null)) match { - case (Array(_), types) => lookupTypes(types) orElse Some(List.empty[SortKeyAndCommandPart]) - case (Array(), types) => lookupTypes(types) - } - result.fold( - throw new RuntimeException(s"could not produce command line parts from input $womValue and types $types"))( - identity - ) - } - } - } -} - -object MyriadInputInnerTypeToSortedCommandParts extends Poly1 { - - type CommandPartBuilder = (Option[InputCommandLineBinding], WomValue, CommandBindingSortingKey, Boolean, ExpressionLib, SchemaDefRequirement) => Option[CommandPartsList] - - import Case._ - - // Primitive type: we just need to create a command part from the binding if there's one here. - implicit def ct: Aux[CwlType, CommandPartBuilder] = { - at { - cwlType => { - case (_, WomOptionalValue(_, None), _, _, _, _) => List.empty.some - case (inputBinding, womValue, key, hasShellCommandRequirement, expressionLib, _) => { - (cwlType, womValue) match { - case (CwlType.Null, _) => List.empty.some - case (_, WomOptionalValue(_, None)) => List.empty.some - case (_,_) => inputBinding.toList.map(_.toCommandPart(key, womValue, hasShellCommandRequirement, expressionLib)).some - } - } - } - } - } - - implicit def irs: Aux[InputRecordSchema, CommandPartBuilder] = at[InputRecordSchema] { irs => { - def go: CommandPartBuilder = { - - //If the value is optional and is supplied, recurse over the value provided - case (inputBinding, WomCoproductValue(_, value), sortingKey, hasShellCommandRequirement, expressionLib, schemaDefRequirement) => - go(inputBinding, value, sortingKey, hasShellCommandRequirement, expressionLib, schemaDefRequirement) - - //If the value is optional and is supplied, recurse over the value provided - case (inputBinding, WomOptionalValue(_, Some(value)), sortingKey, hasShellCommandRequirement, expressionLib, schemaDefRequirement) => - go(inputBinding, value, sortingKey, hasShellCommandRequirement, expressionLib, schemaDefRequirement) - - // If it's optional and there's no value, do nothing - case (_, WomOptionalValue(_, None), _, _, _, _) => List.empty.some - - // If there's no input binding and no input bindings within the irs, do nothing - case (None, _, _, _, _, _) if !irs.fields.exists(_.exists(_.inputBinding.isDefined)) => List.empty.some - - case (inputBinding, objectLike: WomObjectLike, sortingKey, hasShellCommandRequirement, expressionLib, schemaDefRequirement) => - // If there's an input binding, make a SortKeyAndCommandPart for it - val sortingKeyFromInputBindingFromInputParameter: Option[SortKeyAndCommandPart] = - inputBinding.map(_.toCommandPart(sortingKey, objectLike, hasShellCommandRequirement, expressionLib)) - - // iterate through the fields and fold over their type - val partsFromFields:Option[CommandPartsList] = - irs.fields.toList.flatten. - flatTraverse{ - case InputRecordField(name, tpe, _, inputBinding, _) => - // Parse the name to get a clean id - val parsedName = FullyQualifiedName(name)(ParentName.empty).id - - // The field name needs to be added to the key after the input binding (as per the spec) - // Also start from the key from the input binding if there was one - val fieldSortingKey = - sortingKeyFromInputBindingFromInputParameter. - map(_.sortingKey). - getOrElse(sortingKey). - append(inputBinding, Coproduct[StringOrInt](parsedName)) - - val innerValueOption: Option[WomValue] = objectLike.values.get(parsedName) - - val folded = innerValueOption.map(tpe.fold(MyriadInputTypeToSortedCommandParts). - apply(inputBinding, _, fieldSortingKey.asNewKey, hasShellCommandRequirement, expressionLib, schemaDefRequirement)) - folded - } - - sortingKeyFromInputBindingFromInputParameter.fold( - partsFromFields )( - sortingKeyFromInputBindingFromInputParameter => - partsFromFields.map{k => List(sortingKeyFromInputBindingFromInputParameter.copy(nestedCommandParts = k)) } - ) - case (_, other, _, _, _, _) => throw new RuntimeException(s"Value $other cannot be used for an input of type InputRecordSchema") - } - - go - }} - - implicit def ies: Aux[InputEnumSchema, CommandPartBuilder] = at[InputEnumSchema] { inputEnumSchema => - def go: CommandPartBuilder = { - case (_, WomOptionalValue(_, None), _, _, _, _) => List.empty.some - case (inputBinding, value, sortingKey, hasShellCommandRequirement, expressionLib, _) => - // If there's an input binding, make a SortKeyAndCommandPart for it - val fromInputBinding = - inputBinding.map(_.toCommandPart(sortingKey, value, hasShellCommandRequirement, expressionLib)).toList - - val fromIes = inputEnumSchema.inputBinding.map(_.toCommandPart(sortingKey, value, hasShellCommandRequirement, expressionLib)).toList - - (fromInputBinding ++ fromIes).some - } - go - } - - implicit def ias: Aux[InputArraySchema, CommandPartBuilder] = at[InputArraySchema] { ias => - def go: CommandPartBuilder = { - case (inputBindingFromInputParameterParent, WomOptionalValue(_, Some(value)), sortingKey, hasShellCommandRequirement, expressionLib, schemaDefRequirement) => - go(inputBindingFromInputParameterParent, value, sortingKey, hasShellCommandRequirement, expressionLib, schemaDefRequirement) - - //If it's optional and there's no value, do nothing - case (_, WomOptionalValue(_, None), _, _, _, _) => List.empty.some - - // If there's no input binding and no input bindings for the ias, do nothing - case (None, _, _, _, _, _) if ias.inputBinding.isEmpty => List.empty.some - - case (inputBinding, WomArray.WomArrayLike(womArray: WomArray), sortingKey, hasShellCommandRequirement, expressionLib, schemaDefRequirement) => - - - // If there's an input binding, make a SortKeyAndCommandPart for it - val sortKeyFromInputBindingFromInputerParameterParent: Option[SortKeyAndCommandPart] = - inputBinding. - map(_.toCommandPart(sortingKey, womArray, hasShellCommandRequirement, expressionLib)) - - // Now depending on whether we have an itemSeparator and/or valueFrom or not, we're going to recurse over each element of the array (or not). - // See http://www.commonwl.org/v1.0/CommandLineTool.html#CommandLineBinding - if (inputBinding.flatMap(_.itemSeparator).isDefined || inputBinding.flatMap(_.valueFrom).isDefined) { - // If there's an item separator or a valueFrom we can stop here. - // When the command part is instantiated (see CommandLineBindingCommandPart) it will evaluate the valueFrom (if defined) and join the items together (if there's an itemSeparator). - sortKeyFromInputBindingFromInputerParameterParent.toList.some - } else { - // If neither valueFrom nor itemSeparator were defined, we need to process each item of the array - val fromArray: Option[CommandPartsList] = womArray.value.zipWithIndex.toList.flatTraverse({ - case (item, index) => - // Update the sorting key with the binding position (if any), add the index - val itemSortingKey = { - - // The index needs to be added to the key after the input binding (as per the spec) - // Also start from the key from the input binding if there was one - val sortKey = sortKeyFromInputBindingFromInputerParameterParent.fold(sortingKey)(_.sortingKey) - - sortKey.append(ias.inputBinding, Coproduct[StringOrInt](index)) - } - - // Even if the item doesn't have an explicit input binding, it should appear in the command so create a default empty one - //there is an explicit input binding! - val arrayItemInputBinding = - ias. - inputBinding. - orElse(Option(InputCommandLineBinding.default)) - - // Fold over the item type of each array element - Option(ias.items.fold(MyriadInputTypeToSortedCommandParts).apply(arrayItemInputBinding, item, itemSortingKey.asNewKey, hasShellCommandRequirement, expressionLib, schemaDefRequirement)) - }) - sortKeyFromInputBindingFromInputerParameterParent.fold(fromArray){ - sortKeyFromInputBindingFromInputerParameterParent => - fromArray.map{k => List(sortKeyFromInputBindingFromInputerParameterParent.copy(nestedCommandParts = k)) } - } - } - - case (_, other, _, _, _, _) => throw new RuntimeException(s"Value $other cannot be used for an input of type InputArraySchema") - } - - go - } - - implicit def s: Aux[String, CommandPartBuilder] = { - at { - s => { - case (_, WomOptionalValue(_, None), _, _, _, _) => List.empty.some - case (commandLineBindingFromInput, womValue, sortingKey, boolean, expressionLib, schemaDefRequirement) => { - val womType: SchemaDefTypes = schemaDefRequirement.lookupCwlType(s).getOrElse(throw new RuntimeException(s"Looked for type $s in custom types $schemaDefRequirement but no match was found!")) - - womType.fold(this).apply(commandLineBindingFromInput, womValue, sortingKey, boolean, expressionLib, schemaDefRequirement) - } - } - } - } -} diff --git a/cwl/src/main/scala/cwl/MyriadInputTypeToWomType.scala b/cwl/src/main/scala/cwl/MyriadInputTypeToWomType.scala deleted file mode 100644 index d3be698bd33..00000000000 --- a/cwl/src/main/scala/cwl/MyriadInputTypeToWomType.scala +++ /dev/null @@ -1,95 +0,0 @@ -package cwl - -import cats.data.NonEmptyList -import cwl.CwlType.CwlType -import cwl.MyriadInputTypeToWomType.SchemaLookup -import cwl.command.ParentName -import mouse.all._ -import shapeless.Poly1 -import wom.types._ - -object MyriadInputInnerTypeToString extends Poly1 { - implicit def ct = at[CwlType]{ _.toString } - implicit def irs = at[InputRecordSchema]{_.toString} - implicit def ies = at[InputEnumSchema]{ _.toString } - implicit def ias = at[InputArraySchema]{ _.toString} - implicit def s = at[String]{identity} -} - -object MyriadInputTypeToWomType extends Poly1 { - - import Case._ - - type SchemaLookup = SchemaDefRequirement => WomType - - implicit def m:Aux[MyriadInputInnerType, SchemaLookup]= at[MyriadInputInnerType] {_.fold(MyriadInputInnerTypeToWomType)} - - // An array of type means "this input value can be in any of those types." - // Currently we only accept single types or [null, X] to mean Optional[X] - implicit def am: Aux[Array[MyriadInputInnerType], SchemaLookup] = at[Array[MyriadInputInnerType]] { - types => - schemaLookup => - types.partition(_.select[CwlType].contains(CwlType.Null)) match { - // If there's a single non null type, use that - case (Array(), Array(singleNonNullType)) => - singleNonNullType.fold(MyriadInputInnerTypeToWomType).apply(schemaLookup) - case (Array(), array: Array[MyriadInputInnerType]) if array.length > 1 => - val types = array.map(_.fold(MyriadInputInnerTypeToWomType).apply(schemaLookup)) - WomCoproductType(NonEmptyList.fromListUnsafe(types.toList)) - // If there's a null type and a single non null type, it's a WomOptionalType - case (Array(_), Array(singleNonNullType)) => - WomOptionalType(singleNonNullType.fold(MyriadInputInnerTypeToWomType).apply(schemaLookup)) - case (Array(_), array: Array[MyriadInputInnerType]) if array.length > 1 => - val types = array.map(_.fold(MyriadInputInnerTypeToWomType).apply(schemaLookup)) - WomOptionalType(WomCoproductType(NonEmptyList.fromListUnsafe(types.toList))) - case (Array(_), _) => - val readableTypes = types.map(_.fold(MyriadInputInnerTypeToString)).mkString(", ") - throw new UnsupportedOperationException(s"Cromwell only supports single types or optionals (as indicated by [null, X]). Instead we saw: $readableTypes") - } - } -} - -object MyriadInputInnerTypeToWomType extends Poly1 { - import Case._ - - def ex(component: String) = throw new RuntimeException(s"input type $component not yet suported by WOM!") - - implicit def ct: Aux[CwlType, SchemaLookup] = at[CwlType]{ - ct => - cwl.cwlTypeToWomType(ct) |> Function.const - } - - def inputRecordSchemaToWomType(irs: InputRecordSchema): SchemaLookup = { schemaLookup: SchemaDefRequirement => - irs match { - case InputRecordSchema(_, Some(fields), _, _) => - val typeMap = fields.map({ field => - FullyQualifiedName(field.name)(ParentName.empty).id -> field.`type`.fold(MyriadInputTypeToWomType).apply(schemaLookup) - }).toMap - WomCompositeType(typeMap) - case irs => irs.toString |> ex - } - } - - implicit def irs: Aux[InputRecordSchema, SchemaLookup] = at[InputRecordSchema]{ - inputRecordSchemaToWomType - } - - implicit def ies: Aux[InputEnumSchema, SchemaLookup] = at[InputEnumSchema]{ - ies => - ies.toWomEnumerationType |> Function.const - } - - implicit def ias: Aux[InputArraySchema, SchemaLookup] = at[InputArraySchema]{ - ias => - lookup => - val arrayType: WomType = ias.items.fold(MyriadInputTypeToWomType).apply(lookup) - WomArrayType(arrayType) - } - - implicit def s: Aux[String, SchemaLookup] = at[String]{ - string => - schemaReq => - schemaReq.lookupType(string).getOrElse(throw new RuntimeException(s"Custom type $string was referred to but not found in schema def ${schemaReq}.")) - } - -} diff --git a/cwl/src/main/scala/cwl/MyriadOutputTypeToWomFiles.scala b/cwl/src/main/scala/cwl/MyriadOutputTypeToWomFiles.scala deleted file mode 100644 index 7c007d07d46..00000000000 --- a/cwl/src/main/scala/cwl/MyriadOutputTypeToWomFiles.scala +++ /dev/null @@ -1,66 +0,0 @@ -package cwl - -import cats.syntax.traverse._ -import cats.instances.list._ -import common.validation.IOChecked._ -import cwl.CwlType.CwlType -import cwl.MyriadOutputTypeToWomFiles.EvaluationFunction -import mouse.all._ -import shapeless.Poly1 -import wom.expression.FileEvaluation - -object MyriadOutputTypeToWomFiles extends Poly1 { - - type EvaluationFunction = CommandOutputBinding => IOChecked[Set[FileEvaluation]] - - import Case._ - - implicit def cwlType: Aux[MyriadOutputInnerType, EvaluationFunction => IOChecked[Set[FileEvaluation]]] = at[MyriadOutputInnerType]{ - _.fold(MyriadOutputInnerTypeToWomFiles) - } - - implicit def acwl: Aux[Array[MyriadOutputInnerType], EvaluationFunction => IOChecked[Set[FileEvaluation]]] = at[Array[MyriadOutputInnerType]] { types => - evalFunction => - types.toList.traverse(_.fold(MyriadOutputInnerTypeToWomFiles).apply(evalFunction)).map(_.toSet.flatten) - } -} - -object MyriadOutputInnerTypeToWomFiles extends Poly1 { - - import Case._ - - def ex(component: String) = throw new RuntimeException(s"output type $component cannot yield wom files") - - implicit def cwlType: Aux[CwlType, EvaluationFunction => IOChecked[Set[FileEvaluation]]] = at[CwlType] { _ =>_ => - Set.empty[FileEvaluation].validIOChecked - } - - implicit def ors: Aux[OutputRecordSchema, EvaluationFunction => IOChecked[Set[FileEvaluation]]] = at[OutputRecordSchema] { - case OutputRecordSchema(_, Some(fields), _) => - evalFunction => - fields.toList.traverse[IOChecked, Set[FileEvaluation]]({ field => - field.outputBinding match { - case Some(binding) => evalFunction(binding) - case None => field.`type`.fold(MyriadOutputTypeToWomFiles).apply(evalFunction) - } - }).map(_.toSet.flatten) - case ors => ors.toString |> ex - } - - implicit def oes: Aux[OutputEnumSchema, EvaluationFunction => IOChecked[Set[FileEvaluation]]] = at[OutputEnumSchema]{ oes =>_ => - oes.toString |> ex - } - - implicit def oas: Aux[OutputArraySchema, EvaluationFunction => IOChecked[Set[FileEvaluation]]] = at[OutputArraySchema]{ oas => - evalFunction => - import cats.syntax.apply._ - def fromBinding: IOChecked[Set[FileEvaluation]] = oas.outputBinding.map(evalFunction).getOrElse(Set.empty[FileEvaluation].validIOChecked) - def fromType: IOChecked[Set[FileEvaluation]] = oas.items.fold(MyriadOutputTypeToWomFiles).apply(evalFunction) - - (fromBinding, fromType) mapN (_ ++ _) - } - - implicit def s: Aux[String, EvaluationFunction => IOChecked[Set[FileEvaluation]]] = at[String]{ _ =>_ => - Set.empty[FileEvaluation].validIOChecked - } -} diff --git a/cwl/src/main/scala/cwl/MyriadOutputTypeToWomType.scala b/cwl/src/main/scala/cwl/MyriadOutputTypeToWomType.scala deleted file mode 100644 index 61107143006..00000000000 --- a/cwl/src/main/scala/cwl/MyriadOutputTypeToWomType.scala +++ /dev/null @@ -1,105 +0,0 @@ -package cwl - -import cats.data.NonEmptyList -import cwl.CwlType.CwlType -import cwl.command.ParentName -import mouse.all._ -import shapeless.Poly1 -import wom.types._ - -object MyriadOutputInnerTypeToString extends Poly1 { - - implicit def cwlType = at[CwlType] { - _.toString - } - - implicit def ors = at[OutputRecordSchema] { - _.toString - } - - implicit def oes = at[OutputEnumSchema] { - _.toString - } - - implicit def oas = at[OutputArraySchema] { - _.toString - } - - implicit def s = at[String] { - identity - } -} - -object MyriadOutputTypeToWomType extends Poly1{ - - import Case._ - - type SchemaDefToWomType = SchemaDefRequirement => WomType - - implicit def cwlType: Aux[MyriadOutputInnerType, SchemaDefToWomType] = at[MyriadOutputInnerType]{ moit => schemaDefRequirement => - moit.fold(MyriadOutputInnerTypeToWomType).apply(schemaDefRequirement) - } - - implicit def acwl: Aux[Array[MyriadOutputInnerType], SchemaDefToWomType] = at[Array[MyriadOutputInnerType]] { types => schemaDefRequirement => - types.partition(_.select[CwlType].contains(CwlType.Null)) match { - // If there's a single non null type, use that - case (Array(), Array(singleNonNullType)) => - singleNonNullType.fold(MyriadOutputInnerTypeToWomType).apply(schemaDefRequirement) - case (Array(), array: Array[MyriadOutputInnerType]) if array.size > 1 => - val types = array.map(_.fold(MyriadOutputInnerTypeToWomType).apply(schemaDefRequirement)) - WomCoproductType(NonEmptyList.fromListUnsafe(types.toList)) - // If there's a null type and a single non null type, it's a WomOptionalType - case (Array(_), Array(singleNonNullType)) => - WomOptionalType(singleNonNullType.fold(MyriadOutputInnerTypeToWomType).apply(schemaDefRequirement)) - case (Array(_), array: Array[MyriadOutputInnerType]) if array.size > 1 => - val types = array.map(_.fold(MyriadOutputInnerTypeToWomType).apply(schemaDefRequirement)) - WomOptionalType(WomCoproductType(NonEmptyList.fromListUnsafe(types.toList))) - case _ => - val readableTypes = types.map(_.fold(MyriadOutputInnerTypeToString)).mkString(", ") - throw new UnsupportedOperationException(s"Cromwell only supports single types or optionals (as indicated by [null, X]). Instead we saw: $readableTypes") - } - } -} - -object MyriadOutputInnerTypeToWomType extends Poly1 { - - import Case._ - import MyriadOutputTypeToWomType.SchemaDefToWomType - - implicit def cwlType: Aux[CwlType, SchemaDefToWomType] = - at[CwlType]{ - cwl.cwlTypeToWomType andThen Function.const - } - - implicit def ors: Aux[OutputRecordSchema, SchemaDefToWomType] = at[OutputRecordSchema] { - ors => schemaDefRequirement => - ors.fields.fold( - WomCompositeType(Map.empty))( - {fields => - val typeMap = fields.map({ field => - val parsedName = FullyQualifiedName(field.name)(ParentName.empty).id - parsedName -> field.`type`.fold(MyriadOutputTypeToWomType).apply(schemaDefRequirement) - }).toMap - WomCompositeType(typeMap) - } - ) - } - - implicit def oes: Aux[OutputEnumSchema, MyriadOutputTypeToWomType.SchemaDefToWomType] = - at[OutputEnumSchema] { - oes => oes.toWomEnumerationType |> Function.const - } - - implicit def oas: Aux[OutputArraySchema, MyriadOutputTypeToWomType.SchemaDefToWomType] = - at[OutputArraySchema] { - oas => schemaDefRequirement => - val arrayType: WomType = oas.items.fold(MyriadOutputTypeToWomType).apply(schemaDefRequirement) - WomArrayType(arrayType) - } - - implicit def s: Aux[String, MyriadOutputTypeToWomType.SchemaDefToWomType] = - at[String] { - string => schemaReq => - schemaReq.lookupType(string).getOrElse(throw new RuntimeException(s"Custom type $string was referred to but not found in schema def ${schemaReq.types.mkString(", ")}.")) - } -} diff --git a/cwl/src/main/scala/cwl/MyriadOutputTypeToWomValue.scala b/cwl/src/main/scala/cwl/MyriadOutputTypeToWomValue.scala deleted file mode 100644 index 332e4ab3888..00000000000 --- a/cwl/src/main/scala/cwl/MyriadOutputTypeToWomValue.scala +++ /dev/null @@ -1,100 +0,0 @@ -package cwl - -import cats.data.Validated.Valid -import cats.syntax.option._ -import cats.syntax.traverse._ -import cats.instances.list._ -import common.validation.IOChecked.IOChecked -import common.validation.IOChecked._ -import cwl.CwlType.CwlType -import cwl.command.ParentName -import mouse.all._ -import shapeless.Poly1 -import wom.types._ -import wom.values.{WomObject, WomValue} - -/** - * Folds a MyriadOutputType into a WomValue - * This is needed because the type might define the structure of the final output value (for OutputRecordSchemas for example) - */ -object MyriadOutputTypeToWomValue extends Poly1 { - - // We pass in a function that can evaluate a CommandOutputBinding and produce a WomValue. This allows us to recurse into the - // MyriadOutputTypes and evaluate values as we do. - type EvaluationFunction = (CommandOutputBinding, WomType) => IOChecked[WomValue] - - //Our overall return type gives us the evaluator function and custom types; returns WomValues - type Output = (EvaluationFunction, SchemaDefRequirement) => IOChecked[WomValue] - - import Case._ - - implicit def cwlType: Aux[MyriadOutputInnerType, Output] = at[MyriadOutputInnerType]{ - moit => (evalFunction, schemaDefRequirement) => moit.fold(MyriadOutputInnerTypeToWomValue).apply(evalFunction, schemaDefRequirement) - } - - // TODO: Not sure what the right thing to do is here, for now go over the list of types and use the first evaluation that yields success - implicit def acwl: Aux[Array[MyriadOutputInnerType], Output] = at[Array[MyriadOutputInnerType]] { types => - (evalFunction,schemaDefRequirement) => - types.toList.map(_.fold(MyriadOutputInnerTypeToWomValue).apply(evalFunction, schemaDefRequirement)).map(_.toErrorOr).collectFirst({ - case Valid(validValue) => validValue - }).toValidNel(s"Cannot find a suitable type to build a WomValue from in ${types.mkString(", ")}").toIOChecked - } -} - -object MyriadOutputInnerTypeToWomValue extends Poly1 { - - import Case._ - import MyriadOutputTypeToWomValue.Output - - def ex(component: String) = throw new RuntimeException(s"output type $component cannot yield a wom value") - - implicit def cwlType: Aux[CwlType, Output] = at[CwlType] { _ => (_,_) => - "No output binding is defined. Are you expecting the output to be inferred from a cwl.output.json file ? If so please make sure the file was effectively created.".invalidIOChecked - } - - implicit def ors: Aux[OutputRecordSchema, Output] = at[OutputRecordSchema] { ors => (evalFunction, schemaDefRequirement) => ors match { - case OutputRecordSchema(_, Some(fields), _) => - // Go over each field and evaluate the binding if there's one, otherwise keep folding over field types - def evaluateValues = fields.toList.traverse[IOChecked, ((String, WomValue), (String, WomType))]({ field => - val womType = field.`type`.fold(MyriadOutputTypeToWomType).apply(schemaDefRequirement) - val womValue: IOChecked[WomValue] = field.outputBinding match { - case Some(binding) => evalFunction(binding, womType) - case None => field.`type`.fold(MyriadOutputTypeToWomValue).apply(evalFunction, schemaDefRequirement) - } - - // TODO: ParentName might need to be passed in here ? - // return the value and the type with a clean parsedName - womValue map { value => - val parsedName = FullyQualifiedName(field.name)(ParentName.empty).id - (parsedName -> value) -> (parsedName -> womType) - } - }) - - evaluateValues map { evaluatedValues => - val (valueMap, typeMap) = evaluatedValues.unzip - // Create a typed WomObject from the values and the typeMap - WomObject.withTypeUnsafe(valueMap.toMap, WomCompositeType(typeMap.toMap)) - } - case ors => ors.toString |> ex - }} - - implicit def oes: Aux[OutputEnumSchema, Output] = at[OutputEnumSchema]{ - //DB: I tried to do a pattern match as the overall function here but the compiler exploded - oes => (f, schemaDefRequirement) => oes match { - case oes@OutputEnumSchema(_, _, _, _, Some(outputBinding)) => f(outputBinding, oes.toWomEnumerationType) - case _ => s"The enumeration type $oes requires an outputbinding to be evaluated.".invalidIOChecked - } - } - - implicit def oas: Aux[OutputArraySchema, Output] = at[OutputArraySchema]{ oas => (evalFunction, schemaDefRequirement) => oas match { - case OutputArraySchema(itemsType, _, _, outputBinding) => - lazy val itemsWomType = itemsType.fold(MyriadOutputTypeToWomType).apply(schemaDefRequirement) - def fromBinding = outputBinding.map(evalFunction(_, WomArrayType(itemsWomType))) - def fromTypes = itemsType.fold(MyriadOutputTypeToWomValue).apply(evalFunction, schemaDefRequirement) - fromBinding.getOrElse(fromTypes) - }} - - implicit def s: Aux[String, Output] = at[String]{ s => (_, _) => - s.toString |> ex - } -} diff --git a/cwl/src/main/scala/cwl/OutputParameter.scala b/cwl/src/main/scala/cwl/OutputParameter.scala deleted file mode 100644 index 274469da7ee..00000000000 --- a/cwl/src/main/scala/cwl/OutputParameter.scala +++ /dev/null @@ -1,62 +0,0 @@ -package cwl - -import cats.syntax.traverse._ -import cats.syntax.validated._ -import cats.instances.option._ -import common.validation.ErrorOr._ -import shapeless.{:+:, CNil, Poly1} -import wom.values.{WomString, WomValue} - -trait OutputParameter { - def id: String - def label: Option[String] - def secondaryFiles: Option[SecondaryFiles] - def format: Option[OutputParameterFormat] - def streamable: Option[Boolean] - def doc: Option[String :+: Array[String] :+: CNil] - def outputBinding: Option[CommandOutputBinding] - def `type`: Option[MyriadOutputType] - def cacheString: String = toString -} - -object OutputParameter { - object IdAndType { - def unapply(arg: OutputParameter): Option[(String, MyriadOutputType)] = arg.`type`.map((arg.id, _)) - } - - def format(formatOption: Option[StringOrExpression], - parameterContext: ParameterContext, - expressionLib: ExpressionLib): ErrorOr[Option[String]] = { - formatOption.traverse{ - format(_, parameterContext, expressionLib) - } - } - - def format(format: StringOrExpression, parameterContext: ParameterContext, expressionLib: ExpressionLib): ErrorOr[String] = { - format.fold(OutputParameter.FormatPoly).apply(parameterContext, expressionLib) - } - - type FormatFunction = (ParameterContext, ExpressionLib) => ErrorOr[String] - - object FormatPoly extends Poly1 { - implicit def caseStringOrExpression: Case.Aux[StringOrExpression, FormatFunction] = { - at { - _.fold(this) - } - } - - implicit def caseExpression: Case.Aux[Expression, FormatFunction] = { - at { - expression => - (parameterContext, expressionLib) => - val result: ErrorOr[WomValue] = ExpressionEvaluator.eval(expression, parameterContext) - result flatMap { - case womString: WomString => womString.value.valid - case other => s"Not a valid file format: $other".invalidNel - } - } - } - - implicit def caseString: Case.Aux[String, FormatFunction] = at { string => (_,_) => string.valid } - } -} diff --git a/cwl/src/main/scala/cwl/OutputParameterExpression.scala b/cwl/src/main/scala/cwl/OutputParameterExpression.scala deleted file mode 100644 index b044bc36077..00000000000 --- a/cwl/src/main/scala/cwl/OutputParameterExpression.scala +++ /dev/null @@ -1,115 +0,0 @@ -package cwl - -import common.validation.ErrorOr.ErrorOr -import common.validation.IOChecked.{IOChecked, _} -import cwl.CwlType.CwlType -import shapeless.Poly1 -import wom.expression.{EmptyIoFunctionSet, FileEvaluation, IoFunctionSet} -import wom.types._ -import wom.values.WomValue - -import scala.Function.const -import scala.concurrent.{ExecutionContext, Future} - -case class OutputParameterExpression(parameter: OutputParameter, - override val cwlExpressionType: WomType, - override val inputs: Set[String], - override val expressionLib: ExpressionLib, - schemaDefRequirement: SchemaDefRequirement) extends CwlWomExpression { - - override def sourceString = parameter.toString - - override def cacheString: String = parameter.cacheString - - private def evaluateOutputBinding(inputValues: Map[String, WomValue], - ioFunctionSet: IoFunctionSet, - secondaryFilesOption: Option[SecondaryFiles], - formatOption: Option[StringOrExpression] - )(outputBinding: CommandOutputBinding, - cwlExpressionType: WomType): IOChecked[WomValue] = { - CommandOutputBinding.generateOutputWomValue( - inputValues, - ioFunctionSet, - cwlExpressionType, - outputBinding, - secondaryFilesOption, - formatOption, - expressionLib - ) - } - - private def evaluateOutputBindingFiles(inputValues: Map[String, WomValue], - ioFunctionSet: IoFunctionSet, - secondaryFilesOption: Option[SecondaryFiles], - coerceTo: WomType - )(outputBinding: CommandOutputBinding): IOChecked[Set[FileEvaluation]] = { - CommandOutputBinding.getOutputWomFiles( - inputValues, - coerceTo, - outputBinding, - secondaryFilesOption, - ioFunctionSet, - expressionLib - ) - } - - override def evaluateValue(inputValues: Map[String, WomValue], ioFunctionSet: IoFunctionSet): ErrorOr[WomValue] = { - def fromOutputBinding = - parameter.outputBinding.map(evaluateOutputBinding(inputValues, ioFunctionSet, parameter.secondaryFiles, parameter.format)(_, cwlExpressionType)) - - def fromType = - parameter.`type`.map(_.fold(MyriadOutputTypeToWomValue).apply( - evaluateOutputBinding(inputValues, ioFunctionSet, parameter.secondaryFiles, parameter.format), - schemaDefRequirement - )) - - fromOutputBinding.orElse(fromType).getOrElse(s"Cannot evaluate ${parameter.toString}".invalidIOChecked).toErrorOr - } - - /** - * Returns the list of files that _will be_ output after the command is run, unless they are optional and if so _may be_. - * - * In CWL, a list of outputs is specified as glob, say `*.bam`, plus a list of secondary files that may be in the - * form of paths or specified using carets such as `^.bai`. - * - * The coerceTo may be one of four different values: - * - WomMaybePopulatedFileType - * - WomArrayType(WomMaybePopulatedFileType) - * - WomMaybeListedDirectoryType - * - WomArrayType(WomMaybeListedDirectoryType) (Possible according to the way the spec is written, but not likely?) - */ - override def evaluateFiles(inputs: Map[String, WomValue], unused: IoFunctionSet, coerceTo: WomType): ErrorOr[Set[FileEvaluation]] = { - import cats.syntax.apply._ - - // Ignore the supplied ioFunctionSet and use a custom stubbed IoFunctionSet. This is better than a real I/O function set - // because in the context of file evaluation we don't care about the results of these operations. The NoIoFunctionSet - // that otherwise would be used throws for all of its operations which doesn't fly for the way our CWL evaluation works. - val stubbedIoFunctionSet = new EmptyIoFunctionSet { - override def size(path: String): Future[Long] = Future.successful(0L) - override def isDirectory(path: String): Future[Boolean] = Future.successful(false) - override def ec: ExecutionContext = scala.concurrent.ExecutionContext.global - } - def fromOutputBinding: IOChecked[Set[FileEvaluation]] = parameter - .outputBinding - .map(evaluateOutputBindingFiles(inputs, stubbedIoFunctionSet, parameter.secondaryFiles, coerceTo)) - .getOrElse(Set.empty[FileEvaluation].validIOChecked) - - def fromType: IOChecked[Set[FileEvaluation]] = parameter - .`type` - .map(_.fold(MyriadOutputTypeToWomFiles).apply(evaluateOutputBindingFiles(inputs, stubbedIoFunctionSet, parameter.secondaryFiles, coerceTo))) - .getOrElse(Set.empty[FileEvaluation].validIOChecked) - - val optional: Boolean = parameter.`type`.exists(_.fold(OutputTypeIsOptional)) - - ((fromOutputBinding, fromType) mapN (_ ++ _) map { _ map { _.copy(optional = optional) } }).toErrorOr - } -} - -object OutputTypeIsOptional extends Poly1 { - implicit val one: Case.Aux[MyriadOutputInnerType, Boolean] = at[MyriadOutputInnerType] { const(false) } - - implicit val arr: Case.Aux[Array[MyriadOutputInnerType], Boolean] = at[Array[MyriadOutputInnerType]] { - // Possibly too broad, would return true for just single 'null'. - _.exists(_.select[CwlType].contains(CwlType.Null)) - } -} diff --git a/cwl/src/main/scala/cwl/ParameterContext.scala b/cwl/src/main/scala/cwl/ParameterContext.scala deleted file mode 100644 index 4bcb4d132d5..00000000000 --- a/cwl/src/main/scala/cwl/ParameterContext.scala +++ /dev/null @@ -1,17 +0,0 @@ -package cwl - -import wom.callable.RuntimeEnvironment -import wom.expression.IoFunctionSet -import wom.types.WomNothingType -import wom.values.{WomOptionalValue, WomValue} - -object ParameterContext { - val EmptySelf = WomOptionalValue(WomNothingType, None) -} - -case class ParameterContext( - ioFunctionSet: IoFunctionSet, - expressionLib: ExpressionLib, - inputs: Map[String, WomValue] = Map.empty, - self: WomValue = ParameterContext.EmptySelf, - runtimeOption: Option[RuntimeEnvironment] = None) diff --git a/cwl/src/main/scala/cwl/RequirementsAndHints.scala b/cwl/src/main/scala/cwl/RequirementsAndHints.scala deleted file mode 100644 index 46b5c52e410..00000000000 --- a/cwl/src/main/scala/cwl/RequirementsAndHints.scala +++ /dev/null @@ -1,9 +0,0 @@ -package cwl - -case class RequirementsAndHints(list: List[Requirement]) { - - lazy val hasShellCommandRequirement: Boolean = list.exists(_.select[ShellCommandRequirement].nonEmpty) - - //Should we add up all the types instead? This would mean subworkflows can inherit their parent schemas - lazy val schemaDefRequirement = list.flatMap(_.select[SchemaDefRequirement]).headOption.getOrElse(SchemaDefRequirement()) -} diff --git a/cwl/src/main/scala/cwl/RunOutputsToTypeMap.scala b/cwl/src/main/scala/cwl/RunOutputsToTypeMap.scala deleted file mode 100644 index f327b5b03ea..00000000000 --- a/cwl/src/main/scala/cwl/RunOutputsToTypeMap.scala +++ /dev/null @@ -1,45 +0,0 @@ -package cwl - -import shapeless.Poly1 -import wom.types.WomType - -object RunOutputsToTypeMap extends Poly1 { - - import Case.Aux - - type SchemaDefToTypeMap = SchemaDefRequirement => WomTypeMap - - def handleOutputParameters[A <: OutputParameter](outputs: Array[A], schemaDefRequirement: SchemaDefRequirement): Map[String, WomType] = { - outputs.toList.foldLeft(Map.empty[String, WomType]) { - (acc, out) => - acc ++ - out. - `type`. - map(_.fold(MyriadOutputTypeToWomType).apply(schemaDefRequirement)). - map(out.id -> _). - toList. - toMap - } - } - - implicit def commandLineTool : Aux[CommandLineTool, SchemaDefToTypeMap] = - at[CommandLineTool] { - clt => handleOutputParameters(clt.outputs, _) - } - - implicit def string: Aux[String, SchemaDefToTypeMap] = at[String] { - _ => _ => - Map.empty[String, WomType] //should never happen because we dereference/embed CWL subworkflows into one object from the original string references - } - - implicit def expressionTool: Aux[ExpressionTool, SchemaDefToTypeMap] = at[ExpressionTool] { - et => - handleOutputParameters(et.outputs, _) - } - - implicit def workflow: Aux[Workflow, SchemaDefToTypeMap] = at[Workflow] { - wf => - handleOutputParameters(wf.outputs, _) - } -} - diff --git a/cwl/src/main/scala/cwl/RunToEmbeddedCwl.scala b/cwl/src/main/scala/cwl/RunToEmbeddedCwl.scala deleted file mode 100644 index 2547b6333a4..00000000000 --- a/cwl/src/main/scala/cwl/RunToEmbeddedCwl.scala +++ /dev/null @@ -1,36 +0,0 @@ -package cwl - -import shapeless._ -import WorkflowStep.Run - -object RunToEmbeddedCwl extends Poly1 { - implicit def commandLineTool = - at[CommandLineTool] { - clt => - (_: Map[String, Cwl]) => - Coproduct[Run](clt) - } - - implicit def string = at[String] { - fileName => - (cwlMap: Map[String, Cwl]) => { - val cwl = cwlMap(fileName) - cwl.fold(CwlToRun) - } - } - - - implicit def expressionTool = at[ExpressionTool] { - et => - (_: Map[String, Cwl]) => - Coproduct[Run](et) - } - - implicit def workflow = at[Workflow] { - wf => - (_: Map[String, Cwl]) => - Coproduct[Run](wf) - } -} - - diff --git a/cwl/src/main/scala/cwl/RunToInputTypeMap.scala b/cwl/src/main/scala/cwl/RunToInputTypeMap.scala deleted file mode 100644 index 7111e389050..00000000000 --- a/cwl/src/main/scala/cwl/RunToInputTypeMap.scala +++ /dev/null @@ -1,37 +0,0 @@ -package cwl - -import cwl.command.ParentName -import shapeless.Poly1 - -object RunToInputTypeMap extends Poly1 { - - type MyriadInputTypeMap = Map[String, Option[MyriadInputType]] - - type OutputType = ParentName => MyriadInputTypeMap - - import Case.Aux - - implicit def s: Aux[String, OutputType] = - at[String] { - run => _ => throw new RuntimeException(s"Run field $run was not inlined as expected") - } - - implicit def clt: Aux[CommandLineTool, OutputType] = - at[CommandLineTool]{clt => parentName => clt.inputs.map{ - input => - FullyQualifiedName(input.id)(parentName).id -> input.`type` - }.toMap} - - implicit def et: Aux[ExpressionTool, OutputType] = - at[ExpressionTool]{et => parentName => et.inputs.map{ - input => - FullyQualifiedName(input.id)(parentName).id -> input.`type` - }.toMap} - - implicit def wf: Aux[Workflow, OutputType] = - at[Workflow]{ wf => parentName => wf.inputs.map{ - input => - FullyQualifiedName(input.id)(parentName).id -> input.`type` - }.toMap} - -} diff --git a/cwl/src/main/scala/cwl/ScatterLogic.scala b/cwl/src/main/scala/cwl/ScatterLogic.scala deleted file mode 100644 index 97d87daf6df..00000000000 --- a/cwl/src/main/scala/cwl/ScatterLogic.scala +++ /dev/null @@ -1,263 +0,0 @@ -package cwl - -import cats.data.NonEmptyList -import cats.instances.list._ -import common.Checked -import common.validation.Checked._ -import common.validation.ErrorOr.ErrorOr -import cwl.ScatterMethod.ScatterMethod -import shapeless.Poly1 -import wom.graph.CallNode.CallNodeAndNewNodes -import wom.graph.GraphNodePort.ScatterGathererPort -import wom.graph.ScatterNode._ -import wom.graph._ -import wom.graph.expression.ExpressionNode -import wom.types.{WomArrayType, WomMaybeEmptyArrayType, WomType} -import wom.values.{WomArray, WomValue} - -import scala.annotation.tailrec - -/** - * Contains methods used to implement the scatter related logic for CWL workflows. - * The general idea is as follows: - * - * See WorkflowStep for an example of WOM graph for scatters. - * - * # Scattering over multiple variables - * - * When several variables are being scattered over, a scatter method is mandatory to specify how the elements must be combined. - * Let's use the following 2 arrays as an example: - * - * a1: ["one", "two"], a2: ["three", "four"] - * - * * DotProduct: - * DotProduct only works if all the arrays have the same length. - * - * Shard 0: "one" - "three" - * Shard 1: "two" - "four" - * - * * CrossProduct (nested or flat): - * Both cross product methods generate the same shards, the difference is in how they are collected at the end - * - * Shard 0: "one" - "three" - * Shard 1: "one" - "four" - * Shard 2: "two" - "three" - * Shard 3: "two" - "four" - * - * To support this, each SVN will have a method which given a shard index will return the index at which the value should be looked up in the array. - * - * For dot product, this function is the same for all SVN and will be the identity[Int] function. - * For cross product, this function depends on the number of elements in each array, which is known at runtime. - * When we do have this information we update each SVN with it in the ScatterProcessingFunction and returns the number of shards to be generated. - */ -object ScatterLogic { - case class ScatterFunctions(processing: ScatterProcessingFunction, collection: ScatterCollectionFunctionBuilder) - - object ScatterVariablesPoly extends Poly1 { - implicit def fromString: Case.Aux[String, List[String]] = at[String] { s: String => List(s) } - implicit def fromStringList: Case.Aux[Array[String], List[String]] = at[Array[String]] { l: Array[String] => l.toList } - } - - // Validate that the scatter expression is an ArrayLike and return the member type - private [cwl] def scatterExpressionItemType(expressionNode: ExpressionNode) = { - expressionNode.womType match { - case WomArrayType(itemType) => itemType.validNelCheck // Covers maps because this is a custom unapply (see WomArrayType) - case other => s"Cannot scatter over a non-traversable type ${other.stableName}".invalidNelCheck - } - } - - def scatterGatherPortTypeFunction(scatterMethod: Option[ScatterMethod], scatterVariables: NonEmptyList[_]): WomType => WomArrayType = scatterMethod match { - case Some(ScatterMethod.NestedCrossProduct) => - innerType: WomType => - scatterVariables.tail.foldLeft(WomArrayType(innerType))({ case (t, _) => WomArrayType(t) }) - case _ => innerType: WomType => WomArrayType(innerType) - } - - // Build a map (potentially empty) of scatterered input steps with their corresponding SVN node. - def buildScatterVariableNodes(scatter: ScatterVariables, stepInputMappings: Map[WorkflowStepInput, ExpressionNode], stepId: String): Checked[Map[WorkflowStepInput, ScatterVariableNode]] = { - import cats.syntax.all._ - - def buildScatterVariable(scatterVariableName: String): ErrorOr[(WorkflowStepInput, ScatterVariableNode)] = { - // Assume the variable is a step input (is that always true ??). Find the corresponding expression node - - stepInputMappings.find({ case (stepInput, _) => stepInput.id == scatterVariableName }) match { - case Some((stepInput, expressionNode)) => - scatterExpressionItemType(expressionNode).toValidated map { itemType => - // create a scatter variable node for other scattered nodes to point to - stepInput -> ScatterVariableNode(WomIdentifier(scatterVariableName), expressionNode, itemType) - } - case None => s"Could not find a variable $scatterVariableName in the workflow step input to scatter over. Please make sure $scatterVariableName is an input of step $stepId".invalidNel - } - } - // Take the scatter field defining the (list of) input(s) to be scattered over - scatter - // Fold them so we have a List[String] no matter what (a single scattered input becomes a single element list) - .map(_.fold(ScatterVariablesPoly)) - // If there's no scatter make it an empty list - .getOrElse(List.empty) - // Traverse the list to create ScatterVariableNodes that will be used later on to create the ScatterNode - .traverse(buildScatterVariable) - .toEither - .map(_.toMap) - } - - // Prepare the nodes to be returned if this call is being scattered - def buildScatterNode(callNodeAndNewNodes: CallNodeAndNewNodes, - scatterVariableNodes: NonEmptyList[ScatterVariableNode], - ogins: Set[OuterGraphInputNode], - stepExpressionNodes: Set[ExpressionNode], - scatterMethod: Option[ScatterMethod]): Checked[ScatterNode] = { - - val scatterProcessingFunctionCheck = (scatterVariableNodes.size, scatterMethod) match { - // If we scatter over one variable only, the default processing method can handle it - case (1, _) => ScatterFunctions(ScatterNode.DefaultScatterProcessingFunction, ScatterNode.DefaultScatterCollectionFunctionBuilder).validNelCheck - case (_, Some(method)) => ScatterFunctions(processingFunction(method), collectingFunctionBuilder(method)).validNelCheck - case (_, None) => "When scattering over multiple variables, a scatter method needs to be defined. See http://www.commonwl.org/v1.0/Workflow.html#WorkflowStep".invalidNelCheck - } - - val callNode = callNodeAndNewNodes.node - - // We need to generate PBGONs for every output port of the call, so that they can be linked outside the scatter graph - val portBasedGraphOutputNodes = callNode.outputPorts.map(op => PortBasedGraphOutputNode(op.identifier, op.womType, op)) - - def buildScatterNode(innerGraph: Graph, scatterFunctions: ScatterFunctions) = { - val scatterNodeBuilder = new ScatterNodeBuilder - val outputPorts: Set[ScatterGathererPort] = innerGraph.nodes.collect { case gon: PortBasedGraphOutputNode => - scatterNodeBuilder.makeOutputPort(scatterGatherPortTypeFunction(scatterMethod, scatterVariableNodes)(gon.womType), gon) - } - - scatterNodeBuilder.build(innerGraph, outputPorts, scatterVariableNodes.toList, scatterFunctions.processing, scatterFunctions.collection) - } - - for { - scatterProcessingFunctionAndBuilder <- scatterProcessingFunctionCheck - graph <- Graph.validateAndConstruct( - Set(callNodeAndNewNodes.node) ++ - ogins ++ - stepExpressionNodes ++ - scatterVariableNodes.toList ++ - portBasedGraphOutputNodes - ).toEither - scatterNode = buildScatterNode(graph, scatterProcessingFunctionAndBuilder) - } yield scatterNode.node - } - - /* - * Processing function for cross products. Logic is as follows: - * We want 1) know how many shards should be created (which this function will return) - * 2) update each SVN with their relative index length (see ScatterVariableNode for more detail) - * - * We are going to multiply all the array sizes (which will give us 1) at the end) - * Along the way we will update the relative index length for each SVN. - * To do so, we start at the end of the list and recurse. - */ - private [cwl] val CrossProductScatterProcessingFunction: ScatterProcessingFunction = { nodesAndValues: List[ScatterVariableAndValue] => - @tailrec - def updateIndexLengthRec(currentList: List[ScatterVariableAndValue], combinedArraySize: Int): Int = currentList match { - case Nil => combinedArraySize - case ScatterVariableAndValue(variableNode, arrayValue) :: tail => - variableNode.withRelativeIndexLength(combinedArraySize) - updateIndexLengthRec(tail, combinedArraySize * arrayValue.size) - } - - // Reverse the list so that we start from the last variable and make our way back to the first one - updateIndexLengthRec(nodesAndValues.reverse, 1).validNelCheck - } - - private [cwl] val DotProductScatterProcessingFunction: ScatterProcessingFunction = ScatterNode.DefaultScatterProcessingFunction - - // Recursively nest shard results appropriately based on the size of the scatter variable arrays - private [cwl] val NestedCrossProductScatterCollectionFunctionBuilder: ScatterCollectionFunctionBuilder = { arraySizes: List[Int] => - (sortedShards: List[WomValue], valueType: WomArrayType) => - - @tailrec - def buildCollectorArrayRec(currentList: List[Int], currentWomValues: List[WomValue], currentWomType: WomType): WomArray = { - /* We stop right before the last element because it will necessarily be the number of elements in currentWomValues (provided the list is non empty). - * e.g if currentList is (2, 3, 2), then currentWomValues will have 2 * 3 * 2 = 12 elements (this is a cross product) - * As we recurse, we keep grouping currentWomValues by the numbers in currentList: - * - * group by 2: [[_, _], [_, _], [_, _], [_, _], [_, _], [_, _]] - * group by 3: [ - * [[_, _], [_, _], [_, _]], - * [[_, _], [_, _], [_, _]] - * ] - * This list has necessarily 2 elements already because 12 / 2 / 3 = 2. So we can make the final WomArray out of that - */ - currentList match { - case _ :: Nil | Nil => - WomArray(WomMaybeEmptyArrayType(currentWomType), currentWomValues) - case head :: tail => - val arrayType = WomMaybeEmptyArrayType(currentWomType) - val womArrays = currentWomValues.grouped(head).toList map { WomArray(arrayType, _) } - buildCollectorArrayRec(tail, womArrays, arrayType) - } - } - - def mostInnerType(tpe: WomType): WomType = tpe match { - case WomArrayType(t) => mostInnerType(t) - case other => other - } - - def buildEmptyArrays(arraySizeList: List[Int], womType: WomType, womValues: List[WomValue]): WomArray = arraySizeList match { - case Nil => womType match { - case arrayType: WomArrayType => WomArray(arrayType, womValues) - case _ => - // This would mean arraySizeList was empty to begin with (otherwise we would have wrapped womType in a WomArrayType at least once) - // which should be impossible since it would mean there was no scatter variables, so we shouldn't even be here - throw new RuntimeException("Programmer error ! We should not be collecting scatter nodes if there was no scatter !") - } - case 0 :: tail => - buildEmptyArrays(tail, WomArrayType(womType), List.empty) - case head :: tail => - val arrayType = womType match { - case array: WomArrayType => array - case nonArray => WomArrayType(nonArray) - } - val womArrays = (0 until head).toList map { _ => WomArray(arrayType, womValues) } - buildEmptyArrays(tail, WomArrayType(womType), womArrays) - } - - /* - * There are 2 distinct cases. - * If we have no shards, it means that at least one of the scatter expressions evaluated to an empty array. - * In that case we want to create an array structure of empty arrays reflecting the scatter array sizes. - * e.g: - * - * scatter array sizes | collected result - * [0] [] - * [0, 2] [] - * [2, 0] [[], []] - * [2, 0, 3] [[], []] - * [2, 3, 0] [[[], [], []], [[], [], []]] - * - * Note that as soon as we reach a size 0, we return an empty array. - * - * If we have shards, we successively group the shards list to build arrays of the right size. - * - * In both cases we reverse the list so that we can left recurse while still building the result starting from the most nested arrays - */ - if (sortedShards.isEmpty) { - buildEmptyArrays(arraySizes.reverse, mostInnerType(valueType), List.empty) - } else { - // - buildCollectorArrayRec(arraySizes.reverse, sortedShards, mostInnerType(valueType)) - } - } - - // select a processing function based on the scatter method - private def processingFunction(scatterMethod: ScatterMethod) = scatterMethod match { - case ScatterMethod.DotProduct => DotProductScatterProcessingFunction - // Both cross product methods use the same processing function, the difference is in how we collect results (see collectingFunctionBuilder) - case ScatterMethod.FlatCrossProduct => CrossProductScatterProcessingFunction - case ScatterMethod.NestedCrossProduct => CrossProductScatterProcessingFunction - } - - // select a collecting function builder based on the scatter method - private def collectingFunctionBuilder(scatterMethod: ScatterMethod) = scatterMethod match { - // dot product and flat cross product output a flat array, which is the default behavior - case ScatterMethod.DotProduct => ScatterNode.DefaultScatterCollectionFunctionBuilder - case ScatterMethod.FlatCrossProduct => ScatterNode.DefaultScatterCollectionFunctionBuilder - // nested cross product uses a special collecting function to build nested arrays - case ScatterMethod.NestedCrossProduct => NestedCrossProductScatterCollectionFunctionBuilder - } -} diff --git a/cwl/src/main/scala/cwl/ScatterMethod.scala b/cwl/src/main/scala/cwl/ScatterMethod.scala deleted file mode 100644 index 0a60575e0cb..00000000000 --- a/cwl/src/main/scala/cwl/ScatterMethod.scala +++ /dev/null @@ -1,9 +0,0 @@ -package cwl - -object ScatterMethod extends Enumeration { - type ScatterMethod = Value - - val DotProduct = Value("dotproduct") - val NestedCrossProduct = Value("nested_crossproduct") - val FlatCrossProduct = Value("flat_crossproduct") -} diff --git a/cwl/src/main/scala/cwl/StringOrStringArrayToStringList.scala b/cwl/src/main/scala/cwl/StringOrStringArrayToStringList.scala deleted file mode 100644 index 73afab205b7..00000000000 --- a/cwl/src/main/scala/cwl/StringOrStringArrayToStringList.scala +++ /dev/null @@ -1,12 +0,0 @@ -package cwl - -import shapeless.Poly1 - -/** - * From a logic standpoint it's easier to treat sources as a list of strings. - */ -object StringOrStringArrayToStringList extends Poly1{ - import Case._ - implicit def string: Aux[String, List[String]] = at[String]{List(_)} - implicit def array: Aux[Array[String], List[String]] = at[Array[String]]{_.toList} -} diff --git a/cwl/src/main/scala/cwl/Tool.scala b/cwl/src/main/scala/cwl/Tool.scala deleted file mode 100644 index d31b7368aee..00000000000 --- a/cwl/src/main/scala/cwl/Tool.scala +++ /dev/null @@ -1,189 +0,0 @@ -package cwl - -import java.nio.file.Paths - -import cats.instances.list._ -import common.Checked -import common.validation.ErrorOr.ErrorOr -import common.validation.Validation._ -import cwl.CwlVersion.CwlVersion -import cwl.Tool.inlineJavascriptRequirements -import cwl.command.ParentName -import cwl.requirement.RequirementToAttributeMap -import shapeless.Inl -import wom.callable.Callable.{OverridableInputDefinitionWithDefault, OptionalInputDefinition, OutputDefinition, RequiredInputDefinition} -import wom.callable.MetaValueElement.{MetaValueElementBoolean, MetaValueElementObject} -import wom.callable.{Callable, MetaValueElement, TaskDefinition} -import wom.executable.Executable -import wom.expression.{IoFunctionSet, ValueAsAnExpression, WomExpression} -import wom.types.WomOptionalType -import wom.values.{WomInteger, WomLong, WomString} -import wom.{RuntimeAttributes, RuntimeAttributesKeys} - -import scala.util.Try - -object Tool { - def inlineJavascriptRequirements(allRequirementsAndHints: Seq[Requirement]): Vector[String] = { - val inlineJavscriptRequirements: Seq[InlineJavascriptRequirement] = allRequirementsAndHints.toList.collect { - case Inl(ijr:InlineJavascriptRequirement) => ijr - } - - inlineJavscriptRequirements.flatMap(_.expressionLib.toList.flatten).toVector - } -} - -/** - * Abstraction over a Tool (CommandLineTool or ExpressionTool) - * Contains common logic to both, mostly to build the wom task definition. - */ -trait Tool { - def inputs: Array[_ <: InputParameter] - def outputs: Array[_ <: OutputParameter] - def `class`: String - def id: String - def requirements: Option[Array[Requirement]] - def hints: Option[Array[Hint]] - def label: Option[String] - def doc: Option[String] - def cwlVersion: Option[CwlVersion] - - def asCwl: Cwl - - /** Builds an `Executable` directly from a `Tool` CWL with no parent workflow. */ - def womExecutable(validator: RequirementsValidator, inputFile: Option[String] = None, ioFunctions: IoFunctionSet, strictValidation: Boolean): Checked[Executable] = { - val taskDefinition = buildTaskDefinition(validator, Vector.empty) - CwlExecutableValidation.buildWomExecutable(taskDefinition, inputFile, ioFunctions, strictValidation) - } - - protected def buildTaskDefinition(taskName: String, - inputDefinitions: List[_ <: Callable.InputDefinition], - outputDefinitions: List[Callable.OutputDefinition], - runtimeAttributes: RuntimeAttributes, - requirementsAndHints: List[cwl.Requirement], - expressionLib: ExpressionLib): ErrorOr[TaskDefinition] - - private [cwl] implicit val explicitWorkflowName = ParentName(id) - protected val inputNames: Set[String] = this.inputs.map(i => FullyQualifiedName(i.id).id).toSet - - // Circe can't create bidirectional links between workflow steps and runs (including `CommandLineTool`s) so this - // ugly var is here to link back to a possible parent workflow step. This is needed to navigate upward for finding - // requirements in the containment hierarchy. There isn't always a containing workflow step so this is an `Option`. - private[cwl] var parentWorkflowStep: Option[WorkflowStep] = None - - - private def validateRequirementsAndHints(validator: RequirementsValidator): ErrorOr[List[Requirement]] = { - import cats.syntax.traverse._ - - val allRequirements = requirements.toList.flatten ++ parentWorkflowStep.toList.flatMap(_.allRequirements.list) - // All requirements must validate or this fails. - val errorOrValidatedRequirements: ErrorOr[List[Requirement]] = allRequirements traverse validator - - errorOrValidatedRequirements map { validRequirements => - // Only Requirement hints, everything else is thrown out. - // TODO CWL don't throw them out but pass them back to the caller to do with as the caller pleases. - val hintRequirements = hints.toList.flatten.flatMap { _.select[Requirement] } - val parentHintRequirements = parentWorkflowStep.toList.flatMap(_.allHints) - - // Throw out invalid Requirement hints. - // TODO CWL pass invalid hints back to the caller to do with as the caller pleases. - val validHints = (hintRequirements ++ parentHintRequirements).collect { case req if validator(req).isValid => req } - validRequirements ++ validHints - } - } - - private def processRequirement(requirement: Requirement, expressionLib: ExpressionLib): Map[String, WomExpression] = { - requirement.fold(RequirementToAttributeMap).apply(inputNames, expressionLib) - } - - protected def toolAttributes: Map[String, WomExpression] = Map.empty - - def buildTaskDefinition(validator: RequirementsValidator, parentExpressionLib: ExpressionLib): Checked[TaskDefinition] = { - def build(requirementsAndHints: Seq[cwl.Requirement]) = { - val id = this.id - - val expressionLib: ExpressionLib = parentExpressionLib ++ inlineJavascriptRequirements(requirementsAndHints) - - // This is basically doing a `foldMap` but can't actually be a `foldMap` because: - // - There is no monoid instance for `WomExpression`s. - // - We want to fold from the right so the hints and requirements with the lowest precedence are processed first - // and later overridden if there are duplicate hints or requirements of the same type with higher precedence. - val attributesMap: Map[String, WomExpression] = requirementsAndHints.foldRight(Map.empty[String, WomExpression])({ - case (requirement, acc) => acc ++ processRequirement(requirement, expressionLib) - }) - - val runtimeAttributes: RuntimeAttributes = RuntimeAttributes(attributesMap ++ toolAttributes) - - val schemaDefRequirement: SchemaDefRequirement = requirementsAndHints.flatMap{ - _.select[SchemaDefRequirement].toList - }.headOption.getOrElse(SchemaDefRequirement()) - - lazy val localizationOptionalMetaObject: MetaValueElement = MetaValueElementObject( - Map( - "localization_optional" -> MetaValueElementBoolean(true) - ) - ) - - // If input dir min == 0 that's a signal not to localize the input files - // https://github.com/dnanexus/dx-cwl/blob/fca163d825beb62f8a3004f2e0a6742805e6218c/dx-cwl#L426 - val localizationOptional = runtimeAttributes.attributes.get(RuntimeAttributesKeys.DnaNexusInputDirMinKey) match { - case Some(ValueAsAnExpression(WomInteger(0))) => Option(localizationOptionalMetaObject) - case Some(ValueAsAnExpression(WomLong(0L))) => Option(localizationOptionalMetaObject) - case Some(ValueAsAnExpression(WomString("0"))) => Option(localizationOptionalMetaObject) - case _ => None - } - - val inputDefinitions: List[_ <: Callable.InputDefinition] = - this.inputs.map { - case input @ InputParameter.IdDefaultAndType(inputId, default, tpe) => - val inputType = tpe.fold(MyriadInputTypeToWomType).apply(schemaDefRequirement) - val inputName = FullyQualifiedName(inputId).id - val defaultWomValue = default.fold(InputParameter.DefaultToWomValuePoly).apply(inputType).toTry.get - OverridableInputDefinitionWithDefault( - inputName, - inputType, - ValueAsAnExpression(defaultWomValue), - InputParameter.inputValueMapper(input, tpe, expressionLib, asCwl.schemaOption), - localizationOptional - ) - case input @ InputParameter.IdAndType(inputId, tpe) => - val inputType = tpe.fold(MyriadInputTypeToWomType).apply(schemaDefRequirement) - val inputName = FullyQualifiedName(inputId).id - inputType match { - case optional: WomOptionalType => - OptionalInputDefinition( - inputName, - optional, - InputParameter.inputValueMapper(input, tpe, expressionLib, asCwl.schemaOption), - localizationOptional - ) - case _ => - RequiredInputDefinition( - inputName, - inputType, - InputParameter.inputValueMapper(input, tpe, expressionLib, asCwl.schemaOption), - localizationOptional - ) - } - case other => throw new UnsupportedOperationException(s"command input parameters such as $other are not yet supported") - }.toList - - val outputDefinitions: List[Callable.OutputDefinition] = this.outputs.map { - case p @ OutputParameter.IdAndType(cop_id, tpe) => - val womType = tpe.fold(MyriadOutputTypeToWomType).apply(schemaDefRequirement) - OutputDefinition(FullyQualifiedName(cop_id).id, womType, OutputParameterExpression(p, womType, inputNames, expressionLib, schemaDefRequirement)) - case other => throw new UnsupportedOperationException(s"Command output parameters such as $other are not yet supported") - }.toList - - // The try will succeed if this is a task within a step. If it's a standalone file, the ID will be the file, - // so the filename is the fallback. - def taskName = Try(FullyQualifiedName(id).id).getOrElse(Paths.get(id).getFileName.toString) - - buildTaskDefinition(taskName, inputDefinitions, outputDefinitions, runtimeAttributes, requirementsAndHints.toList, expressionLib).toEither - } - - validateRequirementsAndHints(validator).toEither match { - case Right(requirementsAndHints: Seq[cwl.Requirement]) => build(requirementsAndHints) - case Left(errors) => Left(errors) - } - } -} diff --git a/cwl/src/main/scala/cwl/TypeAliases.scala b/cwl/src/main/scala/cwl/TypeAliases.scala deleted file mode 100644 index abbfc9e81e0..00000000000 --- a/cwl/src/main/scala/cwl/TypeAliases.scala +++ /dev/null @@ -1,115 +0,0 @@ -package cwl - -import cwl.CwlType.CwlType -import cwl.ExpressionEvaluator.{ECMAScriptExpression, ECMAScriptFunction} -import io.circe.Json -import shapeless.{:+:, CNil} -import wom.types.WomType - -trait TypeAliases { - - type Expression = ECMAScriptFunction :+: ECMAScriptExpression :+: CNil - - // http://www.commonwl.org/v1.0/Workflow.html#InputParameter - // http://www.commonwl.org/v1.0/CommandLineTool.html#CommandInputParameter - type InputParameterFormat = Expression :+: String :+: Array[String] :+: CNil - - // http://www.commonwl.org/v1.0/Workflow.html#ExpressionToolOutputParameter - // http://www.commonwl.org/v1.0/CommandLineTool.html#CommandOutputParameter - type OutputParameterFormat = StringOrExpression - - type Doc = String :+: Array[String] :+: CNil - - type StringOrExpression = Expression :+: String :+: CNil - - type CwlAny = - FileOrDirectory :+: - Array[FileOrDirectory] :+: - Json :+: - CNil - - type WorkflowStepInputId = String - - type Requirement = - InlineJavascriptRequirement :+: - SchemaDefRequirement :+: - DockerRequirement :+: - SoftwareRequirement :+: - InitialWorkDirRequirement :+: - EnvVarRequirement :+: - ShellCommandRequirement :+: - ResourceRequirement :+: - SubworkflowFeatureRequirement :+: - ScatterFeatureRequirement :+: - MultipleInputFeatureRequirement :+: - StepInputExpressionRequirement :+: - DnaNexusInputResourceRequirement :+: - CNil - - type Hint = - Requirement :+: - CwlAny :+: - CNil - - type MyriadInputType = - MyriadInputInnerType :+: - Array[MyriadInputInnerType] :+: - CNil - - type MyriadInputInnerType = - CwlType :+: - InputRecordSchema :+: - InputEnumSchema :+: - InputArraySchema :+: - String :+: - CNil - - type MyriadOutputType = - MyriadOutputInnerType :+: - Array[MyriadOutputInnerType] :+: - CNil - - type MyriadOutputInnerType = - CwlType :+: - OutputRecordSchema :+: - OutputEnumSchema :+: - OutputArraySchema :+: - String :+: - CNil - - type ResourceRequirementType = Long :+: Expression :+: String :+: CNil - - type SingleOrArrayOfStrings = String :+: Array[String] :+: CNil - - type ScatterVariables = Option[SingleOrArrayOfStrings] - - type FileOrDirectory = File :+: Directory :+: CNil - - type Glob = StringOrExpression :+: Array[String] :+: CNil - - type SecondaryFiles = StringOrExpression :+: Array[StringOrExpression] :+: CNil -} - -object MyriadInputType { - - object InputArraySchema { - def unapply(m: MyriadInputType): Option[InputArraySchema] = m.select[MyriadInputInnerType].flatMap(_.select[InputArraySchema]) - } - - object InputArray { - def unapply(m: MyriadInputType): Option[Array[MyriadInputInnerType]] = m.select[Array[MyriadInputInnerType]] - } - - object CwlType { - def unapply(m: MyriadInputType): Option[CwlType] = { - m.select[MyriadInputInnerType].flatMap(_.select[CwlType]) - } - } - - object WomType { - def unapply(m: MyriadInputType): Option[WomType] = m match { - case CwlType(c) => Option(cwl.cwlTypeToWomType(c)) - case _ => None - } - } -} diff --git a/cwl/src/main/scala/cwl/Workflow.scala b/cwl/src/main/scala/cwl/Workflow.scala deleted file mode 100644 index 22b30019ff1..00000000000 --- a/cwl/src/main/scala/cwl/Workflow.scala +++ /dev/null @@ -1,246 +0,0 @@ -package cwl - -import java.nio.file.Paths - -import cats.data.NonEmptyList -import cats.syntax.either._ -import cats.syntax.traverse._ -import cats.instances.list._ -import common.Checked -import common.validation.ErrorOr._ -import common.validation.Validation._ -import cwl.CwlVersion._ -import cwl.LinkMergeMethod.LinkMergeMethod -import cwl.Workflow.{WorkflowInputParameter, WorkflowOutputParameter} -import cwl.command.ParentName -import shapeless._ -import shapeless.syntax.singleton._ -import wom.SourceFileLocation -import wom.callable.{MetaValueElement, WorkflowDefinition} -import wom.executable.Executable -import wom.expression.{IoFunctionSet, ValueAsAnExpression} -import wom.graph.GraphNodePort.{GraphNodeOutputPort, OutputPort} -import wom.graph._ -import wom.types.{WomOptionalType, WomType} - -case class Workflow private( - cwlVersion: Option[CwlVersion], - `class`: Witness.`"Workflow"`.T, - id: String, - inputs: Array[WorkflowInputParameter], - outputs: Array[WorkflowOutputParameter], - steps: Array[WorkflowStep], - requirements: Option[Array[Requirement]], - hints: Option[Array[Hint]], - `$namespaces`: Option[Map[String, String]], - `$schemas`: Option[Array[String]] - ) { - - steps.foreach { _.parentWorkflow = this } - - /** Builds an `Executable` from a `Workflow` CWL with no parent `Workflow` */ - def womExecutable(validator: RequirementsValidator, inputFile: Option[String] = None, ioFunctions: IoFunctionSet, strictValidation: Boolean): Checked[Executable] = { - CwlExecutableValidation.buildWomExecutableCallable(womDefinition(validator, Vector.empty), inputFile, ioFunctions, strictValidation) - } - - /* Circe can't create bidirectional links between workflow steps and runs (including `Workflow`s) so this - * ugly var is here to link back to a possible parent workflow step. This is needed to navigate upward for finding - * requirements in the containment hierarchy. There isn't always a containing workflow step so this is an `Option`. - */ - private[cwl] var parentWorkflowStep: Option[WorkflowStep] = None - - val allRequirements: RequirementsAndHints = RequirementsAndHints(requirements.toList.flatten ++ parentWorkflowStep.toList.flatMap { _.allRequirements.list }) - - private [cwl] implicit val explicitWorkflowName = ParentName(id) - - lazy val womFqn: Option[wom.graph.FullyQualifiedName] = { - explicitWorkflowName.value map { workflowName => - /* Sometimes the workflow name is of the form "parentWorkflowStepName/UUID" - * In that case, we don't want the fqn to look like "parentWorkflowStepName.parentWorkflowStepName/UUID" - * To avoid that, strip the parentWorkflowStepName from the workflowName - */ - val cleanWorkflowName = parentWorkflowStep - .map(_.womFqn.value + "/") - .map(workflowName.stripPrefix) - .getOrElse(workflowName) - - parentWorkflowStep.map(_.womFqn.combine(cleanWorkflowName)) - .getOrElse(wom.graph.FullyQualifiedName(cleanWorkflowName)) - } - } - - lazy val allHints: List[Requirement] = { - // Just ignore any hint that isn't a Requirement. - val requirementHints = hints.toList.flatten.flatMap { _.select[Requirement] } - requirementHints ++ parentWorkflowStep.toList.flatMap { _.allHints } - } - - val fileNames: List[String] = steps.toList.flatMap(_.run.select[String].toList) - - def outputsTypeMap: WomTypeMap = steps.foldLeft(Map.empty[String, WomType]) { - /* Not implemented as a `foldMap` because there is no semigroup instance for `WomType`s. `foldMap` doesn't know that - * we don't need a semigroup instance since the map keys should be unique and therefore map values would never need - * to be combined under the same key. - */ - (acc, s) => acc ++ s.typedOutputs - } - - def womGraph(workflowName: String, validator: RequirementsValidator, expressionLib: ExpressionLib): Checked[Graph] = { - val workflowNameIdentifier = explicitWorkflowName.value.map(WomIdentifier.apply).getOrElse(WomIdentifier(workflowName)) - - def womTypeForInputParameter(input: InputParameter): Option[WomType] = { - input.`type`.map(_.fold(MyriadInputTypeToWomType).apply(allRequirements.schemaDefRequirement)) - } - - val typeMap: WomTypeMap = - outputsTypeMap ++ - /* Note this is only looking at the workflow inputs and not recursing into steps, because our current thinking - * is that in CWL graph inputs can only be defined at the workflow level. It's possible that's not actually - * correct, but that's the assumption being made here. - */ - inputs.toList.flatMap { i => - womTypeForInputParameter(i).map(i.id -> _).toList - }.toMap - - val externalGraphInputNodes: Set[ExternalGraphInputNode] = inputs.map { wip => - val womType: WomType = womTypeForInputParameter(wip).get - val parsedInputId = FileAndId(wip.id).id - val womId = WomIdentifier(parsedInputId, wip.id) - val valueMapper = - InputParameter.inputValueMapper(wip, wip.`type`.get, expressionLib, asCwl.schemaOption) - - def optionalWithDefault(memberType: WomType): OptionalGraphInputNodeWithDefault = { - val defaultValue = wip.default.get.fold(InputParameter.DefaultToWomValuePoly).apply(womType).toTry.get - OptionalGraphInputNodeWithDefault(womId, memberType, ValueAsAnExpression(defaultValue), parsedInputId, valueMapper) - } - - womType match { - case WomOptionalType(memberType) if wip.default.isDefined => optionalWithDefault(memberType) - case _ if wip.default.isDefined => optionalWithDefault(womType) - case optional @ WomOptionalType(_) => OptionalGraphInputNode(womId, optional, parsedInputId, valueMapper) - case _ => RequiredGraphInputNode(womId, womType, parsedInputId, valueMapper) - } - }.toSet - - val workflowInputs: Map[String, GraphNodeOutputPort] = - externalGraphInputNodes.map { egin => - egin.localName -> egin.singleOutputPort - }.toMap - - val graphFromSteps: Checked[Set[GraphNode]] = - steps. - toList. - foldLeft((Set.empty[GraphNode] ++ externalGraphInputNodes).asRight[NonEmptyList[String]])( - (nodes, step) => nodes.flatMap(step.callWithInputs(typeMap, this, _, workflowInputs, validator, expressionLib))) - - val graphFromOutputs: Checked[Set[GraphNode]] = - outputs.toList.traverse[ErrorOr, GraphNode] { - case WorkflowOutputParameter(id, _, _, _, _, _, _, Some(Inl(outputSource: String)), _, Some(tpe)) => - val womType: WomType = tpe.fold(MyriadOutputTypeToWomType).apply(allRequirements.schemaDefRequirement) - - val parsedWorkflowOutput = FileAndId(id) - val parsedOutputSource = cwl.FullyQualifiedName(outputSource) - - // Try to find an output port for this cwl output in the set of available nodes - def lookupOutputSource(fqn: cwl.FullyQualifiedName): Checked[OutputPort] = { - def isRightOutputPort(op: GraphNodePort.OutputPort) = cwl.FullyQualifiedName.maybeApply(op.name) match { - case Some(f) => f.id == fqn.id - case None => op.internalName == fqn.id - } - - def sourceNode(graph: Set[GraphNode]): Checked[GraphNode] = { - val findSource: PartialFunction[GraphNode, GraphNode] = fqn match { - // A step output becoming a workflow output. - case fsi: FileStepAndId => { - case callNode: CallNode if callNode.localName == fsi.stepId => callNode - case scatterNode: ScatterNode if scatterNode.innerGraph.calls.exists(_.localName == fsi.stepId) => scatterNode - } - // A workflow input recycled back to be an output. - case fi: FileAndId => { - case gin: ExternalGraphInputNode if gin.nameInInputSet == fi.id => gin - } - } - graph collectFirst findSource toRight NonEmptyList.one(s"Call Node by name $fqn was not found in set $graph") - } - - for { - set <- graphFromSteps - node <- sourceNode(set) - output <- node.outputPorts.find(isRightOutputPort).toChecked(s"looking for ${fqn.id} in call $node output ports ${node.outputPorts}") - } yield output - } - - lookupOutputSource(parsedOutputSource).map({ port => - val localName = LocalName(parsedWorkflowOutput.id) - val fullyQualifiedName = workflowNameIdentifier.fullyQualifiedName.combine(parsedWorkflowOutput.id) - val outputIdentifier = WomIdentifier(localName, fullyQualifiedName) - PortBasedGraphOutputNode(outputIdentifier, womType, port) - }).toValidated - case wop => throw new UnsupportedOperationException(s"Workflow output parameters such as $wop are not supported.") - }.map(_.toSet).toEither - - for { - outputs <- graphFromOutputs - steps <- graphFromSteps - ret <- Graph.validateAndConstruct(steps ++ externalGraphInputNodes ++ outputs).toEither - } yield ret - } - - def womDefinition(validator: RequirementsValidator, expressionLib: ExpressionLib): Checked[WorkflowDefinition] = { - val name: String = Paths.get(id).getFileName.toString - val meta: Map[String, MetaValueElement.MetaValueElementString] = Map.empty - val paramMeta: Map[String, MetaValueElement.MetaValueElementString] = Map.empty - val lexInfo : Option[SourceFileLocation] = None - - womGraph(name, validator, expressionLib).map(graph => - WorkflowDefinition( - name, - graph, - meta, - paramMeta, - lexInfo - ) - ) - } - - def asCwl = Coproduct[Cwl](this) -} -object Workflow { - - case class WorkflowInputParameter(id: String, - label: Option[String] = None, - secondaryFiles: Option[SecondaryFiles] = None, - format: Option[InputParameterFormat] = None, - streamable: Option[Boolean] = None, - doc: Option[Doc] = None, - inputBinding: Option[InputCommandLineBinding] = None, - default: Option[CwlAny] = None, - `type`: Option[MyriadInputType] = None) extends InputParameter - - case class WorkflowOutputParameter( - id: String, - label: Option[String] = None, - secondaryFiles: Option[SecondaryFiles] = None, - format: Option[OutputParameterFormat] = None, - streamable: Option[Boolean] = None, - doc: Option[Doc] = None, - outputBinding: Option[CommandOutputBinding] = None, - outputSource: Option[WorkflowOutputParameter#OutputSource] = None, - linkMerge: Option[LinkMergeMethod] = None, - `type`: Option[MyriadOutputType] = None) extends OutputParameter { - - type OutputSource = String :+: Array[String] :+: CNil - } - - def apply(cwlVersion: Option[CwlVersion] = Option(CwlVersion.Version1), - id: String, - inputs: Array[WorkflowInputParameter] = Array.empty, - outputs: Array[WorkflowOutputParameter] = Array.empty, - steps: Array[WorkflowStep] = Array.empty, - requirements: Option[Array[Requirement]] = None, - hints: Option[Array[Hint]] = None, - namespaces: Option[Map[String, String]] = None, - schemas: Option[Array[String]] = None - ): Workflow = - Workflow(cwlVersion, "Workflow".narrow, id, inputs, outputs, steps, requirements, hints, namespaces, schemas) -} diff --git a/cwl/src/main/scala/cwl/WorkflowStep.scala b/cwl/src/main/scala/cwl/WorkflowStep.scala deleted file mode 100644 index cf39f261bae..00000000000 --- a/cwl/src/main/scala/cwl/WorkflowStep.scala +++ /dev/null @@ -1,537 +0,0 @@ -package cwl - -import cats.Monoid -import cats.data.NonEmptyList -import cats.data.Validated._ -import cats.syntax.either._ -import cats.syntax.foldable._ -import cats.syntax.monoid._ -import cats.syntax.traverse._ -import cats.syntax.validated._ -import cats.instances.list._ -import common.Checked -import common.validation.Checked._ -import common.validation.ErrorOr.ErrorOr -import cwl.ScatterLogic.ScatterVariablesPoly -import cwl.ScatterMethod._ -import cwl.WorkflowStep.{WorkflowStepInputFold, _} -import cwl.WorkflowStepInput._ -import cwl.command.ParentName -import shapeless.{:+:, CNil, _} -import wom.callable.Callable -import wom.callable.Callable._ -import wom.graph.CallNode._ -import wom.graph.GraphNodePort.{GraphNodeOutputPort, OutputPort} -import wom.graph._ -import wom.graph.expression.ExpressionNode -import wom.types.WomType -import wom.values.WomValue -import wom.graph.{FullyQualifiedName => WomFullyQualifiedName} - -/** - * An individual job to run. - * - * @see CWL Spec | Workflow Step - * @param run Purposefully not defaulted as it's required in the specification and it is unreasonable to not have something to run. - */ -case class WorkflowStep( - id: String, - in: Array[WorkflowStepInput] = Array.empty, - out: WorkflowStepOutputType, - run: Run, - requirements: Option[Array[Requirement]] = None, - hints: Option[Array[Hint]] = None, - label: Option[String] = None, - doc: Option[String] = None, - scatter: ScatterVariables = None, - scatterMethod: Option[ScatterMethod] = None) { - - run.select[Workflow].foreach(_.parentWorkflowStep = Option(this)) - run.select[CommandLineTool].foreach(_.parentWorkflowStep = Option(this)) - run.select[ExpressionTool].foreach(_.parentWorkflowStep = Option(this)) - - // We're scattering if scatter is defined, and if it's a list of variables the list needs to be non empty - private val isScattered: Boolean = scatter exists { _.select[Array[String]].forall(_.nonEmpty) } - - // Circe can't create bidirectional links between workflows and workflow steps so this ugly var is here to link - // back to the parent workflow. This is needed to navigate upward for finding requirements in the containment - // hierarchy. There is always a workflow containing a workflow step so this is not an `Option`. - private[cwl] var parentWorkflow: Workflow = _ - - lazy val allRequirements = RequirementsAndHints(requirements.toList.flatten ++ parentWorkflow.allRequirements.list) - - lazy val womFqn: WomFullyQualifiedName = { - implicit val parentName = parentWorkflow.explicitWorkflowName - val localFqn = cwl.FullyQualifiedName.maybeApply(id).map(_.id).getOrElse(id) - parentWorkflow.womFqn.map(_.combine(localFqn)).getOrElse(wom.graph.FullyQualifiedName(localFqn)) - } - - lazy val allHints: List[Requirement] = { - // Just ignore any hint that isn't a Requirement. - val requirementHints = hints.toList.flatten.flatMap { _.select[Requirement] } - requirementHints ++ parentWorkflow.allHints - } - - // If the step is being scattered over, apply the necessary transformation to get the final output array type. - lazy val scatterTypeFunction: WomType => WomType = scatter.map(_.fold(ScatterVariablesPoly)) match { - case Some(Nil) => identity[WomType] - case Some(nonEmpty) => ScatterLogic.scatterGatherPortTypeFunction(scatterMethod, NonEmptyList.fromListUnsafe(nonEmpty)) - case _ => identity[WomType] - } - - def typedOutputs: WomTypeMap = { - implicit val parentName = ParentName(id) - // Find the type of the outputs of the run section - val runOutputTypes = run.fold(RunOutputsToTypeMap).apply(allRequirements.schemaDefRequirement) - .map({ - case (runOutputId, womType) => cwl.FullyQualifiedName(runOutputId).id -> womType - }) - // Use them to find get the final type of the workflow outputs, and only the workflow outputs - out.map({ stepOutput => - val stepOutputValue = stepOutput.select[WorkflowStepOutput].map(_.id).getOrElse(stepOutput.select[String].get) - val stepOutputId = cwl.FullyQualifiedName(stepOutputValue) - stepOutputValue -> scatterTypeFunction(runOutputTypes(stepOutputId.id)) - }).toMap - } - - def fileName: Option[String] = run.select[String] - - /** - * Generates all GraphNodes necessary to represent call nodes and input nodes - * Recursive because dependencies are discovered as we iterate through inputs and corresponding - * upstream nodes need to be generated on the fly. - * - * Example: - * - * CWL: - * inputs: - * 
-id: workflow_input_A
 - * type: string[] - * -id: workflow_input_B
 - * type: string[] - * 
-id: workflow_input_C
 - * type: string

 - * steps:
 - * -id: echo - * run: echo.cwl - * scatter: input0 - * in: - * -id:input0 - * source: - * - "#workflow_input_A" - * - "#workflow_input_B"
 - * valueFrom:"bonjour" - * - * -id: input1 - * 
 source: "#workflow_input_C" - * valueFrom:"$(inputs.input0)" - * - * WOM Diagram: - * - * +------------+ +------------+ +------------+ - * | WF Input A | | WF Input B | | WF Input C | - * +--------+---+ +-+------------+ +------+-----+ - * | | | - * | | | - * +-----v-----v+ +------v-----+ - * | StepInput0 | | StepInput1 | - * | MergeLogic | | MergeLogic | - * +-----+------+ +----+-------+ - * | | - * +----------------------------------------------+ - * | | | | - * | +------v----------+ +---v--+ | - * | | ScatterVariable +------------+ | OGIN | | - * | +-----+-----------+ | +--+-+-+ | - * | | | | | | - * | +-----v----+ | | | | - * | |StepInput0<------------------------+ | | - * | |Expression| | | | - * | +----+-----+ +--v------v+ | - * | | |StepInput1| | - * | | |Expression| | - * | | +------+---+ | - * | | +-----------+ | | - * | +----> Call Node <------------+ | - * | +-----------+ | - * | Scatter Node | - * +----------------------------------------------+ - * - * MergeNode: If the step input has one or more sources, a merge node will be created and responsible for merging - * those input sources together. It will NOT evaluate the valueFrom field of the input. - * - * ScatterVariableNode: If the step input is being scattered over, a scatter variable node will be created and will - * act as a proxy inside the scatter graph for the shards of the scatter. It depends on an upstream merge node outputing an array - * and will provide at runtime shard values for other nodes of the scatter graph. - * - * OGIN: If the step has at least one input being scattered over, there will be a scatter node created. - * For inputs that are NOT being scattered over but still have one or more input sources (and hence a merge node), an OGIN - * will be created to act as a proxy to the merge node outside the scatter graph. - * - * ExpressionNode: If an input has a valueFrom field, an expression node will be created to evaluate the expression. - * An important fact to note is that the expression needs access to all other input values - * AFTER their source, default value and shard number has been determined but - * BEFORE their (potential) valueFrom is evaluated (see http://www.commonwl.org/v1.0/Workflow.html#WorkflowStepInput) - * This is why on the above diagram, StepInput0Expression depends on the OGIN, and StepInput1Expression depends on the scatter variable. - */ - def callWithInputs(typeMap: WomTypeMap, - workflow: Workflow, - knownNodes: Set[GraphNode], - workflowInputs: Map[String, GraphNodeOutputPort], - validator: RequirementsValidator, - expressionLib: ExpressionLib): Checked[Set[GraphNode]] = { - - implicit val parentName = workflow.explicitWorkflowName - - val scatterLookupSet = - scatter.toList. - flatMap(_.fold(StringOrStringArrayToStringList)). - map(id => cwl.FullyQualifiedName(id).id) - - def isStepScattered(workflowStepInputId: String) = scatterLookupSet.contains(workflowStepInputId) - - val unqualifiedStepId: WomIdentifier = { - cwl.FullyQualifiedName.maybeApply(id).map({ fqn => - WomIdentifier(LocalName(fqn.id), womFqn) - }).getOrElse(WomIdentifier(id)) - } - - def typedRunInputs: Map[String, Option[MyriadInputType]] = run.fold(RunToInputTypeMap).apply(parentName) - - def allIdentifiersRecursively(nodes: Set[GraphNode]): Set[WomIdentifier] = nodes.flatMap({ - case w: WorkflowCallNode=> Set(w.identifier) - case c: CommandCallNode => Set(c.identifier) - case e: ExpressionCallNode => Set(e.identifier) - // When a node a call node is being scattered over, it is wrapped inside a scatter node. We still don't want to - // duplicate it though so look inside scatter nodes to see if it's there. - case scatter: ScatterNode => allIdentifiersRecursively(scatter.innerGraph.nodes) - case _ => Set.empty[WomIdentifier] - }) - - // To avoid duplicating nodes, return immediately if we've already covered this node - val haveWeSeenThisStep: Boolean = allIdentifiersRecursively(knownNodes).contains(unqualifiedStepId) - - if (haveWeSeenThisStep) Right(knownNodes) - else { - val callable: Checked[Callable] = run match { - case Run.CommandLineTool(clt) => clt.buildTaskDefinition(validator, expressionLib) - case Run.Workflow(wf) => wf.womDefinition(validator, expressionLib) - case Run.ExpressionTool(et) => et.buildTaskDefinition(validator, expressionLib) - case oh => throw new Exception(s"Programmer Error! Unexpected case match: $oh") - } - - val callNodeBuilder = new CallNode.CallNodeBuilder() - - /* - * Method used to fold over the list of inputs declared by this step. - * Note that because we work on saladed CWL, all ids are fully qualified at this point (e.g: file:///path/to/file/three_step.cwl#cgrep/pattern - * The goal of this method is two fold (pardon the pun): - * 1) link each input of the step to an output port (which at this point can be from a different step or from a workflow input) - * 2) accumulate the nodes created along the way to achieve 1) - */ - def foldStepInput(currentFold: Checked[WorkflowStepInputFold], workflowStepInput: WorkflowStepInput): Checked[WorkflowStepInputFold] = currentFold flatMap { - fold => - /* - * Try to find in the given set an output port named stepOutputId in a call node named stepId - * This is useful when we've determined that the input points to an output of a different step and we want - * to get the corresponding output port. - */ - def findThisInputInSet(set: Set[GraphNode], stepId: String, stepOutputId: String): Checked[OutputPort] = { - for { - // We only care for outputPorts of call nodes or scatter nodes - call <- set.collectFirst { - case callNode: CallNode if callNode.localName == stepId => callNode - case scatterNode: ScatterNode if scatterNode.innerGraph.calls.exists(_.localName == stepId) => scatterNode - }. - toRight(NonEmptyList.one(s"stepId $stepId not found in known Nodes $set")) - output <- call.outputPorts.find(_.internalName == stepOutputId). - toRight(NonEmptyList.one(s"step output id $stepOutputId not found in ${call.outputPorts}")) - } yield output - } - - /* - * Build a wom node for the given step and return the newly created nodes - * This is useful when we've determined that the input belongs to an upstream step that we haven't covered yet - */ - def buildUpstreamNodes(upstreamStepId: String, accumulatedNodes: Set[GraphNode]): Checked[Set[GraphNode]] = - // Find the step corresponding to this upstreamStepId in the set of all the steps of this workflow - for { - step <- workflow.steps.find { step => cwl.FullyQualifiedName(step.id).id == upstreamStepId }. - toRight(NonEmptyList.one(s"no step of id $upstreamStepId found in ${workflow.steps.map(_.id).toList}")) - call <- step.callWithInputs(typeMap, workflow, accumulatedNodes, workflowInputs, validator, expressionLib) - } yield call - - def fromWorkflowInput(inputName: String): Checked[Map[String, OutputPort]] = { - // Try to find it in the workflow inputs map, if we can't it's an error - workflowInputs.collectFirst { - case (inputId, port) if inputName == inputId => Map(inputId -> port).asRight[NonEmptyList[String]] - } getOrElse s"Can't find workflow input for $inputName".invalidNelCheck[Map[String, OutputPort]] - } - - def fromStepOutput(stepId: String, stepOutputId: String, accumulatedNodes: Set[GraphNode]): Checked[(Map[String, OutputPort], Set[GraphNode])] = { - // First check if we've already built the WOM node for this step, and if so return the associated output port - findThisInputInSet(accumulatedNodes, stepId, stepOutputId).map(outputPort => (Map(s"$stepId/$stepOutputId" -> outputPort), accumulatedNodes)) - .orElse { - // Otherwise build the upstream nodes and look again in those newly created nodes - for { - newNodes <- buildUpstreamNodes(stepId, accumulatedNodes) - sourceMappings <- findThisInputInSet(newNodes, stepId, stepOutputId).map(outputPort => Map(s"$stepId/$stepOutputId" -> outputPort)) - } yield (sourceMappings, newNodes ++ accumulatedNodes) - } - } - - lazy val workflowStepInputId = cwl.FullyQualifiedName(workflowStepInput.id).id - - def updateFold(sourceMappings: Map[String, OutputPort], newNodes: Set[GraphNode]): Checked[WorkflowStepInputFold] = { - val typeExpectedByRunInput: Option[cwl.MyriadInputType] = typedRunInputs.get(workflowStepInputId).flatten - - val isThisStepScattered = isStepScattered(workflowStepInputId) - - workflowStepInput.toMergeNode(sourceMappings, expressionLib, typeExpectedByRunInput, isThisStepScattered, allRequirements.schemaDefRequirement) match { - // If the input needs a merge node, build it and add it to the input fold - case Some(mergeNode) => - mergeNode.toEither.map({ node => - fold |+| WorkflowStepInputFold( - mergeNodes = Map(workflowStepInput -> node), - generatedNodes = newNodes - ) - }) - case None => (fold |+| WorkflowStepInputFold(generatedNodes = newNodes)).validNelCheck - } - } - - /* - * We intend to validate that all of these sources point to a WOM Outputport that we know about. - * - * If we don't know about them, we find upstream nodes and build them (see "buildUpstreamNodes"). - */ - val baseCase = (Map.empty[String, OutputPort], fold.generatedNodes).asRight[NonEmptyList[String]] - val inputMappingsAndGraphNodes: Checked[(Map[String, OutputPort], Set[GraphNode])] = - workflowStepInput.sources.foldLeft(baseCase) { - case (Right((sourceMappings, graphNodes)), inputSource) => - /* - * Parse the inputSource (what this input is pointing to) - * 2 cases: - * - points to a workflow input - * - points to an upstream step - */ - cwl.FullyQualifiedName(inputSource) match { - // The source points to a workflow input, which means it should be in the workflowInputs map - case FileAndId(_, _, inputId) => fromWorkflowInput(inputId).map(newMap => (sourceMappings ++ newMap, graphNodes)) - // The source points to an output from a different step - case FileStepAndId(_, _, stepId, stepOutputId) => fromStepOutput(stepId, stepOutputId, graphNodes).map({ case (newMap, newNodes) => (sourceMappings ++ newMap, newNodes) }) - } - case (other, _) => other - } - - inputMappingsAndGraphNodes.flatMap((updateFold _).tupled) - } - - /* - * Folds over input definitions and build an InputDefinitionFold - */ - def foldInputDefinition(pointerNode: Map[String, GraphNodeWithSingleOutputPort]) - (inputDefinition: InputDefinition): ErrorOr[InputDefinitionFold] = { - inputDefinition match { - case _ if pointerNode.contains(inputDefinition.name) => - val expressionNode = pointerNode(inputDefinition.name) - InputDefinitionFold( - mappings = List(inputDefinition -> expressionNode.inputDefinitionPointer), - callInputPorts = Set(callNodeBuilder.makeInputPort(inputDefinition, expressionNode.singleOutputPort)) - ).validNel - - // No expression node mapping, use the default - case withDefault @ OverridableInputDefinitionWithDefault(_, _, expression, _, _) => - InputDefinitionFold( - mappings = List(withDefault -> Coproduct[InputDefinitionPointer](expression)) - ).validNel - - // Required input without default value and without mapping, this is a validation error - case RequiredInputDefinition(requiredName, _, _, _) => - s"Input ${requiredName.value} is required and is not bound to any value".invalidNel - - // Optional input without mapping, defaults to empty value - case optional: OptionalInputDefinition => - InputDefinitionFold( - mappings = List(optional -> Coproduct[InputDefinitionPointer](optional.womType.none: WomValue)) - ).validNel - case oh => throw new Exception(s"Programmer Error! Unexpected case match: $oh") - } - } - - /* - If the step is being scattered over, then merge nodes can't directly be referenced because they will be outside the scatter graph. - For inputs that are being scattered over, a scatter variable has already been created, but for the others we need - an OGIN to link the merge node to the inner scatter graph. - */ - def buildOGINs(mergeNodes: Map[WorkflowStepInput, ExpressionNode], - scatterVariables: Map[WorkflowStepInput, ScatterVariableNode]): Map[WorkflowStepInput, OuterGraphInputNode] = if (isScattered) { - mergeNodes - .collect({ - case (input, mergeNode) if !scatterVariables.contains(input) => - val ogin = OuterGraphInputNode( - WomIdentifier(input.parsedId).combine("OGIN"), - mergeNode.singleOutputPort, - preserveScatterIndex = false - ) - input -> ogin - }) - } else Map.empty - - /* - * For inputs that have a valueFrom field, create an ExpressionNode responsible for evaluating the expression. - * Note that this expression might need access to the other input values, so make each expression node depend on all other - * inputs. - */ - def buildStepInputValueFromNodes(sharedInputNodes: Map[WorkflowStepInput, GraphNodeWithSingleOutputPort]): Checked[Map[String, ExpressionNode]] = { - // Add new information to the typeMap from the shard input nodes. - lazy val updatedTypeMap = sharedInputNodes.map({ - // If the input node is a scatter variable, make sure the type is the item type, not the array type, as the expression node - // will operate on shards not on the whole scattered array. - case (stepInput, scatter: ScatterVariableNode) => stepInput.parsedId -> scatter.womType - case (stepInput, node) => stepInput.parsedId -> node.singleOutputPort.womType - }) ++ typeMap - - // Go over each step input and create an expression node for those which have a valueFrom - in.toList.collect({ - case stepInput @ WorkflowStepInput(_, _, _, _, Some(valueFrom)) => - // Transform the shared inputs map into a usable map to create the expression. - lazy val sharedInputMap: Map[String, OutputPort] = sharedInputNodes.map({ - case (siblingStepInput, graphNode) => siblingStepInput.parsedId -> graphNode.singleOutputPort - }) - val typeExpectedByRunInput: Option[cwl.MyriadInputType] = typedRunInputs.get(stepInput.parsedId).flatten - val isThisStepScattered = isStepScattered(stepInput.parsedId) - - stepInput.toExpressionNode(valueFrom, typeExpectedByRunInput, isThisStepScattered, sharedInputMap, updatedTypeMap, expressionLib, allRequirements.schemaDefRequirement).map(stepInput.parsedId -> _) - }) - .sequence[ErrorOr, (String, ExpressionNode)] - .toEither - .map(_.toMap) - } - - //inputs base case consist of the nodes we already know about - val baseCase = WorkflowStepInputFold(generatedNodes = knownNodes).asRight[NonEmptyList[String]] - - // WorkflowStepInputFold contains the mappings from step input to ExpressionNode as well as all created nodes - val stepInputFoldCheck: Checked[WorkflowStepInputFold] = in.foldLeft(baseCase)(foldStepInput) - - /* - * This (big) flatMap builds nodes from top to bottom in the diagram above. - * If necessary, the scatter node is built last as it wraps some of the other nodes. - */ - for { - /* ************************************ */ - /* ************ Merge Nodes *********** */ - /* ************************************ */ - // Build merge nodes and recursively generates other call nodes that we haven't seen so far - stepInputFold <- stepInputFoldCheck - // Extract the merge nodes from the fold - mergeNodes = stepInputFold.mergeNodes - - /* ************************************ */ - /* ****** Scatter Variable Nodes ****** */ - /* ************************************ */ - scatterVariableNodes <- ScatterLogic.buildScatterVariableNodes(scatter, mergeNodes, unqualifiedStepId.localName.value) - - /* ************************************ */ - /* *************** OGINS ************** */ - /* ************************************ */ - ogins = buildOGINs(mergeNodes, scatterVariableNodes) - - /* ************************************ */ - /* ********* Expression Nodes ********* */ - /* ************************************ */ - // Aggregate the generated nodes so far. This map will be used to generate expression nodes, so the order of aggregation matters: - // scatter variables and ogins take precedence over merge nodes (see diagram) - aggregatedMapForValueFromNodes = mergeNodes ++ scatterVariableNodes ++ ogins - // Build expression nodes for inputs that have a valueFrom field - stepInputValueFromNodes <- buildStepInputValueFromNodes(aggregatedMapForValueFromNodes) - - /* ************************************ */ - /* ************* Call Node ************ */ - /* ************************************ */ - // Get the callable object for this step - checkedCallable <- callable - // Aggregate again by adding generated expression nodes. Again order matters here, expression nodes override other nodes. - aggregatedMapForInputDefinitions = aggregatedMapForValueFromNodes.asIdentifierMap ++ stepInputValueFromNodes - // Assign each of the callable's input definition to an output port from the pointer map - inputDefinitionFold <- checkedCallable.inputs.foldMap(foldInputDefinition(aggregatedMapForInputDefinitions)).toEither - // Build the call node - callAndNodes = callNodeBuilder.build(unqualifiedStepId, checkedCallable, inputDefinitionFold, Set.empty, None) - // Depending on whether the step is being scattered, invoke the scatter node builder or not - - /* ************************************ */ - /* ************ Scatter Node ********** */ - /* ************************************ */ - scatterNodeOrExposedNodes <- if (isScattered) { - ScatterLogic.buildScatterNode( - callAndNodes, - NonEmptyList.fromListUnsafe(scatterVariableNodes.values.toList), - ogins.values.toSet, - stepInputValueFromNodes.values.toSet, - scatterMethod).map(Set(_)) - } else { - // If there's no scatter node then we need to return the expression nodes and the call node explicitly - // as they won't be contained in the scatter inner graph - (stepInputValueFromNodes.values.toSet + callAndNodes.node).validNelCheck - } - - /* - * Return all the nodes that need to be made available to the workflow graph: - * knownNodes: this method is used to fold over steps so we don't want to forget to accumulate known nodes - * mergeNodes: they're always outside of the scatter so always return them - * generatedNodes: nodes generated recursively to build this node - * scatterNodeOrExposedNodes: see explanation above - */ - allNodes = knownNodes ++ mergeNodes.values.toSet ++ stepInputFold.generatedNodes ++ scatterNodeOrExposedNodes - } yield allNodes - } - } -} - -/** - * @see WorkflowstepOutput - */ -case class WorkflowStepOutput(id: String) - -object WorkflowStep { - - // A monoid can't be derived automatically for this class because it contains a Map[String, ExpressionNode], - // and there's no monoid defined over ExpressionNode - implicit val workflowStepInputFoldMonoid: Monoid[WorkflowStepInputFold] = new Monoid[WorkflowStepInputFold] { - override def empty: WorkflowStepInputFold = WorkflowStepInputFold() - override def combine(x: WorkflowStepInputFold, y: WorkflowStepInputFold): WorkflowStepInputFold = { - WorkflowStepInputFold( - mergeNodes = x.mergeNodes ++ y.mergeNodes, - generatedNodes = x.generatedNodes ++ y.generatedNodes - ) - } - } - - private [cwl] case class WorkflowStepInputFold(mergeNodes: Map[WorkflowStepInput, ExpressionNode] = Map.empty, - generatedNodes: Set[GraphNode] = Set.empty) - - /** - * Maps input variable (to be scattered over) to their scatter variable node - */ - type ScatterMappings = Map[ExpressionNode, ScatterVariableNode] - - val emptyOutputs: WorkflowStepOutputType = Array.empty - - type Run = - String :+: - CommandLineTool :+: - ExpressionTool :+: - Workflow :+: - CNil - - object Run { - object String { def unapply(run: Run): Option[String] = run.select[String] } - object Workflow { def unapply(run: Run): Option[Workflow] = run.select[Workflow] } - object CommandLineTool { def unapply(run: Run): Option[CommandLineTool] = run.select[CommandLineTool] } - object ExpressionTool { def unapply(run: Run): Option[ExpressionTool] = run.select[ExpressionTool] } - } - - type WorkflowStepOutputInnerType = String :+: WorkflowStepOutput :+: CNil - type WorkflowStepOutputType = Array[WorkflowStepOutputInnerType] -} diff --git a/cwl/src/main/scala/cwl/WorkflowStepInput.scala b/cwl/src/main/scala/cwl/WorkflowStepInput.scala deleted file mode 100644 index a3d028dbdd6..00000000000 --- a/cwl/src/main/scala/cwl/WorkflowStepInput.scala +++ /dev/null @@ -1,155 +0,0 @@ -package cwl - -import cats.data.NonEmptyList -import cats.syntax.either._ -import common.Checked -import common.validation.Checked._ -import common.validation.ErrorOr.ErrorOr -import cwl.InputParameter.DefaultToWomValuePoly -import cwl.LinkMergeMethod.LinkMergeMethod -import cwl.WorkflowStepInput.InputSource -import cwl.command.ParentName -import shapeless.{:+:, CNil} -import wom.expression.ValueAsAnExpression -import wom.graph.GraphNodePort.OutputPort -import wom.graph.WomIdentifier -import wom.graph.expression.{AnonymousExpressionNode, ExposedExpressionNode, ExpressionNode, PlainAnonymousExpressionNode} -import wom.types._ - -case class WorkflowStepInput( - id: String, - source: Option[InputSource] = None, - linkMerge: Option[LinkMergeMethod] = None, - default: Option[CwlAny] = None, - valueFrom: Option[StringOrExpression] = None) { - - def parsedId(implicit parentName: ParentName) = FullyQualifiedName(id).id - - def toExpressionNode(valueFromExpression: StringOrExpression, - runInputExpectedType: Option[cwl.MyriadInputType], - isScattered: Boolean, - sourceMappings:Map[String, OutputPort], - outputTypeMap: Map[String, WomType], - expressionLib: ExpressionLib, - schemaDefRequirement: SchemaDefRequirement - )(implicit parentName: ParentName): ErrorOr[ExpressionNode] = { - val inputs = sourceMappings.keySet - val upstreamMergeType = outputTypeMap.get(parsedId) - - (for { - // we may have several sources, we make sure to have a type common to all of them. - // In the case where there's no input source, we currently wrap the valueFrom value in a WomString (see WorkflowStepInputExpression) - inputType <- WorkflowStepInput.determineValueFromType(upstreamMergeType, runInputExpectedType, isScattered, schemaDefRequirement) - womExpression = WorkflowStepInputExpression(parsedId, valueFromExpression, inputType, inputs, expressionLib) - identifier = WomIdentifier(id).combine("expression") - ret <- ExposedExpressionNode.fromInputMapping(identifier, womExpression, inputType, sourceMappings).toEither - } yield ret).toValidated - } - - /** - * - * @param sourceMappings The outputports to which this source refers - * @param matchingRunInputType This input matches an input declared in the workflowstep's "run". This is that step's declared type - * @return - */ - def toMergeNode(sourceMappings: Map[String, OutputPort], - expressionLib: ExpressionLib, - matchingRunInputType: Option[MyriadInputType], - isScattered: Boolean, - schemaDefRequirement: SchemaDefRequirement - ): Option[ErrorOr[ExpressionNode]] = { - - val identifier = WomIdentifier(id).combine("merge") - val mapType = sourceMappings.map({ case (k, v) => k -> v.womType }) - - val maybeMatchingRunInputWomType: Option[WomType] = matchingRunInputType.map(_.fold(MyriadInputTypeToWomType).apply(schemaDefRequirement)) - - def makeNode(head: (String, OutputPort), tail: List[(String, OutputPort)]) = for { - inputType <- determineMergeType(mapType, maybeMatchingRunInputWomType) - womExpression = WorkflowStepInputMergeExpression(this, inputType, NonEmptyList.of(head, tail: _*), expressionLib) - node <- AnonymousExpressionNode.fromInputMapping(identifier, womExpression, sourceMappings, PlainAnonymousExpressionNode.apply).toEither - } yield node - - val matchingRunInputWomType: WomType = maybeMatchingRunInputWomType.getOrElse(WomAnyType) - lazy val defaultValue = default.map { _.fold(DefaultToWomValuePoly).apply(matchingRunInputWomType) } - - sourceMappings.toList match { - case head :: tail => Option(makeNode(head, tail).toValidated) - case Nil => - defaultValue map { _ map { d => - PlainAnonymousExpressionNode(identifier, ValueAsAnExpression(d), matchingRunInputWomType, Map.empty) - } } - } - } - - def determineMergeType(sources: Map[String, WomType], expectedTypeAsWom: Option[WomType]): Checked[WomType] = { - WorkflowStepInput.determineMergeType(sources, linkMerge, expectedTypeAsWom, default.isDefined) - } - - lazy val sources: List[String] = source.toList.flatMap(_.fold(StringOrStringArrayToStringList)) - - lazy val effectiveLinkMerge: LinkMergeMethod = linkMerge.getOrElse(LinkMergeMethod.MergeNested) -} - -object WorkflowStepInput { - type InputSource = String :+: Array[String] :+: CNil - - implicit class EnhancedStepInputMap[A](val map: Map[WorkflowStepInput, A]) extends AnyVal { - def asIdentifierMap(implicit parentName: ParentName): Map[String, A] = { - map.map({ case (stepInput, value) => stepInput.parsedId -> value }) - } - } - - def determineValueFromType(mergedSourcesType: Option[WomType], - expectedType: Option[MyriadInputType], - isScattered: Boolean, - schemaDefRequirement: SchemaDefRequirement): Checked[WomType] = { - val expectedTypeAsWom: Option[WomType] = expectedType.map(_.fold(MyriadInputTypeToWomType).apply(schemaDefRequirement)) - - expectedTypeAsWom.getOrElse(WomStringType).asRight - } - - def determineMergeType(sources: Map[String, WomType], - linkMerge: Option[LinkMergeMethod], - expectedTypeAsWom: Option[WomType], - hasDefault: Boolean): Checked[WomType] = { - - (sources.toList, expectedTypeAsWom, linkMerge) match { - // If there is a single source and no explicit merge method, use the type of the source (unpacked if it's optional - // and there's a default since the default would be used if there's no supplied value). - case (List((_, WomOptionalType(sourceType))), _, None) if hasDefault => sourceType.asRight - case (List((_, opt @ WomOptionalType(_))), _, None) => opt.asRight - case (List((_, sourceType)), _, None) => sourceType.asRight - // If there is a single source and merge nested is specified, wrap it into an array - case (List((_, sourceType)), _, Some(LinkMergeMethod.MergeNested)) => WomArrayType(sourceType).asRight - // If there are multiple sources and either no merge method or merge nested, find the closest common type. - // TODO: This is technically not enough, cwltool supports sources with totally different types, creating an array with multiple types - // Maybe WomCoproduct can help - case (_, _, Some(LinkMergeMethod.MergeNested) | None) => WomArrayType(WomType.homogeneousTypeFromTypes(sources.values)).asRight - - //If sink parameter is an array and merge_flattened is used, must validate input & output types are equivalent before proceeding - case (_, Some(arrayType @ WomArrayType(itemType)), Some(LinkMergeMethod.MergeFlattened)) if typesToItemMatch(sources.values, itemType) => arrayType.asRight - // If sink parameter is not an array and merge flattened is used, validate that the sources types matche the sink type - case (_, Some(targetType), Some(LinkMergeMethod.MergeFlattened)) if typesToItemMatch(sources.values, targetType) => WomArrayType(targetType).asRight - // If the types are not compatible, fail - case (_, Some(targetType), Some(LinkMergeMethod.MergeFlattened)) => - s"could not verify that types $sources and the items type of the run's InputArraySchema $targetType were compatible".invalidNelCheck - - //We don't have type information from the run input so we gather up the sources and try to determine a common type amongst them. - case _ => WomType.homogeneousTypeFromTypes(sources.values).asRight - } - } - - def typesToItemMatch(lst: Iterable[WomType], target: WomType): Boolean = { - val effectiveInputType = WomType.homogeneousTypeFromTypes(lst) - - typeToItemMatch(effectiveInputType, target) - } - - def typeToItemMatch(upstream: WomType, downstream: WomType): Boolean = { - upstream match { - case WomType.RecursiveType(innerType) => typeToItemMatch(innerType, downstream) - case other => other == downstream - } - } -} diff --git a/cwl/src/main/scala/cwl/WorkflowStepInputExpression.scala b/cwl/src/main/scala/cwl/WorkflowStepInputExpression.scala deleted file mode 100644 index 476d7f44ef5..00000000000 --- a/cwl/src/main/scala/cwl/WorkflowStepInputExpression.scala +++ /dev/null @@ -1,51 +0,0 @@ -package cwl - -import cats.syntax.validated._ -import wom.expression.{FileEvaluation, IoFunctionSet} -import wom.types._ -import wom.values._ - -final case class WorkflowStepInputExpression(inputName: String, - valueFrom: StringOrExpression, - override val cwlExpressionType: WomType, - inputs: Set[String], - override val expressionLib: ExpressionLib) extends CwlWomExpression { - - override def sourceString = inputName - - override def evaluateValue(inputValues: Map[String, WomValue], ioFunctionSet: IoFunctionSet) = { - valueFrom match { - // If valueFrom is a constant string value, use this as the value for this input parameter. - // TODO: need to handle case where this is a parameter reference, it currently looks like a String to us! - case StringOrExpression.String(value) => WomString(value).validNel - - /* - * If valueFrom is a parameter reference or expression, it must be evaluated to yield the actual value to be assiged to the input field. - * - * The self value of in the parameter reference or expression must be the value of the parameter(s) specified in the source field, - * or null if there is no source field. - * - * The value of inputs in the parameter reference or expression must be the input object to the workflow step after - * assigning the source values, applying default, and then scattering. The order of evaluating valueFrom among step - * input parameters is undefined and the result of evaluating valueFrom on a parameter must not be visible to - * evaluation of valueFrom on other parameters. - */ - case StringOrExpression.Expression(expression) => - //used to determine the value of "self" as expected by CWL Spec - def selfValue = inputValues.get(inputName) match { - case Some(value) => value - case None => WomOptionalValue(WomNothingType, None) - } - - val parameterContext = ParameterContext(ioFunctionSet, expressionLib, inputValues, selfValue) - - expression.fold(EvaluateExpression).apply(parameterContext) - case oh => throw new Exception(s"Programmer Error! Unexpected case match: $oh") - } - } - - //this is input, so not producing any output files - override def evaluateFiles(inputTypes: Map[String, WomValue], ioFunctionSet: IoFunctionSet, coerceTo: WomType) = - Set.empty[FileEvaluation].validNel -} - diff --git a/cwl/src/main/scala/cwl/WorkflowStepInputMergeExpression.scala b/cwl/src/main/scala/cwl/WorkflowStepInputMergeExpression.scala deleted file mode 100644 index bc7d4f43d75..00000000000 --- a/cwl/src/main/scala/cwl/WorkflowStepInputMergeExpression.scala +++ /dev/null @@ -1,84 +0,0 @@ -package cwl - -import cats.data.NonEmptyList -import cats.syntax.either._ -import cats.syntax.option._ -import cats.syntax.traverse._ -import cats.syntax.validated._ -import cats.instances.list._ -import common.Checked -import common.validation.ErrorOr.ErrorOr -import cwl.InputParameter.DefaultToWomValuePoly -import wom.expression.{FileEvaluation, IoFunctionSet} -import wom.graph.GraphNodePort.OutputPort -import wom.types.WomType -import wom.values.{WomArray, WomFile, WomOptionalValue, WomValue} - -final case class WorkflowStepInputMergeExpression(input: WorkflowStepInput, - cwlExpressionType: WomType, - stepInputMappingHead: NonEmptyList[(String, OutputPort)], - override val expressionLib: ExpressionLib) extends CwlWomExpression { - - private val allStepInputMappings = stepInputMappingHead.toList - private val allStepInputSources = allStepInputMappings.map(_._1) - - override def sourceString: String = s"${input.id}-Merge-Expression" - override def inputs: Set[String] = allStepInputSources.toSet - - override def evaluateValue(inputValues: Map[String, WomValue], ioFunctionSet: IoFunctionSet): ErrorOr[WomValue] = { - def lookupValue(key: String): ErrorOr[WomValue] = - inputValues. - get(key). - toValidNel(s"source value $key not found in input values ${inputValues.mkString("\n")} and no default value provided. Graph Inputs were ${allStepInputSources.mkString("\n")}") - - def validateSources(sources: List[String]): ErrorOr[List[WomValue]] = - sources. - traverse(lookupValue) - - def isEmptyOptionalValue(womValue: WomValue): Boolean = womValue match { - case WomOptionalValue(_, None) => true - case _ => false - } - - (allStepInputSources, input.effectiveLinkMerge, input.default) match { - // When we have a single source but no value was provided for it and there's a default. - case (List(source), LinkMergeMethod.MergeNested, Some(default)) if isEmptyOptionalValue(inputValues(source)) => - default.fold(DefaultToWomValuePoly).apply(cwlExpressionType) - - case (List(source), LinkMergeMethod.MergeNested, _) => lookupValue(source) - - //When we have several sources, validate they are all present and provide them as a nested array - case (sources, LinkMergeMethod.MergeNested, _) => validateSources(sources).map(WomArray.apply) - - case (sources, LinkMergeMethod.MergeFlattened, _) => - val validatedSourceValues: Checked[List[WomValue]] = - validateSources(sources).toEither - - def flatten: WomValue => List[WomValue] = { - case WomArray(_, value) => value.toList - case WomOptionalValue(_, Some(value)) => flatten(value) - case other => List(other) - } - - //This is the meat of "merge_flattened," where we find arrays and concatenate them to form one array - val flattenedValidatedSourceValues: Checked[List[WomValue]] = validatedSourceValues.map(_.flatMap(flatten)) - - flattenedValidatedSourceValues.map(list => WomArray(list)).toValidated - - case (List(id), _, _) => lookupValue(id) - case oh => throw new Exception(s"Programmer Error! Unexpected case match: $oh") - } - } - - override def evaluateFiles(inputTypes: Map[String, WomValue], ioFunctionSet: IoFunctionSet, coerceTo: WomType): ErrorOr[Set[FileEvaluation]] = { - if (allStepInputMappings.size > 1) { - // TODO add MultipleInputFeatureRequirement logic in here - "MultipleInputFeatureRequirement not supported yet".invalidNel - } else { - val (inputName, _) = allStepInputMappings.head - inputTypes(inputName).collectAsSeq({ - case file: WomFile => file - }).toSet[WomFile].map(FileEvaluation.requiredFile).validNel - } - } -} diff --git a/cwl/src/main/scala/cwl/command/ParentName.scala b/cwl/src/main/scala/cwl/command/ParentName.scala deleted file mode 100644 index 6e34096d4f3..00000000000 --- a/cwl/src/main/scala/cwl/command/ParentName.scala +++ /dev/null @@ -1,10 +0,0 @@ -package cwl.command - -object ParentName { - def empty: ParentName = ParentName(None) - def apply(id: String): ParentName = ParentName(id.split("#").tail.headOption) -} - -case class ParentName(value: Option[String]) { - def stripParent(in: String) = value.map(v => in.stripPrefix(s"$v/")).getOrElse(in) -} diff --git a/cwl/src/main/scala/cwl/command/StringCommandPart.scala b/cwl/src/main/scala/cwl/command/StringCommandPart.scala deleted file mode 100644 index 9ec668231fb..00000000000 --- a/cwl/src/main/scala/cwl/command/StringCommandPart.scala +++ /dev/null @@ -1,18 +0,0 @@ -package cwl.command - -import cats.syntax.validated._ -import common.validation.ErrorOr.ErrorOr -import wom._ -import wom.callable.RuntimeEnvironment -import wom.expression.IoFunctionSet -import wom.graph.LocalName -import wom.values._ - -case class StringCommandPart(literal: String) extends CommandPart { - override def instantiate(inputsMap: Map[LocalName, WomValue], - functions: IoFunctionSet, - valueMapper: (WomValue) => WomValue, - runtimeEnvironment: RuntimeEnvironment): ErrorOr[List[InstantiatedCommand]] = - // TODO CWL shellquotes by default, but this shellquotes everything. Only shellquote what should be shellquoted. - List(InstantiatedCommand(literal.shellQuote)).validNel -} diff --git a/cwl/src/main/scala/cwl/internal/CommandPartSortingAlgorithm.scala b/cwl/src/main/scala/cwl/internal/CommandPartSortingAlgorithm.scala deleted file mode 100644 index eb6ed0410ce..00000000000 --- a/cwl/src/main/scala/cwl/internal/CommandPartSortingAlgorithm.scala +++ /dev/null @@ -1,71 +0,0 @@ -package cwl.internal - -import cats.data.Kleisli._ -import cats.data.ReaderT -import cats.data.Validated._ -import cats.syntax.traverse._ -import cats.syntax.validated._ -import cats.instances.list._ -import cwl.CommandLineTool._ -import cwl.command.ParentName -import cwl.{ArgumentCommandLineBinding, ArgumentToCommandPart, CommandLineTool, CommandPartExpression, FullyQualifiedName, InputParameter, MyriadInputTypeToSortedCommandParts, MyriadInputTypeToWomType} -import shapeless.Coproduct -import wom.types.WomStringType - -object CommandPartSortingAlgorithm { - def argumentCommandParts(arguments: Option[Array[CommandLineTool.Argument]]): CommandPartExpression[List[SortKeyAndCommandPart]] = - // arguments is an Option[Array[Argument]], the toList.flatten gives a List[Argument] - arguments.toList.flatten - // zip the index because we need it in the sorting key - .zipWithIndex.flatTraverse(argumentToCommandPart.tupled) - - def argumentToCommandPart: (Argument, Int) => CommandPartExpression[List[SortKeyAndCommandPart]] = (argument, index) => ReaderT { - case ((requirementsAndHints, expressionLib, _)) => - val part = argument.fold(ArgumentToCommandPart).apply(requirementsAndHints.hasShellCommandRequirement, expressionLib) - // Get the position from the binding if there is one - val position = argument.select[ArgumentCommandLineBinding].flatMap(_.position) - .map(Coproduct[StringOrInt](_)).getOrElse(CommandLineTool.DefaultPosition) - - // The key consists of the position followed by the index - val sortingKey = CommandBindingSortingKey(List(position, Coproduct[StringOrInt](index))) - - List(SortKeyAndCommandPart(sortingKey, part)).validNel - - } - - def inputBindingsCommandParts(inputs: Array[CommandInputParameter]): CommandPartExpression[List[SortKeyAndCommandPart]] = - inputs.toList.flatTraverse(inputBindingsCommandPart) - - def inputBindingsCommandPart(inputParameter: CommandInputParameter): CommandPartExpression[List[SortKeyAndCommandPart]] = - ReaderT{ case ((hintsAndRequirements, expressionLib, inputValues)) => - val parsedName = FullyQualifiedName(inputParameter.id)(ParentName.empty).id - - val womType = inputParameter.`type`.map(_.fold(MyriadInputTypeToWomType).apply(hintsAndRequirements.schemaDefRequirement)).getOrElse(WomStringType) - - val defaultValue = inputParameter.default.map(_.fold(InputParameter.DefaultToWomValuePoly).apply(womType)) - - inputValues - .collectFirst({ case (inputDefinition, womValue) if inputDefinition.name == parsedName => womValue.validNel }) - .orElse(defaultValue) match { - case Some(Valid(value)) => - // See http://www.commonwl.org/v1.0/CommandLineTool.html#Input_binding - lazy val initialKey = CommandBindingSortingKey.empty - .append(inputParameter.inputBinding, Coproduct[StringOrInt](parsedName)) - - inputParameter.`type`.toList. - flatMap{ - _.fold(MyriadInputTypeToSortedCommandParts). - apply( - inputParameter.inputBinding, - value, - initialKey.asNewKey, - hintsAndRequirements.hasShellCommandRequirement, - expressionLib, - hintsAndRequirements.schemaDefRequirement) - }.validNel - case Some(Invalid(errors)) => Invalid(errors) - case None => s"Could not find an input value for input $parsedName in ${inputValues.prettyString}".invalidNel - } - } - -} diff --git a/cwl/src/main/scala/cwl/internal/CwlEcmaScriptDecoder.scala b/cwl/src/main/scala/cwl/internal/CwlEcmaScriptDecoder.scala deleted file mode 100644 index f7f54d3db1c..00000000000 --- a/cwl/src/main/scala/cwl/internal/CwlEcmaScriptDecoder.scala +++ /dev/null @@ -1,124 +0,0 @@ -package cwl.internal - -import cats.syntax.apply._ -import cats.syntax.traverse._ -import cats.syntax.validated._ -import cats.instances.list._ -import cats.instances.option._ -import common.validation.ErrorOr._ -import common.validation.Validation._ -import cwl.{Directory, File, FileOrDirectory} -import org.mozilla.javascript.{ConsString, NativeArray, NativeObject} -import shapeless.Coproduct -import wom.types.WomNothingType -import wom.values._ - -import scala.jdk.CollectionConverters._ - -class CwlEcmaScriptDecoder { - - def decode(value: AnyRef): ErrorOr[WomValue] = - value match { - case map: NativeObject if map.get("class") == "File" => decodeFile(map.asScala.toMap).flatMap(_.asWomValue) - case map: NativeObject if map.get("class") == "Directory" => decodeDirectory(map.asScala.toMap).flatMap(_.asWomValue) - - case map: NativeObject => decodeMap(map.asScala.toMap) - case array: NativeArray => - val anyList = array.asScala.toList - val anyRefArray = anyList.asInstanceOf[List[AnyRef]] - anyRefArray.traverse(decode).map(WomArray.apply) - - //we represent nulls as this type because Wom doesn't have a "null" value, but it does have a nothing type - //If you wish this to be otherwise please tidy up the Expression interpolator as well - case null => WomOptionalValue(WomNothingType, None).valid - - case string: String => WomString(string).valid - case consString: ConsString => WomString(consString.toString).valid - case int: java.lang.Integer => WomInteger(int).valid - case long: java.lang.Long => WomLong(long).valid - case double: java.lang.Double if double == double.doubleValue.floor && !double.isInfinite => - WomInteger(double.intValue).valid - case double: java.lang.Double => WomFloat(double).valid - case boolean: java.lang.Boolean => WomBoolean(boolean).valid - case unknown => s"While decoding the output $value of the Javascript interpreter, we encountered $unknown and were unable to reify it.".invalidNel - } - - def decodeMap(map: Map[Any, Any]): ErrorOr[WomValue] = { - val realMap: Map[AnyRef, AnyRef] = map.asInstanceOf[Map[AnyRef, AnyRef]] - - val tupleList = realMap.toList.traverse{ - case (k,v) => (k.toString.validNel: ErrorOr[String], decode(v)).tupled - } - val mapWomValues = tupleList.map(_.toMap) - mapWomValues.map(WomObject.apply) - } - - /** - * Called to decode a cwl File or Directory. - */ - def decodeDirectoryOrFile(value: Any): ErrorOr[FileOrDirectory] = { - val invalidValue = s"Not a valid CWL map or directory: $value".invalidNel - - value match { - case map: NativeObject if map.get("class") == "File" => decodeFile(map.asScala.toMap).map(Coproduct[FileOrDirectory](_)) - case map: NativeObject if map.get("class") == "Directory" => decodeDirectory(map.asScala.toMap).map(Coproduct[FileOrDirectory](_)) - case _ => invalidValue - } - } - - /** - * Called to decode an array of cwl File or Directory instances. - */ - def decodeDirectoryOrFiles(value: Any): ErrorOr[Array[FileOrDirectory]] = { - value match { - case na: NativeArray => na.asScala.toList.traverse(decodeDirectoryOrFile).map(_.toArray) - } - } - - /** - * Called to decode a map value using a supplied function. - */ - def decodeMapValue[A](map: Map[Any, Any], key: String, f: Any => A): ErrorOr[Option[A]] = { - map.get(key).traverse(anyRef => validate(f(anyRef))) - } - - /** - * Called to decode an array of files or directories from a map value. - */ - def decodeMapDirectoryOrFiles(map: Map[Any, Any], - key: String): ErrorOr[Option[Array[FileOrDirectory]]] = { - map.get(key).traverse(decodeDirectoryOrFiles) - } - - /** - * Called to decode a cwl File. - */ - def decodeFile(map: Map[Any, Any]): ErrorOr[File] = { - val location = decodeMapValue(map, "location", _.toString) - val path = decodeMapValue(map, "path", _.toString) - val basename = decodeMapValue(map, "basename", _.toString) - val checksum = decodeMapValue(map, "checksum", _.toString) - val size = decodeMapValue(map, "size", _.toString.toDouble.toLong) - val secondaryFiles = decodeMapDirectoryOrFiles(map, "secondaryFiles") - val format = decodeMapValue(map, "format", _.toString) - val contents = decodeMapValue(map, "contents", _.toString) - - (location, path, basename, checksum, size, secondaryFiles, format, contents).mapN( - File(_, _, _, _, _, _, _, _) - ) - } - - /** - * Called to decode a cwl Directory. - */ - def decodeDirectory(map: Map[Any, Any]): ErrorOr[Directory] = { - val location = decodeMapValue(map, "location", _.toString) - val path = decodeMapValue(map, "path", _.toString) - val basename = decodeMapValue(map, "basename", _.toString) - val listing = decodeMapDirectoryOrFiles(map, "listing") - - (location, path, basename, listing).mapN( - Directory(_, _, _, _) - ) - } -} diff --git a/cwl/src/main/scala/cwl/internal/EcmaScriptEncoder.scala b/cwl/src/main/scala/cwl/internal/EcmaScriptEncoder.scala deleted file mode 100644 index e4aed51e215..00000000000 --- a/cwl/src/main/scala/cwl/internal/EcmaScriptEncoder.scala +++ /dev/null @@ -1,123 +0,0 @@ -package cwl.internal - -import cats.data.Validated.Valid -import common.validation.ErrorOr.ErrorOr -import cwl.internal.EcmaScriptUtil.{ECMAScriptVariable, ESArray, ESObject, ESPrimitive} -import cwl.{Directory, File} -import mouse.all._ -import wom.values._ - -/** - * Converts a WomValue into a javascript compatible value. - */ -class EcmaScriptEncoder { - - /** - * Base implementation converts any WomPrimitive (except WomFile) into a javascript compatible value. - * - * Inputs, and returned output must be one of: - * - WomString - * - WomBoolean - * - WomFloat - * - WomInteger - * - WomMap - * - WomArray - * - A "WomNull" equal to WomOptionalValue(WomNothingType, None) - * - * The WomMap keys and values, and WomArray elements must be the one of the above, recursively. - * - * Instances of WomFile are not permitted, and must be already converted to one of the above types. - * - * @param value A WOM value. - * @return The javascript equivalent. - */ - def encode(value: WomValue): ECMAScriptVariable = { - value match { - case file: WomFile => encodeFileOrDirectory(file) - case WomOptionalValue(_, None) => ESPrimitive(null) - case WomOptionalValue(_, Some(innerValue)) => encode(innerValue) - case WomString(string) => string |> ESPrimitive - case WomInteger(int) => Int.box(int) |> ESPrimitive - case WomLong(long) => Long.box(long) |> ESPrimitive - case WomFloat(double) => Double.box(double) |> ESPrimitive - case WomBoolean(boolean) => Boolean.box(boolean) |> ESPrimitive - case WomArray(_, array) => array.toList.map(encode).toArray |> ESArray - case WomMap(_, map) => map.map{ - case (mapKey, mapValue) => (encodeString(mapKey), encode(mapValue)) - } |> ESObject - case objectLike: WomObjectLike => objectLike.values.map{ - case (key, innerValue) => (key, encode(innerValue)) - } |> ESObject - case WomCoproductValue(_, womValue) => encode(womValue) - case WomEnumerationValue(_, womValue) => womValue |> ESPrimitive - case _ => throw new RuntimeException(s"$getClass is unable to encode value: $value") - } - } - - def encodeString(value: WomValue): String = { - encode(value) match { - case ESPrimitive(string: String) => string - - //In the case of a non-string, we evaluate a small snippet of Ecma script meant to coerce the object to a string - // http://2ality.com/2012/03/converting-to-string.html - case _ => - val jsString: ErrorOr[WomValue] = EcmaScriptUtil.evalStructish(""""" + other""","other" -> value, encoder = this) - jsString match { - case Valid(WomString(string)) => string - case unexpected => throw new RuntimeException(s"Expected to convert '$value' to a String but ended up with '$unexpected'") - } - } - } - - /** - * Encodes a sequence of wom file or directory values. - */ - def encodeFileOrDirectories(values: Seq[WomFile]): ESArray = { - ESArray(values.toList.map(encodeFileOrDirectory).toArray) - } - - /** - * Encodes a wom file or directory value. - */ - def encodeFileOrDirectory(value: WomFile): ECMAScriptVariable = { - value match { - case directory: WomUnlistedDirectory => encodeDirectory(WomMaybeListedDirectory(directory.value)) - case file: WomSingleFile => encodeFile(WomMaybePopulatedFile(file.value)) - case glob: WomGlobFile => encodeFile(WomMaybePopulatedFile(glob.value)) - case directory: WomMaybeListedDirectory => encodeDirectory(directory) - case file: WomMaybePopulatedFile => encodeFile(file) - } - } - - /** - * Encodes a wom file. - */ - def encodeFile(file: WomMaybePopulatedFile): ECMAScriptVariable = - List( - Option("class" -> ESPrimitive("File")), - file.valueOption.map("location" -> ESPrimitive(_)), - file.valueOption.map("path" -> ESPrimitive(_)), - Option("basename" -> (File.basename(file.value) |> ESPrimitive)), - Option("dirname" -> (File.dirname(file.value) |> ESPrimitive)), - Option("nameroot" -> (File.nameroot(file.value) |> ESPrimitive)), - Option("nameext" -> (File.nameext(file.value) |> ESPrimitive)), - file.checksumOption.map("checksum" -> ESPrimitive(_)), - file.sizeOption.map(Long.box).map("size" -> ESPrimitive(_)), - Option("secondaryFiles" -> encodeFileOrDirectories(file.secondaryFiles)), - file.formatOption.map("format" -> ESPrimitive(_)), - file.contentsOption.map("contents" -> ESPrimitive(_)) - ).flatten.toMap |> ESObject - - /** - * Encodes a wom directory. - */ - def encodeDirectory(directory: WomMaybeListedDirectory): ESObject = { - List( - Option("class" -> ESPrimitive("Directory")), - directory.valueOption.map("location" -> ESPrimitive(_)), - Option(directory.value).map("path" -> ESPrimitive(_)), - Option("basename" -> ESPrimitive(Directory.basename(directory.value))), - directory.listingOption.map(encodeFileOrDirectories).map("listing" -> _) - ).flatten.toMap |> ESObject - } -} diff --git a/cwl/src/main/scala/cwl/internal/EcmaScriptUtil.scala b/cwl/src/main/scala/cwl/internal/EcmaScriptUtil.scala deleted file mode 100644 index d2c5def745f..00000000000 --- a/cwl/src/main/scala/cwl/internal/EcmaScriptUtil.scala +++ /dev/null @@ -1,128 +0,0 @@ -package cwl.internal - -import common.collections.EnhancedCollections._ -import common.validation.ErrorOr._ -import common.validation.Validation._ -import org.mozilla.javascript._ -import wom.values._ - -import scala.concurrent.duration._ -import scala.util.Try - -/** - * This implementation depends on Mozilla Rhino. - * - * Previously we attempted to use Nashorn which is built into the JDK, but we encountered at least 2 situations where - * it didn't work and we found no workarounds to satisfy the use cases. Namely, JSON.stringify of a Map and calling "sort" on an array. - */ -//noinspection VariablePatternShadow -object EcmaScriptUtil { - def writeValue(value: ECMAScriptVariable)(context: Context, scope: Scriptable): AnyRef = - value match { - case ESObject(fields) => - val newObj = context.newObject(scope) - - fields.toList.foreach{ - case (name, value) => - val newerObj = writeValue(value)(context, scope) - ScriptableObject.putProperty(newObj, name, newerObj) - } - newObj - - case ESArray(array) => - val newObj = context.newArray(scope, array.length) - - array.toList.zipWithIndex.foreach { - case (js, index) => - - val newerObj = writeValue(js)(context, scope) - ScriptableObject.putProperty(newObj, index, newerObj) - - } - newObj - - case ESPrimitive(obj) => obj - } - - /** - * Runs ECMAScript as JS1.8 (aka ES5) - * - * Uses RhinoSandbox to reduce chances injected JS wreaks havoc on the JVM. - * - * @see https://en.wikipedia.org/wiki/ECMAScript#Version_correspondence - */ - def evalRaw(expr: String)(block: (Context, Scriptable) => Unit): AnyRef = { - // TODO: Parameterize and update callers to pass in source name, max duration, max instructions, etc.? - // For now, be very liberal with scripts giving 60 seconds of unrestricted CPU usage and unlimited instructions. - val sourceName = "" - val maxDuration: Duration = 60.seconds - val maxInstructionsOption: Option[Int] = None - val strict = true - val languageVersionOption = Option(Context.VERSION_1_8) - - val sandbox = new EnhancedRhinoSandbox(strict, languageVersionOption) - if (maxDuration.isFinite) { - sandbox.setMaxDuration(maxDuration.toMillis.toInt) - } - maxInstructionsOption foreach sandbox.setInstructionLimit - sandbox.setUseSafeStandardObjects(true) - sandbox.setUseSealedScope(true) - - sandbox.eval(sourceName, expr)(block) - } - - sealed trait ECMAScriptVariable - - case class ESObject(fields: Map[String, ECMAScriptVariable]) extends ECMAScriptVariable - case class ESArray(array: Array[ECMAScriptVariable]) extends ECMAScriptVariable { - override def toString: String = s"ESArray(${array.toList})" - } - case class ESPrimitive(anyRef: AnyRef) extends ECMAScriptVariable - - /** - * Evaluates a javascript expression using maps of WOM values. - * - * TODO: Once custom types are supported as WomValue, this custom method won't be required. - * - * @param expr The javascript expression. - * @param rawValues A map filled with WOM values. - * @param mapValues A map of maps filled with WOM values of various types. - * @param encoder Encodes wom values to javascript. - * @param decoder Decodes wom values from javascript. - * @return The result of the expression. - */ - def evalStructish(expr: String, - rawValues: (String, WomValue), - mapValues: Map[String, Map[String, WomValue]] = Map.empty, - encoder: EcmaScriptEncoder, - decoder: CwlEcmaScriptDecoder = new CwlEcmaScriptDecoder): ErrorOr[WomValue] = { - def evaluate = evalRaw(expr) { (context, scope) => - - val (key, value) = rawValues - - val jsValue = encoder.encode(value) - val field = writeValue(jsValue)(context, scope) - ScriptableObject.putProperty(scope, key, field) - - val jsMap = mapValues.safeMapValues{ _.safeMapValues(encoder.encode) } - - jsMap foreach { - case (scopeId, nestedMap) => - - val newObj = context.newObject(scope) - nestedMap.toList foreach { - case (key, value) => - val newerObj = writeValue(value)(context, scope) - ScriptableObject.putProperty(newObj, key, newerObj) - } - ScriptableObject.putProperty(scope, scopeId, newObj) - } - - } - - for { - evaluated <- Try(evaluate).toErrorOr - decoded <- decoder.decode(evaluated) - } yield decoded - } -} diff --git a/cwl/src/main/scala/cwl/internal/EnhancedRhinoSandbox.scala b/cwl/src/main/scala/cwl/internal/EnhancedRhinoSandbox.scala deleted file mode 100644 index 405c04fdaa2..00000000000 --- a/cwl/src/main/scala/cwl/internal/EnhancedRhinoSandbox.scala +++ /dev/null @@ -1,167 +0,0 @@ -package cwl.internal - -import cwl.internal.EnhancedRhinoSandbox._ -import delight.rhinosandox.internal._ -import org.mozilla.javascript._ - -import scala.jdk.CollectionConverters._ -import scala.reflect._ - -/** - * Extends the RhinoSandboxImpl with some fixes and enhancements using java reflection. - * - * @param strict Should evaluation be strict. - * @param languageVersionOption The optional language version to set. - */ -class EnhancedRhinoSandbox(strict: Boolean = true, languageVersionOption: Option[Int] = None) extends RhinoSandboxImpl { - - // Allows easier reflection access to private fields - private lazy val sandboxImpl: RhinoSandboxImpl = this - - /** - * Similar to RhinoSandbox.eval but passes back the context and scope for mutating before evaluation. - * - * With the original RhinoSandbox.eval not sure how to: - * - Create (nested) arrays - * - Create (nested) maps - * - Set JS/ES version - * - * So this uses a copy-port of the original, using reflection to read some of RhinoSandbox's private variables. - * - * Instead of hiding the context and scope as in RhinoSandbox.eval, both are passed back through the block. - * - * TODO: Ask RhinoSandbox if hacks are even needed, and if so contrib back patches so that reflection isn't required. - * - Is there a way to pass nested maps/arrays via a `java.util.Map[String, Object]`, or must we use our `block`? - * - Additionally: can we skip passing `context` to a block as a thread may just call `Context.getCurrentContext()`? - * - * @see https://maxrohde.com/2015/08/06/sandboxing-javascript-in-java-app-link-collection/ - * @see https://github.com/javadelight/delight-rhino-sandbox/blob/9f5a073/src/main/java/delight/rhinosandox/internal/RhinoSandboxImpl.java#L100-L123 - * @see delight.rhinosandox.internal.RhinoSandboxImpl#assertContextFactory() - */ - def eval(sourceName: String, js: String)(block: (Context, Scriptable) => Unit): AnyRef = { - assertContextFactory() - val sandboxImpl_contextFactory = PrivateField(sandboxImpl, "contextFactory").as[ContextFactory] - // RhinoSandbox diff: eval has enterContext inside the try, but Rhino docs say it belongs outside. - // https://www-archive.mozilla.org/rhino/apidocs/org/mozilla/javascript/ContextFactory.html#enterContext%28%29 - val context = sandboxImpl_contextFactory.enterContext - try { - // RhinoSandbox diff: allow setting the language version - languageVersionOption foreach context.setLanguageVersion - assertSafeScope(context) - val sandboxImpl_globalScope = PrivateField(sandboxImpl, "globalScope").as[ScriptableObject] - val sandboxImpl_sealScope = PrivateField(sandboxImpl, "sealScope").as[Boolean] - if (sandboxImpl_sealScope) { - sandboxImpl_globalScope.sealObject() - } - val sandboxImpl_safeScope = PrivateField(sandboxImpl, "safeScope").as[ScriptableObject] - val instanceScope = context.newObject(sandboxImpl_safeScope) - instanceScope.setPrototype(sandboxImpl_safeScope) - instanceScope.setParentScope(null) - - block(context, instanceScope) - - // RhinoSandbox diff: allow strict JS/ES evaluation - // See note at top assertContextFactory as to why we have to put 'use strict'; here. - // All on one line to avoid off-by-one error for javascript error messages that report line numbers. - // Could also pass zero as the line number, but the RhinoSandbox passes hard codes line number one also. - val script = if (strict) s"'use strict';$js" else js - context.evaluateString(instanceScope, script, sourceName, 1, null) - } finally { - Context.exit() - } - } - - /** - * Stricter context factory modified from RhinoSandboxImpl.assertContextFactory(). - * - * ContextFactory.initGlobal() is called within a static synchronized block. - * The globalScope initialized via initSafeStandardObjects instead of initStandardObjects. - * - * The default implementation uses a SafeContext that allows non-strict JS/ES. We would ideally set - * FEATURE_STRICT_MODE to true but that only produces warnings and doesn't return an error. Unfortunately when - * FEATURE_WARNING_AS_ERROR is enabled then non-strict Rhino warnings like "missing ;" throw errors. Instead, - * "'use strict';" is injected before scripts. - */ - override def assertContextFactory(): Unit = { - if (PrivateField(sandboxImpl, "contextFactory").as[ContextFactory] != null) { - return - } - - val _safeContext = new SafeContext - PrivateField(sandboxImpl, "contextFactory") := _safeContext - val _hasExplicitGlobal = ContextFactory.hasExplicitGlobal - val _not = !_hasExplicitGlobal - // RhinoSandbox diff: the global does not like to be initialized twice. Synchronize initialization. - val sandboxImpl_contextFactory = PrivateField(sandboxImpl, "contextFactory").as[SafeContext] - if (_not) initGlobalSynchronized(sandboxImpl_contextFactory) - - val sandboxImpl_instructionLimit = PrivateField(sandboxImpl, "instructionLimit").as[Int] - PrivateField(sandboxImpl_contextFactory, "maxInstructions") := sandboxImpl_instructionLimit - val sandboxImpl_maxDuration = PrivateField(sandboxImpl, "maxDuration").as[Long] - PrivateField(sandboxImpl_contextFactory, "maxRuntimeInMs") := sandboxImpl_maxDuration - // RhinoSandbox diff: assertContextFactory has enterContext inside the try, but Rhino docs say it belongs outside. - // https://www-archive.mozilla.org/rhino/apidocs/org/mozilla/javascript/ContextFactory.html#enterContext%28%29 - val context = sandboxImpl_contextFactory.enterContext - try { - // RhinoSandbox diff: Default globalScope is created via initStandardObjects instead of initSafeStandardObjects. - // initStandardObjects would add the various java packages into the global scope, including `java.io.File`, etc. - PrivateField(sandboxImpl, "globalScope") := context.initSafeStandardObjects(null, false) - val sandboxImpl_inScope = PrivateField(sandboxImpl, "inScope").as[java.util.Map[String, AnyRef]] - val _entrySet = sandboxImpl_inScope.entrySet - val sandboxImpl_globalScope = PrivateField(sandboxImpl, "globalScope").as[ScriptableObject] - for (entry <- _entrySet.asScala) { - sandboxImpl_globalScope.put( - entry.getKey, - sandboxImpl_globalScope, - Context.toObject(entry.getValue, sandboxImpl_globalScope)) - } - val parameters = Array(classOf[String]) - val dealMethod = classOf[RhinoEvalDummy].getMethod("eval", parameters: _*) - val _rhinoEval = new RhinoEval("eval", dealMethod, sandboxImpl_globalScope) - sandboxImpl_globalScope.defineProperty("eval", _rhinoEval, ScriptableObject.DONTENUM) - } finally { - Context.exit() - } - } - -} - -object EnhancedRhinoSandbox { - - /** - * Get or sets a private field. - * - * @param obj The instance to retrieve the field from. - * @param name The name of the field. - * @tparam A The class to retrieve the value from. The field MUST exist on this class, and not a superclass. - */ - final case class PrivateField[A: ClassTag](obj: A, name: String) { - private[this] def field = { - val field = classTag[A].runtimeClass.getDeclaredField(name) - field.setAccessible(true) - field - } - - def as[B]: B = { - field.get(obj).asInstanceOf[B] - } - - def :=(value: Any): Unit = { - field.set(obj, value) - } - } - - /** - * Call ContextFactory.initGlobal in a static synchronized block. - * - * @see [[cwl.internal.EnhancedRhinoSandbox.assertContextFactory]] - */ - private def initGlobalSynchronized(sandboxImpl_contextFactory: ContextFactory) = { - synchronized { - val _hasExplicitGlobal = ContextFactory.hasExplicitGlobal - val _not = !_hasExplicitGlobal - if (_not) ContextFactory.initGlobal(sandboxImpl_contextFactory) - } - } - -} diff --git a/cwl/src/main/scala/cwl/internal/GigabytesToBytes.scala b/cwl/src/main/scala/cwl/internal/GigabytesToBytes.scala deleted file mode 100644 index 7e45aeec5ba..00000000000 --- a/cwl/src/main/scala/cwl/internal/GigabytesToBytes.scala +++ /dev/null @@ -1,26 +0,0 @@ -package cwl.internal - -import shapeless.{Coproduct, Poly1} - -object GigabytesToBytes extends Poly1 { - val toMebibytesMultiplier = Math.pow(2, 20).toLong - - implicit def long = at[Long] { - l => - val value = l * toMebibytesMultiplier - Coproduct[cwl.ResourceRequirementType](value) - } - - implicit def string = at[String] { - s => - //TODO: Scale this by multiplier https://github.com/broadinstitute/cromwell/issues/3382 - Coproduct[cwl.ResourceRequirementType](s) - } - - implicit def expression = at[cwl.Expression] { - e => - //TODO: Scale this by multiplier https://github.com/broadinstitute/cromwell/issues/3382 - Coproduct[cwl.ResourceRequirementType](e) - } -} - diff --git a/cwl/src/main/scala/cwl/model.scala b/cwl/src/main/scala/cwl/model.scala deleted file mode 100644 index 838ee66a98c..00000000000 --- a/cwl/src/main/scala/cwl/model.scala +++ /dev/null @@ -1,310 +0,0 @@ -package cwl - -import cwl.CommandLineTool.{CommandBindingSortingKey, SortKeyAndCommandPart} -import cwl.SchemaDefRequirement.{AsInputEnumSchema, AsInputRecordSchema, SchemaDefTypes} -import cwl.CwlType.CwlType -import cwl.WorkflowStepInput.InputSource -import cwl.command.ParentName -import cwl.internal.GigabytesToBytes -import eu.timepit.refined._ -import eu.timepit.refined.api.Refined -import eu.timepit.refined.string.MatchesRegex -import shapeless.syntax.singleton._ -import shapeless.{:+:, CNil, Coproduct, Poly1, Witness} -import wom.types.WomType -import wom.values.WomValue -import mouse.all._ - -object WorkflowStepInputSource { - object String { - def unapply(arg: InputSource): Option[String] = arg.select[String] - } - object StringArray { - def unapply(arg: InputSource): Option[Array[String]] = arg.select[Array[String]] - } -} - -/** - * Describes a bespoke type. - * - * @param name This field actually does _not_ appear in the v1.0 schema, but it is used anyway in the conformance tests. - * After some consideration it was determined that we should close our eyes and pretend it is in the spec. It - * makes its formal appearance as a required field in v1.1. - */ -case class InputRecordSchema( - name: String, - fields: Option[Array[InputRecordField]] = None, - `type`: W.`"record"`.T = W("record").value, - label: Option[String] = None) { - -} - -case class InputRecordField( - name: String, - `type`: MyriadInputType, - doc: Option[String] = None, - inputBinding: Option[InputCommandLineBinding], - label: Option[String] = None) - -case class InputArraySchema -( - items: MyriadInputType, - `type`: W.`"array"`.T = Witness("array").value, - label: Option[String] = None, - inputBinding: Option[InputCommandLineBinding] = None, - // IAS.secondaryFiles are NOT listed in 1.0 spec, but according to jgentry they will be, maybe - secondaryFiles: Option[SecondaryFiles] = None -) - -trait CommandLineBinding { - def loadContents: Option[Boolean] - def position: Option[Int] - def prefix: Option[String] - def separate: Option[Boolean] - def itemSeparator: Option[String] - def optionalValueFrom: Option[StringOrExpression] - def shellQuote: Option[Boolean] - // separate defaults to true - def effectiveSeparate = separate.getOrElse(true) -} - -object InputCommandLineBinding { - def default = InputCommandLineBinding() -} - -case class InputCommandLineBinding( - loadContents: Option[Boolean] = None, - position: Option[Int] = None, - prefix: Option[String] = None, - separate: Option[Boolean] = None, - itemSeparator: Option[String] = None, - valueFrom: Option[StringOrExpression] = None, - shellQuote: Option[Boolean] = None) extends CommandLineBinding { - override val optionalValueFrom = valueFrom - - def toCommandPart(sortingKey: CommandBindingSortingKey, boundValue: WomValue, hasShellCommandRequirement: Boolean, expressionLib: ExpressionLib) = { - SortKeyAndCommandPart(sortingKey, InputCommandLineBindingCommandPart(this, boundValue)(hasShellCommandRequirement, expressionLib)) - } -} - -// valueFrom is required for command line bindings in the argument section: http://www.commonwl.org/v1.0/CommandLineTool.html#CommandLineBinding -case class ArgumentCommandLineBinding( - valueFrom: StringOrExpression, - loadContents: Option[Boolean] = None, - position: Option[Int] = None, - prefix: Option[String] = None, - separate: Option[Boolean] = None, - itemSeparator: Option[String] = None, - shellQuote: Option[Boolean] = None) extends CommandLineBinding { - override val optionalValueFrom = Option(valueFrom) -} - -case class InputBinding(position: Int, prefix: String) - -object MyriadOutputInnerTypeCacheableString extends Poly1 { - import Case._ - - private def cacheableOutputRecordFieldString(field: OutputRecordField): String = { - val fieldType = field.`type`.fold(MyriadOutputTypeCacheableString) - val lqn = field.name.substring(field.name.lastIndexOf('#') + 1) - s"OutputRecordField($lqn,$fieldType,${field.doc},${field.outputBinding})" - } - - implicit def recordSchema: Aux[OutputRecordSchema, String] = at[OutputRecordSchema] { - s => - val t = s.`type` - s"OutputRecordSchema($t,${s.fields.map(a => "Array(" + a.map(cacheableOutputRecordFieldString).mkString(",") + ")" )},${s.label})" - } - - implicit def arraySchema: Aux[OutputArraySchema, String] = at[OutputArraySchema] { - a => - val is: String = a.items.fold(MyriadOutputTypeCacheableString) - val t = a.`type` - s"OutputArraySchema($is,$t,${a.label},${a.outputBinding})" - } - - implicit def enumSchema: Aux[OutputEnumSchema, String] = at[OutputEnumSchema] { _.toString } - implicit def cwlType: Aux[CwlType, String] = at[CwlType] { _.toString } - implicit def string: Aux[String, String] = at[String] { identity } -} - - -object MyriadOutputTypeCacheableString extends Poly1 { - import Case._ - - implicit def one: Aux[MyriadOutputInnerType, String] = at[MyriadOutputInnerType] { - _.fold(MyriadOutputInnerTypeCacheableString) - } - - implicit def many: Aux[Array[MyriadOutputInnerType], String] = at[Array[MyriadOutputInnerType]] { a => - val strings: Array[String] = a.map(_.fold(MyriadOutputInnerTypeCacheableString)) - "Array(" + strings.mkString(",") + ")" - } -} - -object SecondaryFilesCacheableString extends Poly1 { - import Case._ - - implicit def one: Aux[StringOrExpression, String] = at[StringOrExpression] { - _.toString - } - - implicit def array: Aux[Array[StringOrExpression], String] = at[Array[StringOrExpression]] { - _.mkString("Array(", ",", ")") - } - -} - -case class OutputRecordSchema( - `type`: W.`"record"`.T, - fields: Option[Array[OutputRecordField]], - label: Option[String]) - -case class OutputRecordField( - name: String, - `type`: MyriadOutputType, - doc: Option[String], - outputBinding: Option[CommandOutputBinding]) - -case class OutputArraySchema( - items: MyriadOutputType, - `type`: W.`"array"`.T = Witness("array").value, - label: Option[String] = None, - outputBinding: Option[CommandOutputBinding] = None) - -case class InlineJavascriptRequirement( - `class`: W.`"InlineJavascriptRequirement"`.T = "InlineJavascriptRequirement".narrow, - expressionLib: Option[Array[String]] = None) - -case class SchemaDefRequirement( - types: Array[SchemaDefTypes] = Array.empty, - `class`: W.`"SchemaDefRequirement"`.T = Witness("SchemaDefRequirement").value) { - - def lookupType(tpe: String): Option[WomType] = - lookupCwlType(tpe).flatMap{ - case AsInputRecordSchema(inputRecordSchema: InputRecordSchema) => MyriadInputInnerTypeToWomType.inputRecordSchemaToWomType(inputRecordSchema).apply(this) |> Option.apply - case _ => None - } - - //Currently only InputRecordSchema has a name in the spec, so it is the only thing that can be referenced via string - def lookupCwlType(tpe: String): Option[SchemaDefTypes] = { - - def matchesType(inputEnumSchema: InputEnumSchema): Boolean = { - inputEnumSchema.name.fold(false){name => FileAndId(name)(ParentName.empty).id equalsIgnoreCase FileAndId(tpe)(ParentName.empty).id} - } - - types.toList.flatMap { - case AsInputRecordSchema(inputRecordSchema: InputRecordSchema) if FileAndId(inputRecordSchema.name)(ParentName.empty).id equalsIgnoreCase FileAndId(tpe)(ParentName.empty).id => - List(Coproduct[SchemaDefTypes](inputRecordSchema)) - case AsInputEnumSchema(inputEnumSchema: InputEnumSchema) if matchesType(inputEnumSchema) => - List(Coproduct[SchemaDefTypes](inputEnumSchema)) - case _ => - List() - }.headOption - } -} - -object SchemaDefRequirement { - type SchemaDefTypes = InputRecordSchema :+: InputEnumSchema :+: InputArraySchema :+: CNil - - object AsInputRecordSchema { - def unapply(arg: SchemaDefTypes): Option[InputRecordSchema] = arg.select[InputRecordSchema] - } - - object AsInputEnumSchema { - def unapply(arg: SchemaDefTypes): Option[InputEnumSchema] = arg.select[InputEnumSchema] - } - - object AsInputArraySchema { - def unapply(arg: SchemaDefTypes): Option[InputArraySchema] = arg.select[InputArraySchema] - } -} - -//There is a large potential for regex refinements on these string types -case class DockerRequirement( - `class`: W.`"DockerRequirement"`.T, - dockerPull: Option[String], //TODO Refine to match a docker image regex? - dockerLoad: Option[String], - dockerFile: Option[String], - dockerImport: Option[String], - dockerImageId: Option[String], - dockerOutputDirectory: Option[String] - ) - -case class SoftwareRequirement( - `class`: W.`"SoftwareRequirement"`.T, - packages: Array[SoftwarePackage] = Array.empty - ) - -case class SoftwarePackage( - `package`: String, - version: Option[Array[String]], - specs: Option[Array[String]] // This could be refined to match a regex for IRI. - ) { - type Package = String - type Specs = Array[String] -} - -case class EnvVarRequirement( - `class`: EnvVarRequirement.ClassType = EnvVarRequirement.`class`, - envDef: Array[EnvironmentDef] - ) - -object EnvVarRequirement { - type ClassType = Witness.`"EnvVarRequirement"`.T - - val `class`: ClassType = "EnvVarRequirement".asInstanceOf[ClassType] -} - -case class EnvironmentDef(envName: String, envValue: StringOrExpression) { - type EnvName = String - type EnvValue = String -} - - -case class ShellCommandRequirement(`class`: W.`"ShellCommandRequirement"`.T = "ShellCommandRequirement".narrow) - -case class ResourceRequirement( - `class`: W.`"ResourceRequirement"`.T, - coresMin: Option[ResourceRequirementType], - coresMax: Option[ResourceRequirementType], - ramMin: Option[ResourceRequirementType], - ramMax: Option[ResourceRequirementType], - tmpdirMin: Option[ResourceRequirementType], - tmpdirMax: Option[ResourceRequirementType], - outdirMin: Option[ResourceRequirementType], - outdirMax: Option[ResourceRequirementType]) { - def effectiveCoreMin = coresMin.orElse(coresMax) - def effectiveCoreMax = coresMax.orElse(coresMin) - - def effectiveRamMin = ramMin.orElse(ramMax).map(_.fold(GigabytesToBytes)) - def effectiveRamMax = ramMax.orElse(ramMin).map(_.fold(GigabytesToBytes)) - - def effectiveTmpdirMin = tmpdirMin.orElse(tmpdirMax) - def effectiveTmpdirMax = tmpdirMax.orElse(tmpdirMin) - - def effectiveOutdirMin = outdirMin.orElse(outdirMax) - def effectiveOutdirMax = outdirMax.orElse(outdirMin) -} - -/** - * This promotes DNA Nexus InputResourceRequirement to a first class citizen requirement, which it really isn't. - * Since it's the only one for now it's not a big deal but if more of these pop up we might want to treat custom requirements - * in a different way - */ -case class DnaNexusInputResourceRequirement( - `class`: String Refined MatchesRegex[W.`".*InputResourceRequirement"`.T], - indirMin: Option[Long] - ) - -case class SubworkflowFeatureRequirement( - `class`: W.`"SubworkflowFeatureRequirement"`.T) - -case class ScatterFeatureRequirement( - `class`: W.`"ScatterFeatureRequirement"`.T) - -case class MultipleInputFeatureRequirement( - `class`: W.`"MultipleInputFeatureRequirement"`.T) - -case class StepInputExpressionRequirement( - `class`: W.`"StepInputExpressionRequirement"`.T) diff --git a/cwl/src/main/scala/cwl/ontology/CacheConfiguration.scala b/cwl/src/main/scala/cwl/ontology/CacheConfiguration.scala deleted file mode 100644 index cb82d9db591..00000000000 --- a/cwl/src/main/scala/cwl/ontology/CacheConfiguration.scala +++ /dev/null @@ -1,15 +0,0 @@ -package cwl.ontology - -import com.typesafe.config.Config -import common.validation.Validation._ -import net.ceedubs.ficus.Ficus._ - -case class CacheConfiguration(maxSize: Long) - -object CacheConfiguration { - def apply(config: Config): CacheConfiguration = { - validate(config.getAs[Long]("max-size").getOrElse(0L)) - .map(new CacheConfiguration(_)) - .unsafe("Ontology cache configuration") - } -} diff --git a/cwl/src/main/scala/cwl/ontology/OntologyConfiguration.scala b/cwl/src/main/scala/cwl/ontology/OntologyConfiguration.scala deleted file mode 100644 index 5382ac35b91..00000000000 --- a/cwl/src/main/scala/cwl/ontology/OntologyConfiguration.scala +++ /dev/null @@ -1,27 +0,0 @@ -package cwl.ontology - -import cats.syntax.apply._ -import com.typesafe.config.Config -import common.util.Backoff -import common.validation.ErrorOr.ErrorOr -import common.validation.Validation._ -import net.ceedubs.ficus.Ficus._ - -import scala.concurrent.duration.FiniteDuration - -case class OntologyConfiguration(retries: Option[Int], backoff: Backoff, poolSize: Int) - -object OntologyConfiguration { - def apply(config: Config): OntologyConfiguration = { - val retries = validate { Option(config.as[Int]("retries")) } - val backoff = validate { - Backoff.staticBackoff( - config.as[FiniteDuration]("backoff-time") - ) - } - val poolSize = validate { config.as[Int]("pool-size") } - - val validated: ErrorOr[OntologyConfiguration] = (retries, backoff, poolSize).mapN(OntologyConfiguration.apply) - validated.unsafe("Ontology configuration") - } -} diff --git a/cwl/src/main/scala/cwl/ontology/Schema.scala b/cwl/src/main/scala/cwl/ontology/Schema.scala deleted file mode 100644 index a2ed13d7bf3..00000000000 --- a/cwl/src/main/scala/cwl/ontology/Schema.scala +++ /dev/null @@ -1,180 +0,0 @@ -package cwl.ontology - -import java.util.concurrent.Executors - -import cats.effect.IO -import cats.syntax.traverse._ -import cats.instances.list._ -import com.google.common.cache.{Cache, CacheBuilder} -import com.typesafe.config.{Config, ConfigFactory} -import com.typesafe.scalalogging.Logger -import common.util.IORetry -import common.util.IORetry.StatefulIoError -import common.validation.ErrorOr._ -import common.validation.Validation._ -import cwl.ontology.Schema._ -import mouse.all._ -import net.ceedubs.ficus.Ficus._ -import org.semanticweb.owlapi.apibinding.OWLManager -import org.semanticweb.owlapi.model._ -import org.semanticweb.owlapi.model.parameters.OntologyCopy -import org.semanticweb.owlapi.reasoner.structural.StructuralReasonerFactory -import org.semanticweb.owlapi.reasoner.{OWLReasoner, OWLReasonerFactory} -import org.semanticweb.owlapi.util.OWLAPIStreamUtils -import org.slf4j.LoggerFactory - -import scala.jdk.CollectionConverters._ -import scala.concurrent.ExecutionContext -import scala.util.Try - -/** - * OWL/RDF Schema lookup. - * - * @param schemaIris IRI paths to OWL/RDF schemas. - * @param namespaces Additional/Override namespace prefixes. - */ -case class Schema(schemaIris: Seq[String], - namespaces: Map[String, String], - ontologyManager: OWLOntologyManager = OWLManager.createOWLOntologyManager, - reasonerFactory: OWLReasonerFactory = new StructuralReasonerFactory) { - - /** - * Returns a full IRI based on a full or abbreviated IRI. - * - * A full IRI wrapped in < and > will not be checked for abbreviations but will be returned without the wrapper. - * - * @see https://www.w3.org/TR/owl2-syntax/#IRIs - */ - def fullIri(name: String): String = getIri(name).getIRIString - - /** - * Returns true if child is an equivalent of ancestor, and if not then recursively checks if any of the child's - * super classes are an equivalent of ancestor. - */ - def isSubClass(child: String, ancestor: String): Boolean = { - val childIri: IRI = getIri(child) - val ancestorIri: IRI = getIri(ancestor) - val childClass: OWLClass = dataFactory.getOWLClass(childIri) - val ancestorClass: OWLClass = dataFactory.getOWLClass(ancestorIri) - val schemaReasoner: OWLReasoner = reasonerFactory.createReasoner(schemaOntology) - try { - Schema.isSubClass(schemaReasoner, childClass, ancestorClass) - } finally { - schemaReasoner.dispose() - } - } - - private val dataFactory: OWLDataFactory = ontologyManager.getOWLDataFactory - private val schemaOntology: OWLOntology = ontologyManager.createOntology() - private val schemaPrefixManager: PrefixManager = - ontologyManager.getOntologyFormat(schemaOntology).asPrefixOWLDocumentFormat - - { - def addToSchema(originalOntology: OWLOntology): Unit = { - schemaOntology.addAxioms(originalOntology.axioms()) - val originalOntologyFormat: OWLDocumentFormat = ontologyManager.getOntologyFormat(originalOntology) - if (originalOntologyFormat.isPrefixOWLDocumentFormat) - schemaPrefixManager.copyPrefixesFrom(originalOntologyFormat.asPrefixOWLDocumentFormat) - } - - val errorOr: ErrorOr[Unit] = for { - ontologies <- schemaIris.toList.traverse(loadOntologyFromIri(ontologyManager)) - _ = ontologies.foreach(addToSchema) - } yield () - errorOr.toTry("Error loading schemas").get - - // Add any namespace overrides - namespaces foreach { - case (prefixName, prefix) => schemaPrefixManager.setPrefix(prefixName, prefix) - } - } - - /** - * Returns the full IRI for the name using the prefixManager. - * - * TODO: Not 100% sure why you can't ask owl-api for an iri-with-prefix and have it looked up automatically. - * - * There does seem to be a difference between abbreviated and full IRIs in the official spec, where full IRIs are - * wrapped in < >. But this doesn't seem to be the format used by CWL nor owl-api. - * - * Please update this comment if/when one knows the correct behavior. - * - * @see https://www.w3.org/TR/owl2-syntax/#IRIs - */ - private def getIri(name: String): IRI = { - Try(schemaPrefixManager.getIRI(name)).getOrElse(IRI.create(name)) - } -} - -object Schema { - // Extending StrictLogging creates a circular dependency here for some reason, so making the logger ourselves - private val logger: Logger = Logger(LoggerFactory.getLogger(getClass.getName)) - private [ontology] val ontologyConfig = ConfigFactory.load.as[Config]("ontology") - private val ontologyConfiguration = OntologyConfiguration(ontologyConfig) - private [ontology] val cacheConfig = ontologyConfig.getAs[Config]("cache") - - // Simple cache to avoid reloading the same ontologies too often - private val ontologyCache = cacheConfig.map(makeOntologyCache) - - private implicit val statefulIoError = StatefulIoError.noop[Unit] - private implicit val timer = cats.effect.IO.timer(ExecutionContext.fromExecutor(Executors.newFixedThreadPool(ontologyConfiguration.poolSize))) - - private [ontology] def makeOntologyCache(config: Config): Cache[IRI, OWLOntology] = { - val cacheConfig = CacheConfiguration(config) - logger.info(s"Ontology cache size: ${cacheConfig.maxSize}") - CacheBuilder.newBuilder() - .maximumSize(cacheConfig.maxSize) - .build[IRI, OWLOntology]() - } - - /** - * Returns the absolute path for a file, possibly relative to parent. - */ - def getIriPath(parent: String, path: String): String = IRI.create(parent).resolve(path).getIRIString - - /** - * Load an ontology either from an IRI. - */ - private [ontology] def loadOntologyFromIri(ontologyManager: OWLOntologyManager, cache: Option[Cache[IRI, OWLOntology]] = ontologyCache)(schemaIri: String): ErrorOr[OWLOntology] = { - validate { - val iri = IRI.create(schemaIri) - cache.flatMap(_.getIfPresent(iri) |> Option.apply) match { - case Some(ontology) => - ontologyManager.copyOntology(ontology, OntologyCopy.DEEP) - case _ => - logger.info(s"Loading ${iri.toURI.toString}") - val ontology = loadOntologyFromIri(ontologyManager, iri) - cache.foreach(_.put(iri, ontology)) - ontology - } - } - } - - // Loading the ontology can fail transiently, so put retires around it. See https://github.com/protegeproject/webprotege/issues/298 - private [ontology] def loadOntologyFromIri(ontologyManager: OWLOntologyManager, iri: IRI): OWLOntology = { - val load = IO { ontologyManager.loadOntologyFromOntologyDocument(iri) } - IORetry.withRetry[OWLOntology, Unit](load, (), ontologyConfiguration.retries, ontologyConfiguration.backoff).unsafeRunSync() - } - - /** - * Returns true if child is an equivalent of ancestor, and if not then recursively checks if any of the child's - * super classes are an equivalent of ancestor. - */ - private def isSubClass(reasoner: OWLReasoner, childClass: OWLClass, ancestorClass: OWLClass): Boolean = { - val equivalent: Set[OWLClass] = reasoner.getEquivalentClasses(childClass).asScala.toSet + childClass - if (equivalent.contains(ancestorClass)) { - true - } else { - val parentClasses: Set[OWLClass] = for { - equivalentClass <- equivalent - parentClass <- OWLAPIStreamUtils - .asSet(reasoner.getSuperClasses(equivalentClass).entities) - .asScala - .toSet[OWLClass] - } yield parentClass - parentClasses.collect({ - case superClass: OWLClass if isSubClass(reasoner, superClass, ancestorClass) => superClass - }).nonEmpty - } - } -} diff --git a/cwl/src/main/scala/cwl/package.scala b/cwl/src/main/scala/cwl/package.scala deleted file mode 100644 index 96796712f1e..00000000000 --- a/cwl/src/main/scala/cwl/package.scala +++ /dev/null @@ -1,154 +0,0 @@ - -import cats.data.ReaderT -import common.Checked -import common.validation.Checked._ -import common.validation.ErrorOr._ -import cwl.CwlType._ -import cwl.ExpressionEvaluator.{ECMAScriptExpression, ECMAScriptFunction} -import cwl.command.ParentName -import cwl.ontology.Schema -import shapeless._ -import wom.executable.Executable -import wom.expression.IoFunctionSet -import wom.types._ -import wom.values.WomEvaluatedCallInputs - -import scala.util.{Failure, Success, Try} - -/** - * This package is intended to parse all CWL files. - * - * It makes heavy use of Circe YAML/Json auto derivation feature and - * Circe modules that support the Scala libraries shapeless and Refined. - * - * The [[https://oss.sonatype.org/service/local/repositories/releases/archive/com/chuusai/shapeless_2.12/2.3.2/shapeless_2.12-2.3.2-javadoc.jar/!/shapeless/Coproduct.html shapeless.coproduct]] feature allows us to specify a large - * number of potential types to be parsed. A.k.a. a "disjunction" or "OR" relationship amongst these types. - * - * The [[https://github.com/fthomas/refined/blob/master/modules/core/shared/src/main/scala/eu/timepit/refined/string.scala MatchesRegex]] "refined type" is used - * to enforce structure upon String values in the CWL Yaml. Shapeless' Witness type - * is used to declare a type containing a String literal. - * - * @see CWL Specification - * @see circe - * @see circe-yaml - * @see Refined - * @see Shapeless - */ -package object cwl extends TypeAliases { - - type CwlFile = Array[Cwl] :+: Cwl :+: CNil - type Cwl = Workflow :+: CommandLineTool :+: ExpressionTool :+: CNil - - object Cwl { - object Workflow { def unapply(cwl: Cwl): Option[Workflow] = cwl.select[Workflow] } - object CommandLineTool { def unapply(cwl: Cwl): Option[CommandLineTool] = cwl.select[CommandLineTool] } - object ExpressionTool { def unapply(cwl: Cwl): Option[ExpressionTool] = cwl.select[ExpressionTool] } - } - - def cwlTypeToWomType : CwlType => WomType = { - case CwlType.Any => WomAnyType - case Null => WomNothingType - case Boolean => WomBooleanType - case Int => WomIntegerType - case Long => WomLongType - case Float => WomFloatType - case Double => WomFloatType - case String => WomStringType - case CwlType.File => WomMaybePopulatedFileType - case CwlType.Directory => WomMaybeListedDirectoryType - } - - object StringOrExpression { - object String { - def unapply(soe: StringOrExpression): Option[String] = soe.select[String] - } - object Expression { - def unapply(soe: StringOrExpression): Option[Expression] = soe.select[Expression] - } - object ECMAScriptExpression { - def unapply(soe: StringOrExpression): Option[ECMAScriptExpression] = soe.select[Expression].flatMap(_.select[ECMAScriptExpression]) - } - object ECMAScriptFunction { - def unapply(soe: StringOrExpression): Option[ECMAScriptFunction] = soe.select[Expression].flatMap(_.select[ECMAScriptFunction]) - } - } - - object Expression { - object ECMAScriptExpression { - def unapply(soe: Expression): Option[ECMAScriptExpression] = soe.select[ECMAScriptExpression] - } - object ECMAScriptFunction { - def unapply(soe: Expression): Option[ECMAScriptFunction] = soe.select[ECMAScriptFunction] - } - } - - type WomTypeMap = Map[String, WomType] - - type RequirementsValidator = Requirement => ErrorOr[Requirement] - import cats.syntax.validated._ - val AcceptAllRequirements: RequirementsValidator = _.validNel - - implicit class CwlHelper(val cwl: Cwl) extends AnyVal { - def womExecutable(validator: RequirementsValidator, inputsFile: Option[String], ioFunctions: IoFunctionSet, strictValidation: Boolean): Checked[Executable] = { - def executable = cwl match { - case Cwl.Workflow(w) => w.womExecutable(validator, inputsFile, ioFunctions, strictValidation) - case Cwl.CommandLineTool(clt) => clt.womExecutable(validator, inputsFile, ioFunctions, strictValidation) - case Cwl.ExpressionTool(et) => et.womExecutable(validator, inputsFile, ioFunctions, strictValidation) - case oh => throw new Exception(s"Programmer Error! Unexpected case match: $oh") - } - Try(executable) match { - case Success(s) => s - case Failure(f) => f.getMessage.invalidNelCheck - case oh => throw new Exception(s"Programmer Error! Unexpected case match: $oh") - } - } - - def requiredInputs: Map[String, WomType] = { - implicit val parent = ParentName.empty - - cwl match { - case Cwl.Workflow(w) => selectWomTypeInputs(w.inputs collect { - case i if i.`type`.isDefined => FullyQualifiedName(i.id).id -> i.`type`.get - }) - case Cwl.CommandLineTool(clt) => selectWomTypeInputs(clt.inputs collect { - case i if i.`type`.isDefined => FullyQualifiedName(i.id).id -> i.`type`.get - }) - case Cwl.ExpressionTool(et) => selectWomTypeInputs(et.inputs collect { - case i if i.`type`.isDefined => FullyQualifiedName(i.id).id -> i.`type`.get - }) - case oh => throw new Exception(s"Programmer Error! Unexpected case match: $oh") - } - } - - def schemaOption: Option[Schema] = cwl.fold(CwlSchemaOptionPoly) - - private def selectWomTypeInputs(myriadInputMap: Array[(String, MyriadInputType)]): Map[String, WomType] = { - (myriadInputMap collect { - case (key, MyriadInputType.WomType(w)) => key -> w - }).toMap - } - } - - object CwlSchemaOptionPoly extends Poly1 { - implicit val caseWorkflow: Case.Aux[Workflow, Option[Schema]] = at { - workflow => getSchema(workflow.`$schemas`, workflow.`$namespaces`) - } - implicit val caseCommandLineTool: Case.Aux[CommandLineTool, Option[Schema]] = at { - commandLineTool => getSchema(commandLineTool.`$schemas`, commandLineTool.`$namespaces`) - } - implicit val caseExpressionTool: Case.Aux[ExpressionTool, Option[Schema]] = at { - expressionTool => getSchema(expressionTool.`$schemas`, expressionTool.`$namespaces`) - } - - private def getSchema(schemasOption: Option[Array[String]], - namespacesOption: Option[Map[String, String]]): Option[Schema] = { - schemasOption.map(a => Schema(a.toIndexedSeq, namespacesOption getOrElse Map.empty)) - } - } - - type ExpressionLib = Vector[String] - - type Inputs = (RequirementsAndHints, ExpressionLib, WomEvaluatedCallInputs) - - type CommandPartExpression[A] = ReaderT[ErrorOr, Inputs, A] -} diff --git a/cwl/src/main/scala/cwl/preprocessor/CwlCanonicalizer.scala b/cwl/src/main/scala/cwl/preprocessor/CwlCanonicalizer.scala deleted file mode 100644 index b369d404c18..00000000000 --- a/cwl/src/main/scala/cwl/preprocessor/CwlCanonicalizer.scala +++ /dev/null @@ -1,249 +0,0 @@ -package cwl.preprocessor - -import cats.effect.{ContextShift, IO} -import cats.syntax.parallel._ -import cats.instances.list._ -import common.validation.ErrorOr.ErrorOr -import common.validation.IOChecked._ -import common.validation.Validation._ -import cwl.preprocessor.CwlReference.EnhancedCwlId -import cwl.preprocessor.CwlPreProcessor._ -import io.circe.Json -import io.circe.optics.JsonPath._ -import cwl.preprocessor.CwlCanonicalizer._ - -/** - * The real guts of the CWL pre-processor is taking a CWL reference and producing a single, self-contained JSON from it. - */ -private [preprocessor] class CwlCanonicalizer(saladFunction: SaladFunction)(implicit cs: ContextShift[IO]) { - - def getCanonicalCwl(reference: CwlReference, - namespacesJsonOption: Option[Json] = None, - schemasJsonOption: Option[Json] = None): IOChecked[Json] = { - flattenCwlReferenceInner( - reference, - Map.empty, - Map.empty, - Set.empty, - namespacesJsonOption, - schemasJsonOption).map(_.processedJson) - } - - /** - * Flatten the cwl reference given already known processed references. - */ - private def flattenCwlReferenceInner(cwlReference: CwlReference, - unProcessedReferences: UnProcessedReferences, - processedReferences: ProcessedReferences, - breadCrumbs: Set[CwlReference], - namespacesJsonOption: Option[Json], - schemasJsonOption: Option[Json]): IOChecked[ProcessedJsonAndDependencies] = { - /* - * Salad and parse from a CWL reference into a Json object - */ - def saladAndParse(ref: CwlReference): IOChecked[Json] = for { - saladed <- saladFunction(ref) - saladedJson <- parseJson(saladed) - } yield saladedJson - - for { - // parse the file containing the reference - parsed <- saladAndParse(cwlReference) - // Get a Map[CwlReference, Json] from the parsed file. If the file is a JSON object and only contains one node, the map will only have 1 element - newUnProcessedReferences = mapIdToContent(parsed).toMap - // The reference json in the file - referenceJson <- newUnProcessedReferences - .collectFirst({ case (ref, json) if ref.pointerWithinFile == cwlReference.pointerWithinFile => json }) - .toIOChecked(s"Cannot find a tool or workflow with ID '${cwlReference.pointerWithinFile}' in file ${cwlReference.pathAsString}'s set: [${newUnProcessedReferences.keySet.mkString(", ")}]") - // Process the reference json - processed <- flattenJson( - referenceJson, - newUnProcessedReferences ++ unProcessedReferences, - processedReferences, - breadCrumbs + cwlReference, - namespacesJsonOption, - schemasJsonOption - ) - } yield processed - } - - /** - * Given a Json representing a tool or workflow, flattens it and return the other processed references that were generated. - * - * NB: Flatten here means two things: - * - Find references within the CWL and convert them into 'local' links - * - Create a map of canonical links to JSON CWL content - * - * @param saladedJson json to process - * @param unProcessedReferences references that have been parsed and saladed (we have the json), but not flattened yet - * @param processedReferences references that are fully flattened - * @param breadCrumbs list of references that brought us here - * @param namespacesJsonOption Namespaces from the original json - * @param schemasJsonOption Schemas from the original json - */ - private def flattenJson(saladedJson: Json, - unProcessedReferences: UnProcessedReferences, - processedReferences: ProcessedReferences, - breadCrumbs: Set[CwlReference], - namespacesJsonOption: Option[Json], - schemasJsonOption: Option[Json]): IOChecked[ProcessedJsonAndDependencies] = { - /* - * Given a reference from a step's run field, flattens it and return it - * @param unProcessedReferences references that have been parsed and saladed (we have the json), but not flattened yet. - * @param checkedProcessedReferences references that are fully processed - * @param cwlReference reference being processed - * @return a new ProcessedReferences Map including this cwlReference processed along with all the dependencies - * that might have been processed recursively. - */ - def processCwlRunReference(checkedProcessedReferences: IOChecked[ProcessedReferences], - cwlReference: CwlReference): IOChecked[ProcessedReferences] = { - def processReference(processedReferences: ProcessedReferences) = { - - val result: IOChecked[ProcessedJsonAndDependencies] = unProcessedReferences.get(cwlReference) match { - case Some(unProcessedReferenceJson) => - // Found the json in the unprocessed map, no need to reparse the file, just flatten this json - flattenJson( - unProcessedReferenceJson, - unProcessedReferences, - processedReferences, - breadCrumbs, - namespacesJsonOption, - schemasJsonOption - ) - case None => - // This is the first time we're seeing this reference, we need to parse its file and flatten it - flattenCwlReferenceInner( - cwlReference, - unProcessedReferences, - processedReferences, - breadCrumbs, - namespacesJsonOption, - schemasJsonOption - ) - } - - result map { - // Return everything we've got (the previously known "processedReferences" + our new processed reference + everything that was processed to get to it) - case ProcessedJsonAndDependencies(processed, newReferences) => processedReferences ++ newReferences + (cwlReference -> processed) - } - } - - def processIfNeeded(processedReferences: ProcessedReferences): IOChecked[ProcessedReferences] = { - // If the reference has already been processed, no need to do anything - if (processedReferences.contains(cwlReference)) processedReferences.validIOChecked - // If the reference is in the bread crumbs it means we circled back to it: fail the pre-processing - else if (breadCrumbs.contains(cwlReference)) s"Found a circular dependency on $cwlReference".invalidIOChecked - // Otherwise let's see if we already have the json for it or if we need to process the file - else processReference(processedReferences) - } - - for { - processedReferences <- checkedProcessedReferences - newReferences <- processIfNeeded(processedReferences) - } yield newReferences - } - - def addJsonKeyValue(originalJson: Json, key: String, valueOption: Option[Json]): Json = { - valueOption match { - case Some(value) => originalJson.mapObject(_.add(key, value)) - case None => originalJson - } - } - - val namespacesJson = addJsonKeyValue(saladedJson, JsonKeyNamespaces, namespacesJsonOption) - val schemasJson = addJsonKeyValue(namespacesJson, JsonKeySchemas, schemasJsonOption) - - import cats.syntax.apply._ - - // Take the processed runs and inject them in the json - def inlineProcessedJsons(newKnownReferences: ProcessedReferences, inlinedRunWorkflows: Map[String, ProcessedJsonAndDependencies]) = { - // Provide a function to swap the run reference with its json content - val lookupFunction: Json => Json = { - json: Json => { - val fromRunReferenceMap = for { - asString <- json.asString - reference <- asString.asReference - embeddedJson <- newKnownReferences.get(reference) - } yield embeddedJson - - val fromInlinedWorkflow = for { - asObject <- json.asObject - id <- asObject.kleisli("id") - idAsString <- id.asString - embeddedJson <- inlinedRunWorkflows.get(idAsString) - } yield embeddedJson.processedJson - - fromRunReferenceMap.orElse(fromInlinedWorkflow).getOrElse(json) - } - } - - val flattenedJson = root.steps.each.run.json.modify(lookupFunction)(schemasJson) - - ProcessedJsonAndDependencies(flattenedJson, newKnownReferences ++ inlinedRunWorkflows.values.flatMap(_.processedDependencies)) - } - - /* - * Given a json, collects all "steps.run" values that are JSON Strings, and convert them to CwlReferences. - * A saladed JSON is assumed. - */ - def findRunReferences(json: Json): List[CwlReference] = { - json.asArray match { - case Some(cwls) => cwls.toList.flatMap(findRunReferences) - case _ => root.steps.each.run.string.getAll(json).flatMap(_.asReference).distinct - } - } - - /* - * Given a json, collects all "steps.run" values that are JSON Objects representing a workflow. - * A saladed JSON is assumed. - * @return a Map[String, Json], where the key is the cwl id of the workflow, and the value its content - */ - def findRunInlinedWorkflows(json: Json): ErrorOr[Map[String, Json]] = { - import cats.syntax.traverse._ - - json.asArray match { - case Some(cwls) => cwls.toList - .flatTraverse(findRunInlinedWorkflows(_).map(_.toList)) - .map(_.toMap) - case _ => - // Look for all the "run" steps that are json objects - root.steps.each.run.obj.getAll(json) - .map(Json.fromJsonObject) - // Only keep the workflows (CommandLineTools don't have steps so no need to process them) - .filter(root.`class`.string.exist(_.equalsIgnoreCase("Workflow"))) - .traverse[ErrorOr, (String, Json)]( obj => - // Find the id of the workflow - root.id.string.getOption(obj) - .toErrorOr("Programmer error: Workflow did not contain an id. Make sure the cwl has been saladed") - .map(_ -> obj) - ).map(_.toMap) - } - } - - // Recursively process the run references (where run is a string pointing to another Workflow / Tool) - // TODO: it would be nice to accumulate failures here somehow (while still folding and be able to re-use - // successfully processed references, so I don't know if ErrorOr would work) - val processedRunReferences: IOChecked[ProcessedReferences] = findRunReferences(schemasJson).foldLeft(processedReferences.validIOChecked)(processCwlRunReference) - - // Recursively process the inlined run workflows (where run is a Json object representing a workflow) - val processedInlineReferences: IOChecked[Map[String, ProcessedJsonAndDependencies]] = (for { - inlineWorkflowReferences <- findRunInlinedWorkflows(saladedJson).toIOChecked - flattenedWorkflows <- inlineWorkflowReferences.toList.parTraverse[IOChecked, (String, ProcessedJsonAndDependencies)]({ - case (id, value) => flattenJson(value, unProcessedReferences, processedReferences, breadCrumbs, namespacesJsonOption, schemasJsonOption).map(id -> _) - }) - } yield flattenedWorkflows).map(_.toMap) - - // Replace the unprocessed runs with their processed value - (processedRunReferences, processedInlineReferences).tupled.map(Function.tupled(inlineProcessedJsons)) - } -} - -private object CwlCanonicalizer { - /** - * A Cwl json that has been processed (saladed and flattened), as well as its processed dependencies. - */ - final case class ProcessedJsonAndDependencies(processedJson: Json, processedDependencies: ProcessedReferences) - - final type UnProcessedReferences = Map[CwlReference, Json] - final type ProcessedReferences = Map[CwlReference, Json] -} diff --git a/cwl/src/main/scala/cwl/preprocessor/CwlPreProcessor.scala b/cwl/src/main/scala/cwl/preprocessor/CwlPreProcessor.scala deleted file mode 100644 index 9855ee73953..00000000000 --- a/cwl/src/main/scala/cwl/preprocessor/CwlPreProcessor.scala +++ /dev/null @@ -1,236 +0,0 @@ -package cwl.preprocessor - -import java.util.concurrent.Executors - -import cats.data.NonEmptyList -import cats.effect.{ContextShift, IO} -import cats.syntax.either._ -import common.validation.IOChecked._ -import cwl.CwlDecoder -import cwl.ontology.Schema -import cwl.preprocessor.CwlPreProcessor._ -import cwl.preprocessor.CwlReference.EnhancedCwlId -import io.circe.optics.JsonPath._ -import io.circe.{Json, JsonNumber, JsonObject} -import mouse.all._ -import org.slf4j.LoggerFactory -import wom.util.YamlUtils - -import scala.concurrent.ExecutionContext - -/** - * Class to create a standalone version of a CWL file. - * - * NB: Want to use the pre-processor? Use preProcessCwl(ref: CwlReference) - * - * @param saladFunction function that takes a file and produce a saladed version of the content - */ -class CwlPreProcessor(saladFunction: SaladFunction = saladCwlFile) { - - private val ec: ExecutionContext = ExecutionContext.fromExecutor(Executors.newFixedThreadPool(5)) - private implicit val cs = IO.contextShift(ec) - - /** - * This is THE main entry point into the CWL pre-processor. Takes a CWL reference and - * returns a canonical JSON version with all references resolved. - * - * @param ref The reference to the CWL to pre-process - * @return A canonical JSON representation of the CWL with all internal references expanded in-place - */ - def preProcessCwl(ref: CwlReference): IOChecked[Json] = ref match { - case file: CwlFileReference => preProcessCwlFile(file) - case other => preProcessRemoteCwl(other) - } - - /** - * Convenience method to get the processed workflow as a string. - */ - def preProcessCwlToString(cwlReference: CwlReference): IOChecked[String] = preProcessCwl(cwlReference).map(_.printCompact) - - def preProcessInputFiles(inputContent: String, mappingFunction: String => String): IOChecked[String] = for { - parsed <- parseYaml(inputContent) - mapped = parsed |> mapFilesAndDirectories(mappingFunction) |> mapNumbers - } yield mapped.printCompact - - /** - * Pre-process a CWL file and create a standalone, runnable (given proper inputs), inlined version of its content. - * - * The general idea is to work on CwlReferences, starting from the one coming to this function in the form of file and optional root. - * The goal is to look at the steps in this workflow that point to other references, and recursively flatten them until we can replace the step with - * its flat version. - * - * There are 3 pieces of information that are carried around during this process: - * 1) ProcessedReferences: A Map[CwlReference, Json] of CwlReference for which we have the fully processed (saladed AND flattened) Json value. - * - * 2) UnProcessedReferences: A Map[CwlReference, Json] of CwlReference for which we have the saladed but NOT flattened Json value. - * This can happen because a file can contain multiple tools / workflows. When we salad / parse this file, we get (CwlReference, Json) pairs - * for all the workflow / tools in the file, but they are not flattened yet. - * We keep this to avoid having to re-salad / re-parse files unnecessarily. - * - * 3) BreadCrumb: A Set[CwlReference] used to follow the trail of CwlReferences that we are processing as we recurse down. - * This is used to be able to detect circular dependencies (if the cwlReference being processed is in that set, then we have a circular dependency) . - * - */ - private def preProcessCwlFile(reference: CwlFileReference): IOChecked[Json] = { - - def absoluteSchemaPaths(json: Json): Json = { - json mapArray { - _ map absoluteSchemaPaths - } mapString { - Schema.getIriPath(reference.fullReference, _) - } - } - - // NB the JSON here is only used to decide whether or not to flatten. If we do decide to flatten we throw away the - // json and request a canonical version from the CwlCanonicalizer. - def flattenOrByPass(json: Json): IOChecked[Json] = { - def flatten(json: Json): IOChecked[Json] = { - val cwlReferenceFlattener = new CwlCanonicalizer(saladFunction) - val namespacesJsonOption: Option[Json] = json.asObject.flatMap(_.kleisli(JsonKeyNamespaces)) - val schemasJsonOption: Option[Json] = json.asObject.flatMap(_.kleisli(JsonKeySchemas)).map(absoluteSchemaPaths) - - cwlReferenceFlattener.getCanonicalCwl( - reference, - namespacesJsonOption, - schemasJsonOption - ) - } - - def bypass(alreadyCanonicalJson: Json): IOChecked[Json] = alreadyCanonicalJson.validIOChecked - - val fileContentReference = for { - asObject <- json.asObject - fileContentId <- asObject.kleisli("id") - stringId <- fileContentId.asString - fileContentReference <- CwlReference.fromString(stringId) - } yield fileContentReference - - fileContentReference match { - // This by passes the pre-processing if the file already has an id for which the file part doesn't match the path of the file - // passed to this function, as this would indicate that it has already been saladed and pre-processed. - case Some(CwlFileReference(file, _)) if !file.equals(reference.file) => bypass(json) - case _ => flatten(json) - } - } - - for { - original <- parseYaml(reference.file.contentAsString) - flattened <- flattenOrByPass(original) - } yield flattened - } - - // Like 'File', except that we don't read any contents before passing the path over to cwltool to canonicalize. - private def preProcessRemoteCwl(reference: CwlReference)(implicit cs: ContextShift[IO]): IOChecked[Json] = { - val cwlCanonicalizer = new CwlCanonicalizer(saladFunction) - cwlCanonicalizer.getCanonicalCwl(reference) - } -} - -object CwlPreProcessor { - private val Log = LoggerFactory.getLogger("CwlPreProcessor") - - private [preprocessor] type SaladFunction = CwlReference => IOChecked[String] - private [preprocessor] val JsonKeyNamespaces = s"$$namespaces" - private [preprocessor] val JsonKeySchemas = s"$$schemas" - - private def saladSpinner(doLogging: Boolean): SaladFunction = ref => { - if (doLogging) { - Log.info(s"Pre-Processing ${ref.pathAsString}") - } - - CwlDecoder.saladCwlFile(ref) - } - - private [preprocessor] val saladCwlFile: SaladFunction = saladSpinner(true) - private val saladCwlFileWithoutLogging: SaladFunction = saladSpinner(false) - - implicit class PrintableJson(val json: Json) extends AnyVal { - def printCompact = io.circe.Printer.noSpaces.print(json) - } - - def noLogging = new CwlPreProcessor(saladCwlFileWithoutLogging) - - // Fold over a json recursively and prefix all files - def mapFilesAndDirectories(mappingFunction: String => String)(json: Json): Json = { - // Function to check if the given json has the provided key / value pair - def hasKeyValue(key: String, value: String): Json => Boolean = { - root.selectDynamic(key).string.exist(_.equalsIgnoreCase(value)) - } - - // Return true if the given json object represents a File - def isFile(obj: JsonObject) = hasKeyValue("class", "File")(Json.fromJsonObject(obj)) - - // Return true if the given json object represents a Directory - def isDirectory(obj: JsonObject) = hasKeyValue("class", "Directory")(Json.fromJsonObject(obj)) - - // Modify the string at "key" using the mappingFunction - def mapStringValue(key: String, mappingFunction: String => String): Json => Json = root.selectDynamic(key).string.modify(mappingFunction) - - // Map "location" and "default" - def prefix(mappingFunction: String => String): Json => Json = mapStringValue("location", mappingFunction).compose(mapStringValue("path", mappingFunction)) - - // Prefix the location or path in the json object if it's a file or directory, otherwise recurse over its fields - def prefixObject(mappingFunction: String => String)(obj: JsonObject): Json = { - // If the object is file or a directory, prefix it with the gcs prefix - if (isFile(obj) || isDirectory(obj)) { - prefix(mappingFunction)(Json.fromJsonObject(obj)) - // Even if it's a file it may have secondary files. So keep recursing on its fields - .mapObject(_.mapValues(mapFilesAndDirectories(mappingFunction))) - } - // Otherwise recursively process its fields - else Json.fromJsonObject(obj.mapValues(mapFilesAndDirectories(mappingFunction))) - } - - json.fold( - jsonNull = json, - jsonBoolean = _ => json, - jsonNumber = _ => json, - jsonString = _ => json, - jsonObject = prefixObject(mappingFunction), - jsonArray = arr => Json.arr(arr.map(mapFilesAndDirectories(mappingFunction)): _*) - ) - } - - private [preprocessor] def mapNumbers(json: Json): Json = { - // Circumvent Circe's scientific format for numbers: convert to a JSON String without exponential notation. - def nonScientificNumberFormatting(jsonNumber: JsonNumber): Json = { - val conversions = LazyList[JsonNumber => Option[Any]]( - _.toBigInt.map(_.longValue), - _.toBigDecimal.map(_.doubleValue), - Function.const(Option("null"))) - - // The `get` is safe because `Option("null")` guarantees a match even if the other two Stream elements - // do not satisfy the predicate. - conversions.map(_.apply(jsonNumber)).find(_.isDefined).flatten.get.toString |> Json.fromString - } - - json.fold( - jsonNull = json, - jsonBoolean = _ => json, - jsonNumber = nonScientificNumberFormatting, - jsonString = _ => json, - jsonObject = _.mapValues(mapNumbers) |> Json.fromJsonObject, - jsonArray = _.map(mapNumbers) |> Json.fromValues - ) - } - - private [preprocessor] def parseJson(in: String): IOChecked[Json] = { - io.circe.parser.parse(in).leftMap(error => NonEmptyList.one(error.message)).toIOChecked - } - - private [preprocessor] def parseYaml(in: String): IOChecked[Json] = { - val yaml = YamlUtils.parse(in) - yaml.leftMap(error => NonEmptyList.one(error.message)).toIOChecked - } - - /** - * Given a json, collect all tools or workflows and map them with their reference id. - * A saladed JSON is assumed. - */ - private [preprocessor] def mapIdToContent(json: Json): List[(CwlReference, Json)] = { - json.asArray match { - case Some(cwls) => cwls.toList.flatMap(mapIdToContent) - case None => root.id.string.getOption(json).flatMap(_.asReference).map(_ -> json).toList - } - } -} diff --git a/cwl/src/main/scala/cwl/preprocessor/CwlReference.scala b/cwl/src/main/scala/cwl/preprocessor/CwlReference.scala deleted file mode 100644 index badddf028b4..00000000000 --- a/cwl/src/main/scala/cwl/preprocessor/CwlReference.scala +++ /dev/null @@ -1,82 +0,0 @@ -package cwl.preprocessor - -import better.files.{File => BFile} -import cwl.preprocessor.CwlReference._ -import cwl.{FileAndId, FullyQualifiedName} -import cwl.command.ParentName - -sealed trait CwlReference { - def pathAsString: String - def pointerWithinFile: Option[String] - def changePointer(to: Option[String]): CwlReference - - private def pointerWithHash: String = pointerWithinFile.map(p => s"#$p").getOrElse("") - lazy val fullReference: String = s"$pathAsString$pointerWithHash" - - override def toString: String = fullReference -} - -/** - * Saladed CWLs reference other local CWL "node" (workflow or tool) using a URI as follow: - * file:///path/to/file/containing/node.cwl[#pointer_to_node] - * #pointer_to_node to node is optional, and will specify which workflow or tool is being targeted in the file. - * - * e.g: - * { - * "class": "Workflow", - * "id": "file:///path/to/workflow/workflow.cwl", - * ... - * "steps": [ - * { - * "run": "file:///path/to/workflow/multi_tools.cwl#my_tool", - * ... - * } - * ] - * } - * - * This snippet contains 2 references, one that is the ID of this workflow, the other one is the run step pointing to "my_tool" in "/path/to/workflow/multi_tools.cwl" - * - */ -final case class CwlFileReference(file: BFile, pointerWithinFile: Option[String]) extends CwlReference { - override val pathAsString: String = s"$LocalScheme${file.toString}" - override def changePointer(to: Option[String]): CwlReference = this.copy(pointerWithinFile = to) -} - -final case class CwlHttpReference(pathAsString: String, pointerWithinFile: Option[String]) extends CwlReference { - override def changePointer(to: Option[String]): CwlReference = this.copy(pointerWithinFile = to) -} - -object CwlReference { - val LocalScheme = "file://" - val HttpScheme = "http://" - val HttpsScheme = "https://" - - implicit class EnhancedCwlId(val id: String) extends AnyVal { - def asReference: Option[CwlReference] = CwlReference.fromString(id) - def stripFilePrefix = id.stripPrefix(LocalScheme) - } - - val ReferenceRegex = "(.*://)?([^#]*)(#(.*))?".r - - def fromString(in: String): Option[CwlReference] = { - in match { - case ReferenceRegex(scheme, path, _, pointerWithinFile) => - if (scheme == LocalScheme) { - FullyQualifiedName.maybeApply(in)(ParentName.empty) match { - case Some(FileAndId(file, _, _)) => Option(CwlFileReference(BFile(file.stripFilePrefix), Option(pointerWithinFile))) - case _ => Option(CwlFileReference(BFile(in.stripFilePrefix), Option(pointerWithinFile))) - } - } else if (scheme == HttpScheme || scheme == HttpsScheme) { - Option(CwlHttpReference(s"$scheme$path", Option(pointerWithinFile))) - } else { - None - } - } - } -} - -object CwlFileReference { - def apply(file: BFile, pointer: Option[String]) = { - new CwlFileReference(file, pointer) - } -} diff --git a/cwl/src/main/scala/cwl/requirement/RequirementToAttributeMap.scala b/cwl/src/main/scala/cwl/requirement/RequirementToAttributeMap.scala deleted file mode 100644 index 1a727837172..00000000000 --- a/cwl/src/main/scala/cwl/requirement/RequirementToAttributeMap.scala +++ /dev/null @@ -1,79 +0,0 @@ -package cwl.requirement - -import cwl._ -import shapeless.Poly1 -import wom.RuntimeAttributesKeys._ -import wom.expression.{ValueAsAnExpression, WomExpression} -import wom.values.{WomLong, WomString} - -object RequirementToAttributeMap extends Poly1 { - type ResourcesToExpressionMap = (Set[String], ExpressionLib) => Map[String, WomExpression] - implicit def fromJs: Case.Aux[InlineJavascriptRequirement, ResourcesToExpressionMap] = at[InlineJavascriptRequirement] { - _ => (_,_) => Map.empty - } - - implicit def fromSchemaDef: Case.Aux[SchemaDefRequirement, ResourcesToExpressionMap] = at[SchemaDefRequirement] { - _ => (_,_) => Map.empty - } - - implicit def fromDocker: Case.Aux[DockerRequirement, ResourcesToExpressionMap] = at[DockerRequirement] { - docker => (_,_) => docker.dockerPull.orElse(docker.dockerImageId).map({ pull => - DockerKey -> ValueAsAnExpression(WomString(pull)) - }).toMap - } - - implicit def fromSoftware: Case.Aux[SoftwareRequirement, ResourcesToExpressionMap] = at[SoftwareRequirement] { - _ => (_,_) => Map.empty - } - - implicit def fromInitialWorkDir: Case.Aux[InitialWorkDirRequirement, ResourcesToExpressionMap] = at[InitialWorkDirRequirement] { - _ => (_,_) => Map.empty - } - - implicit def fromEnvVar: Case.Aux[EnvVarRequirement, ResourcesToExpressionMap] = at[EnvVarRequirement] { - _ => (_,_) => Map.empty - } - - implicit def fromShellCommand: Case.Aux[ShellCommandRequirement, ResourcesToExpressionMap] = at[ShellCommandRequirement] { - _ => (_,_) => Map.empty - } - - implicit def fromResource: Case.Aux[ResourceRequirement, ResourcesToExpressionMap] = at[ResourceRequirement] { - resource => (inputNames, expressionLib) => - def toExpression(resourceRequirement: ResourceRequirementType) = - resourceRequirement.fold(ResourceRequirementToWomExpression).apply(inputNames, expressionLib) - - List( - // Map cpuMin to both cpuMin and cpu keys - resource.effectiveCoreMin.toList.map(toExpression).flatMap(min => List(CpuMinKey -> min, CpuKey -> min)), - resource.effectiveCoreMax.toList.map(toExpression).map(CpuMaxKey -> _), - // Map ramMin to both memoryMin and memory keys - resource.effectiveRamMin.toList.map(toExpression).flatMap(min => List(MemoryMinKey -> min, MemoryKey -> min)), - resource.effectiveRamMax.toList.map(toExpression).map(MemoryMaxKey -> _), - resource.effectiveTmpdirMin.toList.map(toExpression).map(TmpDirMinKey -> _), - resource.effectiveTmpdirMax.toList.map(toExpression).map(TmpDirMaxKey -> _), - resource.effectiveOutdirMin.toList.map(toExpression).map(OutDirMinKey -> _), - resource.effectiveOutdirMax.toList.map(toExpression).map(OutDirMaxKey -> _) - ).flatten.toMap - } - - implicit def fromInputResourceRequirement: Case.Aux[DnaNexusInputResourceRequirement, ResourcesToExpressionMap] = at[DnaNexusInputResourceRequirement] { - case DnaNexusInputResourceRequirement(_, indirMin) => (_, _) => indirMin.map(value => ValueAsAnExpression(WomLong(value))).map(DnaNexusInputDirMinKey -> _).toMap - } - - implicit def fromSubWorkflow: Case.Aux[SubworkflowFeatureRequirement, ResourcesToExpressionMap] = at[SubworkflowFeatureRequirement] { - _ => (_,_) => Map.empty - } - - implicit def fromScatter: Case.Aux[ScatterFeatureRequirement, ResourcesToExpressionMap] = at[ScatterFeatureRequirement] { - _ => (_,_) => Map.empty - } - - implicit def fromMultipleInput: Case.Aux[MultipleInputFeatureRequirement, ResourcesToExpressionMap] = at[MultipleInputFeatureRequirement] { - _ => (_,_) => Map.empty - } - - implicit def fromStepInput: Case.Aux[StepInputExpressionRequirement, ResourcesToExpressionMap] = at[StepInputExpressionRequirement] { - _ => (_,_) => Map.empty - } -} diff --git a/cwl/src/main/scala/cwl/requirement/ResourceRequirementToWomExpression.scala b/cwl/src/main/scala/cwl/requirement/ResourceRequirementToWomExpression.scala deleted file mode 100644 index 74727947657..00000000000 --- a/cwl/src/main/scala/cwl/requirement/ResourceRequirementToWomExpression.scala +++ /dev/null @@ -1,15 +0,0 @@ -package cwl.requirement - -import cwl.{Expression, ExpressionLib} -import shapeless.Poly1 -import wom.expression.{ValueAsAnExpression, WomExpression} -import wom.values.{WomLong, WomString} - -object ResourceRequirementToWomExpression extends Poly1 { - type ResourceRequirementStringSetToWomExpression = (Set[String], ExpressionLib) => WomExpression - implicit def fromLong: Case.Aux[Long, ResourceRequirementStringSetToWomExpression] = at[Long] { l => (_, _) => ValueAsAnExpression(WomLong(l)) } - implicit def fromString: Case.Aux[String, ResourceRequirementStringSetToWomExpression] = at[String] { s => (_, _) => ValueAsAnExpression(WomString(s)) } - implicit def fromExpression: Case.Aux[Expression, ResourceRequirementStringSetToWomExpression] = at[Expression] { e => (inputs, expressionLib) => - cwl.ECMAScriptWomExpression(e, inputs, expressionLib) - } -} diff --git a/cwl/src/test/resources/1st-tool.cwl b/cwl/src/test/resources/1st-tool.cwl deleted file mode 100755 index af0c4de297d..00000000000 --- a/cwl/src/test/resources/1st-tool.cwl +++ /dev/null @@ -1,9 +0,0 @@ -cwlVersion: v1.0 -class: CommandLineTool -baseCommand: echo -inputs: - message: - type: string - inputBinding: - position: 1 -outputs: [] diff --git a/cwl/src/test/resources/1st-workflow.cwl b/cwl/src/test/resources/1st-workflow.cwl deleted file mode 100644 index ce4b4b0bd71..00000000000 --- a/cwl/src/test/resources/1st-workflow.cwl +++ /dev/null @@ -1,23 +0,0 @@ -cwlVersion: v1.0 -class: Workflow -inputs: - inp: File - ex: string -outputs: - classout: - type: File - outputSource: compile/classfile - -steps: - untar: - run: tar-param.cwl - in: - tarfile: inp - extractfile: ex - out: [example_out] - - compile: - run: arguments.cwl - in: - src: untar/example_out - out: [classfile] diff --git a/cwl/src/test/resources/application.conf b/cwl/src/test/resources/application.conf deleted file mode 100644 index 8d12b670cd3..00000000000 --- a/cwl/src/test/resources/application.conf +++ /dev/null @@ -1,14 +0,0 @@ -akka { - log-dead-letters = "off" - loggers = ["akka.event.slf4j.Slf4jLogger"] -} - -ontology { - # Uncomment to enable caching of ontologies. Improves performance when loading ontologies from remote IRIs. - #cache { - # max-size = 20 - #} - retries = 3 - pool-size = 3 - backoff-time = 2 seconds -} diff --git a/cwl/src/test/resources/arguments.cwl b/cwl/src/test/resources/arguments.cwl deleted file mode 100644 index 730030e7437..00000000000 --- a/cwl/src/test/resources/arguments.cwl +++ /dev/null @@ -1,22 +0,0 @@ -cwlVersion: "v1.0" -class: "CommandLineTool" -label: "Example trivial wrapper for Java 7 compiler" -hints: - - dockerPull: "java:7-jdk" - class: "DockerRequirement" -baseCommand: "javac" -arguments: - - "-d" - - "$(runtime.outdir)" -inputs: - - type: "File" - inputBinding: - position: 1 - id: "file:///home/dan/wdl4s/arguments.cwl#src" -outputs: - - type: "File" - outputBinding: - glob: "*.class" - id: "file:///home/dan/wdl4s/arguments.cwl#classfile" -id: "file:///home/dan/wdl4s/arguments.cwl" -name: "file:///home/dan/wdl4s/arguments.cwl" diff --git a/cwl/src/test/resources/bad.cwl b/cwl/src/test/resources/bad.cwl deleted file mode 100644 index d38ed298731..00000000000 --- a/cwl/src/test/resources/bad.cwl +++ /dev/null @@ -1 +0,0 @@ -gibberish diff --git a/cwl/src/test/resources/bad2.cwl b/cwl/src/test/resources/bad2.cwl deleted file mode 100644 index d38ed298731..00000000000 --- a/cwl/src/test/resources/bad2.cwl +++ /dev/null @@ -1 +0,0 @@ -gibberish diff --git a/cwl/src/test/resources/brokenlinks.cwl b/cwl/src/test/resources/brokenlinks.cwl deleted file mode 100644 index c710358ce61..00000000000 --- a/cwl/src/test/resources/brokenlinks.cwl +++ /dev/null @@ -1,23 +0,0 @@ -cwlVersion: v1.0 -class: Workflow -inputs: - inp: File - ex: string -outputs: - classout: - type: File - outputSource: compile/classfile - -steps: - untar: - run: wrong.cwl - in: - tarfile: inp - extractfile: ex - out: [example_out] - - compile: - run: wrong2.cwl - in: - src: untar/example_out - out: [classfile] diff --git a/cwl/src/test/resources/cwl/lodash.js b/cwl/src/test/resources/cwl/lodash.js deleted file mode 100644 index 9b95dfefe87..00000000000 --- a/cwl/src/test/resources/cwl/lodash.js +++ /dev/null @@ -1,17112 +0,0 @@ -/** - * @license - * Lodash - * Copyright OpenJS Foundation and other contributors - * Released under MIT license - * Based on Underscore.js 1.8.3 - * Copyright Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors - */ -;(function() { - - /** Used as a safe reference for `undefined` in pre-ES5 environments. */ - var undefined; - - /** Used as the semantic version number. */ - var VERSION = '4.17.15'; - - /** Used as the size to enable large array optimizations. */ - var LARGE_ARRAY_SIZE = 200; - - /** Error message constants. */ - var CORE_ERROR_TEXT = 'Unsupported core-js use. Try https://npms.io/search?q=ponyfill.', - FUNC_ERROR_TEXT = 'Expected a function'; - - /** Used to stand-in for `undefined` hash values. */ - var HASH_UNDEFINED = '__lodash_hash_undefined__'; - - /** Used as the maximum memoize cache size. */ - var MAX_MEMOIZE_SIZE = 500; - - /** Used as the internal argument placeholder. */ - var PLACEHOLDER = '__lodash_placeholder__'; - - /** Used to compose bitmasks for cloning. */ - var CLONE_DEEP_FLAG = 1, - CLONE_FLAT_FLAG = 2, - CLONE_SYMBOLS_FLAG = 4; - - /** Used to compose bitmasks for value comparisons. */ - var COMPARE_PARTIAL_FLAG = 1, - COMPARE_UNORDERED_FLAG = 2; - - /** Used to compose bitmasks for function metadata. */ - var WRAP_BIND_FLAG = 1, - WRAP_BIND_KEY_FLAG = 2, - WRAP_CURRY_BOUND_FLAG = 4, - WRAP_CURRY_FLAG = 8, - WRAP_CURRY_RIGHT_FLAG = 16, - WRAP_PARTIAL_FLAG = 32, - WRAP_PARTIAL_RIGHT_FLAG = 64, - WRAP_ARY_FLAG = 128, - WRAP_REARG_FLAG = 256, - WRAP_FLIP_FLAG = 512; - - /** Used as default options for `_.truncate`. */ - var DEFAULT_TRUNC_LENGTH = 30, - DEFAULT_TRUNC_OMISSION = '...'; - - /** Used to detect hot functions by number of calls within a span of milliseconds. */ - var HOT_COUNT = 800, - HOT_SPAN = 16; - - /** Used to indicate the type of lazy iteratees. */ - var LAZY_FILTER_FLAG = 1, - LAZY_MAP_FLAG = 2, - LAZY_WHILE_FLAG = 3; - - /** Used as references for various `Number` constants. */ - var INFINITY = 1 / 0, - MAX_SAFE_INTEGER = 9007199254740991, - MAX_INTEGER = 1.7976931348623157e+308, - NAN = 0 / 0; - - /** Used as references for the maximum length and index of an array. */ - var MAX_ARRAY_LENGTH = 4294967295, - MAX_ARRAY_INDEX = MAX_ARRAY_LENGTH - 1, - HALF_MAX_ARRAY_LENGTH = MAX_ARRAY_LENGTH >>> 1; - - /** Used to associate wrap methods with their bit flags. */ - var wrapFlags = [ - ['ary', WRAP_ARY_FLAG], - ['bind', WRAP_BIND_FLAG], - ['bindKey', WRAP_BIND_KEY_FLAG], - ['curry', WRAP_CURRY_FLAG], - ['curryRight', WRAP_CURRY_RIGHT_FLAG], - ['flip', WRAP_FLIP_FLAG], - ['partial', WRAP_PARTIAL_FLAG], - ['partialRight', WRAP_PARTIAL_RIGHT_FLAG], - ['rearg', WRAP_REARG_FLAG] - ]; - - /** `Object#toString` result references. */ - var argsTag = '[object Arguments]', - arrayTag = '[object Array]', - asyncTag = '[object AsyncFunction]', - boolTag = '[object Boolean]', - dateTag = '[object Date]', - domExcTag = '[object DOMException]', - errorTag = '[object Error]', - funcTag = '[object Function]', - genTag = '[object GeneratorFunction]', - mapTag = '[object Map]', - numberTag = '[object Number]', - nullTag = '[object Null]', - objectTag = '[object Object]', - promiseTag = '[object Promise]', - proxyTag = '[object Proxy]', - regexpTag = '[object RegExp]', - setTag = '[object Set]', - stringTag = '[object String]', - symbolTag = '[object Symbol]', - undefinedTag = '[object Undefined]', - weakMapTag = '[object WeakMap]', - weakSetTag = '[object WeakSet]'; - - var arrayBufferTag = '[object ArrayBuffer]', - dataViewTag = '[object DataView]', - float32Tag = '[object Float32Array]', - float64Tag = '[object Float64Array]', - int8Tag = '[object Int8Array]', - int16Tag = '[object Int16Array]', - int32Tag = '[object Int32Array]', - uint8Tag = '[object Uint8Array]', - uint8ClampedTag = '[object Uint8ClampedArray]', - uint16Tag = '[object Uint16Array]', - uint32Tag = '[object Uint32Array]'; - - /** Used to match empty string literals in compiled template source. */ - var reEmptyStringLeading = /\b__p \+= '';/g, - reEmptyStringMiddle = /\b(__p \+=) '' \+/g, - reEmptyStringTrailing = /(__e\(.*?\)|\b__t\)) \+\n'';/g; - - /** Used to match HTML entities and HTML characters. */ - var reEscapedHtml = /&(?:amp|lt|gt|quot|#39);/g, - reUnescapedHtml = /[&<>"']/g, - reHasEscapedHtml = RegExp(reEscapedHtml.source), - reHasUnescapedHtml = RegExp(reUnescapedHtml.source); - - /** Used to match template delimiters. */ - var reEscape = /<%-([\s\S]+?)%>/g, - reEvaluate = /<%([\s\S]+?)%>/g, - reInterpolate = /<%=([\s\S]+?)%>/g; - - /** Used to match property names within property paths. */ - var reIsDeepProp = /\.|\[(?:[^[\]]*|(["'])(?:(?!\1)[^\\]|\\.)*?\1)\]/, - reIsPlainProp = /^\w*$/, - rePropName = /[^.[\]]+|\[(?:(-?\d+(?:\.\d+)?)|(["'])((?:(?!\2)[^\\]|\\.)*?)\2)\]|(?=(?:\.|\[\])(?:\.|\[\]|$))/g; - - /** - * Used to match `RegExp` - * [syntax characters](http://ecma-international.org/ecma-262/7.0/#sec-patterns). - */ - var reRegExpChar = /[\\^$.*+?()[\]{}|]/g, - reHasRegExpChar = RegExp(reRegExpChar.source); - - /** Used to match leading and trailing whitespace. */ - var reTrim = /^\s+|\s+$/g, - reTrimStart = /^\s+/, - reTrimEnd = /\s+$/; - - /** Used to match wrap detail comments. */ - var reWrapComment = /\{(?:\n\/\* \[wrapped with .+\] \*\/)?\n?/, - reWrapDetails = /\{\n\/\* \[wrapped with (.+)\] \*/, - reSplitDetails = /,? & /; - - /** Used to match words composed of alphanumeric characters. */ - var reAsciiWord = /[^\x00-\x2f\x3a-\x40\x5b-\x60\x7b-\x7f]+/g; - - /** Used to match backslashes in property paths. */ - var reEscapeChar = /\\(\\)?/g; - - /** - * Used to match - * [ES template delimiters](http://ecma-international.org/ecma-262/7.0/#sec-template-literal-lexical-components). - */ - var reEsTemplate = /\$\{([^\\}]*(?:\\.[^\\}]*)*)\}/g; - - /** Used to match `RegExp` flags from their coerced string values. */ - var reFlags = /\w*$/; - - /** Used to detect bad signed hexadecimal string values. */ - var reIsBadHex = /^[-+]0x[0-9a-f]+$/i; - - /** Used to detect binary string values. */ - var reIsBinary = /^0b[01]+$/i; - - /** Used to detect host constructors (Safari). */ - var reIsHostCtor = /^\[object .+?Constructor\]$/; - - /** Used to detect octal string values. */ - var reIsOctal = /^0o[0-7]+$/i; - - /** Used to detect unsigned integer values. */ - var reIsUint = /^(?:0|[1-9]\d*)$/; - - /** Used to match Latin Unicode letters (excluding mathematical operators). */ - var reLatin = /[\xc0-\xd6\xd8-\xf6\xf8-\xff\u0100-\u017f]/g; - - /** Used to ensure capturing order of template delimiters. */ - var reNoMatch = /($^)/; - - /** Used to match unescaped characters in compiled string literals. */ - var reUnescapedString = /['\n\r\u2028\u2029\\]/g; - - /** Used to compose unicode character classes. */ - var rsAstralRange = '\\ud800-\\udfff', - rsComboMarksRange = '\\u0300-\\u036f', - reComboHalfMarksRange = '\\ufe20-\\ufe2f', - rsComboSymbolsRange = '\\u20d0-\\u20ff', - rsComboRange = rsComboMarksRange + reComboHalfMarksRange + rsComboSymbolsRange, - rsDingbatRange = '\\u2700-\\u27bf', - rsLowerRange = 'a-z\\xdf-\\xf6\\xf8-\\xff', - rsMathOpRange = '\\xac\\xb1\\xd7\\xf7', - rsNonCharRange = '\\x00-\\x2f\\x3a-\\x40\\x5b-\\x60\\x7b-\\xbf', - rsPunctuationRange = '\\u2000-\\u206f', - rsSpaceRange = ' \\t\\x0b\\f\\xa0\\ufeff\\n\\r\\u2028\\u2029\\u1680\\u180e\\u2000\\u2001\\u2002\\u2003\\u2004\\u2005\\u2006\\u2007\\u2008\\u2009\\u200a\\u202f\\u205f\\u3000', - rsUpperRange = 'A-Z\\xc0-\\xd6\\xd8-\\xde', - rsVarRange = '\\ufe0e\\ufe0f', - rsBreakRange = rsMathOpRange + rsNonCharRange + rsPunctuationRange + rsSpaceRange; - - /** Used to compose unicode capture groups. */ - var rsApos = "['\u2019]", - rsAstral = '[' + rsAstralRange + ']', - rsBreak = '[' + rsBreakRange + ']', - rsCombo = '[' + rsComboRange + ']', - rsDigits = '\\d+', - rsDingbat = '[' + rsDingbatRange + ']', - rsLower = '[' + rsLowerRange + ']', - rsMisc = '[^' + rsAstralRange + rsBreakRange + rsDigits + rsDingbatRange + rsLowerRange + rsUpperRange + ']', - rsFitz = '\\ud83c[\\udffb-\\udfff]', - rsModifier = '(?:' + rsCombo + '|' + rsFitz + ')', - rsNonAstral = '[^' + rsAstralRange + ']', - rsRegional = '(?:\\ud83c[\\udde6-\\uddff]){2}', - rsSurrPair = '[\\ud800-\\udbff][\\udc00-\\udfff]', - rsUpper = '[' + rsUpperRange + ']', - rsZWJ = '\\u200d'; - - /** Used to compose unicode regexes. */ - var rsMiscLower = '(?:' + rsLower + '|' + rsMisc + ')', - rsMiscUpper = '(?:' + rsUpper + '|' + rsMisc + ')', - rsOptContrLower = '(?:' + rsApos + '(?:d|ll|m|re|s|t|ve))?', - rsOptContrUpper = '(?:' + rsApos + '(?:D|LL|M|RE|S|T|VE))?', - reOptMod = rsModifier + '?', - rsOptVar = '[' + rsVarRange + ']?', - rsOptJoin = '(?:' + rsZWJ + '(?:' + [rsNonAstral, rsRegional, rsSurrPair].join('|') + ')' + rsOptVar + reOptMod + ')*', - rsOrdLower = '\\d*(?:1st|2nd|3rd|(?![123])\\dth)(?=\\b|[A-Z_])', - rsOrdUpper = '\\d*(?:1ST|2ND|3RD|(?![123])\\dTH)(?=\\b|[a-z_])', - rsSeq = rsOptVar + reOptMod + rsOptJoin, - rsEmoji = '(?:' + [rsDingbat, rsRegional, rsSurrPair].join('|') + ')' + rsSeq, - rsSymbol = '(?:' + [rsNonAstral + rsCombo + '?', rsCombo, rsRegional, rsSurrPair, rsAstral].join('|') + ')'; - - /** Used to match apostrophes. */ - var reApos = RegExp(rsApos, 'g'); - - /** - * Used to match [combining diacritical marks](https://en.wikipedia.org/wiki/Combining_Diacritical_Marks) and - * [combining diacritical marks for symbols](https://en.wikipedia.org/wiki/Combining_Diacritical_Marks_for_Symbols). - */ - var reComboMark = RegExp(rsCombo, 'g'); - - /** Used to match [string symbols](https://mathiasbynens.be/notes/javascript-unicode). */ - var reUnicode = RegExp(rsFitz + '(?=' + rsFitz + ')|' + rsSymbol + rsSeq, 'g'); - - /** Used to match complex or compound words. */ - var reUnicodeWord = RegExp([ - rsUpper + '?' + rsLower + '+' + rsOptContrLower + '(?=' + [rsBreak, rsUpper, '$'].join('|') + ')', - rsMiscUpper + '+' + rsOptContrUpper + '(?=' + [rsBreak, rsUpper + rsMiscLower, '$'].join('|') + ')', - rsUpper + '?' + rsMiscLower + '+' + rsOptContrLower, - rsUpper + '+' + rsOptContrUpper, - rsOrdUpper, - rsOrdLower, - rsDigits, - rsEmoji - ].join('|'), 'g'); - - /** Used to detect strings with [zero-width joiners or code points from the astral planes](http://eev.ee/blog/2015/09/12/dark-corners-of-unicode/). */ - var reHasUnicode = RegExp('[' + rsZWJ + rsAstralRange + rsComboRange + rsVarRange + ']'); - - /** Used to detect strings that need a more robust regexp to match words. */ - var reHasUnicodeWord = /[a-z][A-Z]|[A-Z]{2}[a-z]|[0-9][a-zA-Z]|[a-zA-Z][0-9]|[^a-zA-Z0-9 ]/; - - /** Used to assign default `context` object properties. */ - var contextProps = [ - 'Array', 'Buffer', 'DataView', 'Date', 'Error', 'Float32Array', 'Float64Array', - 'Function', 'Int8Array', 'Int16Array', 'Int32Array', 'Map', 'Math', 'Object', - 'Promise', 'RegExp', 'Set', 'String', 'Symbol', 'TypeError', 'Uint8Array', - 'Uint8ClampedArray', 'Uint16Array', 'Uint32Array', 'WeakMap', - '_', 'clearTimeout', 'isFinite', 'parseInt', 'setTimeout' - ]; - - /** Used to make template sourceURLs easier to identify. */ - var templateCounter = -1; - - /** Used to identify `toStringTag` values of typed arrays. */ - var typedArrayTags = {}; - typedArrayTags[float32Tag] = typedArrayTags[float64Tag] = - typedArrayTags[int8Tag] = typedArrayTags[int16Tag] = - typedArrayTags[int32Tag] = typedArrayTags[uint8Tag] = - typedArrayTags[uint8ClampedTag] = typedArrayTags[uint16Tag] = - typedArrayTags[uint32Tag] = true; - typedArrayTags[argsTag] = typedArrayTags[arrayTag] = - typedArrayTags[arrayBufferTag] = typedArrayTags[boolTag] = - typedArrayTags[dataViewTag] = typedArrayTags[dateTag] = - typedArrayTags[errorTag] = typedArrayTags[funcTag] = - typedArrayTags[mapTag] = typedArrayTags[numberTag] = - typedArrayTags[objectTag] = typedArrayTags[regexpTag] = - typedArrayTags[setTag] = typedArrayTags[stringTag] = - typedArrayTags[weakMapTag] = false; - - /** Used to identify `toStringTag` values supported by `_.clone`. */ - var cloneableTags = {}; - cloneableTags[argsTag] = cloneableTags[arrayTag] = - cloneableTags[arrayBufferTag] = cloneableTags[dataViewTag] = - cloneableTags[boolTag] = cloneableTags[dateTag] = - cloneableTags[float32Tag] = cloneableTags[float64Tag] = - cloneableTags[int8Tag] = cloneableTags[int16Tag] = - cloneableTags[int32Tag] = cloneableTags[mapTag] = - cloneableTags[numberTag] = cloneableTags[objectTag] = - cloneableTags[regexpTag] = cloneableTags[setTag] = - cloneableTags[stringTag] = cloneableTags[symbolTag] = - cloneableTags[uint8Tag] = cloneableTags[uint8ClampedTag] = - cloneableTags[uint16Tag] = cloneableTags[uint32Tag] = true; - cloneableTags[errorTag] = cloneableTags[funcTag] = - cloneableTags[weakMapTag] = false; - - /** Used to map Latin Unicode letters to basic Latin letters. */ - var deburredLetters = { - // Latin-1 Supplement block. - '\xc0': 'A', '\xc1': 'A', '\xc2': 'A', '\xc3': 'A', '\xc4': 'A', '\xc5': 'A', - '\xe0': 'a', '\xe1': 'a', '\xe2': 'a', '\xe3': 'a', '\xe4': 'a', '\xe5': 'a', - '\xc7': 'C', '\xe7': 'c', - '\xd0': 'D', '\xf0': 'd', - '\xc8': 'E', '\xc9': 'E', '\xca': 'E', '\xcb': 'E', - '\xe8': 'e', '\xe9': 'e', '\xea': 'e', '\xeb': 'e', - '\xcc': 'I', '\xcd': 'I', '\xce': 'I', '\xcf': 'I', - '\xec': 'i', '\xed': 'i', '\xee': 'i', '\xef': 'i', - '\xd1': 'N', '\xf1': 'n', - '\xd2': 'O', '\xd3': 'O', '\xd4': 'O', '\xd5': 'O', '\xd6': 'O', '\xd8': 'O', - '\xf2': 'o', '\xf3': 'o', '\xf4': 'o', '\xf5': 'o', '\xf6': 'o', '\xf8': 'o', - '\xd9': 'U', '\xda': 'U', '\xdb': 'U', '\xdc': 'U', - '\xf9': 'u', '\xfa': 'u', '\xfb': 'u', '\xfc': 'u', - '\xdd': 'Y', '\xfd': 'y', '\xff': 'y', - '\xc6': 'Ae', '\xe6': 'ae', - '\xde': 'Th', '\xfe': 'th', - '\xdf': 'ss', - // Latin Extended-A block. - '\u0100': 'A', '\u0102': 'A', '\u0104': 'A', - '\u0101': 'a', '\u0103': 'a', '\u0105': 'a', - '\u0106': 'C', '\u0108': 'C', '\u010a': 'C', '\u010c': 'C', - '\u0107': 'c', '\u0109': 'c', '\u010b': 'c', '\u010d': 'c', - '\u010e': 'D', '\u0110': 'D', '\u010f': 'd', '\u0111': 'd', - '\u0112': 'E', '\u0114': 'E', '\u0116': 'E', '\u0118': 'E', '\u011a': 'E', - '\u0113': 'e', '\u0115': 'e', '\u0117': 'e', '\u0119': 'e', '\u011b': 'e', - '\u011c': 'G', '\u011e': 'G', '\u0120': 'G', '\u0122': 'G', - '\u011d': 'g', '\u011f': 'g', '\u0121': 'g', '\u0123': 'g', - '\u0124': 'H', '\u0126': 'H', '\u0125': 'h', '\u0127': 'h', - '\u0128': 'I', '\u012a': 'I', '\u012c': 'I', '\u012e': 'I', '\u0130': 'I', - '\u0129': 'i', '\u012b': 'i', '\u012d': 'i', '\u012f': 'i', '\u0131': 'i', - '\u0134': 'J', '\u0135': 'j', - '\u0136': 'K', '\u0137': 'k', '\u0138': 'k', - '\u0139': 'L', '\u013b': 'L', '\u013d': 'L', '\u013f': 'L', '\u0141': 'L', - '\u013a': 'l', '\u013c': 'l', '\u013e': 'l', '\u0140': 'l', '\u0142': 'l', - '\u0143': 'N', '\u0145': 'N', '\u0147': 'N', '\u014a': 'N', - '\u0144': 'n', '\u0146': 'n', '\u0148': 'n', '\u014b': 'n', - '\u014c': 'O', '\u014e': 'O', '\u0150': 'O', - '\u014d': 'o', '\u014f': 'o', '\u0151': 'o', - '\u0154': 'R', '\u0156': 'R', '\u0158': 'R', - '\u0155': 'r', '\u0157': 'r', '\u0159': 'r', - '\u015a': 'S', '\u015c': 'S', '\u015e': 'S', '\u0160': 'S', - '\u015b': 's', '\u015d': 's', '\u015f': 's', '\u0161': 's', - '\u0162': 'T', '\u0164': 'T', '\u0166': 'T', - '\u0163': 't', '\u0165': 't', '\u0167': 't', - '\u0168': 'U', '\u016a': 'U', '\u016c': 'U', '\u016e': 'U', '\u0170': 'U', '\u0172': 'U', - '\u0169': 'u', '\u016b': 'u', '\u016d': 'u', '\u016f': 'u', '\u0171': 'u', '\u0173': 'u', - '\u0174': 'W', '\u0175': 'w', - '\u0176': 'Y', '\u0177': 'y', '\u0178': 'Y', - '\u0179': 'Z', '\u017b': 'Z', '\u017d': 'Z', - '\u017a': 'z', '\u017c': 'z', '\u017e': 'z', - '\u0132': 'IJ', '\u0133': 'ij', - '\u0152': 'Oe', '\u0153': 'oe', - '\u0149': "'n", '\u017f': 's' - }; - - /** Used to map characters to HTML entities. */ - var htmlEscapes = { - '&': '&', - '<': '<', - '>': '>', - '"': '"', - "'": ''' - }; - - /** Used to map HTML entities to characters. */ - var htmlUnescapes = { - '&': '&', - '<': '<', - '>': '>', - '"': '"', - ''': "'" - }; - - /** Used to escape characters for inclusion in compiled string literals. */ - var stringEscapes = { - '\\': '\\', - "'": "'", - '\n': 'n', - '\r': 'r', - '\u2028': 'u2028', - '\u2029': 'u2029' - }; - - /** Built-in method references without a dependency on `root`. */ - var freeParseFloat = parseFloat, - freeParseInt = parseInt; - - /** Detect free variable `global` from Node.js. */ - var freeGlobal = typeof global == 'object' && global && global.Object === Object && global; - - /** Detect free variable `self`. */ - var freeSelf = typeof self == 'object' && self && self.Object === Object && self; - - /** Used as a reference to the global object. */ - var root = freeGlobal || freeSelf || Function('return this')(); - - /** Detect free variable `exports`. */ - var freeExports = typeof exports == 'object' && exports && !exports.nodeType && exports; - - /** Detect free variable `module`. */ - var freeModule = freeExports && typeof module == 'object' && module && !module.nodeType && module; - - /** Detect the popular CommonJS extension `module.exports`. */ - var moduleExports = freeModule && freeModule.exports === freeExports; - - /** Detect free variable `process` from Node.js. */ - var freeProcess = moduleExports && freeGlobal.process; - - /** Used to access faster Node.js helpers. */ - var nodeUtil = (function() { - try { - // Use `util.types` for Node.js 10+. - var types = freeModule && freeModule.require && freeModule.require('util').types; - - if (types) { - return types; - } - - // Legacy `process.binding('util')` for Node.js < 10. - return freeProcess && freeProcess.binding && freeProcess.binding('util'); - } catch (e) {} - }()); - - /* Node.js helper references. */ - var nodeIsArrayBuffer = nodeUtil && nodeUtil.isArrayBuffer, - nodeIsDate = nodeUtil && nodeUtil.isDate, - nodeIsMap = nodeUtil && nodeUtil.isMap, - nodeIsRegExp = nodeUtil && nodeUtil.isRegExp, - nodeIsSet = nodeUtil && nodeUtil.isSet, - nodeIsTypedArray = nodeUtil && nodeUtil.isTypedArray; - - /*--------------------------------------------------------------------------*/ - - /** - * A faster alternative to `Function#apply`, this function invokes `func` - * with the `this` binding of `thisArg` and the arguments of `args`. - * - * @private - * @param {Function} func The function to invoke. - * @param {*} thisArg The `this` binding of `func`. - * @param {Array} args The arguments to invoke `func` with. - * @returns {*} Returns the result of `func`. - */ - function apply(func, thisArg, args) { - switch (args.length) { - case 0: return func.call(thisArg); - case 1: return func.call(thisArg, args[0]); - case 2: return func.call(thisArg, args[0], args[1]); - case 3: return func.call(thisArg, args[0], args[1], args[2]); - } - return func.apply(thisArg, args); - } - - /** - * A specialized version of `baseAggregator` for arrays. - * - * @private - * @param {Array} [array] The array to iterate over. - * @param {Function} setter The function to set `accumulator` values. - * @param {Function} iteratee The iteratee to transform keys. - * @param {Object} accumulator The initial aggregated object. - * @returns {Function} Returns `accumulator`. - */ - function arrayAggregator(array, setter, iteratee, accumulator) { - var index = -1, - length = array == null ? 0 : array.length; - - while (++index < length) { - var value = array[index]; - setter(accumulator, value, iteratee(value), array); - } - return accumulator; - } - - /** - * A specialized version of `_.forEach` for arrays without support for - * iteratee shorthands. - * - * @private - * @param {Array} [array] The array to iterate over. - * @param {Function} iteratee The function invoked per iteration. - * @returns {Array} Returns `array`. - */ - function arrayEach(array, iteratee) { - var index = -1, - length = array == null ? 0 : array.length; - - while (++index < length) { - if (iteratee(array[index], index, array) === false) { - break; - } - } - return array; - } - - /** - * A specialized version of `_.forEachRight` for arrays without support for - * iteratee shorthands. - * - * @private - * @param {Array} [array] The array to iterate over. - * @param {Function} iteratee The function invoked per iteration. - * @returns {Array} Returns `array`. - */ - function arrayEachRight(array, iteratee) { - var length = array == null ? 0 : array.length; - - while (length--) { - if (iteratee(array[length], length, array) === false) { - break; - } - } - return array; - } - - /** - * A specialized version of `_.every` for arrays without support for - * iteratee shorthands. - * - * @private - * @param {Array} [array] The array to iterate over. - * @param {Function} predicate The function invoked per iteration. - * @returns {boolean} Returns `true` if all elements pass the predicate check, - * else `false`. - */ - function arrayEvery(array, predicate) { - var index = -1, - length = array == null ? 0 : array.length; - - while (++index < length) { - if (!predicate(array[index], index, array)) { - return false; - } - } - return true; - } - - /** - * A specialized version of `_.filter` for arrays without support for - * iteratee shorthands. - * - * @private - * @param {Array} [array] The array to iterate over. - * @param {Function} predicate The function invoked per iteration. - * @returns {Array} Returns the new filtered array. - */ - function arrayFilter(array, predicate) { - var index = -1, - length = array == null ? 0 : array.length, - resIndex = 0, - result = []; - - while (++index < length) { - var value = array[index]; - if (predicate(value, index, array)) { - result[resIndex++] = value; - } - } - return result; - } - - /** - * A specialized version of `_.includes` for arrays without support for - * specifying an index to search from. - * - * @private - * @param {Array} [array] The array to inspect. - * @param {*} target The value to search for. - * @returns {boolean} Returns `true` if `target` is found, else `false`. - */ - function arrayIncludes(array, value) { - var length = array == null ? 0 : array.length; - return !!length && baseIndexOf(array, value, 0) > -1; - } - - /** - * This function is like `arrayIncludes` except that it accepts a comparator. - * - * @private - * @param {Array} [array] The array to inspect. - * @param {*} target The value to search for. - * @param {Function} comparator The comparator invoked per element. - * @returns {boolean} Returns `true` if `target` is found, else `false`. - */ - function arrayIncludesWith(array, value, comparator) { - var index = -1, - length = array == null ? 0 : array.length; - - while (++index < length) { - if (comparator(value, array[index])) { - return true; - } - } - return false; - } - - /** - * A specialized version of `_.map` for arrays without support for iteratee - * shorthands. - * - * @private - * @param {Array} [array] The array to iterate over. - * @param {Function} iteratee The function invoked per iteration. - * @returns {Array} Returns the new mapped array. - */ - function arrayMap(array, iteratee) { - var index = -1, - length = array == null ? 0 : array.length, - result = Array(length); - - while (++index < length) { - result[index] = iteratee(array[index], index, array); - } - return result; - } - - /** - * Appends the elements of `values` to `array`. - * - * @private - * @param {Array} array The array to modify. - * @param {Array} values The values to append. - * @returns {Array} Returns `array`. - */ - function arrayPush(array, values) { - var index = -1, - length = values.length, - offset = array.length; - - while (++index < length) { - array[offset + index] = values[index]; - } - return array; - } - - /** - * A specialized version of `_.reduce` for arrays without support for - * iteratee shorthands. - * - * @private - * @param {Array} [array] The array to iterate over. - * @param {Function} iteratee The function invoked per iteration. - * @param {*} [accumulator] The initial value. - * @param {boolean} [initAccum] Specify using the first element of `array` as - * the initial value. - * @returns {*} Returns the accumulated value. - */ - function arrayReduce(array, iteratee, accumulator, initAccum) { - var index = -1, - length = array == null ? 0 : array.length; - - if (initAccum && length) { - accumulator = array[++index]; - } - while (++index < length) { - accumulator = iteratee(accumulator, array[index], index, array); - } - return accumulator; - } - - /** - * A specialized version of `_.reduceRight` for arrays without support for - * iteratee shorthands. - * - * @private - * @param {Array} [array] The array to iterate over. - * @param {Function} iteratee The function invoked per iteration. - * @param {*} [accumulator] The initial value. - * @param {boolean} [initAccum] Specify using the last element of `array` as - * the initial value. - * @returns {*} Returns the accumulated value. - */ - function arrayReduceRight(array, iteratee, accumulator, initAccum) { - var length = array == null ? 0 : array.length; - if (initAccum && length) { - accumulator = array[--length]; - } - while (length--) { - accumulator = iteratee(accumulator, array[length], length, array); - } - return accumulator; - } - - /** - * A specialized version of `_.some` for arrays without support for iteratee - * shorthands. - * - * @private - * @param {Array} [array] The array to iterate over. - * @param {Function} predicate The function invoked per iteration. - * @returns {boolean} Returns `true` if any element passes the predicate check, - * else `false`. - */ - function arraySome(array, predicate) { - var index = -1, - length = array == null ? 0 : array.length; - - while (++index < length) { - if (predicate(array[index], index, array)) { - return true; - } - } - return false; - } - - /** - * Gets the size of an ASCII `string`. - * - * @private - * @param {string} string The string inspect. - * @returns {number} Returns the string size. - */ - var asciiSize = baseProperty('length'); - - /** - * Converts an ASCII `string` to an array. - * - * @private - * @param {string} string The string to convert. - * @returns {Array} Returns the converted array. - */ - function asciiToArray(string) { - return string.split(''); - } - - /** - * Splits an ASCII `string` into an array of its words. - * - * @private - * @param {string} The string to inspect. - * @returns {Array} Returns the words of `string`. - */ - function asciiWords(string) { - return string.match(reAsciiWord) || []; - } - - /** - * The base implementation of methods like `_.findKey` and `_.findLastKey`, - * without support for iteratee shorthands, which iterates over `collection` - * using `eachFunc`. - * - * @private - * @param {Array|Object} collection The collection to inspect. - * @param {Function} predicate The function invoked per iteration. - * @param {Function} eachFunc The function to iterate over `collection`. - * @returns {*} Returns the found element or its key, else `undefined`. - */ - function baseFindKey(collection, predicate, eachFunc) { - var result; - eachFunc(collection, function(value, key, collection) { - if (predicate(value, key, collection)) { - result = key; - return false; - } - }); - return result; - } - - /** - * The base implementation of `_.findIndex` and `_.findLastIndex` without - * support for iteratee shorthands. - * - * @private - * @param {Array} array The array to inspect. - * @param {Function} predicate The function invoked per iteration. - * @param {number} fromIndex The index to search from. - * @param {boolean} [fromRight] Specify iterating from right to left. - * @returns {number} Returns the index of the matched value, else `-1`. - */ - function baseFindIndex(array, predicate, fromIndex, fromRight) { - var length = array.length, - index = fromIndex + (fromRight ? 1 : -1); - - while ((fromRight ? index-- : ++index < length)) { - if (predicate(array[index], index, array)) { - return index; - } - } - return -1; - } - - /** - * The base implementation of `_.indexOf` without `fromIndex` bounds checks. - * - * @private - * @param {Array} array The array to inspect. - * @param {*} value The value to search for. - * @param {number} fromIndex The index to search from. - * @returns {number} Returns the index of the matched value, else `-1`. - */ - function baseIndexOf(array, value, fromIndex) { - return value === value - ? strictIndexOf(array, value, fromIndex) - : baseFindIndex(array, baseIsNaN, fromIndex); - } - - /** - * This function is like `baseIndexOf` except that it accepts a comparator. - * - * @private - * @param {Array} array The array to inspect. - * @param {*} value The value to search for. - * @param {number} fromIndex The index to search from. - * @param {Function} comparator The comparator invoked per element. - * @returns {number} Returns the index of the matched value, else `-1`. - */ - function baseIndexOfWith(array, value, fromIndex, comparator) { - var index = fromIndex - 1, - length = array.length; - - while (++index < length) { - if (comparator(array[index], value)) { - return index; - } - } - return -1; - } - - /** - * The base implementation of `_.isNaN` without support for number objects. - * - * @private - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is `NaN`, else `false`. - */ - function baseIsNaN(value) { - return value !== value; - } - - /** - * The base implementation of `_.mean` and `_.meanBy` without support for - * iteratee shorthands. - * - * @private - * @param {Array} array The array to iterate over. - * @param {Function} iteratee The function invoked per iteration. - * @returns {number} Returns the mean. - */ - function baseMean(array, iteratee) { - var length = array == null ? 0 : array.length; - return length ? (baseSum(array, iteratee) / length) : NAN; - } - - /** - * The base implementation of `_.property` without support for deep paths. - * - * @private - * @param {string} key The key of the property to get. - * @returns {Function} Returns the new accessor function. - */ - function baseProperty(key) { - return function(object) { - return object == null ? undefined : object[key]; - }; - } - - /** - * The base implementation of `_.propertyOf` without support for deep paths. - * - * @private - * @param {Object} object The object to query. - * @returns {Function} Returns the new accessor function. - */ - function basePropertyOf(object) { - return function(key) { - return object == null ? undefined : object[key]; - }; - } - - /** - * The base implementation of `_.reduce` and `_.reduceRight`, without support - * for iteratee shorthands, which iterates over `collection` using `eachFunc`. - * - * @private - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} iteratee The function invoked per iteration. - * @param {*} accumulator The initial value. - * @param {boolean} initAccum Specify using the first or last element of - * `collection` as the initial value. - * @param {Function} eachFunc The function to iterate over `collection`. - * @returns {*} Returns the accumulated value. - */ - function baseReduce(collection, iteratee, accumulator, initAccum, eachFunc) { - eachFunc(collection, function(value, index, collection) { - accumulator = initAccum - ? (initAccum = false, value) - : iteratee(accumulator, value, index, collection); - }); - return accumulator; - } - - /** - * The base implementation of `_.sortBy` which uses `comparer` to define the - * sort order of `array` and replaces criteria objects with their corresponding - * values. - * - * @private - * @param {Array} array The array to sort. - * @param {Function} comparer The function to define sort order. - * @returns {Array} Returns `array`. - */ - function baseSortBy(array, comparer) { - var length = array.length; - - array.sort(comparer); - while (length--) { - array[length] = array[length].value; - } - return array; - } - - /** - * The base implementation of `_.sum` and `_.sumBy` without support for - * iteratee shorthands. - * - * @private - * @param {Array} array The array to iterate over. - * @param {Function} iteratee The function invoked per iteration. - * @returns {number} Returns the sum. - */ - function baseSum(array, iteratee) { - var result, - index = -1, - length = array.length; - - while (++index < length) { - var current = iteratee(array[index]); - if (current !== undefined) { - result = result === undefined ? current : (result + current); - } - } - return result; - } - - /** - * The base implementation of `_.times` without support for iteratee shorthands - * or max array length checks. - * - * @private - * @param {number} n The number of times to invoke `iteratee`. - * @param {Function} iteratee The function invoked per iteration. - * @returns {Array} Returns the array of results. - */ - function baseTimes(n, iteratee) { - var index = -1, - result = Array(n); - - while (++index < n) { - result[index] = iteratee(index); - } - return result; - } - - /** - * The base implementation of `_.toPairs` and `_.toPairsIn` which creates an array - * of key-value pairs for `object` corresponding to the property names of `props`. - * - * @private - * @param {Object} object The object to query. - * @param {Array} props The property names to get values for. - * @returns {Object} Returns the key-value pairs. - */ - function baseToPairs(object, props) { - return arrayMap(props, function(key) { - return [key, object[key]]; - }); - } - - /** - * The base implementation of `_.unary` without support for storing metadata. - * - * @private - * @param {Function} func The function to cap arguments for. - * @returns {Function} Returns the new capped function. - */ - function baseUnary(func) { - return function(value) { - return func(value); - }; - } - - /** - * The base implementation of `_.values` and `_.valuesIn` which creates an - * array of `object` property values corresponding to the property names - * of `props`. - * - * @private - * @param {Object} object The object to query. - * @param {Array} props The property names to get values for. - * @returns {Object} Returns the array of property values. - */ - function baseValues(object, props) { - return arrayMap(props, function(key) { - return object[key]; - }); - } - - /** - * Checks if a `cache` value for `key` exists. - * - * @private - * @param {Object} cache The cache to query. - * @param {string} key The key of the entry to check. - * @returns {boolean} Returns `true` if an entry for `key` exists, else `false`. - */ - function cacheHas(cache, key) { - return cache.has(key); - } - - /** - * Used by `_.trim` and `_.trimStart` to get the index of the first string symbol - * that is not found in the character symbols. - * - * @private - * @param {Array} strSymbols The string symbols to inspect. - * @param {Array} chrSymbols The character symbols to find. - * @returns {number} Returns the index of the first unmatched string symbol. - */ - function charsStartIndex(strSymbols, chrSymbols) { - var index = -1, - length = strSymbols.length; - - while (++index < length && baseIndexOf(chrSymbols, strSymbols[index], 0) > -1) {} - return index; - } - - /** - * Used by `_.trim` and `_.trimEnd` to get the index of the last string symbol - * that is not found in the character symbols. - * - * @private - * @param {Array} strSymbols The string symbols to inspect. - * @param {Array} chrSymbols The character symbols to find. - * @returns {number} Returns the index of the last unmatched string symbol. - */ - function charsEndIndex(strSymbols, chrSymbols) { - var index = strSymbols.length; - - while (index-- && baseIndexOf(chrSymbols, strSymbols[index], 0) > -1) {} - return index; - } - - /** - * Gets the number of `placeholder` occurrences in `array`. - * - * @private - * @param {Array} array The array to inspect. - * @param {*} placeholder The placeholder to search for. - * @returns {number} Returns the placeholder count. - */ - function countHolders(array, placeholder) { - var length = array.length, - result = 0; - - while (length--) { - if (array[length] === placeholder) { - ++result; - } - } - return result; - } - - /** - * Used by `_.deburr` to convert Latin-1 Supplement and Latin Extended-A - * letters to basic Latin letters. - * - * @private - * @param {string} letter The matched letter to deburr. - * @returns {string} Returns the deburred letter. - */ - var deburrLetter = basePropertyOf(deburredLetters); - - /** - * Used by `_.escape` to convert characters to HTML entities. - * - * @private - * @param {string} chr The matched character to escape. - * @returns {string} Returns the escaped character. - */ - var escapeHtmlChar = basePropertyOf(htmlEscapes); - - /** - * Used by `_.template` to escape characters for inclusion in compiled string literals. - * - * @private - * @param {string} chr The matched character to escape. - * @returns {string} Returns the escaped character. - */ - function escapeStringChar(chr) { - return '\\' + stringEscapes[chr]; - } - - /** - * Gets the value at `key` of `object`. - * - * @private - * @param {Object} [object] The object to query. - * @param {string} key The key of the property to get. - * @returns {*} Returns the property value. - */ - function getValue(object, key) { - return object == null ? undefined : object[key]; - } - - /** - * Checks if `string` contains Unicode symbols. - * - * @private - * @param {string} string The string to inspect. - * @returns {boolean} Returns `true` if a symbol is found, else `false`. - */ - function hasUnicode(string) { - return reHasUnicode.test(string); - } - - /** - * Checks if `string` contains a word composed of Unicode symbols. - * - * @private - * @param {string} string The string to inspect. - * @returns {boolean} Returns `true` if a word is found, else `false`. - */ - function hasUnicodeWord(string) { - return reHasUnicodeWord.test(string); - } - - /** - * Converts `iterator` to an array. - * - * @private - * @param {Object} iterator The iterator to convert. - * @returns {Array} Returns the converted array. - */ - function iteratorToArray(iterator) { - var data, - result = []; - - while (!(data = iterator.next()).done) { - result.push(data.value); - } - return result; - } - - /** - * Converts `map` to its key-value pairs. - * - * @private - * @param {Object} map The map to convert. - * @returns {Array} Returns the key-value pairs. - */ - function mapToArray(map) { - var index = -1, - result = Array(map.size); - - map.forEach(function(value, key) { - result[++index] = [key, value]; - }); - return result; - } - - /** - * Creates a unary function that invokes `func` with its argument transformed. - * - * @private - * @param {Function} func The function to wrap. - * @param {Function} transform The argument transform. - * @returns {Function} Returns the new function. - */ - function overArg(func, transform) { - return function(arg) { - return func(transform(arg)); - }; - } - - /** - * Replaces all `placeholder` elements in `array` with an internal placeholder - * and returns an array of their indexes. - * - * @private - * @param {Array} array The array to modify. - * @param {*} placeholder The placeholder to replace. - * @returns {Array} Returns the new array of placeholder indexes. - */ - function replaceHolders(array, placeholder) { - var index = -1, - length = array.length, - resIndex = 0, - result = []; - - while (++index < length) { - var value = array[index]; - if (value === placeholder || value === PLACEHOLDER) { - array[index] = PLACEHOLDER; - result[resIndex++] = index; - } - } - return result; - } - - /** - * Converts `set` to an array of its values. - * - * @private - * @param {Object} set The set to convert. - * @returns {Array} Returns the values. - */ - function setToArray(set) { - var index = -1, - result = Array(set.size); - - set.forEach(function(value) { - result[++index] = value; - }); - return result; - } - - /** - * Converts `set` to its value-value pairs. - * - * @private - * @param {Object} set The set to convert. - * @returns {Array} Returns the value-value pairs. - */ - function setToPairs(set) { - var index = -1, - result = Array(set.size); - - set.forEach(function(value) { - result[++index] = [value, value]; - }); - return result; - } - - /** - * A specialized version of `_.indexOf` which performs strict equality - * comparisons of values, i.e. `===`. - * - * @private - * @param {Array} array The array to inspect. - * @param {*} value The value to search for. - * @param {number} fromIndex The index to search from. - * @returns {number} Returns the index of the matched value, else `-1`. - */ - function strictIndexOf(array, value, fromIndex) { - var index = fromIndex - 1, - length = array.length; - - while (++index < length) { - if (array[index] === value) { - return index; - } - } - return -1; - } - - /** - * A specialized version of `_.lastIndexOf` which performs strict equality - * comparisons of values, i.e. `===`. - * - * @private - * @param {Array} array The array to inspect. - * @param {*} value The value to search for. - * @param {number} fromIndex The index to search from. - * @returns {number} Returns the index of the matched value, else `-1`. - */ - function strictLastIndexOf(array, value, fromIndex) { - var index = fromIndex + 1; - while (index--) { - if (array[index] === value) { - return index; - } - } - return index; - } - - /** - * Gets the number of symbols in `string`. - * - * @private - * @param {string} string The string to inspect. - * @returns {number} Returns the string size. - */ - function stringSize(string) { - return hasUnicode(string) - ? unicodeSize(string) - : asciiSize(string); - } - - /** - * Converts `string` to an array. - * - * @private - * @param {string} string The string to convert. - * @returns {Array} Returns the converted array. - */ - function stringToArray(string) { - return hasUnicode(string) - ? unicodeToArray(string) - : asciiToArray(string); - } - - /** - * Used by `_.unescape` to convert HTML entities to characters. - * - * @private - * @param {string} chr The matched character to unescape. - * @returns {string} Returns the unescaped character. - */ - var unescapeHtmlChar = basePropertyOf(htmlUnescapes); - - /** - * Gets the size of a Unicode `string`. - * - * @private - * @param {string} string The string inspect. - * @returns {number} Returns the string size. - */ - function unicodeSize(string) { - var result = reUnicode.lastIndex = 0; - while (reUnicode.test(string)) { - ++result; - } - return result; - } - - /** - * Converts a Unicode `string` to an array. - * - * @private - * @param {string} string The string to convert. - * @returns {Array} Returns the converted array. - */ - function unicodeToArray(string) { - return string.match(reUnicode) || []; - } - - /** - * Splits a Unicode `string` into an array of its words. - * - * @private - * @param {string} The string to inspect. - * @returns {Array} Returns the words of `string`. - */ - function unicodeWords(string) { - return string.match(reUnicodeWord) || []; - } - - /*--------------------------------------------------------------------------*/ - - /** - * Create a new pristine `lodash` function using the `context` object. - * - * @static - * @memberOf _ - * @since 1.1.0 - * @category Util - * @param {Object} [context=root] The context object. - * @returns {Function} Returns a new `lodash` function. - * @example - * - * _.mixin({ 'foo': _.constant('foo') }); - * - * var lodash = _.runInContext(); - * lodash.mixin({ 'bar': lodash.constant('bar') }); - * - * _.isFunction(_.foo); - * // => true - * _.isFunction(_.bar); - * // => false - * - * lodash.isFunction(lodash.foo); - * // => false - * lodash.isFunction(lodash.bar); - * // => true - * - * // Create a suped-up `defer` in Node.js. - * var defer = _.runInContext({ 'setTimeout': setImmediate }).defer; - */ - var runInContext = (function runInContext(context) { - context = context == null ? root : _.defaults(root.Object(), context, _.pick(root, contextProps)); - - /** Built-in constructor references. */ - var Array = context.Array, - Date = context.Date, - Error = context.Error, - Function = context.Function, - Math = context.Math, - Object = context.Object, - RegExp = context.RegExp, - String = context.String, - TypeError = context.TypeError; - - /** Used for built-in method references. */ - var arrayProto = Array.prototype, - funcProto = Function.prototype, - objectProto = Object.prototype; - - /** Used to detect overreaching core-js shims. */ - var coreJsData = context['__core-js_shared__']; - - /** Used to resolve the decompiled source of functions. */ - var funcToString = funcProto.toString; - - /** Used to check objects for own properties. */ - var hasOwnProperty = objectProto.hasOwnProperty; - - /** Used to generate unique IDs. */ - var idCounter = 0; - - /** Used to detect methods masquerading as native. */ - var maskSrcKey = (function() { - var uid = /[^.]+$/.exec(coreJsData && coreJsData.keys && coreJsData.keys.IE_PROTO || ''); - return uid ? ('Symbol(src)_1.' + uid) : ''; - }()); - - /** - * Used to resolve the - * [`toStringTag`](http://ecma-international.org/ecma-262/7.0/#sec-object.prototype.tostring) - * of values. - */ - var nativeObjectToString = objectProto.toString; - - /** Used to infer the `Object` constructor. */ - var objectCtorString = funcToString.call(Object); - - /** Used to restore the original `_` reference in `_.noConflict`. */ - var oldDash = root._; - - /** Used to detect if a method is native. */ - var reIsNative = RegExp('^' + - funcToString.call(hasOwnProperty).replace(reRegExpChar, '\\$&') - .replace(/hasOwnProperty|(function).*?(?=\\\()| for .+?(?=\\\])/g, '$1.*?') + '$' - ); - - /** Built-in value references. */ - var Buffer = moduleExports ? context.Buffer : undefined, - Symbol = context.Symbol, - Uint8Array = context.Uint8Array, - allocUnsafe = Buffer ? Buffer.allocUnsafe : undefined, - getPrototype = overArg(Object.getPrototypeOf, Object), - objectCreate = Object.create, - propertyIsEnumerable = objectProto.propertyIsEnumerable, - splice = arrayProto.splice, - spreadableSymbol = Symbol ? Symbol.isConcatSpreadable : undefined, - symIterator = Symbol ? Symbol.iterator : undefined, - symToStringTag = Symbol ? Symbol.toStringTag : undefined; - - var defineProperty = (function() { - try { - var func = getNative(Object, 'defineProperty'); - func({}, '', {}); - return func; - } catch (e) {} - }()); - - /** Mocked built-ins. */ - var ctxClearTimeout = context.clearTimeout !== root.clearTimeout && context.clearTimeout, - ctxNow = Date && Date.now !== root.Date.now && Date.now, - ctxSetTimeout = context.setTimeout !== root.setTimeout && context.setTimeout; - - /* Built-in method references for those with the same name as other `lodash` methods. */ - var nativeCeil = Math.ceil, - nativeFloor = Math.floor, - nativeGetSymbols = Object.getOwnPropertySymbols, - nativeIsBuffer = Buffer ? Buffer.isBuffer : undefined, - nativeIsFinite = context.isFinite, - nativeJoin = arrayProto.join, - nativeKeys = overArg(Object.keys, Object), - nativeMax = Math.max, - nativeMin = Math.min, - nativeNow = Date.now, - nativeParseInt = context.parseInt, - nativeRandom = Math.random, - nativeReverse = arrayProto.reverse; - - /* Built-in method references that are verified to be native. */ - var DataView = getNative(context, 'DataView'), - Map = getNative(context, 'Map'), - Promise = getNative(context, 'Promise'), - Set = getNative(context, 'Set'), - WeakMap = getNative(context, 'WeakMap'), - nativeCreate = getNative(Object, 'create'); - - /** Used to store function metadata. */ - var metaMap = WeakMap && new WeakMap; - - /** Used to lookup unminified function names. */ - var realNames = {}; - - /** Used to detect maps, sets, and weakmaps. */ - var dataViewCtorString = toSource(DataView), - mapCtorString = toSource(Map), - promiseCtorString = toSource(Promise), - setCtorString = toSource(Set), - weakMapCtorString = toSource(WeakMap); - - /** Used to convert symbols to primitives and strings. */ - var symbolProto = Symbol ? Symbol.prototype : undefined, - symbolValueOf = symbolProto ? symbolProto.valueOf : undefined, - symbolToString = symbolProto ? symbolProto.toString : undefined; - - /*------------------------------------------------------------------------*/ - - /** - * Creates a `lodash` object which wraps `value` to enable implicit method - * chain sequences. Methods that operate on and return arrays, collections, - * and functions can be chained together. Methods that retrieve a single value - * or may return a primitive value will automatically end the chain sequence - * and return the unwrapped value. Otherwise, the value must be unwrapped - * with `_#value`. - * - * Explicit chain sequences, which must be unwrapped with `_#value`, may be - * enabled using `_.chain`. - * - * The execution of chained methods is lazy, that is, it's deferred until - * `_#value` is implicitly or explicitly called. - * - * Lazy evaluation allows several methods to support shortcut fusion. - * Shortcut fusion is an optimization to merge iteratee calls; this avoids - * the creation of intermediate arrays and can greatly reduce the number of - * iteratee executions. Sections of a chain sequence qualify for shortcut - * fusion if the section is applied to an array and iteratees accept only - * one argument. The heuristic for whether a section qualifies for shortcut - * fusion is subject to change. - * - * Chaining is supported in custom builds as long as the `_#value` method is - * directly or indirectly included in the build. - * - * In addition to lodash methods, wrappers have `Array` and `String` methods. - * - * The wrapper `Array` methods are: - * `concat`, `join`, `pop`, `push`, `shift`, `sort`, `splice`, and `unshift` - * - * The wrapper `String` methods are: - * `replace` and `split` - * - * The wrapper methods that support shortcut fusion are: - * `at`, `compact`, `drop`, `dropRight`, `dropWhile`, `filter`, `find`, - * `findLast`, `head`, `initial`, `last`, `map`, `reject`, `reverse`, `slice`, - * `tail`, `take`, `takeRight`, `takeRightWhile`, `takeWhile`, and `toArray` - * - * The chainable wrapper methods are: - * `after`, `ary`, `assign`, `assignIn`, `assignInWith`, `assignWith`, `at`, - * `before`, `bind`, `bindAll`, `bindKey`, `castArray`, `chain`, `chunk`, - * `commit`, `compact`, `concat`, `conforms`, `constant`, `countBy`, `create`, - * `curry`, `debounce`, `defaults`, `defaultsDeep`, `defer`, `delay`, - * `difference`, `differenceBy`, `differenceWith`, `drop`, `dropRight`, - * `dropRightWhile`, `dropWhile`, `extend`, `extendWith`, `fill`, `filter`, - * `flatMap`, `flatMapDeep`, `flatMapDepth`, `flatten`, `flattenDeep`, - * `flattenDepth`, `flip`, `flow`, `flowRight`, `fromPairs`, `functions`, - * `functionsIn`, `groupBy`, `initial`, `intersection`, `intersectionBy`, - * `intersectionWith`, `invert`, `invertBy`, `invokeMap`, `iteratee`, `keyBy`, - * `keys`, `keysIn`, `map`, `mapKeys`, `mapValues`, `matches`, `matchesProperty`, - * `memoize`, `merge`, `mergeWith`, `method`, `methodOf`, `mixin`, `negate`, - * `nthArg`, `omit`, `omitBy`, `once`, `orderBy`, `over`, `overArgs`, - * `overEvery`, `overSome`, `partial`, `partialRight`, `partition`, `pick`, - * `pickBy`, `plant`, `property`, `propertyOf`, `pull`, `pullAll`, `pullAllBy`, - * `pullAllWith`, `pullAt`, `push`, `range`, `rangeRight`, `rearg`, `reject`, - * `remove`, `rest`, `reverse`, `sampleSize`, `set`, `setWith`, `shuffle`, - * `slice`, `sort`, `sortBy`, `splice`, `spread`, `tail`, `take`, `takeRight`, - * `takeRightWhile`, `takeWhile`, `tap`, `throttle`, `thru`, `toArray`, - * `toPairs`, `toPairsIn`, `toPath`, `toPlainObject`, `transform`, `unary`, - * `union`, `unionBy`, `unionWith`, `uniq`, `uniqBy`, `uniqWith`, `unset`, - * `unshift`, `unzip`, `unzipWith`, `update`, `updateWith`, `values`, - * `valuesIn`, `without`, `wrap`, `xor`, `xorBy`, `xorWith`, `zip`, - * `zipObject`, `zipObjectDeep`, and `zipWith` - * - * The wrapper methods that are **not** chainable by default are: - * `add`, `attempt`, `camelCase`, `capitalize`, `ceil`, `clamp`, `clone`, - * `cloneDeep`, `cloneDeepWith`, `cloneWith`, `conformsTo`, `deburr`, - * `defaultTo`, `divide`, `each`, `eachRight`, `endsWith`, `eq`, `escape`, - * `escapeRegExp`, `every`, `find`, `findIndex`, `findKey`, `findLast`, - * `findLastIndex`, `findLastKey`, `first`, `floor`, `forEach`, `forEachRight`, - * `forIn`, `forInRight`, `forOwn`, `forOwnRight`, `get`, `gt`, `gte`, `has`, - * `hasIn`, `head`, `identity`, `includes`, `indexOf`, `inRange`, `invoke`, - * `isArguments`, `isArray`, `isArrayBuffer`, `isArrayLike`, `isArrayLikeObject`, - * `isBoolean`, `isBuffer`, `isDate`, `isElement`, `isEmpty`, `isEqual`, - * `isEqualWith`, `isError`, `isFinite`, `isFunction`, `isInteger`, `isLength`, - * `isMap`, `isMatch`, `isMatchWith`, `isNaN`, `isNative`, `isNil`, `isNull`, - * `isNumber`, `isObject`, `isObjectLike`, `isPlainObject`, `isRegExp`, - * `isSafeInteger`, `isSet`, `isString`, `isUndefined`, `isTypedArray`, - * `isWeakMap`, `isWeakSet`, `join`, `kebabCase`, `last`, `lastIndexOf`, - * `lowerCase`, `lowerFirst`, `lt`, `lte`, `max`, `maxBy`, `mean`, `meanBy`, - * `min`, `minBy`, `multiply`, `noConflict`, `noop`, `now`, `nth`, `pad`, - * `padEnd`, `padStart`, `parseInt`, `pop`, `random`, `reduce`, `reduceRight`, - * `repeat`, `result`, `round`, `runInContext`, `sample`, `shift`, `size`, - * `snakeCase`, `some`, `sortedIndex`, `sortedIndexBy`, `sortedLastIndex`, - * `sortedLastIndexBy`, `startCase`, `startsWith`, `stubArray`, `stubFalse`, - * `stubObject`, `stubString`, `stubTrue`, `subtract`, `sum`, `sumBy`, - * `template`, `times`, `toFinite`, `toInteger`, `toJSON`, `toLength`, - * `toLower`, `toNumber`, `toSafeInteger`, `toString`, `toUpper`, `trim`, - * `trimEnd`, `trimStart`, `truncate`, `unescape`, `uniqueId`, `upperCase`, - * `upperFirst`, `value`, and `words` - * - * @name _ - * @constructor - * @category Seq - * @param {*} value The value to wrap in a `lodash` instance. - * @returns {Object} Returns the new `lodash` wrapper instance. - * @example - * - * function square(n) { - * return n * n; - * } - * - * var wrapped = _([1, 2, 3]); - * - * // Returns an unwrapped value. - * wrapped.reduce(_.add); - * // => 6 - * - * // Returns a wrapped value. - * var squares = wrapped.map(square); - * - * _.isArray(squares); - * // => false - * - * _.isArray(squares.value()); - * // => true - */ - function lodash(value) { - if (isObjectLike(value) && !isArray(value) && !(value instanceof LazyWrapper)) { - if (value instanceof LodashWrapper) { - return value; - } - if (hasOwnProperty.call(value, '__wrapped__')) { - return wrapperClone(value); - } - } - return new LodashWrapper(value); - } - - /** - * The base implementation of `_.create` without support for assigning - * properties to the created object. - * - * @private - * @param {Object} proto The object to inherit from. - * @returns {Object} Returns the new object. - */ - var baseCreate = (function() { - function object() {} - return function(proto) { - if (!isObject(proto)) { - return {}; - } - if (objectCreate) { - return objectCreate(proto); - } - object.prototype = proto; - var result = new object; - object.prototype = undefined; - return result; - }; - }()); - - /** - * The function whose prototype chain sequence wrappers inherit from. - * - * @private - */ - function baseLodash() { - // No operation performed. - } - - /** - * The base constructor for creating `lodash` wrapper objects. - * - * @private - * @param {*} value The value to wrap. - * @param {boolean} [chainAll] Enable explicit method chain sequences. - */ - function LodashWrapper(value, chainAll) { - this.__wrapped__ = value; - this.__actions__ = []; - this.__chain__ = !!chainAll; - this.__index__ = 0; - this.__values__ = undefined; - } - - /** - * By default, the template delimiters used by lodash are like those in - * embedded Ruby (ERB) as well as ES2015 template strings. Change the - * following template settings to use alternative delimiters. - * - * @static - * @memberOf _ - * @type {Object} - */ - lodash.templateSettings = { - - /** - * Used to detect `data` property values to be HTML-escaped. - * - * @memberOf _.templateSettings - * @type {RegExp} - */ - 'escape': reEscape, - - /** - * Used to detect code to be evaluated. - * - * @memberOf _.templateSettings - * @type {RegExp} - */ - 'evaluate': reEvaluate, - - /** - * Used to detect `data` property values to inject. - * - * @memberOf _.templateSettings - * @type {RegExp} - */ - 'interpolate': reInterpolate, - - /** - * Used to reference the data object in the template text. - * - * @memberOf _.templateSettings - * @type {string} - */ - 'variable': '', - - /** - * Used to import variables into the compiled template. - * - * @memberOf _.templateSettings - * @type {Object} - */ - 'imports': { - - /** - * A reference to the `lodash` function. - * - * @memberOf _.templateSettings.imports - * @type {Function} - */ - '_': lodash - } - }; - - // Ensure wrappers are instances of `baseLodash`. - lodash.prototype = baseLodash.prototype; - lodash.prototype.constructor = lodash; - - LodashWrapper.prototype = baseCreate(baseLodash.prototype); - LodashWrapper.prototype.constructor = LodashWrapper; - - /*------------------------------------------------------------------------*/ - - /** - * Creates a lazy wrapper object which wraps `value` to enable lazy evaluation. - * - * @private - * @constructor - * @param {*} value The value to wrap. - */ - function LazyWrapper(value) { - this.__wrapped__ = value; - this.__actions__ = []; - this.__dir__ = 1; - this.__filtered__ = false; - this.__iteratees__ = []; - this.__takeCount__ = MAX_ARRAY_LENGTH; - this.__views__ = []; - } - - /** - * Creates a clone of the lazy wrapper object. - * - * @private - * @name clone - * @memberOf LazyWrapper - * @returns {Object} Returns the cloned `LazyWrapper` object. - */ - function lazyClone() { - var result = new LazyWrapper(this.__wrapped__); - result.__actions__ = copyArray(this.__actions__); - result.__dir__ = this.__dir__; - result.__filtered__ = this.__filtered__; - result.__iteratees__ = copyArray(this.__iteratees__); - result.__takeCount__ = this.__takeCount__; - result.__views__ = copyArray(this.__views__); - return result; - } - - /** - * Reverses the direction of lazy iteration. - * - * @private - * @name reverse - * @memberOf LazyWrapper - * @returns {Object} Returns the new reversed `LazyWrapper` object. - */ - function lazyReverse() { - if (this.__filtered__) { - var result = new LazyWrapper(this); - result.__dir__ = -1; - result.__filtered__ = true; - } else { - result = this.clone(); - result.__dir__ *= -1; - } - return result; - } - - /** - * Extracts the unwrapped value from its lazy wrapper. - * - * @private - * @name value - * @memberOf LazyWrapper - * @returns {*} Returns the unwrapped value. - */ - function lazyValue() { - var array = this.__wrapped__.value(), - dir = this.__dir__, - isArr = isArray(array), - isRight = dir < 0, - arrLength = isArr ? array.length : 0, - view = getView(0, arrLength, this.__views__), - start = view.start, - end = view.end, - length = end - start, - index = isRight ? end : (start - 1), - iteratees = this.__iteratees__, - iterLength = iteratees.length, - resIndex = 0, - takeCount = nativeMin(length, this.__takeCount__); - - if (!isArr || (!isRight && arrLength == length && takeCount == length)) { - return baseWrapperValue(array, this.__actions__); - } - var result = []; - - outer: - while (length-- && resIndex < takeCount) { - index += dir; - - var iterIndex = -1, - value = array[index]; - - while (++iterIndex < iterLength) { - var data = iteratees[iterIndex], - iteratee = data.iteratee, - type = data.type, - computed = iteratee(value); - - if (type == LAZY_MAP_FLAG) { - value = computed; - } else if (!computed) { - if (type == LAZY_FILTER_FLAG) { - continue outer; - } else { - break outer; - } - } - } - result[resIndex++] = value; - } - return result; - } - - // Ensure `LazyWrapper` is an instance of `baseLodash`. - LazyWrapper.prototype = baseCreate(baseLodash.prototype); - LazyWrapper.prototype.constructor = LazyWrapper; - - /*------------------------------------------------------------------------*/ - - /** - * Creates a hash object. - * - * @private - * @constructor - * @param {Array} [entries] The key-value pairs to cache. - */ - function Hash(entries) { - var index = -1, - length = entries == null ? 0 : entries.length; - - this.clear(); - while (++index < length) { - var entry = entries[index]; - this.set(entry[0], entry[1]); - } - } - - /** - * Removes all key-value entries from the hash. - * - * @private - * @name clear - * @memberOf Hash - */ - function hashClear() { - this.__data__ = nativeCreate ? nativeCreate(null) : {}; - this.size = 0; - } - - /** - * Removes `key` and its value from the hash. - * - * @private - * @name delete - * @memberOf Hash - * @param {Object} hash The hash to modify. - * @param {string} key The key of the value to remove. - * @returns {boolean} Returns `true` if the entry was removed, else `false`. - */ - function hashDelete(key) { - var result = this.has(key) && delete this.__data__[key]; - this.size -= result ? 1 : 0; - return result; - } - - /** - * Gets the hash value for `key`. - * - * @private - * @name get - * @memberOf Hash - * @param {string} key The key of the value to get. - * @returns {*} Returns the entry value. - */ - function hashGet(key) { - var data = this.__data__; - if (nativeCreate) { - var result = data[key]; - return result === HASH_UNDEFINED ? undefined : result; - } - return hasOwnProperty.call(data, key) ? data[key] : undefined; - } - - /** - * Checks if a hash value for `key` exists. - * - * @private - * @name has - * @memberOf Hash - * @param {string} key The key of the entry to check. - * @returns {boolean} Returns `true` if an entry for `key` exists, else `false`. - */ - function hashHas(key) { - var data = this.__data__; - return nativeCreate ? (data[key] !== undefined) : hasOwnProperty.call(data, key); - } - - /** - * Sets the hash `key` to `value`. - * - * @private - * @name set - * @memberOf Hash - * @param {string} key The key of the value to set. - * @param {*} value The value to set. - * @returns {Object} Returns the hash instance. - */ - function hashSet(key, value) { - var data = this.__data__; - this.size += this.has(key) ? 0 : 1; - data[key] = (nativeCreate && value === undefined) ? HASH_UNDEFINED : value; - return this; - } - - // Add methods to `Hash`. - Hash.prototype.clear = hashClear; - Hash.prototype['delete'] = hashDelete; - Hash.prototype.get = hashGet; - Hash.prototype.has = hashHas; - Hash.prototype.set = hashSet; - - /*------------------------------------------------------------------------*/ - - /** - * Creates an list cache object. - * - * @private - * @constructor - * @param {Array} [entries] The key-value pairs to cache. - */ - function ListCache(entries) { - var index = -1, - length = entries == null ? 0 : entries.length; - - this.clear(); - while (++index < length) { - var entry = entries[index]; - this.set(entry[0], entry[1]); - } - } - - /** - * Removes all key-value entries from the list cache. - * - * @private - * @name clear - * @memberOf ListCache - */ - function listCacheClear() { - this.__data__ = []; - this.size = 0; - } - - /** - * Removes `key` and its value from the list cache. - * - * @private - * @name delete - * @memberOf ListCache - * @param {string} key The key of the value to remove. - * @returns {boolean} Returns `true` if the entry was removed, else `false`. - */ - function listCacheDelete(key) { - var data = this.__data__, - index = assocIndexOf(data, key); - - if (index < 0) { - return false; - } - var lastIndex = data.length - 1; - if (index == lastIndex) { - data.pop(); - } else { - splice.call(data, index, 1); - } - --this.size; - return true; - } - - /** - * Gets the list cache value for `key`. - * - * @private - * @name get - * @memberOf ListCache - * @param {string} key The key of the value to get. - * @returns {*} Returns the entry value. - */ - function listCacheGet(key) { - var data = this.__data__, - index = assocIndexOf(data, key); - - return index < 0 ? undefined : data[index][1]; - } - - /** - * Checks if a list cache value for `key` exists. - * - * @private - * @name has - * @memberOf ListCache - * @param {string} key The key of the entry to check. - * @returns {boolean} Returns `true` if an entry for `key` exists, else `false`. - */ - function listCacheHas(key) { - return assocIndexOf(this.__data__, key) > -1; - } - - /** - * Sets the list cache `key` to `value`. - * - * @private - * @name set - * @memberOf ListCache - * @param {string} key The key of the value to set. - * @param {*} value The value to set. - * @returns {Object} Returns the list cache instance. - */ - function listCacheSet(key, value) { - var data = this.__data__, - index = assocIndexOf(data, key); - - if (index < 0) { - ++this.size; - data.push([key, value]); - } else { - data[index][1] = value; - } - return this; - } - - // Add methods to `ListCache`. - ListCache.prototype.clear = listCacheClear; - ListCache.prototype['delete'] = listCacheDelete; - ListCache.prototype.get = listCacheGet; - ListCache.prototype.has = listCacheHas; - ListCache.prototype.set = listCacheSet; - - /*------------------------------------------------------------------------*/ - - /** - * Creates a map cache object to store key-value pairs. - * - * @private - * @constructor - * @param {Array} [entries] The key-value pairs to cache. - */ - function MapCache(entries) { - var index = -1, - length = entries == null ? 0 : entries.length; - - this.clear(); - while (++index < length) { - var entry = entries[index]; - this.set(entry[0], entry[1]); - } - } - - /** - * Removes all key-value entries from the map. - * - * @private - * @name clear - * @memberOf MapCache - */ - function mapCacheClear() { - this.size = 0; - this.__data__ = { - 'hash': new Hash, - 'map': new (Map || ListCache), - 'string': new Hash - }; - } - - /** - * Removes `key` and its value from the map. - * - * @private - * @name delete - * @memberOf MapCache - * @param {string} key The key of the value to remove. - * @returns {boolean} Returns `true` if the entry was removed, else `false`. - */ - function mapCacheDelete(key) { - var result = getMapData(this, key)['delete'](key); - this.size -= result ? 1 : 0; - return result; - } - - /** - * Gets the map value for `key`. - * - * @private - * @name get - * @memberOf MapCache - * @param {string} key The key of the value to get. - * @returns {*} Returns the entry value. - */ - function mapCacheGet(key) { - return getMapData(this, key).get(key); - } - - /** - * Checks if a map value for `key` exists. - * - * @private - * @name has - * @memberOf MapCache - * @param {string} key The key of the entry to check. - * @returns {boolean} Returns `true` if an entry for `key` exists, else `false`. - */ - function mapCacheHas(key) { - return getMapData(this, key).has(key); - } - - /** - * Sets the map `key` to `value`. - * - * @private - * @name set - * @memberOf MapCache - * @param {string} key The key of the value to set. - * @param {*} value The value to set. - * @returns {Object} Returns the map cache instance. - */ - function mapCacheSet(key, value) { - var data = getMapData(this, key), - size = data.size; - - data.set(key, value); - this.size += data.size == size ? 0 : 1; - return this; - } - - // Add methods to `MapCache`. - MapCache.prototype.clear = mapCacheClear; - MapCache.prototype['delete'] = mapCacheDelete; - MapCache.prototype.get = mapCacheGet; - MapCache.prototype.has = mapCacheHas; - MapCache.prototype.set = mapCacheSet; - - /*------------------------------------------------------------------------*/ - - /** - * - * Creates an array cache object to store unique values. - * - * @private - * @constructor - * @param {Array} [values] The values to cache. - */ - function SetCache(values) { - var index = -1, - length = values == null ? 0 : values.length; - - this.__data__ = new MapCache; - while (++index < length) { - this.add(values[index]); - } - } - - /** - * Adds `value` to the array cache. - * - * @private - * @name add - * @memberOf SetCache - * @alias push - * @param {*} value The value to cache. - * @returns {Object} Returns the cache instance. - */ - function setCacheAdd(value) { - this.__data__.set(value, HASH_UNDEFINED); - return this; - } - - /** - * Checks if `value` is in the array cache. - * - * @private - * @name has - * @memberOf SetCache - * @param {*} value The value to search for. - * @returns {number} Returns `true` if `value` is found, else `false`. - */ - function setCacheHas(value) { - return this.__data__.has(value); - } - - // Add methods to `SetCache`. - SetCache.prototype.add = SetCache.prototype.push = setCacheAdd; - SetCache.prototype.has = setCacheHas; - - /*------------------------------------------------------------------------*/ - - /** - * Creates a stack cache object to store key-value pairs. - * - * @private - * @constructor - * @param {Array} [entries] The key-value pairs to cache. - */ - function Stack(entries) { - var data = this.__data__ = new ListCache(entries); - this.size = data.size; - } - - /** - * Removes all key-value entries from the stack. - * - * @private - * @name clear - * @memberOf Stack - */ - function stackClear() { - this.__data__ = new ListCache; - this.size = 0; - } - - /** - * Removes `key` and its value from the stack. - * - * @private - * @name delete - * @memberOf Stack - * @param {string} key The key of the value to remove. - * @returns {boolean} Returns `true` if the entry was removed, else `false`. - */ - function stackDelete(key) { - var data = this.__data__, - result = data['delete'](key); - - this.size = data.size; - return result; - } - - /** - * Gets the stack value for `key`. - * - * @private - * @name get - * @memberOf Stack - * @param {string} key The key of the value to get. - * @returns {*} Returns the entry value. - */ - function stackGet(key) { - return this.__data__.get(key); - } - - /** - * Checks if a stack value for `key` exists. - * - * @private - * @name has - * @memberOf Stack - * @param {string} key The key of the entry to check. - * @returns {boolean} Returns `true` if an entry for `key` exists, else `false`. - */ - function stackHas(key) { - return this.__data__.has(key); - } - - /** - * Sets the stack `key` to `value`. - * - * @private - * @name set - * @memberOf Stack - * @param {string} key The key of the value to set. - * @param {*} value The value to set. - * @returns {Object} Returns the stack cache instance. - */ - function stackSet(key, value) { - var data = this.__data__; - if (data instanceof ListCache) { - var pairs = data.__data__; - if (!Map || (pairs.length < LARGE_ARRAY_SIZE - 1)) { - pairs.push([key, value]); - this.size = ++data.size; - return this; - } - data = this.__data__ = new MapCache(pairs); - } - data.set(key, value); - this.size = data.size; - return this; - } - - // Add methods to `Stack`. - Stack.prototype.clear = stackClear; - Stack.prototype['delete'] = stackDelete; - Stack.prototype.get = stackGet; - Stack.prototype.has = stackHas; - Stack.prototype.set = stackSet; - - /*------------------------------------------------------------------------*/ - - /** - * Creates an array of the enumerable property names of the array-like `value`. - * - * @private - * @param {*} value The value to query. - * @param {boolean} inherited Specify returning inherited property names. - * @returns {Array} Returns the array of property names. - */ - function arrayLikeKeys(value, inherited) { - var isArr = isArray(value), - isArg = !isArr && isArguments(value), - isBuff = !isArr && !isArg && isBuffer(value), - isType = !isArr && !isArg && !isBuff && isTypedArray(value), - skipIndexes = isArr || isArg || isBuff || isType, - result = skipIndexes ? baseTimes(value.length, String) : [], - length = result.length; - - for (var key in value) { - if ((inherited || hasOwnProperty.call(value, key)) && - !(skipIndexes && ( - // Safari 9 has enumerable `arguments.length` in strict mode. - key == 'length' || - // Node.js 0.10 has enumerable non-index properties on buffers. - (isBuff && (key == 'offset' || key == 'parent')) || - // PhantomJS 2 has enumerable non-index properties on typed arrays. - (isType && (key == 'buffer' || key == 'byteLength' || key == 'byteOffset')) || - // Skip index properties. - isIndex(key, length) - ))) { - result.push(key); - } - } - return result; - } - - /** - * A specialized version of `_.sample` for arrays. - * - * @private - * @param {Array} array The array to sample. - * @returns {*} Returns the random element. - */ - function arraySample(array) { - var length = array.length; - return length ? array[baseRandom(0, length - 1)] : undefined; - } - - /** - * A specialized version of `_.sampleSize` for arrays. - * - * @private - * @param {Array} array The array to sample. - * @param {number} n The number of elements to sample. - * @returns {Array} Returns the random elements. - */ - function arraySampleSize(array, n) { - return shuffleSelf(copyArray(array), baseClamp(n, 0, array.length)); - } - - /** - * A specialized version of `_.shuffle` for arrays. - * - * @private - * @param {Array} array The array to shuffle. - * @returns {Array} Returns the new shuffled array. - */ - function arrayShuffle(array) { - return shuffleSelf(copyArray(array)); - } - - /** - * This function is like `assignValue` except that it doesn't assign - * `undefined` values. - * - * @private - * @param {Object} object The object to modify. - * @param {string} key The key of the property to assign. - * @param {*} value The value to assign. - */ - function assignMergeValue(object, key, value) { - if ((value !== undefined && !eq(object[key], value)) || - (value === undefined && !(key in object))) { - baseAssignValue(object, key, value); - } - } - - /** - * Assigns `value` to `key` of `object` if the existing value is not equivalent - * using [`SameValueZero`](http://ecma-international.org/ecma-262/7.0/#sec-samevaluezero) - * for equality comparisons. - * - * @private - * @param {Object} object The object to modify. - * @param {string} key The key of the property to assign. - * @param {*} value The value to assign. - */ - function assignValue(object, key, value) { - var objValue = object[key]; - if (!(hasOwnProperty.call(object, key) && eq(objValue, value)) || - (value === undefined && !(key in object))) { - baseAssignValue(object, key, value); - } - } - - /** - * Gets the index at which the `key` is found in `array` of key-value pairs. - * - * @private - * @param {Array} array The array to inspect. - * @param {*} key The key to search for. - * @returns {number} Returns the index of the matched value, else `-1`. - */ - function assocIndexOf(array, key) { - var length = array.length; - while (length--) { - if (eq(array[length][0], key)) { - return length; - } - } - return -1; - } - - /** - * Aggregates elements of `collection` on `accumulator` with keys transformed - * by `iteratee` and values set by `setter`. - * - * @private - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} setter The function to set `accumulator` values. - * @param {Function} iteratee The iteratee to transform keys. - * @param {Object} accumulator The initial aggregated object. - * @returns {Function} Returns `accumulator`. - */ - function baseAggregator(collection, setter, iteratee, accumulator) { - baseEach(collection, function(value, key, collection) { - setter(accumulator, value, iteratee(value), collection); - }); - return accumulator; - } - - /** - * The base implementation of `_.assign` without support for multiple sources - * or `customizer` functions. - * - * @private - * @param {Object} object The destination object. - * @param {Object} source The source object. - * @returns {Object} Returns `object`. - */ - function baseAssign(object, source) { - return object && copyObject(source, keys(source), object); - } - - /** - * The base implementation of `_.assignIn` without support for multiple sources - * or `customizer` functions. - * - * @private - * @param {Object} object The destination object. - * @param {Object} source The source object. - * @returns {Object} Returns `object`. - */ - function baseAssignIn(object, source) { - return object && copyObject(source, keysIn(source), object); - } - - /** - * The base implementation of `assignValue` and `assignMergeValue` without - * value checks. - * - * @private - * @param {Object} object The object to modify. - * @param {string} key The key of the property to assign. - * @param {*} value The value to assign. - */ - function baseAssignValue(object, key, value) { - if (key == '__proto__' && defineProperty) { - defineProperty(object, key, { - 'configurable': true, - 'enumerable': true, - 'value': value, - 'writable': true - }); - } else { - object[key] = value; - } - } - - /** - * The base implementation of `_.at` without support for individual paths. - * - * @private - * @param {Object} object The object to iterate over. - * @param {string[]} paths The property paths to pick. - * @returns {Array} Returns the picked elements. - */ - function baseAt(object, paths) { - var index = -1, - length = paths.length, - result = Array(length), - skip = object == null; - - while (++index < length) { - result[index] = skip ? undefined : get(object, paths[index]); - } - return result; - } - - /** - * The base implementation of `_.clamp` which doesn't coerce arguments. - * - * @private - * @param {number} number The number to clamp. - * @param {number} [lower] The lower bound. - * @param {number} upper The upper bound. - * @returns {number} Returns the clamped number. - */ - function baseClamp(number, lower, upper) { - if (number === number) { - if (upper !== undefined) { - number = number <= upper ? number : upper; - } - if (lower !== undefined) { - number = number >= lower ? number : lower; - } - } - return number; - } - - /** - * The base implementation of `_.clone` and `_.cloneDeep` which tracks - * traversed objects. - * - * @private - * @param {*} value The value to clone. - * @param {boolean} bitmask The bitmask flags. - * 1 - Deep clone - * 2 - Flatten inherited properties - * 4 - Clone symbols - * @param {Function} [customizer] The function to customize cloning. - * @param {string} [key] The key of `value`. - * @param {Object} [object] The parent object of `value`. - * @param {Object} [stack] Tracks traversed objects and their clone counterparts. - * @returns {*} Returns the cloned value. - */ - function baseClone(value, bitmask, customizer, key, object, stack) { - var result, - isDeep = bitmask & CLONE_DEEP_FLAG, - isFlat = bitmask & CLONE_FLAT_FLAG, - isFull = bitmask & CLONE_SYMBOLS_FLAG; - - if (customizer) { - result = object ? customizer(value, key, object, stack) : customizer(value); - } - if (result !== undefined) { - return result; - } - if (!isObject(value)) { - return value; - } - var isArr = isArray(value); - if (isArr) { - result = initCloneArray(value); - if (!isDeep) { - return copyArray(value, result); - } - } else { - var tag = getTag(value), - isFunc = tag == funcTag || tag == genTag; - - if (isBuffer(value)) { - return cloneBuffer(value, isDeep); - } - if (tag == objectTag || tag == argsTag || (isFunc && !object)) { - result = (isFlat || isFunc) ? {} : initCloneObject(value); - if (!isDeep) { - return isFlat - ? copySymbolsIn(value, baseAssignIn(result, value)) - : copySymbols(value, baseAssign(result, value)); - } - } else { - if (!cloneableTags[tag]) { - return object ? value : {}; - } - result = initCloneByTag(value, tag, isDeep); - } - } - // Check for circular references and return its corresponding clone. - stack || (stack = new Stack); - var stacked = stack.get(value); - if (stacked) { - return stacked; - } - stack.set(value, result); - - if (isSet(value)) { - value.forEach(function(subValue) { - result.add(baseClone(subValue, bitmask, customizer, subValue, value, stack)); - }); - } else if (isMap(value)) { - value.forEach(function(subValue, key) { - result.set(key, baseClone(subValue, bitmask, customizer, key, value, stack)); - }); - } - - var keysFunc = isFull - ? (isFlat ? getAllKeysIn : getAllKeys) - : (isFlat ? keysIn : keys); - - var props = isArr ? undefined : keysFunc(value); - arrayEach(props || value, function(subValue, key) { - if (props) { - key = subValue; - subValue = value[key]; - } - // Recursively populate clone (susceptible to call stack limits). - assignValue(result, key, baseClone(subValue, bitmask, customizer, key, value, stack)); - }); - return result; - } - - /** - * The base implementation of `_.conforms` which doesn't clone `source`. - * - * @private - * @param {Object} source The object of property predicates to conform to. - * @returns {Function} Returns the new spec function. - */ - function baseConforms(source) { - var props = keys(source); - return function(object) { - return baseConformsTo(object, source, props); - }; - } - - /** - * The base implementation of `_.conformsTo` which accepts `props` to check. - * - * @private - * @param {Object} object The object to inspect. - * @param {Object} source The object of property predicates to conform to. - * @returns {boolean} Returns `true` if `object` conforms, else `false`. - */ - function baseConformsTo(object, source, props) { - var length = props.length; - if (object == null) { - return !length; - } - object = Object(object); - while (length--) { - var key = props[length], - predicate = source[key], - value = object[key]; - - if ((value === undefined && !(key in object)) || !predicate(value)) { - return false; - } - } - return true; - } - - /** - * The base implementation of `_.delay` and `_.defer` which accepts `args` - * to provide to `func`. - * - * @private - * @param {Function} func The function to delay. - * @param {number} wait The number of milliseconds to delay invocation. - * @param {Array} args The arguments to provide to `func`. - * @returns {number|Object} Returns the timer id or timeout object. - */ - function baseDelay(func, wait, args) { - if (typeof func != 'function') { - throw new TypeError(FUNC_ERROR_TEXT); - } - return setTimeout(function() { func.apply(undefined, args); }, wait); - } - - /** - * The base implementation of methods like `_.difference` without support - * for excluding multiple arrays or iteratee shorthands. - * - * @private - * @param {Array} array The array to inspect. - * @param {Array} values The values to exclude. - * @param {Function} [iteratee] The iteratee invoked per element. - * @param {Function} [comparator] The comparator invoked per element. - * @returns {Array} Returns the new array of filtered values. - */ - function baseDifference(array, values, iteratee, comparator) { - var index = -1, - includes = arrayIncludes, - isCommon = true, - length = array.length, - result = [], - valuesLength = values.length; - - if (!length) { - return result; - } - if (iteratee) { - values = arrayMap(values, baseUnary(iteratee)); - } - if (comparator) { - includes = arrayIncludesWith; - isCommon = false; - } - else if (values.length >= LARGE_ARRAY_SIZE) { - includes = cacheHas; - isCommon = false; - values = new SetCache(values); - } - outer: - while (++index < length) { - var value = array[index], - computed = iteratee == null ? value : iteratee(value); - - value = (comparator || value !== 0) ? value : 0; - if (isCommon && computed === computed) { - var valuesIndex = valuesLength; - while (valuesIndex--) { - if (values[valuesIndex] === computed) { - continue outer; - } - } - result.push(value); - } - else if (!includes(values, computed, comparator)) { - result.push(value); - } - } - return result; - } - - /** - * The base implementation of `_.forEach` without support for iteratee shorthands. - * - * @private - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} iteratee The function invoked per iteration. - * @returns {Array|Object} Returns `collection`. - */ - var baseEach = createBaseEach(baseForOwn); - - /** - * The base implementation of `_.forEachRight` without support for iteratee shorthands. - * - * @private - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} iteratee The function invoked per iteration. - * @returns {Array|Object} Returns `collection`. - */ - var baseEachRight = createBaseEach(baseForOwnRight, true); - - /** - * The base implementation of `_.every` without support for iteratee shorthands. - * - * @private - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} predicate The function invoked per iteration. - * @returns {boolean} Returns `true` if all elements pass the predicate check, - * else `false` - */ - function baseEvery(collection, predicate) { - var result = true; - baseEach(collection, function(value, index, collection) { - result = !!predicate(value, index, collection); - return result; - }); - return result; - } - - /** - * The base implementation of methods like `_.max` and `_.min` which accepts a - * `comparator` to determine the extremum value. - * - * @private - * @param {Array} array The array to iterate over. - * @param {Function} iteratee The iteratee invoked per iteration. - * @param {Function} comparator The comparator used to compare values. - * @returns {*} Returns the extremum value. - */ - function baseExtremum(array, iteratee, comparator) { - var index = -1, - length = array.length; - - while (++index < length) { - var value = array[index], - current = iteratee(value); - - if (current != null && (computed === undefined - ? (current === current && !isSymbol(current)) - : comparator(current, computed) - )) { - var computed = current, - result = value; - } - } - return result; - } - - /** - * The base implementation of `_.fill` without an iteratee call guard. - * - * @private - * @param {Array} array The array to fill. - * @param {*} value The value to fill `array` with. - * @param {number} [start=0] The start position. - * @param {number} [end=array.length] The end position. - * @returns {Array} Returns `array`. - */ - function baseFill(array, value, start, end) { - var length = array.length; - - start = toInteger(start); - if (start < 0) { - start = -start > length ? 0 : (length + start); - } - end = (end === undefined || end > length) ? length : toInteger(end); - if (end < 0) { - end += length; - } - end = start > end ? 0 : toLength(end); - while (start < end) { - array[start++] = value; - } - return array; - } - - /** - * The base implementation of `_.filter` without support for iteratee shorthands. - * - * @private - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} predicate The function invoked per iteration. - * @returns {Array} Returns the new filtered array. - */ - function baseFilter(collection, predicate) { - var result = []; - baseEach(collection, function(value, index, collection) { - if (predicate(value, index, collection)) { - result.push(value); - } - }); - return result; - } - - /** - * The base implementation of `_.flatten` with support for restricting flattening. - * - * @private - * @param {Array} array The array to flatten. - * @param {number} depth The maximum recursion depth. - * @param {boolean} [predicate=isFlattenable] The function invoked per iteration. - * @param {boolean} [isStrict] Restrict to values that pass `predicate` checks. - * @param {Array} [result=[]] The initial result value. - * @returns {Array} Returns the new flattened array. - */ - function baseFlatten(array, depth, predicate, isStrict, result) { - var index = -1, - length = array.length; - - predicate || (predicate = isFlattenable); - result || (result = []); - - while (++index < length) { - var value = array[index]; - if (depth > 0 && predicate(value)) { - if (depth > 1) { - // Recursively flatten arrays (susceptible to call stack limits). - baseFlatten(value, depth - 1, predicate, isStrict, result); - } else { - arrayPush(result, value); - } - } else if (!isStrict) { - result[result.length] = value; - } - } - return result; - } - - /** - * The base implementation of `baseForOwn` which iterates over `object` - * properties returned by `keysFunc` and invokes `iteratee` for each property. - * Iteratee functions may exit iteration early by explicitly returning `false`. - * - * @private - * @param {Object} object The object to iterate over. - * @param {Function} iteratee The function invoked per iteration. - * @param {Function} keysFunc The function to get the keys of `object`. - * @returns {Object} Returns `object`. - */ - var baseFor = createBaseFor(); - - /** - * This function is like `baseFor` except that it iterates over properties - * in the opposite order. - * - * @private - * @param {Object} object The object to iterate over. - * @param {Function} iteratee The function invoked per iteration. - * @param {Function} keysFunc The function to get the keys of `object`. - * @returns {Object} Returns `object`. - */ - var baseForRight = createBaseFor(true); - - /** - * The base implementation of `_.forOwn` without support for iteratee shorthands. - * - * @private - * @param {Object} object The object to iterate over. - * @param {Function} iteratee The function invoked per iteration. - * @returns {Object} Returns `object`. - */ - function baseForOwn(object, iteratee) { - return object && baseFor(object, iteratee, keys); - } - - /** - * The base implementation of `_.forOwnRight` without support for iteratee shorthands. - * - * @private - * @param {Object} object The object to iterate over. - * @param {Function} iteratee The function invoked per iteration. - * @returns {Object} Returns `object`. - */ - function baseForOwnRight(object, iteratee) { - return object && baseForRight(object, iteratee, keys); - } - - /** - * The base implementation of `_.functions` which creates an array of - * `object` function property names filtered from `props`. - * - * @private - * @param {Object} object The object to inspect. - * @param {Array} props The property names to filter. - * @returns {Array} Returns the function names. - */ - function baseFunctions(object, props) { - return arrayFilter(props, function(key) { - return isFunction(object[key]); - }); - } - - /** - * The base implementation of `_.get` without support for default values. - * - * @private - * @param {Object} object The object to query. - * @param {Array|string} path The path of the property to get. - * @returns {*} Returns the resolved value. - */ - function baseGet(object, path) { - path = castPath(path, object); - - var index = 0, - length = path.length; - - while (object != null && index < length) { - object = object[toKey(path[index++])]; - } - return (index && index == length) ? object : undefined; - } - - /** - * The base implementation of `getAllKeys` and `getAllKeysIn` which uses - * `keysFunc` and `symbolsFunc` to get the enumerable property names and - * symbols of `object`. - * - * @private - * @param {Object} object The object to query. - * @param {Function} keysFunc The function to get the keys of `object`. - * @param {Function} symbolsFunc The function to get the symbols of `object`. - * @returns {Array} Returns the array of property names and symbols. - */ - function baseGetAllKeys(object, keysFunc, symbolsFunc) { - var result = keysFunc(object); - return isArray(object) ? result : arrayPush(result, symbolsFunc(object)); - } - - /** - * The base implementation of `getTag` without fallbacks for buggy environments. - * - * @private - * @param {*} value The value to query. - * @returns {string} Returns the `toStringTag`. - */ - function baseGetTag(value) { - if (value == null) { - return value === undefined ? undefinedTag : nullTag; - } - return (symToStringTag && symToStringTag in Object(value)) - ? getRawTag(value) - : objectToString(value); - } - - /** - * The base implementation of `_.gt` which doesn't coerce arguments. - * - * @private - * @param {*} value The value to compare. - * @param {*} other The other value to compare. - * @returns {boolean} Returns `true` if `value` is greater than `other`, - * else `false`. - */ - function baseGt(value, other) { - return value > other; - } - - /** - * The base implementation of `_.has` without support for deep paths. - * - * @private - * @param {Object} [object] The object to query. - * @param {Array|string} key The key to check. - * @returns {boolean} Returns `true` if `key` exists, else `false`. - */ - function baseHas(object, key) { - return object != null && hasOwnProperty.call(object, key); - } - - /** - * The base implementation of `_.hasIn` without support for deep paths. - * - * @private - * @param {Object} [object] The object to query. - * @param {Array|string} key The key to check. - * @returns {boolean} Returns `true` if `key` exists, else `false`. - */ - function baseHasIn(object, key) { - return object != null && key in Object(object); - } - - /** - * The base implementation of `_.inRange` which doesn't coerce arguments. - * - * @private - * @param {number} number The number to check. - * @param {number} start The start of the range. - * @param {number} end The end of the range. - * @returns {boolean} Returns `true` if `number` is in the range, else `false`. - */ - function baseInRange(number, start, end) { - return number >= nativeMin(start, end) && number < nativeMax(start, end); - } - - /** - * The base implementation of methods like `_.intersection`, without support - * for iteratee shorthands, that accepts an array of arrays to inspect. - * - * @private - * @param {Array} arrays The arrays to inspect. - * @param {Function} [iteratee] The iteratee invoked per element. - * @param {Function} [comparator] The comparator invoked per element. - * @returns {Array} Returns the new array of shared values. - */ - function baseIntersection(arrays, iteratee, comparator) { - var includes = comparator ? arrayIncludesWith : arrayIncludes, - length = arrays[0].length, - othLength = arrays.length, - othIndex = othLength, - caches = Array(othLength), - maxLength = Infinity, - result = []; - - while (othIndex--) { - var array = arrays[othIndex]; - if (othIndex && iteratee) { - array = arrayMap(array, baseUnary(iteratee)); - } - maxLength = nativeMin(array.length, maxLength); - caches[othIndex] = !comparator && (iteratee || (length >= 120 && array.length >= 120)) - ? new SetCache(othIndex && array) - : undefined; - } - array = arrays[0]; - - var index = -1, - seen = caches[0]; - - outer: - while (++index < length && result.length < maxLength) { - var value = array[index], - computed = iteratee ? iteratee(value) : value; - - value = (comparator || value !== 0) ? value : 0; - if (!(seen - ? cacheHas(seen, computed) - : includes(result, computed, comparator) - )) { - othIndex = othLength; - while (--othIndex) { - var cache = caches[othIndex]; - if (!(cache - ? cacheHas(cache, computed) - : includes(arrays[othIndex], computed, comparator)) - ) { - continue outer; - } - } - if (seen) { - seen.push(computed); - } - result.push(value); - } - } - return result; - } - - /** - * The base implementation of `_.invert` and `_.invertBy` which inverts - * `object` with values transformed by `iteratee` and set by `setter`. - * - * @private - * @param {Object} object The object to iterate over. - * @param {Function} setter The function to set `accumulator` values. - * @param {Function} iteratee The iteratee to transform values. - * @param {Object} accumulator The initial inverted object. - * @returns {Function} Returns `accumulator`. - */ - function baseInverter(object, setter, iteratee, accumulator) { - baseForOwn(object, function(value, key, object) { - setter(accumulator, iteratee(value), key, object); - }); - return accumulator; - } - - /** - * The base implementation of `_.invoke` without support for individual - * method arguments. - * - * @private - * @param {Object} object The object to query. - * @param {Array|string} path The path of the method to invoke. - * @param {Array} args The arguments to invoke the method with. - * @returns {*} Returns the result of the invoked method. - */ - function baseInvoke(object, path, args) { - path = castPath(path, object); - object = parent(object, path); - var func = object == null ? object : object[toKey(last(path))]; - return func == null ? undefined : apply(func, object, args); - } - - /** - * The base implementation of `_.isArguments`. - * - * @private - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is an `arguments` object, - */ - function baseIsArguments(value) { - return isObjectLike(value) && baseGetTag(value) == argsTag; - } - - /** - * The base implementation of `_.isArrayBuffer` without Node.js optimizations. - * - * @private - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is an array buffer, else `false`. - */ - function baseIsArrayBuffer(value) { - return isObjectLike(value) && baseGetTag(value) == arrayBufferTag; - } - - /** - * The base implementation of `_.isDate` without Node.js optimizations. - * - * @private - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a date object, else `false`. - */ - function baseIsDate(value) { - return isObjectLike(value) && baseGetTag(value) == dateTag; - } - - /** - * The base implementation of `_.isEqual` which supports partial comparisons - * and tracks traversed objects. - * - * @private - * @param {*} value The value to compare. - * @param {*} other The other value to compare. - * @param {boolean} bitmask The bitmask flags. - * 1 - Unordered comparison - * 2 - Partial comparison - * @param {Function} [customizer] The function to customize comparisons. - * @param {Object} [stack] Tracks traversed `value` and `other` objects. - * @returns {boolean} Returns `true` if the values are equivalent, else `false`. - */ - function baseIsEqual(value, other, bitmask, customizer, stack) { - if (value === other) { - return true; - } - if (value == null || other == null || (!isObjectLike(value) && !isObjectLike(other))) { - return value !== value && other !== other; - } - return baseIsEqualDeep(value, other, bitmask, customizer, baseIsEqual, stack); - } - - /** - * A specialized version of `baseIsEqual` for arrays and objects which performs - * deep comparisons and tracks traversed objects enabling objects with circular - * references to be compared. - * - * @private - * @param {Object} object The object to compare. - * @param {Object} other The other object to compare. - * @param {number} bitmask The bitmask flags. See `baseIsEqual` for more details. - * @param {Function} customizer The function to customize comparisons. - * @param {Function} equalFunc The function to determine equivalents of values. - * @param {Object} [stack] Tracks traversed `object` and `other` objects. - * @returns {boolean} Returns `true` if the objects are equivalent, else `false`. - */ - function baseIsEqualDeep(object, other, bitmask, customizer, equalFunc, stack) { - var objIsArr = isArray(object), - othIsArr = isArray(other), - objTag = objIsArr ? arrayTag : getTag(object), - othTag = othIsArr ? arrayTag : getTag(other); - - objTag = objTag == argsTag ? objectTag : objTag; - othTag = othTag == argsTag ? objectTag : othTag; - - var objIsObj = objTag == objectTag, - othIsObj = othTag == objectTag, - isSameTag = objTag == othTag; - - if (isSameTag && isBuffer(object)) { - if (!isBuffer(other)) { - return false; - } - objIsArr = true; - objIsObj = false; - } - if (isSameTag && !objIsObj) { - stack || (stack = new Stack); - return (objIsArr || isTypedArray(object)) - ? equalArrays(object, other, bitmask, customizer, equalFunc, stack) - : equalByTag(object, other, objTag, bitmask, customizer, equalFunc, stack); - } - if (!(bitmask & COMPARE_PARTIAL_FLAG)) { - var objIsWrapped = objIsObj && hasOwnProperty.call(object, '__wrapped__'), - othIsWrapped = othIsObj && hasOwnProperty.call(other, '__wrapped__'); - - if (objIsWrapped || othIsWrapped) { - var objUnwrapped = objIsWrapped ? object.value() : object, - othUnwrapped = othIsWrapped ? other.value() : other; - - stack || (stack = new Stack); - return equalFunc(objUnwrapped, othUnwrapped, bitmask, customizer, stack); - } - } - if (!isSameTag) { - return false; - } - stack || (stack = new Stack); - return equalObjects(object, other, bitmask, customizer, equalFunc, stack); - } - - /** - * The base implementation of `_.isMap` without Node.js optimizations. - * - * @private - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a map, else `false`. - */ - function baseIsMap(value) { - return isObjectLike(value) && getTag(value) == mapTag; - } - - /** - * The base implementation of `_.isMatch` without support for iteratee shorthands. - * - * @private - * @param {Object} object The object to inspect. - * @param {Object} source The object of property values to match. - * @param {Array} matchData The property names, values, and compare flags to match. - * @param {Function} [customizer] The function to customize comparisons. - * @returns {boolean} Returns `true` if `object` is a match, else `false`. - */ - function baseIsMatch(object, source, matchData, customizer) { - var index = matchData.length, - length = index, - noCustomizer = !customizer; - - if (object == null) { - return !length; - } - object = Object(object); - while (index--) { - var data = matchData[index]; - if ((noCustomizer && data[2]) - ? data[1] !== object[data[0]] - : !(data[0] in object) - ) { - return false; - } - } - while (++index < length) { - data = matchData[index]; - var key = data[0], - objValue = object[key], - srcValue = data[1]; - - if (noCustomizer && data[2]) { - if (objValue === undefined && !(key in object)) { - return false; - } - } else { - var stack = new Stack; - if (customizer) { - var result = customizer(objValue, srcValue, key, object, source, stack); - } - if (!(result === undefined - ? baseIsEqual(srcValue, objValue, COMPARE_PARTIAL_FLAG | COMPARE_UNORDERED_FLAG, customizer, stack) - : result - )) { - return false; - } - } - } - return true; - } - - /** - * The base implementation of `_.isNative` without bad shim checks. - * - * @private - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a native function, - * else `false`. - */ - function baseIsNative(value) { - if (!isObject(value) || isMasked(value)) { - return false; - } - var pattern = isFunction(value) ? reIsNative : reIsHostCtor; - return pattern.test(toSource(value)); - } - - /** - * The base implementation of `_.isRegExp` without Node.js optimizations. - * - * @private - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a regexp, else `false`. - */ - function baseIsRegExp(value) { - return isObjectLike(value) && baseGetTag(value) == regexpTag; - } - - /** - * The base implementation of `_.isSet` without Node.js optimizations. - * - * @private - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a set, else `false`. - */ - function baseIsSet(value) { - return isObjectLike(value) && getTag(value) == setTag; - } - - /** - * The base implementation of `_.isTypedArray` without Node.js optimizations. - * - * @private - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a typed array, else `false`. - */ - function baseIsTypedArray(value) { - return isObjectLike(value) && - isLength(value.length) && !!typedArrayTags[baseGetTag(value)]; - } - - /** - * The base implementation of `_.iteratee`. - * - * @private - * @param {*} [value=_.identity] The value to convert to an iteratee. - * @returns {Function} Returns the iteratee. - */ - function baseIteratee(value) { - // Don't store the `typeof` result in a variable to avoid a JIT bug in Safari 9. - // See https://bugs.webkit.org/show_bug.cgi?id=156034 for more details. - if (typeof value == 'function') { - return value; - } - if (value == null) { - return identity; - } - if (typeof value == 'object') { - return isArray(value) - ? baseMatchesProperty(value[0], value[1]) - : baseMatches(value); - } - return property(value); - } - - /** - * The base implementation of `_.keys` which doesn't treat sparse arrays as dense. - * - * @private - * @param {Object} object The object to query. - * @returns {Array} Returns the array of property names. - */ - function baseKeys(object) { - if (!isPrototype(object)) { - return nativeKeys(object); - } - var result = []; - for (var key in Object(object)) { - if (hasOwnProperty.call(object, key) && key != 'constructor') { - result.push(key); - } - } - return result; - } - - /** - * The base implementation of `_.keysIn` which doesn't treat sparse arrays as dense. - * - * @private - * @param {Object} object The object to query. - * @returns {Array} Returns the array of property names. - */ - function baseKeysIn(object) { - if (!isObject(object)) { - return nativeKeysIn(object); - } - var isProto = isPrototype(object), - result = []; - - for (var key in object) { - if (!(key == 'constructor' && (isProto || !hasOwnProperty.call(object, key)))) { - result.push(key); - } - } - return result; - } - - /** - * The base implementation of `_.lt` which doesn't coerce arguments. - * - * @private - * @param {*} value The value to compare. - * @param {*} other The other value to compare. - * @returns {boolean} Returns `true` if `value` is less than `other`, - * else `false`. - */ - function baseLt(value, other) { - return value < other; - } - - /** - * The base implementation of `_.map` without support for iteratee shorthands. - * - * @private - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} iteratee The function invoked per iteration. - * @returns {Array} Returns the new mapped array. - */ - function baseMap(collection, iteratee) { - var index = -1, - result = isArrayLike(collection) ? Array(collection.length) : []; - - baseEach(collection, function(value, key, collection) { - result[++index] = iteratee(value, key, collection); - }); - return result; - } - - /** - * The base implementation of `_.matches` which doesn't clone `source`. - * - * @private - * @param {Object} source The object of property values to match. - * @returns {Function} Returns the new spec function. - */ - function baseMatches(source) { - var matchData = getMatchData(source); - if (matchData.length == 1 && matchData[0][2]) { - return matchesStrictComparable(matchData[0][0], matchData[0][1]); - } - return function(object) { - return object === source || baseIsMatch(object, source, matchData); - }; - } - - /** - * The base implementation of `_.matchesProperty` which doesn't clone `srcValue`. - * - * @private - * @param {string} path The path of the property to get. - * @param {*} srcValue The value to match. - * @returns {Function} Returns the new spec function. - */ - function baseMatchesProperty(path, srcValue) { - if (isKey(path) && isStrictComparable(srcValue)) { - return matchesStrictComparable(toKey(path), srcValue); - } - return function(object) { - var objValue = get(object, path); - return (objValue === undefined && objValue === srcValue) - ? hasIn(object, path) - : baseIsEqual(srcValue, objValue, COMPARE_PARTIAL_FLAG | COMPARE_UNORDERED_FLAG); - }; - } - - /** - * The base implementation of `_.merge` without support for multiple sources. - * - * @private - * @param {Object} object The destination object. - * @param {Object} source The source object. - * @param {number} srcIndex The index of `source`. - * @param {Function} [customizer] The function to customize merged values. - * @param {Object} [stack] Tracks traversed source values and their merged - * counterparts. - */ - function baseMerge(object, source, srcIndex, customizer, stack) { - if (object === source) { - return; - } - baseFor(source, function(srcValue, key) { - stack || (stack = new Stack); - if (isObject(srcValue)) { - baseMergeDeep(object, source, key, srcIndex, baseMerge, customizer, stack); - } - else { - var newValue = customizer - ? customizer(safeGet(object, key), srcValue, (key + ''), object, source, stack) - : undefined; - - if (newValue === undefined) { - newValue = srcValue; - } - assignMergeValue(object, key, newValue); - } - }, keysIn); - } - - /** - * A specialized version of `baseMerge` for arrays and objects which performs - * deep merges and tracks traversed objects enabling objects with circular - * references to be merged. - * - * @private - * @param {Object} object The destination object. - * @param {Object} source The source object. - * @param {string} key The key of the value to merge. - * @param {number} srcIndex The index of `source`. - * @param {Function} mergeFunc The function to merge values. - * @param {Function} [customizer] The function to customize assigned values. - * @param {Object} [stack] Tracks traversed source values and their merged - * counterparts. - */ - function baseMergeDeep(object, source, key, srcIndex, mergeFunc, customizer, stack) { - var objValue = safeGet(object, key), - srcValue = safeGet(source, key), - stacked = stack.get(srcValue); - - if (stacked) { - assignMergeValue(object, key, stacked); - return; - } - var newValue = customizer - ? customizer(objValue, srcValue, (key + ''), object, source, stack) - : undefined; - - var isCommon = newValue === undefined; - - if (isCommon) { - var isArr = isArray(srcValue), - isBuff = !isArr && isBuffer(srcValue), - isTyped = !isArr && !isBuff && isTypedArray(srcValue); - - newValue = srcValue; - if (isArr || isBuff || isTyped) { - if (isArray(objValue)) { - newValue = objValue; - } - else if (isArrayLikeObject(objValue)) { - newValue = copyArray(objValue); - } - else if (isBuff) { - isCommon = false; - newValue = cloneBuffer(srcValue, true); - } - else if (isTyped) { - isCommon = false; - newValue = cloneTypedArray(srcValue, true); - } - else { - newValue = []; - } - } - else if (isPlainObject(srcValue) || isArguments(srcValue)) { - newValue = objValue; - if (isArguments(objValue)) { - newValue = toPlainObject(objValue); - } - else if (!isObject(objValue) || isFunction(objValue)) { - newValue = initCloneObject(srcValue); - } - } - else { - isCommon = false; - } - } - if (isCommon) { - // Recursively merge objects and arrays (susceptible to call stack limits). - stack.set(srcValue, newValue); - mergeFunc(newValue, srcValue, srcIndex, customizer, stack); - stack['delete'](srcValue); - } - assignMergeValue(object, key, newValue); - } - - /** - * The base implementation of `_.nth` which doesn't coerce arguments. - * - * @private - * @param {Array} array The array to query. - * @param {number} n The index of the element to return. - * @returns {*} Returns the nth element of `array`. - */ - function baseNth(array, n) { - var length = array.length; - if (!length) { - return; - } - n += n < 0 ? length : 0; - return isIndex(n, length) ? array[n] : undefined; - } - - /** - * The base implementation of `_.orderBy` without param guards. - * - * @private - * @param {Array|Object} collection The collection to iterate over. - * @param {Function[]|Object[]|string[]} iteratees The iteratees to sort by. - * @param {string[]} orders The sort orders of `iteratees`. - * @returns {Array} Returns the new sorted array. - */ - function baseOrderBy(collection, iteratees, orders) { - var index = -1; - iteratees = arrayMap(iteratees.length ? iteratees : [identity], baseUnary(getIteratee())); - - var result = baseMap(collection, function(value, key, collection) { - var criteria = arrayMap(iteratees, function(iteratee) { - return iteratee(value); - }); - return { 'criteria': criteria, 'index': ++index, 'value': value }; - }); - - return baseSortBy(result, function(object, other) { - return compareMultiple(object, other, orders); - }); - } - - /** - * The base implementation of `_.pick` without support for individual - * property identifiers. - * - * @private - * @param {Object} object The source object. - * @param {string[]} paths The property paths to pick. - * @returns {Object} Returns the new object. - */ - function basePick(object, paths) { - return basePickBy(object, paths, function(value, path) { - return hasIn(object, path); - }); - } - - /** - * The base implementation of `_.pickBy` without support for iteratee shorthands. - * - * @private - * @param {Object} object The source object. - * @param {string[]} paths The property paths to pick. - * @param {Function} predicate The function invoked per property. - * @returns {Object} Returns the new object. - */ - function basePickBy(object, paths, predicate) { - var index = -1, - length = paths.length, - result = {}; - - while (++index < length) { - var path = paths[index], - value = baseGet(object, path); - - if (predicate(value, path)) { - baseSet(result, castPath(path, object), value); - } - } - return result; - } - - /** - * A specialized version of `baseProperty` which supports deep paths. - * - * @private - * @param {Array|string} path The path of the property to get. - * @returns {Function} Returns the new accessor function. - */ - function basePropertyDeep(path) { - return function(object) { - return baseGet(object, path); - }; - } - - /** - * The base implementation of `_.pullAllBy` without support for iteratee - * shorthands. - * - * @private - * @param {Array} array The array to modify. - * @param {Array} values The values to remove. - * @param {Function} [iteratee] The iteratee invoked per element. - * @param {Function} [comparator] The comparator invoked per element. - * @returns {Array} Returns `array`. - */ - function basePullAll(array, values, iteratee, comparator) { - var indexOf = comparator ? baseIndexOfWith : baseIndexOf, - index = -1, - length = values.length, - seen = array; - - if (array === values) { - values = copyArray(values); - } - if (iteratee) { - seen = arrayMap(array, baseUnary(iteratee)); - } - while (++index < length) { - var fromIndex = 0, - value = values[index], - computed = iteratee ? iteratee(value) : value; - - while ((fromIndex = indexOf(seen, computed, fromIndex, comparator)) > -1) { - if (seen !== array) { - splice.call(seen, fromIndex, 1); - } - splice.call(array, fromIndex, 1); - } - } - return array; - } - - /** - * The base implementation of `_.pullAt` without support for individual - * indexes or capturing the removed elements. - * - * @private - * @param {Array} array The array to modify. - * @param {number[]} indexes The indexes of elements to remove. - * @returns {Array} Returns `array`. - */ - function basePullAt(array, indexes) { - var length = array ? indexes.length : 0, - lastIndex = length - 1; - - while (length--) { - var index = indexes[length]; - if (length == lastIndex || index !== previous) { - var previous = index; - if (isIndex(index)) { - splice.call(array, index, 1); - } else { - baseUnset(array, index); - } - } - } - return array; - } - - /** - * The base implementation of `_.random` without support for returning - * floating-point numbers. - * - * @private - * @param {number} lower The lower bound. - * @param {number} upper The upper bound. - * @returns {number} Returns the random number. - */ - function baseRandom(lower, upper) { - return lower + nativeFloor(nativeRandom() * (upper - lower + 1)); - } - - /** - * The base implementation of `_.range` and `_.rangeRight` which doesn't - * coerce arguments. - * - * @private - * @param {number} start The start of the range. - * @param {number} end The end of the range. - * @param {number} step The value to increment or decrement by. - * @param {boolean} [fromRight] Specify iterating from right to left. - * @returns {Array} Returns the range of numbers. - */ - function baseRange(start, end, step, fromRight) { - var index = -1, - length = nativeMax(nativeCeil((end - start) / (step || 1)), 0), - result = Array(length); - - while (length--) { - result[fromRight ? length : ++index] = start; - start += step; - } - return result; - } - - /** - * The base implementation of `_.repeat` which doesn't coerce arguments. - * - * @private - * @param {string} string The string to repeat. - * @param {number} n The number of times to repeat the string. - * @returns {string} Returns the repeated string. - */ - function baseRepeat(string, n) { - var result = ''; - if (!string || n < 1 || n > MAX_SAFE_INTEGER) { - return result; - } - // Leverage the exponentiation by squaring algorithm for a faster repeat. - // See https://en.wikipedia.org/wiki/Exponentiation_by_squaring for more details. - do { - if (n % 2) { - result += string; - } - n = nativeFloor(n / 2); - if (n) { - string += string; - } - } while (n); - - return result; - } - - /** - * The base implementation of `_.rest` which doesn't validate or coerce arguments. - * - * @private - * @param {Function} func The function to apply a rest parameter to. - * @param {number} [start=func.length-1] The start position of the rest parameter. - * @returns {Function} Returns the new function. - */ - function baseRest(func, start) { - return setToString(overRest(func, start, identity), func + ''); - } - - /** - * The base implementation of `_.sample`. - * - * @private - * @param {Array|Object} collection The collection to sample. - * @returns {*} Returns the random element. - */ - function baseSample(collection) { - return arraySample(values(collection)); - } - - /** - * The base implementation of `_.sampleSize` without param guards. - * - * @private - * @param {Array|Object} collection The collection to sample. - * @param {number} n The number of elements to sample. - * @returns {Array} Returns the random elements. - */ - function baseSampleSize(collection, n) { - var array = values(collection); - return shuffleSelf(array, baseClamp(n, 0, array.length)); - } - - /** - * The base implementation of `_.set`. - * - * @private - * @param {Object} object The object to modify. - * @param {Array|string} path The path of the property to set. - * @param {*} value The value to set. - * @param {Function} [customizer] The function to customize path creation. - * @returns {Object} Returns `object`. - */ - function baseSet(object, path, value, customizer) { - if (!isObject(object)) { - return object; - } - path = castPath(path, object); - - var index = -1, - length = path.length, - lastIndex = length - 1, - nested = object; - - while (nested != null && ++index < length) { - var key = toKey(path[index]), - newValue = value; - - if (index != lastIndex) { - var objValue = nested[key]; - newValue = customizer ? customizer(objValue, key, nested) : undefined; - if (newValue === undefined) { - newValue = isObject(objValue) - ? objValue - : (isIndex(path[index + 1]) ? [] : {}); - } - } - assignValue(nested, key, newValue); - nested = nested[key]; - } - return object; - } - - /** - * The base implementation of `setData` without support for hot loop shorting. - * - * @private - * @param {Function} func The function to associate metadata with. - * @param {*} data The metadata. - * @returns {Function} Returns `func`. - */ - var baseSetData = !metaMap ? identity : function(func, data) { - metaMap.set(func, data); - return func; - }; - - /** - * The base implementation of `setToString` without support for hot loop shorting. - * - * @private - * @param {Function} func The function to modify. - * @param {Function} string The `toString` result. - * @returns {Function} Returns `func`. - */ - var baseSetToString = !defineProperty ? identity : function(func, string) { - return defineProperty(func, 'toString', { - 'configurable': true, - 'enumerable': false, - 'value': constant(string), - 'writable': true - }); - }; - - /** - * The base implementation of `_.shuffle`. - * - * @private - * @param {Array|Object} collection The collection to shuffle. - * @returns {Array} Returns the new shuffled array. - */ - function baseShuffle(collection) { - return shuffleSelf(values(collection)); - } - - /** - * The base implementation of `_.slice` without an iteratee call guard. - * - * @private - * @param {Array} array The array to slice. - * @param {number} [start=0] The start position. - * @param {number} [end=array.length] The end position. - * @returns {Array} Returns the slice of `array`. - */ - function baseSlice(array, start, end) { - var index = -1, - length = array.length; - - if (start < 0) { - start = -start > length ? 0 : (length + start); - } - end = end > length ? length : end; - if (end < 0) { - end += length; - } - length = start > end ? 0 : ((end - start) >>> 0); - start >>>= 0; - - var result = Array(length); - while (++index < length) { - result[index] = array[index + start]; - } - return result; - } - - /** - * The base implementation of `_.some` without support for iteratee shorthands. - * - * @private - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} predicate The function invoked per iteration. - * @returns {boolean} Returns `true` if any element passes the predicate check, - * else `false`. - */ - function baseSome(collection, predicate) { - var result; - - baseEach(collection, function(value, index, collection) { - result = predicate(value, index, collection); - return !result; - }); - return !!result; - } - - /** - * The base implementation of `_.sortedIndex` and `_.sortedLastIndex` which - * performs a binary search of `array` to determine the index at which `value` - * should be inserted into `array` in order to maintain its sort order. - * - * @private - * @param {Array} array The sorted array to inspect. - * @param {*} value The value to evaluate. - * @param {boolean} [retHighest] Specify returning the highest qualified index. - * @returns {number} Returns the index at which `value` should be inserted - * into `array`. - */ - function baseSortedIndex(array, value, retHighest) { - var low = 0, - high = array == null ? low : array.length; - - if (typeof value == 'number' && value === value && high <= HALF_MAX_ARRAY_LENGTH) { - while (low < high) { - var mid = (low + high) >>> 1, - computed = array[mid]; - - if (computed !== null && !isSymbol(computed) && - (retHighest ? (computed <= value) : (computed < value))) { - low = mid + 1; - } else { - high = mid; - } - } - return high; - } - return baseSortedIndexBy(array, value, identity, retHighest); - } - - /** - * The base implementation of `_.sortedIndexBy` and `_.sortedLastIndexBy` - * which invokes `iteratee` for `value` and each element of `array` to compute - * their sort ranking. The iteratee is invoked with one argument; (value). - * - * @private - * @param {Array} array The sorted array to inspect. - * @param {*} value The value to evaluate. - * @param {Function} iteratee The iteratee invoked per element. - * @param {boolean} [retHighest] Specify returning the highest qualified index. - * @returns {number} Returns the index at which `value` should be inserted - * into `array`. - */ - function baseSortedIndexBy(array, value, iteratee, retHighest) { - value = iteratee(value); - - var low = 0, - high = array == null ? 0 : array.length, - valIsNaN = value !== value, - valIsNull = value === null, - valIsSymbol = isSymbol(value), - valIsUndefined = value === undefined; - - while (low < high) { - var mid = nativeFloor((low + high) / 2), - computed = iteratee(array[mid]), - othIsDefined = computed !== undefined, - othIsNull = computed === null, - othIsReflexive = computed === computed, - othIsSymbol = isSymbol(computed); - - if (valIsNaN) { - var setLow = retHighest || othIsReflexive; - } else if (valIsUndefined) { - setLow = othIsReflexive && (retHighest || othIsDefined); - } else if (valIsNull) { - setLow = othIsReflexive && othIsDefined && (retHighest || !othIsNull); - } else if (valIsSymbol) { - setLow = othIsReflexive && othIsDefined && !othIsNull && (retHighest || !othIsSymbol); - } else if (othIsNull || othIsSymbol) { - setLow = false; - } else { - setLow = retHighest ? (computed <= value) : (computed < value); - } - if (setLow) { - low = mid + 1; - } else { - high = mid; - } - } - return nativeMin(high, MAX_ARRAY_INDEX); - } - - /** - * The base implementation of `_.sortedUniq` and `_.sortedUniqBy` without - * support for iteratee shorthands. - * - * @private - * @param {Array} array The array to inspect. - * @param {Function} [iteratee] The iteratee invoked per element. - * @returns {Array} Returns the new duplicate free array. - */ - function baseSortedUniq(array, iteratee) { - var index = -1, - length = array.length, - resIndex = 0, - result = []; - - while (++index < length) { - var value = array[index], - computed = iteratee ? iteratee(value) : value; - - if (!index || !eq(computed, seen)) { - var seen = computed; - result[resIndex++] = value === 0 ? 0 : value; - } - } - return result; - } - - /** - * The base implementation of `_.toNumber` which doesn't ensure correct - * conversions of binary, hexadecimal, or octal string values. - * - * @private - * @param {*} value The value to process. - * @returns {number} Returns the number. - */ - function baseToNumber(value) { - if (typeof value == 'number') { - return value; - } - if (isSymbol(value)) { - return NAN; - } - return +value; - } - - /** - * The base implementation of `_.toString` which doesn't convert nullish - * values to empty strings. - * - * @private - * @param {*} value The value to process. - * @returns {string} Returns the string. - */ - function baseToString(value) { - // Exit early for strings to avoid a performance hit in some environments. - if (typeof value == 'string') { - return value; - } - if (isArray(value)) { - // Recursively convert values (susceptible to call stack limits). - return arrayMap(value, baseToString) + ''; - } - if (isSymbol(value)) { - return symbolToString ? symbolToString.call(value) : ''; - } - var result = (value + ''); - return (result == '0' && (1 / value) == -INFINITY) ? '-0' : result; - } - - /** - * The base implementation of `_.uniqBy` without support for iteratee shorthands. - * - * @private - * @param {Array} array The array to inspect. - * @param {Function} [iteratee] The iteratee invoked per element. - * @param {Function} [comparator] The comparator invoked per element. - * @returns {Array} Returns the new duplicate free array. - */ - function baseUniq(array, iteratee, comparator) { - var index = -1, - includes = arrayIncludes, - length = array.length, - isCommon = true, - result = [], - seen = result; - - if (comparator) { - isCommon = false; - includes = arrayIncludesWith; - } - else if (length >= LARGE_ARRAY_SIZE) { - var set = iteratee ? null : createSet(array); - if (set) { - return setToArray(set); - } - isCommon = false; - includes = cacheHas; - seen = new SetCache; - } - else { - seen = iteratee ? [] : result; - } - outer: - while (++index < length) { - var value = array[index], - computed = iteratee ? iteratee(value) : value; - - value = (comparator || value !== 0) ? value : 0; - if (isCommon && computed === computed) { - var seenIndex = seen.length; - while (seenIndex--) { - if (seen[seenIndex] === computed) { - continue outer; - } - } - if (iteratee) { - seen.push(computed); - } - result.push(value); - } - else if (!includes(seen, computed, comparator)) { - if (seen !== result) { - seen.push(computed); - } - result.push(value); - } - } - return result; - } - - /** - * The base implementation of `_.unset`. - * - * @private - * @param {Object} object The object to modify. - * @param {Array|string} path The property path to unset. - * @returns {boolean} Returns `true` if the property is deleted, else `false`. - */ - function baseUnset(object, path) { - path = castPath(path, object); - object = parent(object, path); - return object == null || delete object[toKey(last(path))]; - } - - /** - * The base implementation of `_.update`. - * - * @private - * @param {Object} object The object to modify. - * @param {Array|string} path The path of the property to update. - * @param {Function} updater The function to produce the updated value. - * @param {Function} [customizer] The function to customize path creation. - * @returns {Object} Returns `object`. - */ - function baseUpdate(object, path, updater, customizer) { - return baseSet(object, path, updater(baseGet(object, path)), customizer); - } - - /** - * The base implementation of methods like `_.dropWhile` and `_.takeWhile` - * without support for iteratee shorthands. - * - * @private - * @param {Array} array The array to query. - * @param {Function} predicate The function invoked per iteration. - * @param {boolean} [isDrop] Specify dropping elements instead of taking them. - * @param {boolean} [fromRight] Specify iterating from right to left. - * @returns {Array} Returns the slice of `array`. - */ - function baseWhile(array, predicate, isDrop, fromRight) { - var length = array.length, - index = fromRight ? length : -1; - - while ((fromRight ? index-- : ++index < length) && - predicate(array[index], index, array)) {} - - return isDrop - ? baseSlice(array, (fromRight ? 0 : index), (fromRight ? index + 1 : length)) - : baseSlice(array, (fromRight ? index + 1 : 0), (fromRight ? length : index)); - } - - /** - * The base implementation of `wrapperValue` which returns the result of - * performing a sequence of actions on the unwrapped `value`, where each - * successive action is supplied the return value of the previous. - * - * @private - * @param {*} value The unwrapped value. - * @param {Array} actions Actions to perform to resolve the unwrapped value. - * @returns {*} Returns the resolved value. - */ - function baseWrapperValue(value, actions) { - var result = value; - if (result instanceof LazyWrapper) { - result = result.value(); - } - return arrayReduce(actions, function(result, action) { - return action.func.apply(action.thisArg, arrayPush([result], action.args)); - }, result); - } - - /** - * The base implementation of methods like `_.xor`, without support for - * iteratee shorthands, that accepts an array of arrays to inspect. - * - * @private - * @param {Array} arrays The arrays to inspect. - * @param {Function} [iteratee] The iteratee invoked per element. - * @param {Function} [comparator] The comparator invoked per element. - * @returns {Array} Returns the new array of values. - */ - function baseXor(arrays, iteratee, comparator) { - var length = arrays.length; - if (length < 2) { - return length ? baseUniq(arrays[0]) : []; - } - var index = -1, - result = Array(length); - - while (++index < length) { - var array = arrays[index], - othIndex = -1; - - while (++othIndex < length) { - if (othIndex != index) { - result[index] = baseDifference(result[index] || array, arrays[othIndex], iteratee, comparator); - } - } - } - return baseUniq(baseFlatten(result, 1), iteratee, comparator); - } - - /** - * This base implementation of `_.zipObject` which assigns values using `assignFunc`. - * - * @private - * @param {Array} props The property identifiers. - * @param {Array} values The property values. - * @param {Function} assignFunc The function to assign values. - * @returns {Object} Returns the new object. - */ - function baseZipObject(props, values, assignFunc) { - var index = -1, - length = props.length, - valsLength = values.length, - result = {}; - - while (++index < length) { - var value = index < valsLength ? values[index] : undefined; - assignFunc(result, props[index], value); - } - return result; - } - - /** - * Casts `value` to an empty array if it's not an array like object. - * - * @private - * @param {*} value The value to inspect. - * @returns {Array|Object} Returns the cast array-like object. - */ - function castArrayLikeObject(value) { - return isArrayLikeObject(value) ? value : []; - } - - /** - * Casts `value` to `identity` if it's not a function. - * - * @private - * @param {*} value The value to inspect. - * @returns {Function} Returns cast function. - */ - function castFunction(value) { - return typeof value == 'function' ? value : identity; - } - - /** - * Casts `value` to a path array if it's not one. - * - * @private - * @param {*} value The value to inspect. - * @param {Object} [object] The object to query keys on. - * @returns {Array} Returns the cast property path array. - */ - function castPath(value, object) { - if (isArray(value)) { - return value; - } - return isKey(value, object) ? [value] : stringToPath(toString(value)); - } - - /** - * A `baseRest` alias which can be replaced with `identity` by module - * replacement plugins. - * - * @private - * @type {Function} - * @param {Function} func The function to apply a rest parameter to. - * @returns {Function} Returns the new function. - */ - var castRest = baseRest; - - /** - * Casts `array` to a slice if it's needed. - * - * @private - * @param {Array} array The array to inspect. - * @param {number} start The start position. - * @param {number} [end=array.length] The end position. - * @returns {Array} Returns the cast slice. - */ - function castSlice(array, start, end) { - var length = array.length; - end = end === undefined ? length : end; - return (!start && end >= length) ? array : baseSlice(array, start, end); - } - - /** - * A simple wrapper around the global [`clearTimeout`](https://mdn.io/clearTimeout). - * - * @private - * @param {number|Object} id The timer id or timeout object of the timer to clear. - */ - var clearTimeout = ctxClearTimeout || function(id) { - return root.clearTimeout(id); - }; - - /** - * Creates a clone of `buffer`. - * - * @private - * @param {Buffer} buffer The buffer to clone. - * @param {boolean} [isDeep] Specify a deep clone. - * @returns {Buffer} Returns the cloned buffer. - */ - function cloneBuffer(buffer, isDeep) { - if (isDeep) { - return buffer.slice(); - } - var length = buffer.length, - result = allocUnsafe ? allocUnsafe(length) : new buffer.constructor(length); - - buffer.copy(result); - return result; - } - - /** - * Creates a clone of `arrayBuffer`. - * - * @private - * @param {ArrayBuffer} arrayBuffer The array buffer to clone. - * @returns {ArrayBuffer} Returns the cloned array buffer. - */ - function cloneArrayBuffer(arrayBuffer) { - var result = new arrayBuffer.constructor(arrayBuffer.byteLength); - new Uint8Array(result).set(new Uint8Array(arrayBuffer)); - return result; - } - - /** - * Creates a clone of `dataView`. - * - * @private - * @param {Object} dataView The data view to clone. - * @param {boolean} [isDeep] Specify a deep clone. - * @returns {Object} Returns the cloned data view. - */ - function cloneDataView(dataView, isDeep) { - var buffer = isDeep ? cloneArrayBuffer(dataView.buffer) : dataView.buffer; - return new dataView.constructor(buffer, dataView.byteOffset, dataView.byteLength); - } - - /** - * Creates a clone of `regexp`. - * - * @private - * @param {Object} regexp The regexp to clone. - * @returns {Object} Returns the cloned regexp. - */ - function cloneRegExp(regexp) { - var result = new regexp.constructor(regexp.source, reFlags.exec(regexp)); - result.lastIndex = regexp.lastIndex; - return result; - } - - /** - * Creates a clone of the `symbol` object. - * - * @private - * @param {Object} symbol The symbol object to clone. - * @returns {Object} Returns the cloned symbol object. - */ - function cloneSymbol(symbol) { - return symbolValueOf ? Object(symbolValueOf.call(symbol)) : {}; - } - - /** - * Creates a clone of `typedArray`. - * - * @private - * @param {Object} typedArray The typed array to clone. - * @param {boolean} [isDeep] Specify a deep clone. - * @returns {Object} Returns the cloned typed array. - */ - function cloneTypedArray(typedArray, isDeep) { - var buffer = isDeep ? cloneArrayBuffer(typedArray.buffer) : typedArray.buffer; - return new typedArray.constructor(buffer, typedArray.byteOffset, typedArray.length); - } - - /** - * Compares values to sort them in ascending order. - * - * @private - * @param {*} value The value to compare. - * @param {*} other The other value to compare. - * @returns {number} Returns the sort order indicator for `value`. - */ - function compareAscending(value, other) { - if (value !== other) { - var valIsDefined = value !== undefined, - valIsNull = value === null, - valIsReflexive = value === value, - valIsSymbol = isSymbol(value); - - var othIsDefined = other !== undefined, - othIsNull = other === null, - othIsReflexive = other === other, - othIsSymbol = isSymbol(other); - - if ((!othIsNull && !othIsSymbol && !valIsSymbol && value > other) || - (valIsSymbol && othIsDefined && othIsReflexive && !othIsNull && !othIsSymbol) || - (valIsNull && othIsDefined && othIsReflexive) || - (!valIsDefined && othIsReflexive) || - !valIsReflexive) { - return 1; - } - if ((!valIsNull && !valIsSymbol && !othIsSymbol && value < other) || - (othIsSymbol && valIsDefined && valIsReflexive && !valIsNull && !valIsSymbol) || - (othIsNull && valIsDefined && valIsReflexive) || - (!othIsDefined && valIsReflexive) || - !othIsReflexive) { - return -1; - } - } - return 0; - } - - /** - * Used by `_.orderBy` to compare multiple properties of a value to another - * and stable sort them. - * - * If `orders` is unspecified, all values are sorted in ascending order. Otherwise, - * specify an order of "desc" for descending or "asc" for ascending sort order - * of corresponding values. - * - * @private - * @param {Object} object The object to compare. - * @param {Object} other The other object to compare. - * @param {boolean[]|string[]} orders The order to sort by for each property. - * @returns {number} Returns the sort order indicator for `object`. - */ - function compareMultiple(object, other, orders) { - var index = -1, - objCriteria = object.criteria, - othCriteria = other.criteria, - length = objCriteria.length, - ordersLength = orders.length; - - while (++index < length) { - var result = compareAscending(objCriteria[index], othCriteria[index]); - if (result) { - if (index >= ordersLength) { - return result; - } - var order = orders[index]; - return result * (order == 'desc' ? -1 : 1); - } - } - // Fixes an `Array#sort` bug in the JS engine embedded in Adobe applications - // that causes it, under certain circumstances, to provide the same value for - // `object` and `other`. See https://github.com/jashkenas/underscore/pull/1247 - // for more details. - // - // This also ensures a stable sort in V8 and other engines. - // See https://bugs.chromium.org/p/v8/issues/detail?id=90 for more details. - return object.index - other.index; - } - - /** - * Creates an array that is the composition of partially applied arguments, - * placeholders, and provided arguments into a single array of arguments. - * - * @private - * @param {Array} args The provided arguments. - * @param {Array} partials The arguments to prepend to those provided. - * @param {Array} holders The `partials` placeholder indexes. - * @params {boolean} [isCurried] Specify composing for a curried function. - * @returns {Array} Returns the new array of composed arguments. - */ - function composeArgs(args, partials, holders, isCurried) { - var argsIndex = -1, - argsLength = args.length, - holdersLength = holders.length, - leftIndex = -1, - leftLength = partials.length, - rangeLength = nativeMax(argsLength - holdersLength, 0), - result = Array(leftLength + rangeLength), - isUncurried = !isCurried; - - while (++leftIndex < leftLength) { - result[leftIndex] = partials[leftIndex]; - } - while (++argsIndex < holdersLength) { - if (isUncurried || argsIndex < argsLength) { - result[holders[argsIndex]] = args[argsIndex]; - } - } - while (rangeLength--) { - result[leftIndex++] = args[argsIndex++]; - } - return result; - } - - /** - * This function is like `composeArgs` except that the arguments composition - * is tailored for `_.partialRight`. - * - * @private - * @param {Array} args The provided arguments. - * @param {Array} partials The arguments to append to those provided. - * @param {Array} holders The `partials` placeholder indexes. - * @params {boolean} [isCurried] Specify composing for a curried function. - * @returns {Array} Returns the new array of composed arguments. - */ - function composeArgsRight(args, partials, holders, isCurried) { - var argsIndex = -1, - argsLength = args.length, - holdersIndex = -1, - holdersLength = holders.length, - rightIndex = -1, - rightLength = partials.length, - rangeLength = nativeMax(argsLength - holdersLength, 0), - result = Array(rangeLength + rightLength), - isUncurried = !isCurried; - - while (++argsIndex < rangeLength) { - result[argsIndex] = args[argsIndex]; - } - var offset = argsIndex; - while (++rightIndex < rightLength) { - result[offset + rightIndex] = partials[rightIndex]; - } - while (++holdersIndex < holdersLength) { - if (isUncurried || argsIndex < argsLength) { - result[offset + holders[holdersIndex]] = args[argsIndex++]; - } - } - return result; - } - - /** - * Copies the values of `source` to `array`. - * - * @private - * @param {Array} source The array to copy values from. - * @param {Array} [array=[]] The array to copy values to. - * @returns {Array} Returns `array`. - */ - function copyArray(source, array) { - var index = -1, - length = source.length; - - array || (array = Array(length)); - while (++index < length) { - array[index] = source[index]; - } - return array; - } - - /** - * Copies properties of `source` to `object`. - * - * @private - * @param {Object} source The object to copy properties from. - * @param {Array} props The property identifiers to copy. - * @param {Object} [object={}] The object to copy properties to. - * @param {Function} [customizer] The function to customize copied values. - * @returns {Object} Returns `object`. - */ - function copyObject(source, props, object, customizer) { - var isNew = !object; - object || (object = {}); - - var index = -1, - length = props.length; - - while (++index < length) { - var key = props[index]; - - var newValue = customizer - ? customizer(object[key], source[key], key, object, source) - : undefined; - - if (newValue === undefined) { - newValue = source[key]; - } - if (isNew) { - baseAssignValue(object, key, newValue); - } else { - assignValue(object, key, newValue); - } - } - return object; - } - - /** - * Copies own symbols of `source` to `object`. - * - * @private - * @param {Object} source The object to copy symbols from. - * @param {Object} [object={}] The object to copy symbols to. - * @returns {Object} Returns `object`. - */ - function copySymbols(source, object) { - return copyObject(source, getSymbols(source), object); - } - - /** - * Copies own and inherited symbols of `source` to `object`. - * - * @private - * @param {Object} source The object to copy symbols from. - * @param {Object} [object={}] The object to copy symbols to. - * @returns {Object} Returns `object`. - */ - function copySymbolsIn(source, object) { - return copyObject(source, getSymbolsIn(source), object); - } - - /** - * Creates a function like `_.groupBy`. - * - * @private - * @param {Function} setter The function to set accumulator values. - * @param {Function} [initializer] The accumulator object initializer. - * @returns {Function} Returns the new aggregator function. - */ - function createAggregator(setter, initializer) { - return function(collection, iteratee) { - var func = isArray(collection) ? arrayAggregator : baseAggregator, - accumulator = initializer ? initializer() : {}; - - return func(collection, setter, getIteratee(iteratee, 2), accumulator); - }; - } - - /** - * Creates a function like `_.assign`. - * - * @private - * @param {Function} assigner The function to assign values. - * @returns {Function} Returns the new assigner function. - */ - function createAssigner(assigner) { - return baseRest(function(object, sources) { - var index = -1, - length = sources.length, - customizer = length > 1 ? sources[length - 1] : undefined, - guard = length > 2 ? sources[2] : undefined; - - customizer = (assigner.length > 3 && typeof customizer == 'function') - ? (length--, customizer) - : undefined; - - if (guard && isIterateeCall(sources[0], sources[1], guard)) { - customizer = length < 3 ? undefined : customizer; - length = 1; - } - object = Object(object); - while (++index < length) { - var source = sources[index]; - if (source) { - assigner(object, source, index, customizer); - } - } - return object; - }); - } - - /** - * Creates a `baseEach` or `baseEachRight` function. - * - * @private - * @param {Function} eachFunc The function to iterate over a collection. - * @param {boolean} [fromRight] Specify iterating from right to left. - * @returns {Function} Returns the new base function. - */ - function createBaseEach(eachFunc, fromRight) { - return function(collection, iteratee) { - if (collection == null) { - return collection; - } - if (!isArrayLike(collection)) { - return eachFunc(collection, iteratee); - } - var length = collection.length, - index = fromRight ? length : -1, - iterable = Object(collection); - - while ((fromRight ? index-- : ++index < length)) { - if (iteratee(iterable[index], index, iterable) === false) { - break; - } - } - return collection; - }; - } - - /** - * Creates a base function for methods like `_.forIn` and `_.forOwn`. - * - * @private - * @param {boolean} [fromRight] Specify iterating from right to left. - * @returns {Function} Returns the new base function. - */ - function createBaseFor(fromRight) { - return function(object, iteratee, keysFunc) { - var index = -1, - iterable = Object(object), - props = keysFunc(object), - length = props.length; - - while (length--) { - var key = props[fromRight ? length : ++index]; - if (iteratee(iterable[key], key, iterable) === false) { - break; - } - } - return object; - }; - } - - /** - * Creates a function that wraps `func` to invoke it with the optional `this` - * binding of `thisArg`. - * - * @private - * @param {Function} func The function to wrap. - * @param {number} bitmask The bitmask flags. See `createWrap` for more details. - * @param {*} [thisArg] The `this` binding of `func`. - * @returns {Function} Returns the new wrapped function. - */ - function createBind(func, bitmask, thisArg) { - var isBind = bitmask & WRAP_BIND_FLAG, - Ctor = createCtor(func); - - function wrapper() { - var fn = (this && this !== root && this instanceof wrapper) ? Ctor : func; - return fn.apply(isBind ? thisArg : this, arguments); - } - return wrapper; - } - - /** - * Creates a function like `_.lowerFirst`. - * - * @private - * @param {string} methodName The name of the `String` case method to use. - * @returns {Function} Returns the new case function. - */ - function createCaseFirst(methodName) { - return function(string) { - string = toString(string); - - var strSymbols = hasUnicode(string) - ? stringToArray(string) - : undefined; - - var chr = strSymbols - ? strSymbols[0] - : string.charAt(0); - - var trailing = strSymbols - ? castSlice(strSymbols, 1).join('') - : string.slice(1); - - return chr[methodName]() + trailing; - }; - } - - /** - * Creates a function like `_.camelCase`. - * - * @private - * @param {Function} callback The function to combine each word. - * @returns {Function} Returns the new compounder function. - */ - function createCompounder(callback) { - return function(string) { - return arrayReduce(words(deburr(string).replace(reApos, '')), callback, ''); - }; - } - - /** - * Creates a function that produces an instance of `Ctor` regardless of - * whether it was invoked as part of a `new` expression or by `call` or `apply`. - * - * @private - * @param {Function} Ctor The constructor to wrap. - * @returns {Function} Returns the new wrapped function. - */ - function createCtor(Ctor) { - return function() { - // Use a `switch` statement to work with class constructors. See - // http://ecma-international.org/ecma-262/7.0/#sec-ecmascript-function-objects-call-thisargument-argumentslist - // for more details. - var args = arguments; - switch (args.length) { - case 0: return new Ctor; - case 1: return new Ctor(args[0]); - case 2: return new Ctor(args[0], args[1]); - case 3: return new Ctor(args[0], args[1], args[2]); - case 4: return new Ctor(args[0], args[1], args[2], args[3]); - case 5: return new Ctor(args[0], args[1], args[2], args[3], args[4]); - case 6: return new Ctor(args[0], args[1], args[2], args[3], args[4], args[5]); - case 7: return new Ctor(args[0], args[1], args[2], args[3], args[4], args[5], args[6]); - } - var thisBinding = baseCreate(Ctor.prototype), - result = Ctor.apply(thisBinding, args); - - // Mimic the constructor's `return` behavior. - // See https://es5.github.io/#x13.2.2 for more details. - return isObject(result) ? result : thisBinding; - }; - } - - /** - * Creates a function that wraps `func` to enable currying. - * - * @private - * @param {Function} func The function to wrap. - * @param {number} bitmask The bitmask flags. See `createWrap` for more details. - * @param {number} arity The arity of `func`. - * @returns {Function} Returns the new wrapped function. - */ - function createCurry(func, bitmask, arity) { - var Ctor = createCtor(func); - - function wrapper() { - var length = arguments.length, - args = Array(length), - index = length, - placeholder = getHolder(wrapper); - - while (index--) { - args[index] = arguments[index]; - } - var holders = (length < 3 && args[0] !== placeholder && args[length - 1] !== placeholder) - ? [] - : replaceHolders(args, placeholder); - - length -= holders.length; - if (length < arity) { - return createRecurry( - func, bitmask, createHybrid, wrapper.placeholder, undefined, - args, holders, undefined, undefined, arity - length); - } - var fn = (this && this !== root && this instanceof wrapper) ? Ctor : func; - return apply(fn, this, args); - } - return wrapper; - } - - /** - * Creates a `_.find` or `_.findLast` function. - * - * @private - * @param {Function} findIndexFunc The function to find the collection index. - * @returns {Function} Returns the new find function. - */ - function createFind(findIndexFunc) { - return function(collection, predicate, fromIndex) { - var iterable = Object(collection); - if (!isArrayLike(collection)) { - var iteratee = getIteratee(predicate, 3); - collection = keys(collection); - predicate = function(key) { return iteratee(iterable[key], key, iterable); }; - } - var index = findIndexFunc(collection, predicate, fromIndex); - return index > -1 ? iterable[iteratee ? collection[index] : index] : undefined; - }; - } - - /** - * Creates a `_.flow` or `_.flowRight` function. - * - * @private - * @param {boolean} [fromRight] Specify iterating from right to left. - * @returns {Function} Returns the new flow function. - */ - function createFlow(fromRight) { - return flatRest(function(funcs) { - var length = funcs.length, - index = length, - prereq = LodashWrapper.prototype.thru; - - if (fromRight) { - funcs.reverse(); - } - while (index--) { - var func = funcs[index]; - if (typeof func != 'function') { - throw new TypeError(FUNC_ERROR_TEXT); - } - if (prereq && !wrapper && getFuncName(func) == 'wrapper') { - var wrapper = new LodashWrapper([], true); - } - } - index = wrapper ? index : length; - while (++index < length) { - func = funcs[index]; - - var funcName = getFuncName(func), - data = funcName == 'wrapper' ? getData(func) : undefined; - - if (data && isLaziable(data[0]) && - data[1] == (WRAP_ARY_FLAG | WRAP_CURRY_FLAG | WRAP_PARTIAL_FLAG | WRAP_REARG_FLAG) && - !data[4].length && data[9] == 1 - ) { - wrapper = wrapper[getFuncName(data[0])].apply(wrapper, data[3]); - } else { - wrapper = (func.length == 1 && isLaziable(func)) - ? wrapper[funcName]() - : wrapper.thru(func); - } - } - return function() { - var args = arguments, - value = args[0]; - - if (wrapper && args.length == 1 && isArray(value)) { - return wrapper.plant(value).value(); - } - var index = 0, - result = length ? funcs[index].apply(this, args) : value; - - while (++index < length) { - result = funcs[index].call(this, result); - } - return result; - }; - }); - } - - /** - * Creates a function that wraps `func` to invoke it with optional `this` - * binding of `thisArg`, partial application, and currying. - * - * @private - * @param {Function|string} func The function or method name to wrap. - * @param {number} bitmask The bitmask flags. See `createWrap` for more details. - * @param {*} [thisArg] The `this` binding of `func`. - * @param {Array} [partials] The arguments to prepend to those provided to - * the new function. - * @param {Array} [holders] The `partials` placeholder indexes. - * @param {Array} [partialsRight] The arguments to append to those provided - * to the new function. - * @param {Array} [holdersRight] The `partialsRight` placeholder indexes. - * @param {Array} [argPos] The argument positions of the new function. - * @param {number} [ary] The arity cap of `func`. - * @param {number} [arity] The arity of `func`. - * @returns {Function} Returns the new wrapped function. - */ - function createHybrid(func, bitmask, thisArg, partials, holders, partialsRight, holdersRight, argPos, ary, arity) { - var isAry = bitmask & WRAP_ARY_FLAG, - isBind = bitmask & WRAP_BIND_FLAG, - isBindKey = bitmask & WRAP_BIND_KEY_FLAG, - isCurried = bitmask & (WRAP_CURRY_FLAG | WRAP_CURRY_RIGHT_FLAG), - isFlip = bitmask & WRAP_FLIP_FLAG, - Ctor = isBindKey ? undefined : createCtor(func); - - function wrapper() { - var length = arguments.length, - args = Array(length), - index = length; - - while (index--) { - args[index] = arguments[index]; - } - if (isCurried) { - var placeholder = getHolder(wrapper), - holdersCount = countHolders(args, placeholder); - } - if (partials) { - args = composeArgs(args, partials, holders, isCurried); - } - if (partialsRight) { - args = composeArgsRight(args, partialsRight, holdersRight, isCurried); - } - length -= holdersCount; - if (isCurried && length < arity) { - var newHolders = replaceHolders(args, placeholder); - return createRecurry( - func, bitmask, createHybrid, wrapper.placeholder, thisArg, - args, newHolders, argPos, ary, arity - length - ); - } - var thisBinding = isBind ? thisArg : this, - fn = isBindKey ? thisBinding[func] : func; - - length = args.length; - if (argPos) { - args = reorder(args, argPos); - } else if (isFlip && length > 1) { - args.reverse(); - } - if (isAry && ary < length) { - args.length = ary; - } - if (this && this !== root && this instanceof wrapper) { - fn = Ctor || createCtor(fn); - } - return fn.apply(thisBinding, args); - } - return wrapper; - } - - /** - * Creates a function like `_.invertBy`. - * - * @private - * @param {Function} setter The function to set accumulator values. - * @param {Function} toIteratee The function to resolve iteratees. - * @returns {Function} Returns the new inverter function. - */ - function createInverter(setter, toIteratee) { - return function(object, iteratee) { - return baseInverter(object, setter, toIteratee(iteratee), {}); - }; - } - - /** - * Creates a function that performs a mathematical operation on two values. - * - * @private - * @param {Function} operator The function to perform the operation. - * @param {number} [defaultValue] The value used for `undefined` arguments. - * @returns {Function} Returns the new mathematical operation function. - */ - function createMathOperation(operator, defaultValue) { - return function(value, other) { - var result; - if (value === undefined && other === undefined) { - return defaultValue; - } - if (value !== undefined) { - result = value; - } - if (other !== undefined) { - if (result === undefined) { - return other; - } - if (typeof value == 'string' || typeof other == 'string') { - value = baseToString(value); - other = baseToString(other); - } else { - value = baseToNumber(value); - other = baseToNumber(other); - } - result = operator(value, other); - } - return result; - }; - } - - /** - * Creates a function like `_.over`. - * - * @private - * @param {Function} arrayFunc The function to iterate over iteratees. - * @returns {Function} Returns the new over function. - */ - function createOver(arrayFunc) { - return flatRest(function(iteratees) { - iteratees = arrayMap(iteratees, baseUnary(getIteratee())); - return baseRest(function(args) { - var thisArg = this; - return arrayFunc(iteratees, function(iteratee) { - return apply(iteratee, thisArg, args); - }); - }); - }); - } - - /** - * Creates the padding for `string` based on `length`. The `chars` string - * is truncated if the number of characters exceeds `length`. - * - * @private - * @param {number} length The padding length. - * @param {string} [chars=' '] The string used as padding. - * @returns {string} Returns the padding for `string`. - */ - function createPadding(length, chars) { - chars = chars === undefined ? ' ' : baseToString(chars); - - var charsLength = chars.length; - if (charsLength < 2) { - return charsLength ? baseRepeat(chars, length) : chars; - } - var result = baseRepeat(chars, nativeCeil(length / stringSize(chars))); - return hasUnicode(chars) - ? castSlice(stringToArray(result), 0, length).join('') - : result.slice(0, length); - } - - /** - * Creates a function that wraps `func` to invoke it with the `this` binding - * of `thisArg` and `partials` prepended to the arguments it receives. - * - * @private - * @param {Function} func The function to wrap. - * @param {number} bitmask The bitmask flags. See `createWrap` for more details. - * @param {*} thisArg The `this` binding of `func`. - * @param {Array} partials The arguments to prepend to those provided to - * the new function. - * @returns {Function} Returns the new wrapped function. - */ - function createPartial(func, bitmask, thisArg, partials) { - var isBind = bitmask & WRAP_BIND_FLAG, - Ctor = createCtor(func); - - function wrapper() { - var argsIndex = -1, - argsLength = arguments.length, - leftIndex = -1, - leftLength = partials.length, - args = Array(leftLength + argsLength), - fn = (this && this !== root && this instanceof wrapper) ? Ctor : func; - - while (++leftIndex < leftLength) { - args[leftIndex] = partials[leftIndex]; - } - while (argsLength--) { - args[leftIndex++] = arguments[++argsIndex]; - } - return apply(fn, isBind ? thisArg : this, args); - } - return wrapper; - } - - /** - * Creates a `_.range` or `_.rangeRight` function. - * - * @private - * @param {boolean} [fromRight] Specify iterating from right to left. - * @returns {Function} Returns the new range function. - */ - function createRange(fromRight) { - return function(start, end, step) { - if (step && typeof step != 'number' && isIterateeCall(start, end, step)) { - end = step = undefined; - } - // Ensure the sign of `-0` is preserved. - start = toFinite(start); - if (end === undefined) { - end = start; - start = 0; - } else { - end = toFinite(end); - } - step = step === undefined ? (start < end ? 1 : -1) : toFinite(step); - return baseRange(start, end, step, fromRight); - }; - } - - /** - * Creates a function that performs a relational operation on two values. - * - * @private - * @param {Function} operator The function to perform the operation. - * @returns {Function} Returns the new relational operation function. - */ - function createRelationalOperation(operator) { - return function(value, other) { - if (!(typeof value == 'string' && typeof other == 'string')) { - value = toNumber(value); - other = toNumber(other); - } - return operator(value, other); - }; - } - - /** - * Creates a function that wraps `func` to continue currying. - * - * @private - * @param {Function} func The function to wrap. - * @param {number} bitmask The bitmask flags. See `createWrap` for more details. - * @param {Function} wrapFunc The function to create the `func` wrapper. - * @param {*} placeholder The placeholder value. - * @param {*} [thisArg] The `this` binding of `func`. - * @param {Array} [partials] The arguments to prepend to those provided to - * the new function. - * @param {Array} [holders] The `partials` placeholder indexes. - * @param {Array} [argPos] The argument positions of the new function. - * @param {number} [ary] The arity cap of `func`. - * @param {number} [arity] The arity of `func`. - * @returns {Function} Returns the new wrapped function. - */ - function createRecurry(func, bitmask, wrapFunc, placeholder, thisArg, partials, holders, argPos, ary, arity) { - var isCurry = bitmask & WRAP_CURRY_FLAG, - newHolders = isCurry ? holders : undefined, - newHoldersRight = isCurry ? undefined : holders, - newPartials = isCurry ? partials : undefined, - newPartialsRight = isCurry ? undefined : partials; - - bitmask |= (isCurry ? WRAP_PARTIAL_FLAG : WRAP_PARTIAL_RIGHT_FLAG); - bitmask &= ~(isCurry ? WRAP_PARTIAL_RIGHT_FLAG : WRAP_PARTIAL_FLAG); - - if (!(bitmask & WRAP_CURRY_BOUND_FLAG)) { - bitmask &= ~(WRAP_BIND_FLAG | WRAP_BIND_KEY_FLAG); - } - var newData = [ - func, bitmask, thisArg, newPartials, newHolders, newPartialsRight, - newHoldersRight, argPos, ary, arity - ]; - - var result = wrapFunc.apply(undefined, newData); - if (isLaziable(func)) { - setData(result, newData); - } - result.placeholder = placeholder; - return setWrapToString(result, func, bitmask); - } - - /** - * Creates a function like `_.round`. - * - * @private - * @param {string} methodName The name of the `Math` method to use when rounding. - * @returns {Function} Returns the new round function. - */ - function createRound(methodName) { - var func = Math[methodName]; - return function(number, precision) { - number = toNumber(number); - precision = precision == null ? 0 : nativeMin(toInteger(precision), 292); - if (precision && nativeIsFinite(number)) { - // Shift with exponential notation to avoid floating-point issues. - // See [MDN](https://mdn.io/round#Examples) for more details. - var pair = (toString(number) + 'e').split('e'), - value = func(pair[0] + 'e' + (+pair[1] + precision)); - - pair = (toString(value) + 'e').split('e'); - return +(pair[0] + 'e' + (+pair[1] - precision)); - } - return func(number); - }; - } - - /** - * Creates a set object of `values`. - * - * @private - * @param {Array} values The values to add to the set. - * @returns {Object} Returns the new set. - */ - var createSet = !(Set && (1 / setToArray(new Set([,-0]))[1]) == INFINITY) ? noop : function(values) { - return new Set(values); - }; - - /** - * Creates a `_.toPairs` or `_.toPairsIn` function. - * - * @private - * @param {Function} keysFunc The function to get the keys of a given object. - * @returns {Function} Returns the new pairs function. - */ - function createToPairs(keysFunc) { - return function(object) { - var tag = getTag(object); - if (tag == mapTag) { - return mapToArray(object); - } - if (tag == setTag) { - return setToPairs(object); - } - return baseToPairs(object, keysFunc(object)); - }; - } - - /** - * Creates a function that either curries or invokes `func` with optional - * `this` binding and partially applied arguments. - * - * @private - * @param {Function|string} func The function or method name to wrap. - * @param {number} bitmask The bitmask flags. - * 1 - `_.bind` - * 2 - `_.bindKey` - * 4 - `_.curry` or `_.curryRight` of a bound function - * 8 - `_.curry` - * 16 - `_.curryRight` - * 32 - `_.partial` - * 64 - `_.partialRight` - * 128 - `_.rearg` - * 256 - `_.ary` - * 512 - `_.flip` - * @param {*} [thisArg] The `this` binding of `func`. - * @param {Array} [partials] The arguments to be partially applied. - * @param {Array} [holders] The `partials` placeholder indexes. - * @param {Array} [argPos] The argument positions of the new function. - * @param {number} [ary] The arity cap of `func`. - * @param {number} [arity] The arity of `func`. - * @returns {Function} Returns the new wrapped function. - */ - function createWrap(func, bitmask, thisArg, partials, holders, argPos, ary, arity) { - var isBindKey = bitmask & WRAP_BIND_KEY_FLAG; - if (!isBindKey && typeof func != 'function') { - throw new TypeError(FUNC_ERROR_TEXT); - } - var length = partials ? partials.length : 0; - if (!length) { - bitmask &= ~(WRAP_PARTIAL_FLAG | WRAP_PARTIAL_RIGHT_FLAG); - partials = holders = undefined; - } - ary = ary === undefined ? ary : nativeMax(toInteger(ary), 0); - arity = arity === undefined ? arity : toInteger(arity); - length -= holders ? holders.length : 0; - - if (bitmask & WRAP_PARTIAL_RIGHT_FLAG) { - var partialsRight = partials, - holdersRight = holders; - - partials = holders = undefined; - } - var data = isBindKey ? undefined : getData(func); - - var newData = [ - func, bitmask, thisArg, partials, holders, partialsRight, holdersRight, - argPos, ary, arity - ]; - - if (data) { - mergeData(newData, data); - } - func = newData[0]; - bitmask = newData[1]; - thisArg = newData[2]; - partials = newData[3]; - holders = newData[4]; - arity = newData[9] = newData[9] === undefined - ? (isBindKey ? 0 : func.length) - : nativeMax(newData[9] - length, 0); - - if (!arity && bitmask & (WRAP_CURRY_FLAG | WRAP_CURRY_RIGHT_FLAG)) { - bitmask &= ~(WRAP_CURRY_FLAG | WRAP_CURRY_RIGHT_FLAG); - } - if (!bitmask || bitmask == WRAP_BIND_FLAG) { - var result = createBind(func, bitmask, thisArg); - } else if (bitmask == WRAP_CURRY_FLAG || bitmask == WRAP_CURRY_RIGHT_FLAG) { - result = createCurry(func, bitmask, arity); - } else if ((bitmask == WRAP_PARTIAL_FLAG || bitmask == (WRAP_BIND_FLAG | WRAP_PARTIAL_FLAG)) && !holders.length) { - result = createPartial(func, bitmask, thisArg, partials); - } else { - result = createHybrid.apply(undefined, newData); - } - var setter = data ? baseSetData : setData; - return setWrapToString(setter(result, newData), func, bitmask); - } - - /** - * Used by `_.defaults` to customize its `_.assignIn` use to assign properties - * of source objects to the destination object for all destination properties - * that resolve to `undefined`. - * - * @private - * @param {*} objValue The destination value. - * @param {*} srcValue The source value. - * @param {string} key The key of the property to assign. - * @param {Object} object The parent object of `objValue`. - * @returns {*} Returns the value to assign. - */ - function customDefaultsAssignIn(objValue, srcValue, key, object) { - if (objValue === undefined || - (eq(objValue, objectProto[key]) && !hasOwnProperty.call(object, key))) { - return srcValue; - } - return objValue; - } - - /** - * Used by `_.defaultsDeep` to customize its `_.merge` use to merge source - * objects into destination objects that are passed thru. - * - * @private - * @param {*} objValue The destination value. - * @param {*} srcValue The source value. - * @param {string} key The key of the property to merge. - * @param {Object} object The parent object of `objValue`. - * @param {Object} source The parent object of `srcValue`. - * @param {Object} [stack] Tracks traversed source values and their merged - * counterparts. - * @returns {*} Returns the value to assign. - */ - function customDefaultsMerge(objValue, srcValue, key, object, source, stack) { - if (isObject(objValue) && isObject(srcValue)) { - // Recursively merge objects and arrays (susceptible to call stack limits). - stack.set(srcValue, objValue); - baseMerge(objValue, srcValue, undefined, customDefaultsMerge, stack); - stack['delete'](srcValue); - } - return objValue; - } - - /** - * Used by `_.omit` to customize its `_.cloneDeep` use to only clone plain - * objects. - * - * @private - * @param {*} value The value to inspect. - * @param {string} key The key of the property to inspect. - * @returns {*} Returns the uncloned value or `undefined` to defer cloning to `_.cloneDeep`. - */ - function customOmitClone(value) { - return isPlainObject(value) ? undefined : value; - } - - /** - * A specialized version of `baseIsEqualDeep` for arrays with support for - * partial deep comparisons. - * - * @private - * @param {Array} array The array to compare. - * @param {Array} other The other array to compare. - * @param {number} bitmask The bitmask flags. See `baseIsEqual` for more details. - * @param {Function} customizer The function to customize comparisons. - * @param {Function} equalFunc The function to determine equivalents of values. - * @param {Object} stack Tracks traversed `array` and `other` objects. - * @returns {boolean} Returns `true` if the arrays are equivalent, else `false`. - */ - function equalArrays(array, other, bitmask, customizer, equalFunc, stack) { - var isPartial = bitmask & COMPARE_PARTIAL_FLAG, - arrLength = array.length, - othLength = other.length; - - if (arrLength != othLength && !(isPartial && othLength > arrLength)) { - return false; - } - // Assume cyclic values are equal. - var stacked = stack.get(array); - if (stacked && stack.get(other)) { - return stacked == other; - } - var index = -1, - result = true, - seen = (bitmask & COMPARE_UNORDERED_FLAG) ? new SetCache : undefined; - - stack.set(array, other); - stack.set(other, array); - - // Ignore non-index properties. - while (++index < arrLength) { - var arrValue = array[index], - othValue = other[index]; - - if (customizer) { - var compared = isPartial - ? customizer(othValue, arrValue, index, other, array, stack) - : customizer(arrValue, othValue, index, array, other, stack); - } - if (compared !== undefined) { - if (compared) { - continue; - } - result = false; - break; - } - // Recursively compare arrays (susceptible to call stack limits). - if (seen) { - if (!arraySome(other, function(othValue, othIndex) { - if (!cacheHas(seen, othIndex) && - (arrValue === othValue || equalFunc(arrValue, othValue, bitmask, customizer, stack))) { - return seen.push(othIndex); - } - })) { - result = false; - break; - } - } else if (!( - arrValue === othValue || - equalFunc(arrValue, othValue, bitmask, customizer, stack) - )) { - result = false; - break; - } - } - stack['delete'](array); - stack['delete'](other); - return result; - } - - /** - * A specialized version of `baseIsEqualDeep` for comparing objects of - * the same `toStringTag`. - * - * **Note:** This function only supports comparing values with tags of - * `Boolean`, `Date`, `Error`, `Number`, `RegExp`, or `String`. - * - * @private - * @param {Object} object The object to compare. - * @param {Object} other The other object to compare. - * @param {string} tag The `toStringTag` of the objects to compare. - * @param {number} bitmask The bitmask flags. See `baseIsEqual` for more details. - * @param {Function} customizer The function to customize comparisons. - * @param {Function} equalFunc The function to determine equivalents of values. - * @param {Object} stack Tracks traversed `object` and `other` objects. - * @returns {boolean} Returns `true` if the objects are equivalent, else `false`. - */ - function equalByTag(object, other, tag, bitmask, customizer, equalFunc, stack) { - switch (tag) { - case dataViewTag: - if ((object.byteLength != other.byteLength) || - (object.byteOffset != other.byteOffset)) { - return false; - } - object = object.buffer; - other = other.buffer; - - case arrayBufferTag: - if ((object.byteLength != other.byteLength) || - !equalFunc(new Uint8Array(object), new Uint8Array(other))) { - return false; - } - return true; - - case boolTag: - case dateTag: - case numberTag: - // Coerce booleans to `1` or `0` and dates to milliseconds. - // Invalid dates are coerced to `NaN`. - return eq(+object, +other); - - case errorTag: - return object.name == other.name && object.message == other.message; - - case regexpTag: - case stringTag: - // Coerce regexes to strings and treat strings, primitives and objects, - // as equal. See http://www.ecma-international.org/ecma-262/7.0/#sec-regexp.prototype.tostring - // for more details. - return object == (other + ''); - - case mapTag: - var convert = mapToArray; - - case setTag: - var isPartial = bitmask & COMPARE_PARTIAL_FLAG; - convert || (convert = setToArray); - - if (object.size != other.size && !isPartial) { - return false; - } - // Assume cyclic values are equal. - var stacked = stack.get(object); - if (stacked) { - return stacked == other; - } - bitmask |= COMPARE_UNORDERED_FLAG; - - // Recursively compare objects (susceptible to call stack limits). - stack.set(object, other); - var result = equalArrays(convert(object), convert(other), bitmask, customizer, equalFunc, stack); - stack['delete'](object); - return result; - - case symbolTag: - if (symbolValueOf) { - return symbolValueOf.call(object) == symbolValueOf.call(other); - } - } - return false; - } - - /** - * A specialized version of `baseIsEqualDeep` for objects with support for - * partial deep comparisons. - * - * @private - * @param {Object} object The object to compare. - * @param {Object} other The other object to compare. - * @param {number} bitmask The bitmask flags. See `baseIsEqual` for more details. - * @param {Function} customizer The function to customize comparisons. - * @param {Function} equalFunc The function to determine equivalents of values. - * @param {Object} stack Tracks traversed `object` and `other` objects. - * @returns {boolean} Returns `true` if the objects are equivalent, else `false`. - */ - function equalObjects(object, other, bitmask, customizer, equalFunc, stack) { - var isPartial = bitmask & COMPARE_PARTIAL_FLAG, - objProps = getAllKeys(object), - objLength = objProps.length, - othProps = getAllKeys(other), - othLength = othProps.length; - - if (objLength != othLength && !isPartial) { - return false; - } - var index = objLength; - while (index--) { - var key = objProps[index]; - if (!(isPartial ? key in other : hasOwnProperty.call(other, key))) { - return false; - } - } - // Assume cyclic values are equal. - var stacked = stack.get(object); - if (stacked && stack.get(other)) { - return stacked == other; - } - var result = true; - stack.set(object, other); - stack.set(other, object); - - var skipCtor = isPartial; - while (++index < objLength) { - key = objProps[index]; - var objValue = object[key], - othValue = other[key]; - - if (customizer) { - var compared = isPartial - ? customizer(othValue, objValue, key, other, object, stack) - : customizer(objValue, othValue, key, object, other, stack); - } - // Recursively compare objects (susceptible to call stack limits). - if (!(compared === undefined - ? (objValue === othValue || equalFunc(objValue, othValue, bitmask, customizer, stack)) - : compared - )) { - result = false; - break; - } - skipCtor || (skipCtor = key == 'constructor'); - } - if (result && !skipCtor) { - var objCtor = object.constructor, - othCtor = other.constructor; - - // Non `Object` object instances with different constructors are not equal. - if (objCtor != othCtor && - ('constructor' in object && 'constructor' in other) && - !(typeof objCtor == 'function' && objCtor instanceof objCtor && - typeof othCtor == 'function' && othCtor instanceof othCtor)) { - result = false; - } - } - stack['delete'](object); - stack['delete'](other); - return result; - } - - /** - * A specialized version of `baseRest` which flattens the rest array. - * - * @private - * @param {Function} func The function to apply a rest parameter to. - * @returns {Function} Returns the new function. - */ - function flatRest(func) { - return setToString(overRest(func, undefined, flatten), func + ''); - } - - /** - * Creates an array of own enumerable property names and symbols of `object`. - * - * @private - * @param {Object} object The object to query. - * @returns {Array} Returns the array of property names and symbols. - */ - function getAllKeys(object) { - return baseGetAllKeys(object, keys, getSymbols); - } - - /** - * Creates an array of own and inherited enumerable property names and - * symbols of `object`. - * - * @private - * @param {Object} object The object to query. - * @returns {Array} Returns the array of property names and symbols. - */ - function getAllKeysIn(object) { - return baseGetAllKeys(object, keysIn, getSymbolsIn); - } - - /** - * Gets metadata for `func`. - * - * @private - * @param {Function} func The function to query. - * @returns {*} Returns the metadata for `func`. - */ - var getData = !metaMap ? noop : function(func) { - return metaMap.get(func); - }; - - /** - * Gets the name of `func`. - * - * @private - * @param {Function} func The function to query. - * @returns {string} Returns the function name. - */ - function getFuncName(func) { - var result = (func.name + ''), - array = realNames[result], - length = hasOwnProperty.call(realNames, result) ? array.length : 0; - - while (length--) { - var data = array[length], - otherFunc = data.func; - if (otherFunc == null || otherFunc == func) { - return data.name; - } - } - return result; - } - - /** - * Gets the argument placeholder value for `func`. - * - * @private - * @param {Function} func The function to inspect. - * @returns {*} Returns the placeholder value. - */ - function getHolder(func) { - var object = hasOwnProperty.call(lodash, 'placeholder') ? lodash : func; - return object.placeholder; - } - - /** - * Gets the appropriate "iteratee" function. If `_.iteratee` is customized, - * this function returns the custom method, otherwise it returns `baseIteratee`. - * If arguments are provided, the chosen function is invoked with them and - * its result is returned. - * - * @private - * @param {*} [value] The value to convert to an iteratee. - * @param {number} [arity] The arity of the created iteratee. - * @returns {Function} Returns the chosen function or its result. - */ - function getIteratee() { - var result = lodash.iteratee || iteratee; - result = result === iteratee ? baseIteratee : result; - return arguments.length ? result(arguments[0], arguments[1]) : result; - } - - /** - * Gets the data for `map`. - * - * @private - * @param {Object} map The map to query. - * @param {string} key The reference key. - * @returns {*} Returns the map data. - */ - function getMapData(map, key) { - var data = map.__data__; - return isKeyable(key) - ? data[typeof key == 'string' ? 'string' : 'hash'] - : data.map; - } - - /** - * Gets the property names, values, and compare flags of `object`. - * - * @private - * @param {Object} object The object to query. - * @returns {Array} Returns the match data of `object`. - */ - function getMatchData(object) { - var result = keys(object), - length = result.length; - - while (length--) { - var key = result[length], - value = object[key]; - - result[length] = [key, value, isStrictComparable(value)]; - } - return result; - } - - /** - * Gets the native function at `key` of `object`. - * - * @private - * @param {Object} object The object to query. - * @param {string} key The key of the method to get. - * @returns {*} Returns the function if it's native, else `undefined`. - */ - function getNative(object, key) { - var value = getValue(object, key); - return baseIsNative(value) ? value : undefined; - } - - /** - * A specialized version of `baseGetTag` which ignores `Symbol.toStringTag` values. - * - * @private - * @param {*} value The value to query. - * @returns {string} Returns the raw `toStringTag`. - */ - function getRawTag(value) { - var isOwn = hasOwnProperty.call(value, symToStringTag), - tag = value[symToStringTag]; - - try { - value[symToStringTag] = undefined; - var unmasked = true; - } catch (e) {} - - var result = nativeObjectToString.call(value); - if (unmasked) { - if (isOwn) { - value[symToStringTag] = tag; - } else { - delete value[symToStringTag]; - } - } - return result; - } - - /** - * Creates an array of the own enumerable symbols of `object`. - * - * @private - * @param {Object} object The object to query. - * @returns {Array} Returns the array of symbols. - */ - var getSymbols = !nativeGetSymbols ? stubArray : function(object) { - if (object == null) { - return []; - } - object = Object(object); - return arrayFilter(nativeGetSymbols(object), function(symbol) { - return propertyIsEnumerable.call(object, symbol); - }); - }; - - /** - * Creates an array of the own and inherited enumerable symbols of `object`. - * - * @private - * @param {Object} object The object to query. - * @returns {Array} Returns the array of symbols. - */ - var getSymbolsIn = !nativeGetSymbols ? stubArray : function(object) { - var result = []; - while (object) { - arrayPush(result, getSymbols(object)); - object = getPrototype(object); - } - return result; - }; - - /** - * Gets the `toStringTag` of `value`. - * - * @private - * @param {*} value The value to query. - * @returns {string} Returns the `toStringTag`. - */ - var getTag = baseGetTag; - - // Fallback for data views, maps, sets, and weak maps in IE 11 and promises in Node.js < 6. - if ((DataView && getTag(new DataView(new ArrayBuffer(1))) != dataViewTag) || - (Map && getTag(new Map) != mapTag) || - (Promise && getTag(Promise.resolve()) != promiseTag) || - (Set && getTag(new Set) != setTag) || - (WeakMap && getTag(new WeakMap) != weakMapTag)) { - getTag = function(value) { - var result = baseGetTag(value), - Ctor = result == objectTag ? value.constructor : undefined, - ctorString = Ctor ? toSource(Ctor) : ''; - - if (ctorString) { - switch (ctorString) { - case dataViewCtorString: return dataViewTag; - case mapCtorString: return mapTag; - case promiseCtorString: return promiseTag; - case setCtorString: return setTag; - case weakMapCtorString: return weakMapTag; - } - } - return result; - }; - } - - /** - * Gets the view, applying any `transforms` to the `start` and `end` positions. - * - * @private - * @param {number} start The start of the view. - * @param {number} end The end of the view. - * @param {Array} transforms The transformations to apply to the view. - * @returns {Object} Returns an object containing the `start` and `end` - * positions of the view. - */ - function getView(start, end, transforms) { - var index = -1, - length = transforms.length; - - while (++index < length) { - var data = transforms[index], - size = data.size; - - switch (data.type) { - case 'drop': start += size; break; - case 'dropRight': end -= size; break; - case 'take': end = nativeMin(end, start + size); break; - case 'takeRight': start = nativeMax(start, end - size); break; - } - } - return { 'start': start, 'end': end }; - } - - /** - * Extracts wrapper details from the `source` body comment. - * - * @private - * @param {string} source The source to inspect. - * @returns {Array} Returns the wrapper details. - */ - function getWrapDetails(source) { - var match = source.match(reWrapDetails); - return match ? match[1].split(reSplitDetails) : []; - } - - /** - * Checks if `path` exists on `object`. - * - * @private - * @param {Object} object The object to query. - * @param {Array|string} path The path to check. - * @param {Function} hasFunc The function to check properties. - * @returns {boolean} Returns `true` if `path` exists, else `false`. - */ - function hasPath(object, path, hasFunc) { - path = castPath(path, object); - - var index = -1, - length = path.length, - result = false; - - while (++index < length) { - var key = toKey(path[index]); - if (!(result = object != null && hasFunc(object, key))) { - break; - } - object = object[key]; - } - if (result || ++index != length) { - return result; - } - length = object == null ? 0 : object.length; - return !!length && isLength(length) && isIndex(key, length) && - (isArray(object) || isArguments(object)); - } - - /** - * Initializes an array clone. - * - * @private - * @param {Array} array The array to clone. - * @returns {Array} Returns the initialized clone. - */ - function initCloneArray(array) { - var length = array.length, - result = new array.constructor(length); - - // Add properties assigned by `RegExp#exec`. - if (length && typeof array[0] == 'string' && hasOwnProperty.call(array, 'index')) { - result.index = array.index; - result.input = array.input; - } - return result; - } - - /** - * Initializes an object clone. - * - * @private - * @param {Object} object The object to clone. - * @returns {Object} Returns the initialized clone. - */ - function initCloneObject(object) { - return (typeof object.constructor == 'function' && !isPrototype(object)) - ? baseCreate(getPrototype(object)) - : {}; - } - - /** - * Initializes an object clone based on its `toStringTag`. - * - * **Note:** This function only supports cloning values with tags of - * `Boolean`, `Date`, `Error`, `Map`, `Number`, `RegExp`, `Set`, or `String`. - * - * @private - * @param {Object} object The object to clone. - * @param {string} tag The `toStringTag` of the object to clone. - * @param {boolean} [isDeep] Specify a deep clone. - * @returns {Object} Returns the initialized clone. - */ - function initCloneByTag(object, tag, isDeep) { - var Ctor = object.constructor; - switch (tag) { - case arrayBufferTag: - return cloneArrayBuffer(object); - - case boolTag: - case dateTag: - return new Ctor(+object); - - case dataViewTag: - return cloneDataView(object, isDeep); - - case float32Tag: case float64Tag: - case int8Tag: case int16Tag: case int32Tag: - case uint8Tag: case uint8ClampedTag: case uint16Tag: case uint32Tag: - return cloneTypedArray(object, isDeep); - - case mapTag: - return new Ctor; - - case numberTag: - case stringTag: - return new Ctor(object); - - case regexpTag: - return cloneRegExp(object); - - case setTag: - return new Ctor; - - case symbolTag: - return cloneSymbol(object); - } - } - - /** - * Inserts wrapper `details` in a comment at the top of the `source` body. - * - * @private - * @param {string} source The source to modify. - * @returns {Array} details The details to insert. - * @returns {string} Returns the modified source. - */ - function insertWrapDetails(source, details) { - var length = details.length; - if (!length) { - return source; - } - var lastIndex = length - 1; - details[lastIndex] = (length > 1 ? '& ' : '') + details[lastIndex]; - details = details.join(length > 2 ? ', ' : ' '); - return source.replace(reWrapComment, '{\n/* [wrapped with ' + details + '] */\n'); - } - - /** - * Checks if `value` is a flattenable `arguments` object or array. - * - * @private - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is flattenable, else `false`. - */ - function isFlattenable(value) { - return isArray(value) || isArguments(value) || - !!(spreadableSymbol && value && value[spreadableSymbol]); - } - - /** - * Checks if `value` is a valid array-like index. - * - * @private - * @param {*} value The value to check. - * @param {number} [length=MAX_SAFE_INTEGER] The upper bounds of a valid index. - * @returns {boolean} Returns `true` if `value` is a valid index, else `false`. - */ - function isIndex(value, length) { - var type = typeof value; - length = length == null ? MAX_SAFE_INTEGER : length; - - return !!length && - (type == 'number' || - (type != 'symbol' && reIsUint.test(value))) && - (value > -1 && value % 1 == 0 && value < length); - } - - /** - * Checks if the given arguments are from an iteratee call. - * - * @private - * @param {*} value The potential iteratee value argument. - * @param {*} index The potential iteratee index or key argument. - * @param {*} object The potential iteratee object argument. - * @returns {boolean} Returns `true` if the arguments are from an iteratee call, - * else `false`. - */ - function isIterateeCall(value, index, object) { - if (!isObject(object)) { - return false; - } - var type = typeof index; - if (type == 'number' - ? (isArrayLike(object) && isIndex(index, object.length)) - : (type == 'string' && index in object) - ) { - return eq(object[index], value); - } - return false; - } - - /** - * Checks if `value` is a property name and not a property path. - * - * @private - * @param {*} value The value to check. - * @param {Object} [object] The object to query keys on. - * @returns {boolean} Returns `true` if `value` is a property name, else `false`. - */ - function isKey(value, object) { - if (isArray(value)) { - return false; - } - var type = typeof value; - if (type == 'number' || type == 'symbol' || type == 'boolean' || - value == null || isSymbol(value)) { - return true; - } - return reIsPlainProp.test(value) || !reIsDeepProp.test(value) || - (object != null && value in Object(object)); - } - - /** - * Checks if `value` is suitable for use as unique object key. - * - * @private - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is suitable, else `false`. - */ - function isKeyable(value) { - var type = typeof value; - return (type == 'string' || type == 'number' || type == 'symbol' || type == 'boolean') - ? (value !== '__proto__') - : (value === null); - } - - /** - * Checks if `func` has a lazy counterpart. - * - * @private - * @param {Function} func The function to check. - * @returns {boolean} Returns `true` if `func` has a lazy counterpart, - * else `false`. - */ - function isLaziable(func) { - var funcName = getFuncName(func), - other = lodash[funcName]; - - if (typeof other != 'function' || !(funcName in LazyWrapper.prototype)) { - return false; - } - if (func === other) { - return true; - } - var data = getData(other); - return !!data && func === data[0]; - } - - /** - * Checks if `func` has its source masked. - * - * @private - * @param {Function} func The function to check. - * @returns {boolean} Returns `true` if `func` is masked, else `false`. - */ - function isMasked(func) { - return !!maskSrcKey && (maskSrcKey in func); - } - - /** - * Checks if `func` is capable of being masked. - * - * @private - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `func` is maskable, else `false`. - */ - var isMaskable = coreJsData ? isFunction : stubFalse; - - /** - * Checks if `value` is likely a prototype object. - * - * @private - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a prototype, else `false`. - */ - function isPrototype(value) { - var Ctor = value && value.constructor, - proto = (typeof Ctor == 'function' && Ctor.prototype) || objectProto; - - return value === proto; - } - - /** - * Checks if `value` is suitable for strict equality comparisons, i.e. `===`. - * - * @private - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` if suitable for strict - * equality comparisons, else `false`. - */ - function isStrictComparable(value) { - return value === value && !isObject(value); - } - - /** - * A specialized version of `matchesProperty` for source values suitable - * for strict equality comparisons, i.e. `===`. - * - * @private - * @param {string} key The key of the property to get. - * @param {*} srcValue The value to match. - * @returns {Function} Returns the new spec function. - */ - function matchesStrictComparable(key, srcValue) { - return function(object) { - if (object == null) { - return false; - } - return object[key] === srcValue && - (srcValue !== undefined || (key in Object(object))); - }; - } - - /** - * A specialized version of `_.memoize` which clears the memoized function's - * cache when it exceeds `MAX_MEMOIZE_SIZE`. - * - * @private - * @param {Function} func The function to have its output memoized. - * @returns {Function} Returns the new memoized function. - */ - function memoizeCapped(func) { - var result = memoize(func, function(key) { - if (cache.size === MAX_MEMOIZE_SIZE) { - cache.clear(); - } - return key; - }); - - var cache = result.cache; - return result; - } - - /** - * Merges the function metadata of `source` into `data`. - * - * Merging metadata reduces the number of wrappers used to invoke a function. - * This is possible because methods like `_.bind`, `_.curry`, and `_.partial` - * may be applied regardless of execution order. Methods like `_.ary` and - * `_.rearg` modify function arguments, making the order in which they are - * executed important, preventing the merging of metadata. However, we make - * an exception for a safe combined case where curried functions have `_.ary` - * and or `_.rearg` applied. - * - * @private - * @param {Array} data The destination metadata. - * @param {Array} source The source metadata. - * @returns {Array} Returns `data`. - */ - function mergeData(data, source) { - var bitmask = data[1], - srcBitmask = source[1], - newBitmask = bitmask | srcBitmask, - isCommon = newBitmask < (WRAP_BIND_FLAG | WRAP_BIND_KEY_FLAG | WRAP_ARY_FLAG); - - var isCombo = - ((srcBitmask == WRAP_ARY_FLAG) && (bitmask == WRAP_CURRY_FLAG)) || - ((srcBitmask == WRAP_ARY_FLAG) && (bitmask == WRAP_REARG_FLAG) && (data[7].length <= source[8])) || - ((srcBitmask == (WRAP_ARY_FLAG | WRAP_REARG_FLAG)) && (source[7].length <= source[8]) && (bitmask == WRAP_CURRY_FLAG)); - - // Exit early if metadata can't be merged. - if (!(isCommon || isCombo)) { - return data; - } - // Use source `thisArg` if available. - if (srcBitmask & WRAP_BIND_FLAG) { - data[2] = source[2]; - // Set when currying a bound function. - newBitmask |= bitmask & WRAP_BIND_FLAG ? 0 : WRAP_CURRY_BOUND_FLAG; - } - // Compose partial arguments. - var value = source[3]; - if (value) { - var partials = data[3]; - data[3] = partials ? composeArgs(partials, value, source[4]) : value; - data[4] = partials ? replaceHolders(data[3], PLACEHOLDER) : source[4]; - } - // Compose partial right arguments. - value = source[5]; - if (value) { - partials = data[5]; - data[5] = partials ? composeArgsRight(partials, value, source[6]) : value; - data[6] = partials ? replaceHolders(data[5], PLACEHOLDER) : source[6]; - } - // Use source `argPos` if available. - value = source[7]; - if (value) { - data[7] = value; - } - // Use source `ary` if it's smaller. - if (srcBitmask & WRAP_ARY_FLAG) { - data[8] = data[8] == null ? source[8] : nativeMin(data[8], source[8]); - } - // Use source `arity` if one is not provided. - if (data[9] == null) { - data[9] = source[9]; - } - // Use source `func` and merge bitmasks. - data[0] = source[0]; - data[1] = newBitmask; - - return data; - } - - /** - * This function is like - * [`Object.keys`](http://ecma-international.org/ecma-262/7.0/#sec-object.keys) - * except that it includes inherited enumerable properties. - * - * @private - * @param {Object} object The object to query. - * @returns {Array} Returns the array of property names. - */ - function nativeKeysIn(object) { - var result = []; - if (object != null) { - for (var key in Object(object)) { - result.push(key); - } - } - return result; - } - - /** - * Converts `value` to a string using `Object.prototype.toString`. - * - * @private - * @param {*} value The value to convert. - * @returns {string} Returns the converted string. - */ - function objectToString(value) { - return nativeObjectToString.call(value); - } - - /** - * A specialized version of `baseRest` which transforms the rest array. - * - * @private - * @param {Function} func The function to apply a rest parameter to. - * @param {number} [start=func.length-1] The start position of the rest parameter. - * @param {Function} transform The rest array transform. - * @returns {Function} Returns the new function. - */ - function overRest(func, start, transform) { - start = nativeMax(start === undefined ? (func.length - 1) : start, 0); - return function() { - var args = arguments, - index = -1, - length = nativeMax(args.length - start, 0), - array = Array(length); - - while (++index < length) { - array[index] = args[start + index]; - } - index = -1; - var otherArgs = Array(start + 1); - while (++index < start) { - otherArgs[index] = args[index]; - } - otherArgs[start] = transform(array); - return apply(func, this, otherArgs); - }; - } - - /** - * Gets the parent value at `path` of `object`. - * - * @private - * @param {Object} object The object to query. - * @param {Array} path The path to get the parent value of. - * @returns {*} Returns the parent value. - */ - function parent(object, path) { - return path.length < 2 ? object : baseGet(object, baseSlice(path, 0, -1)); - } - - /** - * Reorder `array` according to the specified indexes where the element at - * the first index is assigned as the first element, the element at - * the second index is assigned as the second element, and so on. - * - * @private - * @param {Array} array The array to reorder. - * @param {Array} indexes The arranged array indexes. - * @returns {Array} Returns `array`. - */ - function reorder(array, indexes) { - var arrLength = array.length, - length = nativeMin(indexes.length, arrLength), - oldArray = copyArray(array); - - while (length--) { - var index = indexes[length]; - array[length] = isIndex(index, arrLength) ? oldArray[index] : undefined; - } - return array; - } - - /** - * Gets the value at `key`, unless `key` is "__proto__" or "constructor". - * - * @private - * @param {Object} object The object to query. - * @param {string} key The key of the property to get. - * @returns {*} Returns the property value. - */ - function safeGet(object, key) { - if (key === 'constructor' && typeof object[key] === 'function') { - return; - } - - if (key == '__proto__') { - return; - } - - return object[key]; - } - - /** - * Sets metadata for `func`. - * - * **Note:** If this function becomes hot, i.e. is invoked a lot in a short - * period of time, it will trip its breaker and transition to an identity - * function to avoid garbage collection pauses in V8. See - * [V8 issue 2070](https://bugs.chromium.org/p/v8/issues/detail?id=2070) - * for more details. - * - * @private - * @param {Function} func The function to associate metadata with. - * @param {*} data The metadata. - * @returns {Function} Returns `func`. - */ - var setData = shortOut(baseSetData); - - /** - * A simple wrapper around the global [`setTimeout`](https://mdn.io/setTimeout). - * - * @private - * @param {Function} func The function to delay. - * @param {number} wait The number of milliseconds to delay invocation. - * @returns {number|Object} Returns the timer id or timeout object. - */ - var setTimeout = ctxSetTimeout || function(func, wait) { - return root.setTimeout(func, wait); - }; - - /** - * Sets the `toString` method of `func` to return `string`. - * - * @private - * @param {Function} func The function to modify. - * @param {Function} string The `toString` result. - * @returns {Function} Returns `func`. - */ - var setToString = shortOut(baseSetToString); - - /** - * Sets the `toString` method of `wrapper` to mimic the source of `reference` - * with wrapper details in a comment at the top of the source body. - * - * @private - * @param {Function} wrapper The function to modify. - * @param {Function} reference The reference function. - * @param {number} bitmask The bitmask flags. See `createWrap` for more details. - * @returns {Function} Returns `wrapper`. - */ - function setWrapToString(wrapper, reference, bitmask) { - var source = (reference + ''); - return setToString(wrapper, insertWrapDetails(source, updateWrapDetails(getWrapDetails(source), bitmask))); - } - - /** - * Creates a function that'll short out and invoke `identity` instead - * of `func` when it's called `HOT_COUNT` or more times in `HOT_SPAN` - * milliseconds. - * - * @private - * @param {Function} func The function to restrict. - * @returns {Function} Returns the new shortable function. - */ - function shortOut(func) { - var count = 0, - lastCalled = 0; - - return function() { - var stamp = nativeNow(), - remaining = HOT_SPAN - (stamp - lastCalled); - - lastCalled = stamp; - if (remaining > 0) { - if (++count >= HOT_COUNT) { - return arguments[0]; - } - } else { - count = 0; - } - return func.apply(undefined, arguments); - }; - } - - /** - * A specialized version of `_.shuffle` which mutates and sets the size of `array`. - * - * @private - * @param {Array} array The array to shuffle. - * @param {number} [size=array.length] The size of `array`. - * @returns {Array} Returns `array`. - */ - function shuffleSelf(array, size) { - var index = -1, - length = array.length, - lastIndex = length - 1; - - size = size === undefined ? length : size; - while (++index < size) { - var rand = baseRandom(index, lastIndex), - value = array[rand]; - - array[rand] = array[index]; - array[index] = value; - } - array.length = size; - return array; - } - - /** - * Converts `string` to a property path array. - * - * @private - * @param {string} string The string to convert. - * @returns {Array} Returns the property path array. - */ - var stringToPath = memoizeCapped(function(string) { - var result = []; - if (string.charCodeAt(0) === 46 /* . */) { - result.push(''); - } - string.replace(rePropName, function(match, number, quote, subString) { - result.push(quote ? subString.replace(reEscapeChar, '$1') : (number || match)); - }); - return result; - }); - - /** - * Converts `value` to a string key if it's not a string or symbol. - * - * @private - * @param {*} value The value to inspect. - * @returns {string|symbol} Returns the key. - */ - function toKey(value) { - if (typeof value == 'string' || isSymbol(value)) { - return value; - } - var result = (value + ''); - return (result == '0' && (1 / value) == -INFINITY) ? '-0' : result; - } - - /** - * Converts `func` to its source code. - * - * @private - * @param {Function} func The function to convert. - * @returns {string} Returns the source code. - */ - function toSource(func) { - if (func != null) { - try { - return funcToString.call(func); - } catch (e) {} - try { - return (func + ''); - } catch (e) {} - } - return ''; - } - - /** - * Updates wrapper `details` based on `bitmask` flags. - * - * @private - * @returns {Array} details The details to modify. - * @param {number} bitmask The bitmask flags. See `createWrap` for more details. - * @returns {Array} Returns `details`. - */ - function updateWrapDetails(details, bitmask) { - arrayEach(wrapFlags, function(pair) { - var value = '_.' + pair[0]; - if ((bitmask & pair[1]) && !arrayIncludes(details, value)) { - details.push(value); - } - }); - return details.sort(); - } - - /** - * Creates a clone of `wrapper`. - * - * @private - * @param {Object} wrapper The wrapper to clone. - * @returns {Object} Returns the cloned wrapper. - */ - function wrapperClone(wrapper) { - if (wrapper instanceof LazyWrapper) { - return wrapper.clone(); - } - var result = new LodashWrapper(wrapper.__wrapped__, wrapper.__chain__); - result.__actions__ = copyArray(wrapper.__actions__); - result.__index__ = wrapper.__index__; - result.__values__ = wrapper.__values__; - return result; - } - - /*------------------------------------------------------------------------*/ - - /** - * Creates an array of elements split into groups the length of `size`. - * If `array` can't be split evenly, the final chunk will be the remaining - * elements. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Array - * @param {Array} array The array to process. - * @param {number} [size=1] The length of each chunk - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.map`. - * @returns {Array} Returns the new array of chunks. - * @example - * - * _.chunk(['a', 'b', 'c', 'd'], 2); - * // => [['a', 'b'], ['c', 'd']] - * - * _.chunk(['a', 'b', 'c', 'd'], 3); - * // => [['a', 'b', 'c'], ['d']] - */ - function chunk(array, size, guard) { - if ((guard ? isIterateeCall(array, size, guard) : size === undefined)) { - size = 1; - } else { - size = nativeMax(toInteger(size), 0); - } - var length = array == null ? 0 : array.length; - if (!length || size < 1) { - return []; - } - var index = 0, - resIndex = 0, - result = Array(nativeCeil(length / size)); - - while (index < length) { - result[resIndex++] = baseSlice(array, index, (index += size)); - } - return result; - } - - /** - * Creates an array with all falsey values removed. The values `false`, `null`, - * `0`, `""`, `undefined`, and `NaN` are falsey. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Array - * @param {Array} array The array to compact. - * @returns {Array} Returns the new array of filtered values. - * @example - * - * _.compact([0, 1, false, 2, '', 3]); - * // => [1, 2, 3] - */ - function compact(array) { - var index = -1, - length = array == null ? 0 : array.length, - resIndex = 0, - result = []; - - while (++index < length) { - var value = array[index]; - if (value) { - result[resIndex++] = value; - } - } - return result; - } - - /** - * Creates a new array concatenating `array` with any additional arrays - * and/or values. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The array to concatenate. - * @param {...*} [values] The values to concatenate. - * @returns {Array} Returns the new concatenated array. - * @example - * - * var array = [1]; - * var other = _.concat(array, 2, [3], [[4]]); - * - * console.log(other); - * // => [1, 2, 3, [4]] - * - * console.log(array); - * // => [1] - */ - function concat() { - var length = arguments.length; - if (!length) { - return []; - } - var args = Array(length - 1), - array = arguments[0], - index = length; - - while (index--) { - args[index - 1] = arguments[index]; - } - return arrayPush(isArray(array) ? copyArray(array) : [array], baseFlatten(args, 1)); - } - - /** - * Creates an array of `array` values not included in the other given arrays - * using [`SameValueZero`](http://ecma-international.org/ecma-262/7.0/#sec-samevaluezero) - * for equality comparisons. The order and references of result values are - * determined by the first array. - * - * **Note:** Unlike `_.pullAll`, this method returns a new array. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Array - * @param {Array} array The array to inspect. - * @param {...Array} [values] The values to exclude. - * @returns {Array} Returns the new array of filtered values. - * @see _.without, _.xor - * @example - * - * _.difference([2, 1], [2, 3]); - * // => [1] - */ - var difference = baseRest(function(array, values) { - return isArrayLikeObject(array) - ? baseDifference(array, baseFlatten(values, 1, isArrayLikeObject, true)) - : []; - }); - - /** - * This method is like `_.difference` except that it accepts `iteratee` which - * is invoked for each element of `array` and `values` to generate the criterion - * by which they're compared. The order and references of result values are - * determined by the first array. The iteratee is invoked with one argument: - * (value). - * - * **Note:** Unlike `_.pullAllBy`, this method returns a new array. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The array to inspect. - * @param {...Array} [values] The values to exclude. - * @param {Function} [iteratee=_.identity] The iteratee invoked per element. - * @returns {Array} Returns the new array of filtered values. - * @example - * - * _.differenceBy([2.1, 1.2], [2.3, 3.4], Math.floor); - * // => [1.2] - * - * // The `_.property` iteratee shorthand. - * _.differenceBy([{ 'x': 2 }, { 'x': 1 }], [{ 'x': 1 }], 'x'); - * // => [{ 'x': 2 }] - */ - var differenceBy = baseRest(function(array, values) { - var iteratee = last(values); - if (isArrayLikeObject(iteratee)) { - iteratee = undefined; - } - return isArrayLikeObject(array) - ? baseDifference(array, baseFlatten(values, 1, isArrayLikeObject, true), getIteratee(iteratee, 2)) - : []; - }); - - /** - * This method is like `_.difference` except that it accepts `comparator` - * which is invoked to compare elements of `array` to `values`. The order and - * references of result values are determined by the first array. The comparator - * is invoked with two arguments: (arrVal, othVal). - * - * **Note:** Unlike `_.pullAllWith`, this method returns a new array. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The array to inspect. - * @param {...Array} [values] The values to exclude. - * @param {Function} [comparator] The comparator invoked per element. - * @returns {Array} Returns the new array of filtered values. - * @example - * - * var objects = [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }]; - * - * _.differenceWith(objects, [{ 'x': 1, 'y': 2 }], _.isEqual); - * // => [{ 'x': 2, 'y': 1 }] - */ - var differenceWith = baseRest(function(array, values) { - var comparator = last(values); - if (isArrayLikeObject(comparator)) { - comparator = undefined; - } - return isArrayLikeObject(array) - ? baseDifference(array, baseFlatten(values, 1, isArrayLikeObject, true), undefined, comparator) - : []; - }); - - /** - * Creates a slice of `array` with `n` elements dropped from the beginning. - * - * @static - * @memberOf _ - * @since 0.5.0 - * @category Array - * @param {Array} array The array to query. - * @param {number} [n=1] The number of elements to drop. - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.map`. - * @returns {Array} Returns the slice of `array`. - * @example - * - * _.drop([1, 2, 3]); - * // => [2, 3] - * - * _.drop([1, 2, 3], 2); - * // => [3] - * - * _.drop([1, 2, 3], 5); - * // => [] - * - * _.drop([1, 2, 3], 0); - * // => [1, 2, 3] - */ - function drop(array, n, guard) { - var length = array == null ? 0 : array.length; - if (!length) { - return []; - } - n = (guard || n === undefined) ? 1 : toInteger(n); - return baseSlice(array, n < 0 ? 0 : n, length); - } - - /** - * Creates a slice of `array` with `n` elements dropped from the end. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Array - * @param {Array} array The array to query. - * @param {number} [n=1] The number of elements to drop. - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.map`. - * @returns {Array} Returns the slice of `array`. - * @example - * - * _.dropRight([1, 2, 3]); - * // => [1, 2] - * - * _.dropRight([1, 2, 3], 2); - * // => [1] - * - * _.dropRight([1, 2, 3], 5); - * // => [] - * - * _.dropRight([1, 2, 3], 0); - * // => [1, 2, 3] - */ - function dropRight(array, n, guard) { - var length = array == null ? 0 : array.length; - if (!length) { - return []; - } - n = (guard || n === undefined) ? 1 : toInteger(n); - n = length - n; - return baseSlice(array, 0, n < 0 ? 0 : n); - } - - /** - * Creates a slice of `array` excluding elements dropped from the end. - * Elements are dropped until `predicate` returns falsey. The predicate is - * invoked with three arguments: (value, index, array). - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Array - * @param {Array} array The array to query. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @returns {Array} Returns the slice of `array`. - * @example - * - * var users = [ - * { 'user': 'barney', 'active': true }, - * { 'user': 'fred', 'active': false }, - * { 'user': 'pebbles', 'active': false } - * ]; - * - * _.dropRightWhile(users, function(o) { return !o.active; }); - * // => objects for ['barney'] - * - * // The `_.matches` iteratee shorthand. - * _.dropRightWhile(users, { 'user': 'pebbles', 'active': false }); - * // => objects for ['barney', 'fred'] - * - * // The `_.matchesProperty` iteratee shorthand. - * _.dropRightWhile(users, ['active', false]); - * // => objects for ['barney'] - * - * // The `_.property` iteratee shorthand. - * _.dropRightWhile(users, 'active'); - * // => objects for ['barney', 'fred', 'pebbles'] - */ - function dropRightWhile(array, predicate) { - return (array && array.length) - ? baseWhile(array, getIteratee(predicate, 3), true, true) - : []; - } - - /** - * Creates a slice of `array` excluding elements dropped from the beginning. - * Elements are dropped until `predicate` returns falsey. The predicate is - * invoked with three arguments: (value, index, array). - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Array - * @param {Array} array The array to query. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @returns {Array} Returns the slice of `array`. - * @example - * - * var users = [ - * { 'user': 'barney', 'active': false }, - * { 'user': 'fred', 'active': false }, - * { 'user': 'pebbles', 'active': true } - * ]; - * - * _.dropWhile(users, function(o) { return !o.active; }); - * // => objects for ['pebbles'] - * - * // The `_.matches` iteratee shorthand. - * _.dropWhile(users, { 'user': 'barney', 'active': false }); - * // => objects for ['fred', 'pebbles'] - * - * // The `_.matchesProperty` iteratee shorthand. - * _.dropWhile(users, ['active', false]); - * // => objects for ['pebbles'] - * - * // The `_.property` iteratee shorthand. - * _.dropWhile(users, 'active'); - * // => objects for ['barney', 'fred', 'pebbles'] - */ - function dropWhile(array, predicate) { - return (array && array.length) - ? baseWhile(array, getIteratee(predicate, 3), true) - : []; - } - - /** - * Fills elements of `array` with `value` from `start` up to, but not - * including, `end`. - * - * **Note:** This method mutates `array`. - * - * @static - * @memberOf _ - * @since 3.2.0 - * @category Array - * @param {Array} array The array to fill. - * @param {*} value The value to fill `array` with. - * @param {number} [start=0] The start position. - * @param {number} [end=array.length] The end position. - * @returns {Array} Returns `array`. - * @example - * - * var array = [1, 2, 3]; - * - * _.fill(array, 'a'); - * console.log(array); - * // => ['a', 'a', 'a'] - * - * _.fill(Array(3), 2); - * // => [2, 2, 2] - * - * _.fill([4, 6, 8, 10], '*', 1, 3); - * // => [4, '*', '*', 10] - */ - function fill(array, value, start, end) { - var length = array == null ? 0 : array.length; - if (!length) { - return []; - } - if (start && typeof start != 'number' && isIterateeCall(array, value, start)) { - start = 0; - end = length; - } - return baseFill(array, value, start, end); - } - - /** - * This method is like `_.find` except that it returns the index of the first - * element `predicate` returns truthy for instead of the element itself. - * - * @static - * @memberOf _ - * @since 1.1.0 - * @category Array - * @param {Array} array The array to inspect. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @param {number} [fromIndex=0] The index to search from. - * @returns {number} Returns the index of the found element, else `-1`. - * @example - * - * var users = [ - * { 'user': 'barney', 'active': false }, - * { 'user': 'fred', 'active': false }, - * { 'user': 'pebbles', 'active': true } - * ]; - * - * _.findIndex(users, function(o) { return o.user == 'barney'; }); - * // => 0 - * - * // The `_.matches` iteratee shorthand. - * _.findIndex(users, { 'user': 'fred', 'active': false }); - * // => 1 - * - * // The `_.matchesProperty` iteratee shorthand. - * _.findIndex(users, ['active', false]); - * // => 0 - * - * // The `_.property` iteratee shorthand. - * _.findIndex(users, 'active'); - * // => 2 - */ - function findIndex(array, predicate, fromIndex) { - var length = array == null ? 0 : array.length; - if (!length) { - return -1; - } - var index = fromIndex == null ? 0 : toInteger(fromIndex); - if (index < 0) { - index = nativeMax(length + index, 0); - } - return baseFindIndex(array, getIteratee(predicate, 3), index); - } - - /** - * This method is like `_.findIndex` except that it iterates over elements - * of `collection` from right to left. - * - * @static - * @memberOf _ - * @since 2.0.0 - * @category Array - * @param {Array} array The array to inspect. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @param {number} [fromIndex=array.length-1] The index to search from. - * @returns {number} Returns the index of the found element, else `-1`. - * @example - * - * var users = [ - * { 'user': 'barney', 'active': true }, - * { 'user': 'fred', 'active': false }, - * { 'user': 'pebbles', 'active': false } - * ]; - * - * _.findLastIndex(users, function(o) { return o.user == 'pebbles'; }); - * // => 2 - * - * // The `_.matches` iteratee shorthand. - * _.findLastIndex(users, { 'user': 'barney', 'active': true }); - * // => 0 - * - * // The `_.matchesProperty` iteratee shorthand. - * _.findLastIndex(users, ['active', false]); - * // => 2 - * - * // The `_.property` iteratee shorthand. - * _.findLastIndex(users, 'active'); - * // => 0 - */ - function findLastIndex(array, predicate, fromIndex) { - var length = array == null ? 0 : array.length; - if (!length) { - return -1; - } - var index = length - 1; - if (fromIndex !== undefined) { - index = toInteger(fromIndex); - index = fromIndex < 0 - ? nativeMax(length + index, 0) - : nativeMin(index, length - 1); - } - return baseFindIndex(array, getIteratee(predicate, 3), index, true); - } - - /** - * Flattens `array` a single level deep. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Array - * @param {Array} array The array to flatten. - * @returns {Array} Returns the new flattened array. - * @example - * - * _.flatten([1, [2, [3, [4]], 5]]); - * // => [1, 2, [3, [4]], 5] - */ - function flatten(array) { - var length = array == null ? 0 : array.length; - return length ? baseFlatten(array, 1) : []; - } - - /** - * Recursively flattens `array`. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Array - * @param {Array} array The array to flatten. - * @returns {Array} Returns the new flattened array. - * @example - * - * _.flattenDeep([1, [2, [3, [4]], 5]]); - * // => [1, 2, 3, 4, 5] - */ - function flattenDeep(array) { - var length = array == null ? 0 : array.length; - return length ? baseFlatten(array, INFINITY) : []; - } - - /** - * Recursively flatten `array` up to `depth` times. - * - * @static - * @memberOf _ - * @since 4.4.0 - * @category Array - * @param {Array} array The array to flatten. - * @param {number} [depth=1] The maximum recursion depth. - * @returns {Array} Returns the new flattened array. - * @example - * - * var array = [1, [2, [3, [4]], 5]]; - * - * _.flattenDepth(array, 1); - * // => [1, 2, [3, [4]], 5] - * - * _.flattenDepth(array, 2); - * // => [1, 2, 3, [4], 5] - */ - function flattenDepth(array, depth) { - var length = array == null ? 0 : array.length; - if (!length) { - return []; - } - depth = depth === undefined ? 1 : toInteger(depth); - return baseFlatten(array, depth); - } - - /** - * The inverse of `_.toPairs`; this method returns an object composed - * from key-value `pairs`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} pairs The key-value pairs. - * @returns {Object} Returns the new object. - * @example - * - * _.fromPairs([['a', 1], ['b', 2]]); - * // => { 'a': 1, 'b': 2 } - */ - function fromPairs(pairs) { - var index = -1, - length = pairs == null ? 0 : pairs.length, - result = {}; - - while (++index < length) { - var pair = pairs[index]; - result[pair[0]] = pair[1]; - } - return result; - } - - /** - * Gets the first element of `array`. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @alias first - * @category Array - * @param {Array} array The array to query. - * @returns {*} Returns the first element of `array`. - * @example - * - * _.head([1, 2, 3]); - * // => 1 - * - * _.head([]); - * // => undefined - */ - function head(array) { - return (array && array.length) ? array[0] : undefined; - } - - /** - * Gets the index at which the first occurrence of `value` is found in `array` - * using [`SameValueZero`](http://ecma-international.org/ecma-262/7.0/#sec-samevaluezero) - * for equality comparisons. If `fromIndex` is negative, it's used as the - * offset from the end of `array`. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Array - * @param {Array} array The array to inspect. - * @param {*} value The value to search for. - * @param {number} [fromIndex=0] The index to search from. - * @returns {number} Returns the index of the matched value, else `-1`. - * @example - * - * _.indexOf([1, 2, 1, 2], 2); - * // => 1 - * - * // Search from the `fromIndex`. - * _.indexOf([1, 2, 1, 2], 2, 2); - * // => 3 - */ - function indexOf(array, value, fromIndex) { - var length = array == null ? 0 : array.length; - if (!length) { - return -1; - } - var index = fromIndex == null ? 0 : toInteger(fromIndex); - if (index < 0) { - index = nativeMax(length + index, 0); - } - return baseIndexOf(array, value, index); - } - - /** - * Gets all but the last element of `array`. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Array - * @param {Array} array The array to query. - * @returns {Array} Returns the slice of `array`. - * @example - * - * _.initial([1, 2, 3]); - * // => [1, 2] - */ - function initial(array) { - var length = array == null ? 0 : array.length; - return length ? baseSlice(array, 0, -1) : []; - } - - /** - * Creates an array of unique values that are included in all given arrays - * using [`SameValueZero`](http://ecma-international.org/ecma-262/7.0/#sec-samevaluezero) - * for equality comparisons. The order and references of result values are - * determined by the first array. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Array - * @param {...Array} [arrays] The arrays to inspect. - * @returns {Array} Returns the new array of intersecting values. - * @example - * - * _.intersection([2, 1], [2, 3]); - * // => [2] - */ - var intersection = baseRest(function(arrays) { - var mapped = arrayMap(arrays, castArrayLikeObject); - return (mapped.length && mapped[0] === arrays[0]) - ? baseIntersection(mapped) - : []; - }); - - /** - * This method is like `_.intersection` except that it accepts `iteratee` - * which is invoked for each element of each `arrays` to generate the criterion - * by which they're compared. The order and references of result values are - * determined by the first array. The iteratee is invoked with one argument: - * (value). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {...Array} [arrays] The arrays to inspect. - * @param {Function} [iteratee=_.identity] The iteratee invoked per element. - * @returns {Array} Returns the new array of intersecting values. - * @example - * - * _.intersectionBy([2.1, 1.2], [2.3, 3.4], Math.floor); - * // => [2.1] - * - * // The `_.property` iteratee shorthand. - * _.intersectionBy([{ 'x': 1 }], [{ 'x': 2 }, { 'x': 1 }], 'x'); - * // => [{ 'x': 1 }] - */ - var intersectionBy = baseRest(function(arrays) { - var iteratee = last(arrays), - mapped = arrayMap(arrays, castArrayLikeObject); - - if (iteratee === last(mapped)) { - iteratee = undefined; - } else { - mapped.pop(); - } - return (mapped.length && mapped[0] === arrays[0]) - ? baseIntersection(mapped, getIteratee(iteratee, 2)) - : []; - }); - - /** - * This method is like `_.intersection` except that it accepts `comparator` - * which is invoked to compare elements of `arrays`. The order and references - * of result values are determined by the first array. The comparator is - * invoked with two arguments: (arrVal, othVal). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {...Array} [arrays] The arrays to inspect. - * @param {Function} [comparator] The comparator invoked per element. - * @returns {Array} Returns the new array of intersecting values. - * @example - * - * var objects = [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }]; - * var others = [{ 'x': 1, 'y': 1 }, { 'x': 1, 'y': 2 }]; - * - * _.intersectionWith(objects, others, _.isEqual); - * // => [{ 'x': 1, 'y': 2 }] - */ - var intersectionWith = baseRest(function(arrays) { - var comparator = last(arrays), - mapped = arrayMap(arrays, castArrayLikeObject); - - comparator = typeof comparator == 'function' ? comparator : undefined; - if (comparator) { - mapped.pop(); - } - return (mapped.length && mapped[0] === arrays[0]) - ? baseIntersection(mapped, undefined, comparator) - : []; - }); - - /** - * Converts all elements in `array` into a string separated by `separator`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The array to convert. - * @param {string} [separator=','] The element separator. - * @returns {string} Returns the joined string. - * @example - * - * _.join(['a', 'b', 'c'], '~'); - * // => 'a~b~c' - */ - function join(array, separator) { - return array == null ? '' : nativeJoin.call(array, separator); - } - - /** - * Gets the last element of `array`. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Array - * @param {Array} array The array to query. - * @returns {*} Returns the last element of `array`. - * @example - * - * _.last([1, 2, 3]); - * // => 3 - */ - function last(array) { - var length = array == null ? 0 : array.length; - return length ? array[length - 1] : undefined; - } - - /** - * This method is like `_.indexOf` except that it iterates over elements of - * `array` from right to left. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Array - * @param {Array} array The array to inspect. - * @param {*} value The value to search for. - * @param {number} [fromIndex=array.length-1] The index to search from. - * @returns {number} Returns the index of the matched value, else `-1`. - * @example - * - * _.lastIndexOf([1, 2, 1, 2], 2); - * // => 3 - * - * // Search from the `fromIndex`. - * _.lastIndexOf([1, 2, 1, 2], 2, 2); - * // => 1 - */ - function lastIndexOf(array, value, fromIndex) { - var length = array == null ? 0 : array.length; - if (!length) { - return -1; - } - var index = length; - if (fromIndex !== undefined) { - index = toInteger(fromIndex); - index = index < 0 ? nativeMax(length + index, 0) : nativeMin(index, length - 1); - } - return value === value - ? strictLastIndexOf(array, value, index) - : baseFindIndex(array, baseIsNaN, index, true); - } - - /** - * Gets the element at index `n` of `array`. If `n` is negative, the nth - * element from the end is returned. - * - * @static - * @memberOf _ - * @since 4.11.0 - * @category Array - * @param {Array} array The array to query. - * @param {number} [n=0] The index of the element to return. - * @returns {*} Returns the nth element of `array`. - * @example - * - * var array = ['a', 'b', 'c', 'd']; - * - * _.nth(array, 1); - * // => 'b' - * - * _.nth(array, -2); - * // => 'c'; - */ - function nth(array, n) { - return (array && array.length) ? baseNth(array, toInteger(n)) : undefined; - } - - /** - * Removes all given values from `array` using - * [`SameValueZero`](http://ecma-international.org/ecma-262/7.0/#sec-samevaluezero) - * for equality comparisons. - * - * **Note:** Unlike `_.without`, this method mutates `array`. Use `_.remove` - * to remove elements from an array by predicate. - * - * @static - * @memberOf _ - * @since 2.0.0 - * @category Array - * @param {Array} array The array to modify. - * @param {...*} [values] The values to remove. - * @returns {Array} Returns `array`. - * @example - * - * var array = ['a', 'b', 'c', 'a', 'b', 'c']; - * - * _.pull(array, 'a', 'c'); - * console.log(array); - * // => ['b', 'b'] - */ - var pull = baseRest(pullAll); - - /** - * This method is like `_.pull` except that it accepts an array of values to remove. - * - * **Note:** Unlike `_.difference`, this method mutates `array`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The array to modify. - * @param {Array} values The values to remove. - * @returns {Array} Returns `array`. - * @example - * - * var array = ['a', 'b', 'c', 'a', 'b', 'c']; - * - * _.pullAll(array, ['a', 'c']); - * console.log(array); - * // => ['b', 'b'] - */ - function pullAll(array, values) { - return (array && array.length && values && values.length) - ? basePullAll(array, values) - : array; - } - - /** - * This method is like `_.pullAll` except that it accepts `iteratee` which is - * invoked for each element of `array` and `values` to generate the criterion - * by which they're compared. The iteratee is invoked with one argument: (value). - * - * **Note:** Unlike `_.differenceBy`, this method mutates `array`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The array to modify. - * @param {Array} values The values to remove. - * @param {Function} [iteratee=_.identity] The iteratee invoked per element. - * @returns {Array} Returns `array`. - * @example - * - * var array = [{ 'x': 1 }, { 'x': 2 }, { 'x': 3 }, { 'x': 1 }]; - * - * _.pullAllBy(array, [{ 'x': 1 }, { 'x': 3 }], 'x'); - * console.log(array); - * // => [{ 'x': 2 }] - */ - function pullAllBy(array, values, iteratee) { - return (array && array.length && values && values.length) - ? basePullAll(array, values, getIteratee(iteratee, 2)) - : array; - } - - /** - * This method is like `_.pullAll` except that it accepts `comparator` which - * is invoked to compare elements of `array` to `values`. The comparator is - * invoked with two arguments: (arrVal, othVal). - * - * **Note:** Unlike `_.differenceWith`, this method mutates `array`. - * - * @static - * @memberOf _ - * @since 4.6.0 - * @category Array - * @param {Array} array The array to modify. - * @param {Array} values The values to remove. - * @param {Function} [comparator] The comparator invoked per element. - * @returns {Array} Returns `array`. - * @example - * - * var array = [{ 'x': 1, 'y': 2 }, { 'x': 3, 'y': 4 }, { 'x': 5, 'y': 6 }]; - * - * _.pullAllWith(array, [{ 'x': 3, 'y': 4 }], _.isEqual); - * console.log(array); - * // => [{ 'x': 1, 'y': 2 }, { 'x': 5, 'y': 6 }] - */ - function pullAllWith(array, values, comparator) { - return (array && array.length && values && values.length) - ? basePullAll(array, values, undefined, comparator) - : array; - } - - /** - * Removes elements from `array` corresponding to `indexes` and returns an - * array of removed elements. - * - * **Note:** Unlike `_.at`, this method mutates `array`. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Array - * @param {Array} array The array to modify. - * @param {...(number|number[])} [indexes] The indexes of elements to remove. - * @returns {Array} Returns the new array of removed elements. - * @example - * - * var array = ['a', 'b', 'c', 'd']; - * var pulled = _.pullAt(array, [1, 3]); - * - * console.log(array); - * // => ['a', 'c'] - * - * console.log(pulled); - * // => ['b', 'd'] - */ - var pullAt = flatRest(function(array, indexes) { - var length = array == null ? 0 : array.length, - result = baseAt(array, indexes); - - basePullAt(array, arrayMap(indexes, function(index) { - return isIndex(index, length) ? +index : index; - }).sort(compareAscending)); - - return result; - }); - - /** - * Removes all elements from `array` that `predicate` returns truthy for - * and returns an array of the removed elements. The predicate is invoked - * with three arguments: (value, index, array). - * - * **Note:** Unlike `_.filter`, this method mutates `array`. Use `_.pull` - * to pull elements from an array by value. - * - * @static - * @memberOf _ - * @since 2.0.0 - * @category Array - * @param {Array} array The array to modify. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @returns {Array} Returns the new array of removed elements. - * @example - * - * var array = [1, 2, 3, 4]; - * var evens = _.remove(array, function(n) { - * return n % 2 == 0; - * }); - * - * console.log(array); - * // => [1, 3] - * - * console.log(evens); - * // => [2, 4] - */ - function remove(array, predicate) { - var result = []; - if (!(array && array.length)) { - return result; - } - var index = -1, - indexes = [], - length = array.length; - - predicate = getIteratee(predicate, 3); - while (++index < length) { - var value = array[index]; - if (predicate(value, index, array)) { - result.push(value); - indexes.push(index); - } - } - basePullAt(array, indexes); - return result; - } - - /** - * Reverses `array` so that the first element becomes the last, the second - * element becomes the second to last, and so on. - * - * **Note:** This method mutates `array` and is based on - * [`Array#reverse`](https://mdn.io/Array/reverse). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The array to modify. - * @returns {Array} Returns `array`. - * @example - * - * var array = [1, 2, 3]; - * - * _.reverse(array); - * // => [3, 2, 1] - * - * console.log(array); - * // => [3, 2, 1] - */ - function reverse(array) { - return array == null ? array : nativeReverse.call(array); - } - - /** - * Creates a slice of `array` from `start` up to, but not including, `end`. - * - * **Note:** This method is used instead of - * [`Array#slice`](https://mdn.io/Array/slice) to ensure dense arrays are - * returned. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Array - * @param {Array} array The array to slice. - * @param {number} [start=0] The start position. - * @param {number} [end=array.length] The end position. - * @returns {Array} Returns the slice of `array`. - */ - function slice(array, start, end) { - var length = array == null ? 0 : array.length; - if (!length) { - return []; - } - if (end && typeof end != 'number' && isIterateeCall(array, start, end)) { - start = 0; - end = length; - } - else { - start = start == null ? 0 : toInteger(start); - end = end === undefined ? length : toInteger(end); - } - return baseSlice(array, start, end); - } - - /** - * Uses a binary search to determine the lowest index at which `value` - * should be inserted into `array` in order to maintain its sort order. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Array - * @param {Array} array The sorted array to inspect. - * @param {*} value The value to evaluate. - * @returns {number} Returns the index at which `value` should be inserted - * into `array`. - * @example - * - * _.sortedIndex([30, 50], 40); - * // => 1 - */ - function sortedIndex(array, value) { - return baseSortedIndex(array, value); - } - - /** - * This method is like `_.sortedIndex` except that it accepts `iteratee` - * which is invoked for `value` and each element of `array` to compute their - * sort ranking. The iteratee is invoked with one argument: (value). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The sorted array to inspect. - * @param {*} value The value to evaluate. - * @param {Function} [iteratee=_.identity] The iteratee invoked per element. - * @returns {number} Returns the index at which `value` should be inserted - * into `array`. - * @example - * - * var objects = [{ 'x': 4 }, { 'x': 5 }]; - * - * _.sortedIndexBy(objects, { 'x': 4 }, function(o) { return o.x; }); - * // => 0 - * - * // The `_.property` iteratee shorthand. - * _.sortedIndexBy(objects, { 'x': 4 }, 'x'); - * // => 0 - */ - function sortedIndexBy(array, value, iteratee) { - return baseSortedIndexBy(array, value, getIteratee(iteratee, 2)); - } - - /** - * This method is like `_.indexOf` except that it performs a binary - * search on a sorted `array`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The array to inspect. - * @param {*} value The value to search for. - * @returns {number} Returns the index of the matched value, else `-1`. - * @example - * - * _.sortedIndexOf([4, 5, 5, 5, 6], 5); - * // => 1 - */ - function sortedIndexOf(array, value) { - var length = array == null ? 0 : array.length; - if (length) { - var index = baseSortedIndex(array, value); - if (index < length && eq(array[index], value)) { - return index; - } - } - return -1; - } - - /** - * This method is like `_.sortedIndex` except that it returns the highest - * index at which `value` should be inserted into `array` in order to - * maintain its sort order. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Array - * @param {Array} array The sorted array to inspect. - * @param {*} value The value to evaluate. - * @returns {number} Returns the index at which `value` should be inserted - * into `array`. - * @example - * - * _.sortedLastIndex([4, 5, 5, 5, 6], 5); - * // => 4 - */ - function sortedLastIndex(array, value) { - return baseSortedIndex(array, value, true); - } - - /** - * This method is like `_.sortedLastIndex` except that it accepts `iteratee` - * which is invoked for `value` and each element of `array` to compute their - * sort ranking. The iteratee is invoked with one argument: (value). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The sorted array to inspect. - * @param {*} value The value to evaluate. - * @param {Function} [iteratee=_.identity] The iteratee invoked per element. - * @returns {number} Returns the index at which `value` should be inserted - * into `array`. - * @example - * - * var objects = [{ 'x': 4 }, { 'x': 5 }]; - * - * _.sortedLastIndexBy(objects, { 'x': 4 }, function(o) { return o.x; }); - * // => 1 - * - * // The `_.property` iteratee shorthand. - * _.sortedLastIndexBy(objects, { 'x': 4 }, 'x'); - * // => 1 - */ - function sortedLastIndexBy(array, value, iteratee) { - return baseSortedIndexBy(array, value, getIteratee(iteratee, 2), true); - } - - /** - * This method is like `_.lastIndexOf` except that it performs a binary - * search on a sorted `array`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The array to inspect. - * @param {*} value The value to search for. - * @returns {number} Returns the index of the matched value, else `-1`. - * @example - * - * _.sortedLastIndexOf([4, 5, 5, 5, 6], 5); - * // => 3 - */ - function sortedLastIndexOf(array, value) { - var length = array == null ? 0 : array.length; - if (length) { - var index = baseSortedIndex(array, value, true) - 1; - if (eq(array[index], value)) { - return index; - } - } - return -1; - } - - /** - * This method is like `_.uniq` except that it's designed and optimized - * for sorted arrays. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The array to inspect. - * @returns {Array} Returns the new duplicate free array. - * @example - * - * _.sortedUniq([1, 1, 2]); - * // => [1, 2] - */ - function sortedUniq(array) { - return (array && array.length) - ? baseSortedUniq(array) - : []; - } - - /** - * This method is like `_.uniqBy` except that it's designed and optimized - * for sorted arrays. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The array to inspect. - * @param {Function} [iteratee] The iteratee invoked per element. - * @returns {Array} Returns the new duplicate free array. - * @example - * - * _.sortedUniqBy([1.1, 1.2, 2.3, 2.4], Math.floor); - * // => [1.1, 2.3] - */ - function sortedUniqBy(array, iteratee) { - return (array && array.length) - ? baseSortedUniq(array, getIteratee(iteratee, 2)) - : []; - } - - /** - * Gets all but the first element of `array`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The array to query. - * @returns {Array} Returns the slice of `array`. - * @example - * - * _.tail([1, 2, 3]); - * // => [2, 3] - */ - function tail(array) { - var length = array == null ? 0 : array.length; - return length ? baseSlice(array, 1, length) : []; - } - - /** - * Creates a slice of `array` with `n` elements taken from the beginning. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Array - * @param {Array} array The array to query. - * @param {number} [n=1] The number of elements to take. - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.map`. - * @returns {Array} Returns the slice of `array`. - * @example - * - * _.take([1, 2, 3]); - * // => [1] - * - * _.take([1, 2, 3], 2); - * // => [1, 2] - * - * _.take([1, 2, 3], 5); - * // => [1, 2, 3] - * - * _.take([1, 2, 3], 0); - * // => [] - */ - function take(array, n, guard) { - if (!(array && array.length)) { - return []; - } - n = (guard || n === undefined) ? 1 : toInteger(n); - return baseSlice(array, 0, n < 0 ? 0 : n); - } - - /** - * Creates a slice of `array` with `n` elements taken from the end. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Array - * @param {Array} array The array to query. - * @param {number} [n=1] The number of elements to take. - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.map`. - * @returns {Array} Returns the slice of `array`. - * @example - * - * _.takeRight([1, 2, 3]); - * // => [3] - * - * _.takeRight([1, 2, 3], 2); - * // => [2, 3] - * - * _.takeRight([1, 2, 3], 5); - * // => [1, 2, 3] - * - * _.takeRight([1, 2, 3], 0); - * // => [] - */ - function takeRight(array, n, guard) { - var length = array == null ? 0 : array.length; - if (!length) { - return []; - } - n = (guard || n === undefined) ? 1 : toInteger(n); - n = length - n; - return baseSlice(array, n < 0 ? 0 : n, length); - } - - /** - * Creates a slice of `array` with elements taken from the end. Elements are - * taken until `predicate` returns falsey. The predicate is invoked with - * three arguments: (value, index, array). - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Array - * @param {Array} array The array to query. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @returns {Array} Returns the slice of `array`. - * @example - * - * var users = [ - * { 'user': 'barney', 'active': true }, - * { 'user': 'fred', 'active': false }, - * { 'user': 'pebbles', 'active': false } - * ]; - * - * _.takeRightWhile(users, function(o) { return !o.active; }); - * // => objects for ['fred', 'pebbles'] - * - * // The `_.matches` iteratee shorthand. - * _.takeRightWhile(users, { 'user': 'pebbles', 'active': false }); - * // => objects for ['pebbles'] - * - * // The `_.matchesProperty` iteratee shorthand. - * _.takeRightWhile(users, ['active', false]); - * // => objects for ['fred', 'pebbles'] - * - * // The `_.property` iteratee shorthand. - * _.takeRightWhile(users, 'active'); - * // => [] - */ - function takeRightWhile(array, predicate) { - return (array && array.length) - ? baseWhile(array, getIteratee(predicate, 3), false, true) - : []; - } - - /** - * Creates a slice of `array` with elements taken from the beginning. Elements - * are taken until `predicate` returns falsey. The predicate is invoked with - * three arguments: (value, index, array). - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Array - * @param {Array} array The array to query. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @returns {Array} Returns the slice of `array`. - * @example - * - * var users = [ - * { 'user': 'barney', 'active': false }, - * { 'user': 'fred', 'active': false }, - * { 'user': 'pebbles', 'active': true } - * ]; - * - * _.takeWhile(users, function(o) { return !o.active; }); - * // => objects for ['barney', 'fred'] - * - * // The `_.matches` iteratee shorthand. - * _.takeWhile(users, { 'user': 'barney', 'active': false }); - * // => objects for ['barney'] - * - * // The `_.matchesProperty` iteratee shorthand. - * _.takeWhile(users, ['active', false]); - * // => objects for ['barney', 'fred'] - * - * // The `_.property` iteratee shorthand. - * _.takeWhile(users, 'active'); - * // => [] - */ - function takeWhile(array, predicate) { - return (array && array.length) - ? baseWhile(array, getIteratee(predicate, 3)) - : []; - } - - /** - * Creates an array of unique values, in order, from all given arrays using - * [`SameValueZero`](http://ecma-international.org/ecma-262/7.0/#sec-samevaluezero) - * for equality comparisons. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Array - * @param {...Array} [arrays] The arrays to inspect. - * @returns {Array} Returns the new array of combined values. - * @example - * - * _.union([2], [1, 2]); - * // => [2, 1] - */ - var union = baseRest(function(arrays) { - return baseUniq(baseFlatten(arrays, 1, isArrayLikeObject, true)); - }); - - /** - * This method is like `_.union` except that it accepts `iteratee` which is - * invoked for each element of each `arrays` to generate the criterion by - * which uniqueness is computed. Result values are chosen from the first - * array in which the value occurs. The iteratee is invoked with one argument: - * (value). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {...Array} [arrays] The arrays to inspect. - * @param {Function} [iteratee=_.identity] The iteratee invoked per element. - * @returns {Array} Returns the new array of combined values. - * @example - * - * _.unionBy([2.1], [1.2, 2.3], Math.floor); - * // => [2.1, 1.2] - * - * // The `_.property` iteratee shorthand. - * _.unionBy([{ 'x': 1 }], [{ 'x': 2 }, { 'x': 1 }], 'x'); - * // => [{ 'x': 1 }, { 'x': 2 }] - */ - var unionBy = baseRest(function(arrays) { - var iteratee = last(arrays); - if (isArrayLikeObject(iteratee)) { - iteratee = undefined; - } - return baseUniq(baseFlatten(arrays, 1, isArrayLikeObject, true), getIteratee(iteratee, 2)); - }); - - /** - * This method is like `_.union` except that it accepts `comparator` which - * is invoked to compare elements of `arrays`. Result values are chosen from - * the first array in which the value occurs. The comparator is invoked - * with two arguments: (arrVal, othVal). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {...Array} [arrays] The arrays to inspect. - * @param {Function} [comparator] The comparator invoked per element. - * @returns {Array} Returns the new array of combined values. - * @example - * - * var objects = [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }]; - * var others = [{ 'x': 1, 'y': 1 }, { 'x': 1, 'y': 2 }]; - * - * _.unionWith(objects, others, _.isEqual); - * // => [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }, { 'x': 1, 'y': 1 }] - */ - var unionWith = baseRest(function(arrays) { - var comparator = last(arrays); - comparator = typeof comparator == 'function' ? comparator : undefined; - return baseUniq(baseFlatten(arrays, 1, isArrayLikeObject, true), undefined, comparator); - }); - - /** - * Creates a duplicate-free version of an array, using - * [`SameValueZero`](http://ecma-international.org/ecma-262/7.0/#sec-samevaluezero) - * for equality comparisons, in which only the first occurrence of each element - * is kept. The order of result values is determined by the order they occur - * in the array. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Array - * @param {Array} array The array to inspect. - * @returns {Array} Returns the new duplicate free array. - * @example - * - * _.uniq([2, 1, 2]); - * // => [2, 1] - */ - function uniq(array) { - return (array && array.length) ? baseUniq(array) : []; - } - - /** - * This method is like `_.uniq` except that it accepts `iteratee` which is - * invoked for each element in `array` to generate the criterion by which - * uniqueness is computed. The order of result values is determined by the - * order they occur in the array. The iteratee is invoked with one argument: - * (value). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The array to inspect. - * @param {Function} [iteratee=_.identity] The iteratee invoked per element. - * @returns {Array} Returns the new duplicate free array. - * @example - * - * _.uniqBy([2.1, 1.2, 2.3], Math.floor); - * // => [2.1, 1.2] - * - * // The `_.property` iteratee shorthand. - * _.uniqBy([{ 'x': 1 }, { 'x': 2 }, { 'x': 1 }], 'x'); - * // => [{ 'x': 1 }, { 'x': 2 }] - */ - function uniqBy(array, iteratee) { - return (array && array.length) ? baseUniq(array, getIteratee(iteratee, 2)) : []; - } - - /** - * This method is like `_.uniq` except that it accepts `comparator` which - * is invoked to compare elements of `array`. The order of result values is - * determined by the order they occur in the array.The comparator is invoked - * with two arguments: (arrVal, othVal). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {Array} array The array to inspect. - * @param {Function} [comparator] The comparator invoked per element. - * @returns {Array} Returns the new duplicate free array. - * @example - * - * var objects = [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }, { 'x': 1, 'y': 2 }]; - * - * _.uniqWith(objects, _.isEqual); - * // => [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }] - */ - function uniqWith(array, comparator) { - comparator = typeof comparator == 'function' ? comparator : undefined; - return (array && array.length) ? baseUniq(array, undefined, comparator) : []; - } - - /** - * This method is like `_.zip` except that it accepts an array of grouped - * elements and creates an array regrouping the elements to their pre-zip - * configuration. - * - * @static - * @memberOf _ - * @since 1.2.0 - * @category Array - * @param {Array} array The array of grouped elements to process. - * @returns {Array} Returns the new array of regrouped elements. - * @example - * - * var zipped = _.zip(['a', 'b'], [1, 2], [true, false]); - * // => [['a', 1, true], ['b', 2, false]] - * - * _.unzip(zipped); - * // => [['a', 'b'], [1, 2], [true, false]] - */ - function unzip(array) { - if (!(array && array.length)) { - return []; - } - var length = 0; - array = arrayFilter(array, function(group) { - if (isArrayLikeObject(group)) { - length = nativeMax(group.length, length); - return true; - } - }); - return baseTimes(length, function(index) { - return arrayMap(array, baseProperty(index)); - }); - } - - /** - * This method is like `_.unzip` except that it accepts `iteratee` to specify - * how regrouped values should be combined. The iteratee is invoked with the - * elements of each group: (...group). - * - * @static - * @memberOf _ - * @since 3.8.0 - * @category Array - * @param {Array} array The array of grouped elements to process. - * @param {Function} [iteratee=_.identity] The function to combine - * regrouped values. - * @returns {Array} Returns the new array of regrouped elements. - * @example - * - * var zipped = _.zip([1, 2], [10, 20], [100, 200]); - * // => [[1, 10, 100], [2, 20, 200]] - * - * _.unzipWith(zipped, _.add); - * // => [3, 30, 300] - */ - function unzipWith(array, iteratee) { - if (!(array && array.length)) { - return []; - } - var result = unzip(array); - if (iteratee == null) { - return result; - } - return arrayMap(result, function(group) { - return apply(iteratee, undefined, group); - }); - } - - /** - * Creates an array excluding all given values using - * [`SameValueZero`](http://ecma-international.org/ecma-262/7.0/#sec-samevaluezero) - * for equality comparisons. - * - * **Note:** Unlike `_.pull`, this method returns a new array. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Array - * @param {Array} array The array to inspect. - * @param {...*} [values] The values to exclude. - * @returns {Array} Returns the new array of filtered values. - * @see _.difference, _.xor - * @example - * - * _.without([2, 1, 2, 3], 1, 2); - * // => [3] - */ - var without = baseRest(function(array, values) { - return isArrayLikeObject(array) - ? baseDifference(array, values) - : []; - }); - - /** - * Creates an array of unique values that is the - * [symmetric difference](https://en.wikipedia.org/wiki/Symmetric_difference) - * of the given arrays. The order of result values is determined by the order - * they occur in the arrays. - * - * @static - * @memberOf _ - * @since 2.4.0 - * @category Array - * @param {...Array} [arrays] The arrays to inspect. - * @returns {Array} Returns the new array of filtered values. - * @see _.difference, _.without - * @example - * - * _.xor([2, 1], [2, 3]); - * // => [1, 3] - */ - var xor = baseRest(function(arrays) { - return baseXor(arrayFilter(arrays, isArrayLikeObject)); - }); - - /** - * This method is like `_.xor` except that it accepts `iteratee` which is - * invoked for each element of each `arrays` to generate the criterion by - * which by which they're compared. The order of result values is determined - * by the order they occur in the arrays. The iteratee is invoked with one - * argument: (value). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {...Array} [arrays] The arrays to inspect. - * @param {Function} [iteratee=_.identity] The iteratee invoked per element. - * @returns {Array} Returns the new array of filtered values. - * @example - * - * _.xorBy([2.1, 1.2], [2.3, 3.4], Math.floor); - * // => [1.2, 3.4] - * - * // The `_.property` iteratee shorthand. - * _.xorBy([{ 'x': 1 }], [{ 'x': 2 }, { 'x': 1 }], 'x'); - * // => [{ 'x': 2 }] - */ - var xorBy = baseRest(function(arrays) { - var iteratee = last(arrays); - if (isArrayLikeObject(iteratee)) { - iteratee = undefined; - } - return baseXor(arrayFilter(arrays, isArrayLikeObject), getIteratee(iteratee, 2)); - }); - - /** - * This method is like `_.xor` except that it accepts `comparator` which is - * invoked to compare elements of `arrays`. The order of result values is - * determined by the order they occur in the arrays. The comparator is invoked - * with two arguments: (arrVal, othVal). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Array - * @param {...Array} [arrays] The arrays to inspect. - * @param {Function} [comparator] The comparator invoked per element. - * @returns {Array} Returns the new array of filtered values. - * @example - * - * var objects = [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }]; - * var others = [{ 'x': 1, 'y': 1 }, { 'x': 1, 'y': 2 }]; - * - * _.xorWith(objects, others, _.isEqual); - * // => [{ 'x': 2, 'y': 1 }, { 'x': 1, 'y': 1 }] - */ - var xorWith = baseRest(function(arrays) { - var comparator = last(arrays); - comparator = typeof comparator == 'function' ? comparator : undefined; - return baseXor(arrayFilter(arrays, isArrayLikeObject), undefined, comparator); - }); - - /** - * Creates an array of grouped elements, the first of which contains the - * first elements of the given arrays, the second of which contains the - * second elements of the given arrays, and so on. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Array - * @param {...Array} [arrays] The arrays to process. - * @returns {Array} Returns the new array of grouped elements. - * @example - * - * _.zip(['a', 'b'], [1, 2], [true, false]); - * // => [['a', 1, true], ['b', 2, false]] - */ - var zip = baseRest(unzip); - - /** - * This method is like `_.fromPairs` except that it accepts two arrays, - * one of property identifiers and one of corresponding values. - * - * @static - * @memberOf _ - * @since 0.4.0 - * @category Array - * @param {Array} [props=[]] The property identifiers. - * @param {Array} [values=[]] The property values. - * @returns {Object} Returns the new object. - * @example - * - * _.zipObject(['a', 'b'], [1, 2]); - * // => { 'a': 1, 'b': 2 } - */ - function zipObject(props, values) { - return baseZipObject(props || [], values || [], assignValue); - } - - /** - * This method is like `_.zipObject` except that it supports property paths. - * - * @static - * @memberOf _ - * @since 4.1.0 - * @category Array - * @param {Array} [props=[]] The property identifiers. - * @param {Array} [values=[]] The property values. - * @returns {Object} Returns the new object. - * @example - * - * _.zipObjectDeep(['a.b[0].c', 'a.b[1].d'], [1, 2]); - * // => { 'a': { 'b': [{ 'c': 1 }, { 'd': 2 }] } } - */ - function zipObjectDeep(props, values) { - return baseZipObject(props || [], values || [], baseSet); - } - - /** - * This method is like `_.zip` except that it accepts `iteratee` to specify - * how grouped values should be combined. The iteratee is invoked with the - * elements of each group: (...group). - * - * @static - * @memberOf _ - * @since 3.8.0 - * @category Array - * @param {...Array} [arrays] The arrays to process. - * @param {Function} [iteratee=_.identity] The function to combine - * grouped values. - * @returns {Array} Returns the new array of grouped elements. - * @example - * - * _.zipWith([1, 2], [10, 20], [100, 200], function(a, b, c) { - * return a + b + c; - * }); - * // => [111, 222] - */ - var zipWith = baseRest(function(arrays) { - var length = arrays.length, - iteratee = length > 1 ? arrays[length - 1] : undefined; - - iteratee = typeof iteratee == 'function' ? (arrays.pop(), iteratee) : undefined; - return unzipWith(arrays, iteratee); - }); - - /*------------------------------------------------------------------------*/ - - /** - * Creates a `lodash` wrapper instance that wraps `value` with explicit method - * chain sequences enabled. The result of such sequences must be unwrapped - * with `_#value`. - * - * @static - * @memberOf _ - * @since 1.3.0 - * @category Seq - * @param {*} value The value to wrap. - * @returns {Object} Returns the new `lodash` wrapper instance. - * @example - * - * var users = [ - * { 'user': 'barney', 'age': 36 }, - * { 'user': 'fred', 'age': 40 }, - * { 'user': 'pebbles', 'age': 1 } - * ]; - * - * var youngest = _ - * .chain(users) - * .sortBy('age') - * .map(function(o) { - * return o.user + ' is ' + o.age; - * }) - * .head() - * .value(); - * // => 'pebbles is 1' - */ - function chain(value) { - var result = lodash(value); - result.__chain__ = true; - return result; - } - - /** - * This method invokes `interceptor` and returns `value`. The interceptor - * is invoked with one argument; (value). The purpose of this method is to - * "tap into" a method chain sequence in order to modify intermediate results. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Seq - * @param {*} value The value to provide to `interceptor`. - * @param {Function} interceptor The function to invoke. - * @returns {*} Returns `value`. - * @example - * - * _([1, 2, 3]) - * .tap(function(array) { - * // Mutate input array. - * array.pop(); - * }) - * .reverse() - * .value(); - * // => [2, 1] - */ - function tap(value, interceptor) { - interceptor(value); - return value; - } - - /** - * This method is like `_.tap` except that it returns the result of `interceptor`. - * The purpose of this method is to "pass thru" values replacing intermediate - * results in a method chain sequence. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Seq - * @param {*} value The value to provide to `interceptor`. - * @param {Function} interceptor The function to invoke. - * @returns {*} Returns the result of `interceptor`. - * @example - * - * _(' abc ') - * .chain() - * .trim() - * .thru(function(value) { - * return [value]; - * }) - * .value(); - * // => ['abc'] - */ - function thru(value, interceptor) { - return interceptor(value); - } - - /** - * This method is the wrapper version of `_.at`. - * - * @name at - * @memberOf _ - * @since 1.0.0 - * @category Seq - * @param {...(string|string[])} [paths] The property paths to pick. - * @returns {Object} Returns the new `lodash` wrapper instance. - * @example - * - * var object = { 'a': [{ 'b': { 'c': 3 } }, 4] }; - * - * _(object).at(['a[0].b.c', 'a[1]']).value(); - * // => [3, 4] - */ - var wrapperAt = flatRest(function(paths) { - var length = paths.length, - start = length ? paths[0] : 0, - value = this.__wrapped__, - interceptor = function(object) { return baseAt(object, paths); }; - - if (length > 1 || this.__actions__.length || - !(value instanceof LazyWrapper) || !isIndex(start)) { - return this.thru(interceptor); - } - value = value.slice(start, +start + (length ? 1 : 0)); - value.__actions__.push({ - 'func': thru, - 'args': [interceptor], - 'thisArg': undefined - }); - return new LodashWrapper(value, this.__chain__).thru(function(array) { - if (length && !array.length) { - array.push(undefined); - } - return array; - }); - }); - - /** - * Creates a `lodash` wrapper instance with explicit method chain sequences enabled. - * - * @name chain - * @memberOf _ - * @since 0.1.0 - * @category Seq - * @returns {Object} Returns the new `lodash` wrapper instance. - * @example - * - * var users = [ - * { 'user': 'barney', 'age': 36 }, - * { 'user': 'fred', 'age': 40 } - * ]; - * - * // A sequence without explicit chaining. - * _(users).head(); - * // => { 'user': 'barney', 'age': 36 } - * - * // A sequence with explicit chaining. - * _(users) - * .chain() - * .head() - * .pick('user') - * .value(); - * // => { 'user': 'barney' } - */ - function wrapperChain() { - return chain(this); - } - - /** - * Executes the chain sequence and returns the wrapped result. - * - * @name commit - * @memberOf _ - * @since 3.2.0 - * @category Seq - * @returns {Object} Returns the new `lodash` wrapper instance. - * @example - * - * var array = [1, 2]; - * var wrapped = _(array).push(3); - * - * console.log(array); - * // => [1, 2] - * - * wrapped = wrapped.commit(); - * console.log(array); - * // => [1, 2, 3] - * - * wrapped.last(); - * // => 3 - * - * console.log(array); - * // => [1, 2, 3] - */ - function wrapperCommit() { - return new LodashWrapper(this.value(), this.__chain__); - } - - /** - * Gets the next value on a wrapped object following the - * [iterator protocol](https://mdn.io/iteration_protocols#iterator). - * - * @name next - * @memberOf _ - * @since 4.0.0 - * @category Seq - * @returns {Object} Returns the next iterator value. - * @example - * - * var wrapped = _([1, 2]); - * - * wrapped.next(); - * // => { 'done': false, 'value': 1 } - * - * wrapped.next(); - * // => { 'done': false, 'value': 2 } - * - * wrapped.next(); - * // => { 'done': true, 'value': undefined } - */ - function wrapperNext() { - if (this.__values__ === undefined) { - this.__values__ = toArray(this.value()); - } - var done = this.__index__ >= this.__values__.length, - value = done ? undefined : this.__values__[this.__index__++]; - - return { 'done': done, 'value': value }; - } - - /** - * Enables the wrapper to be iterable. - * - * @name Symbol.iterator - * @memberOf _ - * @since 4.0.0 - * @category Seq - * @returns {Object} Returns the wrapper object. - * @example - * - * var wrapped = _([1, 2]); - * - * wrapped[Symbol.iterator]() === wrapped; - * // => true - * - * Array.from(wrapped); - * // => [1, 2] - */ - function wrapperToIterator() { - return this; - } - - /** - * Creates a clone of the chain sequence planting `value` as the wrapped value. - * - * @name plant - * @memberOf _ - * @since 3.2.0 - * @category Seq - * @param {*} value The value to plant. - * @returns {Object} Returns the new `lodash` wrapper instance. - * @example - * - * function square(n) { - * return n * n; - * } - * - * var wrapped = _([1, 2]).map(square); - * var other = wrapped.plant([3, 4]); - * - * other.value(); - * // => [9, 16] - * - * wrapped.value(); - * // => [1, 4] - */ - function wrapperPlant(value) { - var result, - parent = this; - - while (parent instanceof baseLodash) { - var clone = wrapperClone(parent); - clone.__index__ = 0; - clone.__values__ = undefined; - if (result) { - previous.__wrapped__ = clone; - } else { - result = clone; - } - var previous = clone; - parent = parent.__wrapped__; - } - previous.__wrapped__ = value; - return result; - } - - /** - * This method is the wrapper version of `_.reverse`. - * - * **Note:** This method mutates the wrapped array. - * - * @name reverse - * @memberOf _ - * @since 0.1.0 - * @category Seq - * @returns {Object} Returns the new `lodash` wrapper instance. - * @example - * - * var array = [1, 2, 3]; - * - * _(array).reverse().value() - * // => [3, 2, 1] - * - * console.log(array); - * // => [3, 2, 1] - */ - function wrapperReverse() { - var value = this.__wrapped__; - if (value instanceof LazyWrapper) { - var wrapped = value; - if (this.__actions__.length) { - wrapped = new LazyWrapper(this); - } - wrapped = wrapped.reverse(); - wrapped.__actions__.push({ - 'func': thru, - 'args': [reverse], - 'thisArg': undefined - }); - return new LodashWrapper(wrapped, this.__chain__); - } - return this.thru(reverse); - } - - /** - * Executes the chain sequence to resolve the unwrapped value. - * - * @name value - * @memberOf _ - * @since 0.1.0 - * @alias toJSON, valueOf - * @category Seq - * @returns {*} Returns the resolved unwrapped value. - * @example - * - * _([1, 2, 3]).value(); - * // => [1, 2, 3] - */ - function wrapperValue() { - return baseWrapperValue(this.__wrapped__, this.__actions__); - } - - /*------------------------------------------------------------------------*/ - - /** - * Creates an object composed of keys generated from the results of running - * each element of `collection` thru `iteratee`. The corresponding value of - * each key is the number of times the key was returned by `iteratee`. The - * iteratee is invoked with one argument: (value). - * - * @static - * @memberOf _ - * @since 0.5.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [iteratee=_.identity] The iteratee to transform keys. - * @returns {Object} Returns the composed aggregate object. - * @example - * - * _.countBy([6.1, 4.2, 6.3], Math.floor); - * // => { '4': 1, '6': 2 } - * - * // The `_.property` iteratee shorthand. - * _.countBy(['one', 'two', 'three'], 'length'); - * // => { '3': 2, '5': 1 } - */ - var countBy = createAggregator(function(result, value, key) { - if (hasOwnProperty.call(result, key)) { - ++result[key]; - } else { - baseAssignValue(result, key, 1); - } - }); - - /** - * Checks if `predicate` returns truthy for **all** elements of `collection`. - * Iteration is stopped once `predicate` returns falsey. The predicate is - * invoked with three arguments: (value, index|key, collection). - * - * **Note:** This method returns `true` for - * [empty collections](https://en.wikipedia.org/wiki/Empty_set) because - * [everything is true](https://en.wikipedia.org/wiki/Vacuous_truth) of - * elements of empty collections. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.map`. - * @returns {boolean} Returns `true` if all elements pass the predicate check, - * else `false`. - * @example - * - * _.every([true, 1, null, 'yes'], Boolean); - * // => false - * - * var users = [ - * { 'user': 'barney', 'age': 36, 'active': false }, - * { 'user': 'fred', 'age': 40, 'active': false } - * ]; - * - * // The `_.matches` iteratee shorthand. - * _.every(users, { 'user': 'barney', 'active': false }); - * // => false - * - * // The `_.matchesProperty` iteratee shorthand. - * _.every(users, ['active', false]); - * // => true - * - * // The `_.property` iteratee shorthand. - * _.every(users, 'active'); - * // => false - */ - function every(collection, predicate, guard) { - var func = isArray(collection) ? arrayEvery : baseEvery; - if (guard && isIterateeCall(collection, predicate, guard)) { - predicate = undefined; - } - return func(collection, getIteratee(predicate, 3)); - } - - /** - * Iterates over elements of `collection`, returning an array of all elements - * `predicate` returns truthy for. The predicate is invoked with three - * arguments: (value, index|key, collection). - * - * **Note:** Unlike `_.remove`, this method returns a new array. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @returns {Array} Returns the new filtered array. - * @see _.reject - * @example - * - * var users = [ - * { 'user': 'barney', 'age': 36, 'active': true }, - * { 'user': 'fred', 'age': 40, 'active': false } - * ]; - * - * _.filter(users, function(o) { return !o.active; }); - * // => objects for ['fred'] - * - * // The `_.matches` iteratee shorthand. - * _.filter(users, { 'age': 36, 'active': true }); - * // => objects for ['barney'] - * - * // The `_.matchesProperty` iteratee shorthand. - * _.filter(users, ['active', false]); - * // => objects for ['fred'] - * - * // The `_.property` iteratee shorthand. - * _.filter(users, 'active'); - * // => objects for ['barney'] - */ - function filter(collection, predicate) { - var func = isArray(collection) ? arrayFilter : baseFilter; - return func(collection, getIteratee(predicate, 3)); - } - - /** - * Iterates over elements of `collection`, returning the first element - * `predicate` returns truthy for. The predicate is invoked with three - * arguments: (value, index|key, collection). - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Collection - * @param {Array|Object} collection The collection to inspect. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @param {number} [fromIndex=0] The index to search from. - * @returns {*} Returns the matched element, else `undefined`. - * @example - * - * var users = [ - * { 'user': 'barney', 'age': 36, 'active': true }, - * { 'user': 'fred', 'age': 40, 'active': false }, - * { 'user': 'pebbles', 'age': 1, 'active': true } - * ]; - * - * _.find(users, function(o) { return o.age < 40; }); - * // => object for 'barney' - * - * // The `_.matches` iteratee shorthand. - * _.find(users, { 'age': 1, 'active': true }); - * // => object for 'pebbles' - * - * // The `_.matchesProperty` iteratee shorthand. - * _.find(users, ['active', false]); - * // => object for 'fred' - * - * // The `_.property` iteratee shorthand. - * _.find(users, 'active'); - * // => object for 'barney' - */ - var find = createFind(findIndex); - - /** - * This method is like `_.find` except that it iterates over elements of - * `collection` from right to left. - * - * @static - * @memberOf _ - * @since 2.0.0 - * @category Collection - * @param {Array|Object} collection The collection to inspect. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @param {number} [fromIndex=collection.length-1] The index to search from. - * @returns {*} Returns the matched element, else `undefined`. - * @example - * - * _.findLast([1, 2, 3, 4], function(n) { - * return n % 2 == 1; - * }); - * // => 3 - */ - var findLast = createFind(findLastIndex); - - /** - * Creates a flattened array of values by running each element in `collection` - * thru `iteratee` and flattening the mapped results. The iteratee is invoked - * with three arguments: (value, index|key, collection). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [iteratee=_.identity] The function invoked per iteration. - * @returns {Array} Returns the new flattened array. - * @example - * - * function duplicate(n) { - * return [n, n]; - * } - * - * _.flatMap([1, 2], duplicate); - * // => [1, 1, 2, 2] - */ - function flatMap(collection, iteratee) { - return baseFlatten(map(collection, iteratee), 1); - } - - /** - * This method is like `_.flatMap` except that it recursively flattens the - * mapped results. - * - * @static - * @memberOf _ - * @since 4.7.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [iteratee=_.identity] The function invoked per iteration. - * @returns {Array} Returns the new flattened array. - * @example - * - * function duplicate(n) { - * return [[[n, n]]]; - * } - * - * _.flatMapDeep([1, 2], duplicate); - * // => [1, 1, 2, 2] - */ - function flatMapDeep(collection, iteratee) { - return baseFlatten(map(collection, iteratee), INFINITY); - } - - /** - * This method is like `_.flatMap` except that it recursively flattens the - * mapped results up to `depth` times. - * - * @static - * @memberOf _ - * @since 4.7.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [iteratee=_.identity] The function invoked per iteration. - * @param {number} [depth=1] The maximum recursion depth. - * @returns {Array} Returns the new flattened array. - * @example - * - * function duplicate(n) { - * return [[[n, n]]]; - * } - * - * _.flatMapDepth([1, 2], duplicate, 2); - * // => [[1, 1], [2, 2]] - */ - function flatMapDepth(collection, iteratee, depth) { - depth = depth === undefined ? 1 : toInteger(depth); - return baseFlatten(map(collection, iteratee), depth); - } - - /** - * Iterates over elements of `collection` and invokes `iteratee` for each element. - * The iteratee is invoked with three arguments: (value, index|key, collection). - * Iteratee functions may exit iteration early by explicitly returning `false`. - * - * **Note:** As with other "Collections" methods, objects with a "length" - * property are iterated like arrays. To avoid this behavior use `_.forIn` - * or `_.forOwn` for object iteration. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @alias each - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [iteratee=_.identity] The function invoked per iteration. - * @returns {Array|Object} Returns `collection`. - * @see _.forEachRight - * @example - * - * _.forEach([1, 2], function(value) { - * console.log(value); - * }); - * // => Logs `1` then `2`. - * - * _.forEach({ 'a': 1, 'b': 2 }, function(value, key) { - * console.log(key); - * }); - * // => Logs 'a' then 'b' (iteration order is not guaranteed). - */ - function forEach(collection, iteratee) { - var func = isArray(collection) ? arrayEach : baseEach; - return func(collection, getIteratee(iteratee, 3)); - } - - /** - * This method is like `_.forEach` except that it iterates over elements of - * `collection` from right to left. - * - * @static - * @memberOf _ - * @since 2.0.0 - * @alias eachRight - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [iteratee=_.identity] The function invoked per iteration. - * @returns {Array|Object} Returns `collection`. - * @see _.forEach - * @example - * - * _.forEachRight([1, 2], function(value) { - * console.log(value); - * }); - * // => Logs `2` then `1`. - */ - function forEachRight(collection, iteratee) { - var func = isArray(collection) ? arrayEachRight : baseEachRight; - return func(collection, getIteratee(iteratee, 3)); - } - - /** - * Creates an object composed of keys generated from the results of running - * each element of `collection` thru `iteratee`. The order of grouped values - * is determined by the order they occur in `collection`. The corresponding - * value of each key is an array of elements responsible for generating the - * key. The iteratee is invoked with one argument: (value). - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [iteratee=_.identity] The iteratee to transform keys. - * @returns {Object} Returns the composed aggregate object. - * @example - * - * _.groupBy([6.1, 4.2, 6.3], Math.floor); - * // => { '4': [4.2], '6': [6.1, 6.3] } - * - * // The `_.property` iteratee shorthand. - * _.groupBy(['one', 'two', 'three'], 'length'); - * // => { '3': ['one', 'two'], '5': ['three'] } - */ - var groupBy = createAggregator(function(result, value, key) { - if (hasOwnProperty.call(result, key)) { - result[key].push(value); - } else { - baseAssignValue(result, key, [value]); - } - }); - - /** - * Checks if `value` is in `collection`. If `collection` is a string, it's - * checked for a substring of `value`, otherwise - * [`SameValueZero`](http://ecma-international.org/ecma-262/7.0/#sec-samevaluezero) - * is used for equality comparisons. If `fromIndex` is negative, it's used as - * the offset from the end of `collection`. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Collection - * @param {Array|Object|string} collection The collection to inspect. - * @param {*} value The value to search for. - * @param {number} [fromIndex=0] The index to search from. - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.reduce`. - * @returns {boolean} Returns `true` if `value` is found, else `false`. - * @example - * - * _.includes([1, 2, 3], 1); - * // => true - * - * _.includes([1, 2, 3], 1, 2); - * // => false - * - * _.includes({ 'a': 1, 'b': 2 }, 1); - * // => true - * - * _.includes('abcd', 'bc'); - * // => true - */ - function includes(collection, value, fromIndex, guard) { - collection = isArrayLike(collection) ? collection : values(collection); - fromIndex = (fromIndex && !guard) ? toInteger(fromIndex) : 0; - - var length = collection.length; - if (fromIndex < 0) { - fromIndex = nativeMax(length + fromIndex, 0); - } - return isString(collection) - ? (fromIndex <= length && collection.indexOf(value, fromIndex) > -1) - : (!!length && baseIndexOf(collection, value, fromIndex) > -1); - } - - /** - * Invokes the method at `path` of each element in `collection`, returning - * an array of the results of each invoked method. Any additional arguments - * are provided to each invoked method. If `path` is a function, it's invoked - * for, and `this` bound to, each element in `collection`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Array|Function|string} path The path of the method to invoke or - * the function invoked per iteration. - * @param {...*} [args] The arguments to invoke each method with. - * @returns {Array} Returns the array of results. - * @example - * - * _.invokeMap([[5, 1, 7], [3, 2, 1]], 'sort'); - * // => [[1, 5, 7], [1, 2, 3]] - * - * _.invokeMap([123, 456], String.prototype.split, ''); - * // => [['1', '2', '3'], ['4', '5', '6']] - */ - var invokeMap = baseRest(function(collection, path, args) { - var index = -1, - isFunc = typeof path == 'function', - result = isArrayLike(collection) ? Array(collection.length) : []; - - baseEach(collection, function(value) { - result[++index] = isFunc ? apply(path, value, args) : baseInvoke(value, path, args); - }); - return result; - }); - - /** - * Creates an object composed of keys generated from the results of running - * each element of `collection` thru `iteratee`. The corresponding value of - * each key is the last element responsible for generating the key. The - * iteratee is invoked with one argument: (value). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [iteratee=_.identity] The iteratee to transform keys. - * @returns {Object} Returns the composed aggregate object. - * @example - * - * var array = [ - * { 'dir': 'left', 'code': 97 }, - * { 'dir': 'right', 'code': 100 } - * ]; - * - * _.keyBy(array, function(o) { - * return String.fromCharCode(o.code); - * }); - * // => { 'a': { 'dir': 'left', 'code': 97 }, 'd': { 'dir': 'right', 'code': 100 } } - * - * _.keyBy(array, 'dir'); - * // => { 'left': { 'dir': 'left', 'code': 97 }, 'right': { 'dir': 'right', 'code': 100 } } - */ - var keyBy = createAggregator(function(result, value, key) { - baseAssignValue(result, key, value); - }); - - /** - * Creates an array of values by running each element in `collection` thru - * `iteratee`. The iteratee is invoked with three arguments: - * (value, index|key, collection). - * - * Many lodash methods are guarded to work as iteratees for methods like - * `_.every`, `_.filter`, `_.map`, `_.mapValues`, `_.reject`, and `_.some`. - * - * The guarded methods are: - * `ary`, `chunk`, `curry`, `curryRight`, `drop`, `dropRight`, `every`, - * `fill`, `invert`, `parseInt`, `random`, `range`, `rangeRight`, `repeat`, - * `sampleSize`, `slice`, `some`, `sortBy`, `split`, `take`, `takeRight`, - * `template`, `trim`, `trimEnd`, `trimStart`, and `words` - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [iteratee=_.identity] The function invoked per iteration. - * @returns {Array} Returns the new mapped array. - * @example - * - * function square(n) { - * return n * n; - * } - * - * _.map([4, 8], square); - * // => [16, 64] - * - * _.map({ 'a': 4, 'b': 8 }, square); - * // => [16, 64] (iteration order is not guaranteed) - * - * var users = [ - * { 'user': 'barney' }, - * { 'user': 'fred' } - * ]; - * - * // The `_.property` iteratee shorthand. - * _.map(users, 'user'); - * // => ['barney', 'fred'] - */ - function map(collection, iteratee) { - var func = isArray(collection) ? arrayMap : baseMap; - return func(collection, getIteratee(iteratee, 3)); - } - - /** - * This method is like `_.sortBy` except that it allows specifying the sort - * orders of the iteratees to sort by. If `orders` is unspecified, all values - * are sorted in ascending order. Otherwise, specify an order of "desc" for - * descending or "asc" for ascending sort order of corresponding values. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Array[]|Function[]|Object[]|string[]} [iteratees=[_.identity]] - * The iteratees to sort by. - * @param {string[]} [orders] The sort orders of `iteratees`. - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.reduce`. - * @returns {Array} Returns the new sorted array. - * @example - * - * var users = [ - * { 'user': 'fred', 'age': 48 }, - * { 'user': 'barney', 'age': 34 }, - * { 'user': 'fred', 'age': 40 }, - * { 'user': 'barney', 'age': 36 } - * ]; - * - * // Sort by `user` in ascending order and by `age` in descending order. - * _.orderBy(users, ['user', 'age'], ['asc', 'desc']); - * // => objects for [['barney', 36], ['barney', 34], ['fred', 48], ['fred', 40]] - */ - function orderBy(collection, iteratees, orders, guard) { - if (collection == null) { - return []; - } - if (!isArray(iteratees)) { - iteratees = iteratees == null ? [] : [iteratees]; - } - orders = guard ? undefined : orders; - if (!isArray(orders)) { - orders = orders == null ? [] : [orders]; - } - return baseOrderBy(collection, iteratees, orders); - } - - /** - * Creates an array of elements split into two groups, the first of which - * contains elements `predicate` returns truthy for, the second of which - * contains elements `predicate` returns falsey for. The predicate is - * invoked with one argument: (value). - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @returns {Array} Returns the array of grouped elements. - * @example - * - * var users = [ - * { 'user': 'barney', 'age': 36, 'active': false }, - * { 'user': 'fred', 'age': 40, 'active': true }, - * { 'user': 'pebbles', 'age': 1, 'active': false } - * ]; - * - * _.partition(users, function(o) { return o.active; }); - * // => objects for [['fred'], ['barney', 'pebbles']] - * - * // The `_.matches` iteratee shorthand. - * _.partition(users, { 'age': 1, 'active': false }); - * // => objects for [['pebbles'], ['barney', 'fred']] - * - * // The `_.matchesProperty` iteratee shorthand. - * _.partition(users, ['active', false]); - * // => objects for [['barney', 'pebbles'], ['fred']] - * - * // The `_.property` iteratee shorthand. - * _.partition(users, 'active'); - * // => objects for [['fred'], ['barney', 'pebbles']] - */ - var partition = createAggregator(function(result, value, key) { - result[key ? 0 : 1].push(value); - }, function() { return [[], []]; }); - - /** - * Reduces `collection` to a value which is the accumulated result of running - * each element in `collection` thru `iteratee`, where each successive - * invocation is supplied the return value of the previous. If `accumulator` - * is not given, the first element of `collection` is used as the initial - * value. The iteratee is invoked with four arguments: - * (accumulator, value, index|key, collection). - * - * Many lodash methods are guarded to work as iteratees for methods like - * `_.reduce`, `_.reduceRight`, and `_.transform`. - * - * The guarded methods are: - * `assign`, `defaults`, `defaultsDeep`, `includes`, `merge`, `orderBy`, - * and `sortBy` - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [iteratee=_.identity] The function invoked per iteration. - * @param {*} [accumulator] The initial value. - * @returns {*} Returns the accumulated value. - * @see _.reduceRight - * @example - * - * _.reduce([1, 2], function(sum, n) { - * return sum + n; - * }, 0); - * // => 3 - * - * _.reduce({ 'a': 1, 'b': 2, 'c': 1 }, function(result, value, key) { - * (result[value] || (result[value] = [])).push(key); - * return result; - * }, {}); - * // => { '1': ['a', 'c'], '2': ['b'] } (iteration order is not guaranteed) - */ - function reduce(collection, iteratee, accumulator) { - var func = isArray(collection) ? arrayReduce : baseReduce, - initAccum = arguments.length < 3; - - return func(collection, getIteratee(iteratee, 4), accumulator, initAccum, baseEach); - } - - /** - * This method is like `_.reduce` except that it iterates over elements of - * `collection` from right to left. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [iteratee=_.identity] The function invoked per iteration. - * @param {*} [accumulator] The initial value. - * @returns {*} Returns the accumulated value. - * @see _.reduce - * @example - * - * var array = [[0, 1], [2, 3], [4, 5]]; - * - * _.reduceRight(array, function(flattened, other) { - * return flattened.concat(other); - * }, []); - * // => [4, 5, 2, 3, 0, 1] - */ - function reduceRight(collection, iteratee, accumulator) { - var func = isArray(collection) ? arrayReduceRight : baseReduce, - initAccum = arguments.length < 3; - - return func(collection, getIteratee(iteratee, 4), accumulator, initAccum, baseEachRight); - } - - /** - * The opposite of `_.filter`; this method returns the elements of `collection` - * that `predicate` does **not** return truthy for. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @returns {Array} Returns the new filtered array. - * @see _.filter - * @example - * - * var users = [ - * { 'user': 'barney', 'age': 36, 'active': false }, - * { 'user': 'fred', 'age': 40, 'active': true } - * ]; - * - * _.reject(users, function(o) { return !o.active; }); - * // => objects for ['fred'] - * - * // The `_.matches` iteratee shorthand. - * _.reject(users, { 'age': 40, 'active': true }); - * // => objects for ['barney'] - * - * // The `_.matchesProperty` iteratee shorthand. - * _.reject(users, ['active', false]); - * // => objects for ['fred'] - * - * // The `_.property` iteratee shorthand. - * _.reject(users, 'active'); - * // => objects for ['barney'] - */ - function reject(collection, predicate) { - var func = isArray(collection) ? arrayFilter : baseFilter; - return func(collection, negate(getIteratee(predicate, 3))); - } - - /** - * Gets a random element from `collection`. - * - * @static - * @memberOf _ - * @since 2.0.0 - * @category Collection - * @param {Array|Object} collection The collection to sample. - * @returns {*} Returns the random element. - * @example - * - * _.sample([1, 2, 3, 4]); - * // => 2 - */ - function sample(collection) { - var func = isArray(collection) ? arraySample : baseSample; - return func(collection); - } - - /** - * Gets `n` random elements at unique keys from `collection` up to the - * size of `collection`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Collection - * @param {Array|Object} collection The collection to sample. - * @param {number} [n=1] The number of elements to sample. - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.map`. - * @returns {Array} Returns the random elements. - * @example - * - * _.sampleSize([1, 2, 3], 2); - * // => [3, 1] - * - * _.sampleSize([1, 2, 3], 4); - * // => [2, 3, 1] - */ - function sampleSize(collection, n, guard) { - if ((guard ? isIterateeCall(collection, n, guard) : n === undefined)) { - n = 1; - } else { - n = toInteger(n); - } - var func = isArray(collection) ? arraySampleSize : baseSampleSize; - return func(collection, n); - } - - /** - * Creates an array of shuffled values, using a version of the - * [Fisher-Yates shuffle](https://en.wikipedia.org/wiki/Fisher-Yates_shuffle). - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Collection - * @param {Array|Object} collection The collection to shuffle. - * @returns {Array} Returns the new shuffled array. - * @example - * - * _.shuffle([1, 2, 3, 4]); - * // => [4, 1, 3, 2] - */ - function shuffle(collection) { - var func = isArray(collection) ? arrayShuffle : baseShuffle; - return func(collection); - } - - /** - * Gets the size of `collection` by returning its length for array-like - * values or the number of own enumerable string keyed properties for objects. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Collection - * @param {Array|Object|string} collection The collection to inspect. - * @returns {number} Returns the collection size. - * @example - * - * _.size([1, 2, 3]); - * // => 3 - * - * _.size({ 'a': 1, 'b': 2 }); - * // => 2 - * - * _.size('pebbles'); - * // => 7 - */ - function size(collection) { - if (collection == null) { - return 0; - } - if (isArrayLike(collection)) { - return isString(collection) ? stringSize(collection) : collection.length; - } - var tag = getTag(collection); - if (tag == mapTag || tag == setTag) { - return collection.size; - } - return baseKeys(collection).length; - } - - /** - * Checks if `predicate` returns truthy for **any** element of `collection`. - * Iteration is stopped once `predicate` returns truthy. The predicate is - * invoked with three arguments: (value, index|key, collection). - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.map`. - * @returns {boolean} Returns `true` if any element passes the predicate check, - * else `false`. - * @example - * - * _.some([null, 0, 'yes', false], Boolean); - * // => true - * - * var users = [ - * { 'user': 'barney', 'active': true }, - * { 'user': 'fred', 'active': false } - * ]; - * - * // The `_.matches` iteratee shorthand. - * _.some(users, { 'user': 'barney', 'active': false }); - * // => false - * - * // The `_.matchesProperty` iteratee shorthand. - * _.some(users, ['active', false]); - * // => true - * - * // The `_.property` iteratee shorthand. - * _.some(users, 'active'); - * // => true - */ - function some(collection, predicate, guard) { - var func = isArray(collection) ? arraySome : baseSome; - if (guard && isIterateeCall(collection, predicate, guard)) { - predicate = undefined; - } - return func(collection, getIteratee(predicate, 3)); - } - - /** - * Creates an array of elements, sorted in ascending order by the results of - * running each element in a collection thru each iteratee. This method - * performs a stable sort, that is, it preserves the original sort order of - * equal elements. The iteratees are invoked with one argument: (value). - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Collection - * @param {Array|Object} collection The collection to iterate over. - * @param {...(Function|Function[])} [iteratees=[_.identity]] - * The iteratees to sort by. - * @returns {Array} Returns the new sorted array. - * @example - * - * var users = [ - * { 'user': 'fred', 'age': 48 }, - * { 'user': 'barney', 'age': 36 }, - * { 'user': 'fred', 'age': 40 }, - * { 'user': 'barney', 'age': 34 } - * ]; - * - * _.sortBy(users, [function(o) { return o.user; }]); - * // => objects for [['barney', 36], ['barney', 34], ['fred', 48], ['fred', 40]] - * - * _.sortBy(users, ['user', 'age']); - * // => objects for [['barney', 34], ['barney', 36], ['fred', 40], ['fred', 48]] - */ - var sortBy = baseRest(function(collection, iteratees) { - if (collection == null) { - return []; - } - var length = iteratees.length; - if (length > 1 && isIterateeCall(collection, iteratees[0], iteratees[1])) { - iteratees = []; - } else if (length > 2 && isIterateeCall(iteratees[0], iteratees[1], iteratees[2])) { - iteratees = [iteratees[0]]; - } - return baseOrderBy(collection, baseFlatten(iteratees, 1), []); - }); - - /*------------------------------------------------------------------------*/ - - /** - * Gets the timestamp of the number of milliseconds that have elapsed since - * the Unix epoch (1 January 1970 00:00:00 UTC). - * - * @static - * @memberOf _ - * @since 2.4.0 - * @category Date - * @returns {number} Returns the timestamp. - * @example - * - * _.defer(function(stamp) { - * console.log(_.now() - stamp); - * }, _.now()); - * // => Logs the number of milliseconds it took for the deferred invocation. - */ - var now = ctxNow || function() { - return root.Date.now(); - }; - - /*------------------------------------------------------------------------*/ - - /** - * The opposite of `_.before`; this method creates a function that invokes - * `func` once it's called `n` or more times. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Function - * @param {number} n The number of calls before `func` is invoked. - * @param {Function} func The function to restrict. - * @returns {Function} Returns the new restricted function. - * @example - * - * var saves = ['profile', 'settings']; - * - * var done = _.after(saves.length, function() { - * console.log('done saving!'); - * }); - * - * _.forEach(saves, function(type) { - * asyncSave({ 'type': type, 'complete': done }); - * }); - * // => Logs 'done saving!' after the two async saves have completed. - */ - function after(n, func) { - if (typeof func != 'function') { - throw new TypeError(FUNC_ERROR_TEXT); - } - n = toInteger(n); - return function() { - if (--n < 1) { - return func.apply(this, arguments); - } - }; - } - - /** - * Creates a function that invokes `func`, with up to `n` arguments, - * ignoring any additional arguments. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Function - * @param {Function} func The function to cap arguments for. - * @param {number} [n=func.length] The arity cap. - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.map`. - * @returns {Function} Returns the new capped function. - * @example - * - * _.map(['6', '8', '10'], _.ary(parseInt, 1)); - * // => [6, 8, 10] - */ - function ary(func, n, guard) { - n = guard ? undefined : n; - n = (func && n == null) ? func.length : n; - return createWrap(func, WRAP_ARY_FLAG, undefined, undefined, undefined, undefined, n); - } - - /** - * Creates a function that invokes `func`, with the `this` binding and arguments - * of the created function, while it's called less than `n` times. Subsequent - * calls to the created function return the result of the last `func` invocation. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Function - * @param {number} n The number of calls at which `func` is no longer invoked. - * @param {Function} func The function to restrict. - * @returns {Function} Returns the new restricted function. - * @example - * - * jQuery(element).on('click', _.before(5, addContactToList)); - * // => Allows adding up to 4 contacts to the list. - */ - function before(n, func) { - var result; - if (typeof func != 'function') { - throw new TypeError(FUNC_ERROR_TEXT); - } - n = toInteger(n); - return function() { - if (--n > 0) { - result = func.apply(this, arguments); - } - if (n <= 1) { - func = undefined; - } - return result; - }; - } - - /** - * Creates a function that invokes `func` with the `this` binding of `thisArg` - * and `partials` prepended to the arguments it receives. - * - * The `_.bind.placeholder` value, which defaults to `_` in monolithic builds, - * may be used as a placeholder for partially applied arguments. - * - * **Note:** Unlike native `Function#bind`, this method doesn't set the "length" - * property of bound functions. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Function - * @param {Function} func The function to bind. - * @param {*} thisArg The `this` binding of `func`. - * @param {...*} [partials] The arguments to be partially applied. - * @returns {Function} Returns the new bound function. - * @example - * - * function greet(greeting, punctuation) { - * return greeting + ' ' + this.user + punctuation; - * } - * - * var object = { 'user': 'fred' }; - * - * var bound = _.bind(greet, object, 'hi'); - * bound('!'); - * // => 'hi fred!' - * - * // Bound with placeholders. - * var bound = _.bind(greet, object, _, '!'); - * bound('hi'); - * // => 'hi fred!' - */ - var bind = baseRest(function(func, thisArg, partials) { - var bitmask = WRAP_BIND_FLAG; - if (partials.length) { - var holders = replaceHolders(partials, getHolder(bind)); - bitmask |= WRAP_PARTIAL_FLAG; - } - return createWrap(func, bitmask, thisArg, partials, holders); - }); - - /** - * Creates a function that invokes the method at `object[key]` with `partials` - * prepended to the arguments it receives. - * - * This method differs from `_.bind` by allowing bound functions to reference - * methods that may be redefined or don't yet exist. See - * [Peter Michaux's article](http://peter.michaux.ca/articles/lazy-function-definition-pattern) - * for more details. - * - * The `_.bindKey.placeholder` value, which defaults to `_` in monolithic - * builds, may be used as a placeholder for partially applied arguments. - * - * @static - * @memberOf _ - * @since 0.10.0 - * @category Function - * @param {Object} object The object to invoke the method on. - * @param {string} key The key of the method. - * @param {...*} [partials] The arguments to be partially applied. - * @returns {Function} Returns the new bound function. - * @example - * - * var object = { - * 'user': 'fred', - * 'greet': function(greeting, punctuation) { - * return greeting + ' ' + this.user + punctuation; - * } - * }; - * - * var bound = _.bindKey(object, 'greet', 'hi'); - * bound('!'); - * // => 'hi fred!' - * - * object.greet = function(greeting, punctuation) { - * return greeting + 'ya ' + this.user + punctuation; - * }; - * - * bound('!'); - * // => 'hiya fred!' - * - * // Bound with placeholders. - * var bound = _.bindKey(object, 'greet', _, '!'); - * bound('hi'); - * // => 'hiya fred!' - */ - var bindKey = baseRest(function(object, key, partials) { - var bitmask = WRAP_BIND_FLAG | WRAP_BIND_KEY_FLAG; - if (partials.length) { - var holders = replaceHolders(partials, getHolder(bindKey)); - bitmask |= WRAP_PARTIAL_FLAG; - } - return createWrap(key, bitmask, object, partials, holders); - }); - - /** - * Creates a function that accepts arguments of `func` and either invokes - * `func` returning its result, if at least `arity` number of arguments have - * been provided, or returns a function that accepts the remaining `func` - * arguments, and so on. The arity of `func` may be specified if `func.length` - * is not sufficient. - * - * The `_.curry.placeholder` value, which defaults to `_` in monolithic builds, - * may be used as a placeholder for provided arguments. - * - * **Note:** This method doesn't set the "length" property of curried functions. - * - * @static - * @memberOf _ - * @since 2.0.0 - * @category Function - * @param {Function} func The function to curry. - * @param {number} [arity=func.length] The arity of `func`. - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.map`. - * @returns {Function} Returns the new curried function. - * @example - * - * var abc = function(a, b, c) { - * return [a, b, c]; - * }; - * - * var curried = _.curry(abc); - * - * curried(1)(2)(3); - * // => [1, 2, 3] - * - * curried(1, 2)(3); - * // => [1, 2, 3] - * - * curried(1, 2, 3); - * // => [1, 2, 3] - * - * // Curried with placeholders. - * curried(1)(_, 3)(2); - * // => [1, 2, 3] - */ - function curry(func, arity, guard) { - arity = guard ? undefined : arity; - var result = createWrap(func, WRAP_CURRY_FLAG, undefined, undefined, undefined, undefined, undefined, arity); - result.placeholder = curry.placeholder; - return result; - } - - /** - * This method is like `_.curry` except that arguments are applied to `func` - * in the manner of `_.partialRight` instead of `_.partial`. - * - * The `_.curryRight.placeholder` value, which defaults to `_` in monolithic - * builds, may be used as a placeholder for provided arguments. - * - * **Note:** This method doesn't set the "length" property of curried functions. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Function - * @param {Function} func The function to curry. - * @param {number} [arity=func.length] The arity of `func`. - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.map`. - * @returns {Function} Returns the new curried function. - * @example - * - * var abc = function(a, b, c) { - * return [a, b, c]; - * }; - * - * var curried = _.curryRight(abc); - * - * curried(3)(2)(1); - * // => [1, 2, 3] - * - * curried(2, 3)(1); - * // => [1, 2, 3] - * - * curried(1, 2, 3); - * // => [1, 2, 3] - * - * // Curried with placeholders. - * curried(3)(1, _)(2); - * // => [1, 2, 3] - */ - function curryRight(func, arity, guard) { - arity = guard ? undefined : arity; - var result = createWrap(func, WRAP_CURRY_RIGHT_FLAG, undefined, undefined, undefined, undefined, undefined, arity); - result.placeholder = curryRight.placeholder; - return result; - } - - /** - * Creates a debounced function that delays invoking `func` until after `wait` - * milliseconds have elapsed since the last time the debounced function was - * invoked. The debounced function comes with a `cancel` method to cancel - * delayed `func` invocations and a `flush` method to immediately invoke them. - * Provide `options` to indicate whether `func` should be invoked on the - * leading and/or trailing edge of the `wait` timeout. The `func` is invoked - * with the last arguments provided to the debounced function. Subsequent - * calls to the debounced function return the result of the last `func` - * invocation. - * - * **Note:** If `leading` and `trailing` options are `true`, `func` is - * invoked on the trailing edge of the timeout only if the debounced function - * is invoked more than once during the `wait` timeout. - * - * If `wait` is `0` and `leading` is `false`, `func` invocation is deferred - * until to the next tick, similar to `setTimeout` with a timeout of `0`. - * - * See [David Corbacho's article](https://css-tricks.com/debouncing-throttling-explained-examples/) - * for details over the differences between `_.debounce` and `_.throttle`. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Function - * @param {Function} func The function to debounce. - * @param {number} [wait=0] The number of milliseconds to delay. - * @param {Object} [options={}] The options object. - * @param {boolean} [options.leading=false] - * Specify invoking on the leading edge of the timeout. - * @param {number} [options.maxWait] - * The maximum time `func` is allowed to be delayed before it's invoked. - * @param {boolean} [options.trailing=true] - * Specify invoking on the trailing edge of the timeout. - * @returns {Function} Returns the new debounced function. - * @example - * - * // Avoid costly calculations while the window size is in flux. - * jQuery(window).on('resize', _.debounce(calculateLayout, 150)); - * - * // Invoke `sendMail` when clicked, debouncing subsequent calls. - * jQuery(element).on('click', _.debounce(sendMail, 300, { - * 'leading': true, - * 'trailing': false - * })); - * - * // Ensure `batchLog` is invoked once after 1 second of debounced calls. - * var debounced = _.debounce(batchLog, 250, { 'maxWait': 1000 }); - * var source = new EventSource('/stream'); - * jQuery(source).on('message', debounced); - * - * // Cancel the trailing debounced invocation. - * jQuery(window).on('popstate', debounced.cancel); - */ - function debounce(func, wait, options) { - var lastArgs, - lastThis, - maxWait, - result, - timerId, - lastCallTime, - lastInvokeTime = 0, - leading = false, - maxing = false, - trailing = true; - - if (typeof func != 'function') { - throw new TypeError(FUNC_ERROR_TEXT); - } - wait = toNumber(wait) || 0; - if (isObject(options)) { - leading = !!options.leading; - maxing = 'maxWait' in options; - maxWait = maxing ? nativeMax(toNumber(options.maxWait) || 0, wait) : maxWait; - trailing = 'trailing' in options ? !!options.trailing : trailing; - } - - function invokeFunc(time) { - var args = lastArgs, - thisArg = lastThis; - - lastArgs = lastThis = undefined; - lastInvokeTime = time; - result = func.apply(thisArg, args); - return result; - } - - function leadingEdge(time) { - // Reset any `maxWait` timer. - lastInvokeTime = time; - // Start the timer for the trailing edge. - timerId = setTimeout(timerExpired, wait); - // Invoke the leading edge. - return leading ? invokeFunc(time) : result; - } - - function remainingWait(time) { - var timeSinceLastCall = time - lastCallTime, - timeSinceLastInvoke = time - lastInvokeTime, - timeWaiting = wait - timeSinceLastCall; - - return maxing - ? nativeMin(timeWaiting, maxWait - timeSinceLastInvoke) - : timeWaiting; - } - - function shouldInvoke(time) { - var timeSinceLastCall = time - lastCallTime, - timeSinceLastInvoke = time - lastInvokeTime; - - // Either this is the first call, activity has stopped and we're at the - // trailing edge, the system time has gone backwards and we're treating - // it as the trailing edge, or we've hit the `maxWait` limit. - return (lastCallTime === undefined || (timeSinceLastCall >= wait) || - (timeSinceLastCall < 0) || (maxing && timeSinceLastInvoke >= maxWait)); - } - - function timerExpired() { - var time = now(); - if (shouldInvoke(time)) { - return trailingEdge(time); - } - // Restart the timer. - timerId = setTimeout(timerExpired, remainingWait(time)); - } - - function trailingEdge(time) { - timerId = undefined; - - // Only invoke if we have `lastArgs` which means `func` has been - // debounced at least once. - if (trailing && lastArgs) { - return invokeFunc(time); - } - lastArgs = lastThis = undefined; - return result; - } - - function cancel() { - if (timerId !== undefined) { - clearTimeout(timerId); - } - lastInvokeTime = 0; - lastArgs = lastCallTime = lastThis = timerId = undefined; - } - - function flush() { - return timerId === undefined ? result : trailingEdge(now()); - } - - function debounced() { - var time = now(), - isInvoking = shouldInvoke(time); - - lastArgs = arguments; - lastThis = this; - lastCallTime = time; - - if (isInvoking) { - if (timerId === undefined) { - return leadingEdge(lastCallTime); - } - if (maxing) { - // Handle invocations in a tight loop. - clearTimeout(timerId); - timerId = setTimeout(timerExpired, wait); - return invokeFunc(lastCallTime); - } - } - if (timerId === undefined) { - timerId = setTimeout(timerExpired, wait); - } - return result; - } - debounced.cancel = cancel; - debounced.flush = flush; - return debounced; - } - - /** - * Defers invoking the `func` until the current call stack has cleared. Any - * additional arguments are provided to `func` when it's invoked. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Function - * @param {Function} func The function to defer. - * @param {...*} [args] The arguments to invoke `func` with. - * @returns {number} Returns the timer id. - * @example - * - * _.defer(function(text) { - * console.log(text); - * }, 'deferred'); - * // => Logs 'deferred' after one millisecond. - */ - var defer = baseRest(function(func, args) { - return baseDelay(func, 1, args); - }); - - /** - * Invokes `func` after `wait` milliseconds. Any additional arguments are - * provided to `func` when it's invoked. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Function - * @param {Function} func The function to delay. - * @param {number} wait The number of milliseconds to delay invocation. - * @param {...*} [args] The arguments to invoke `func` with. - * @returns {number} Returns the timer id. - * @example - * - * _.delay(function(text) { - * console.log(text); - * }, 1000, 'later'); - * // => Logs 'later' after one second. - */ - var delay = baseRest(function(func, wait, args) { - return baseDelay(func, toNumber(wait) || 0, args); - }); - - /** - * Creates a function that invokes `func` with arguments reversed. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Function - * @param {Function} func The function to flip arguments for. - * @returns {Function} Returns the new flipped function. - * @example - * - * var flipped = _.flip(function() { - * return _.toArray(arguments); - * }); - * - * flipped('a', 'b', 'c', 'd'); - * // => ['d', 'c', 'b', 'a'] - */ - function flip(func) { - return createWrap(func, WRAP_FLIP_FLAG); - } - - /** - * Creates a function that memoizes the result of `func`. If `resolver` is - * provided, it determines the cache key for storing the result based on the - * arguments provided to the memoized function. By default, the first argument - * provided to the memoized function is used as the map cache key. The `func` - * is invoked with the `this` binding of the memoized function. - * - * **Note:** The cache is exposed as the `cache` property on the memoized - * function. Its creation may be customized by replacing the `_.memoize.Cache` - * constructor with one whose instances implement the - * [`Map`](http://ecma-international.org/ecma-262/7.0/#sec-properties-of-the-map-prototype-object) - * method interface of `clear`, `delete`, `get`, `has`, and `set`. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Function - * @param {Function} func The function to have its output memoized. - * @param {Function} [resolver] The function to resolve the cache key. - * @returns {Function} Returns the new memoized function. - * @example - * - * var object = { 'a': 1, 'b': 2 }; - * var other = { 'c': 3, 'd': 4 }; - * - * var values = _.memoize(_.values); - * values(object); - * // => [1, 2] - * - * values(other); - * // => [3, 4] - * - * object.a = 2; - * values(object); - * // => [1, 2] - * - * // Modify the result cache. - * values.cache.set(object, ['a', 'b']); - * values(object); - * // => ['a', 'b'] - * - * // Replace `_.memoize.Cache`. - * _.memoize.Cache = WeakMap; - */ - function memoize(func, resolver) { - if (typeof func != 'function' || (resolver != null && typeof resolver != 'function')) { - throw new TypeError(FUNC_ERROR_TEXT); - } - var memoized = function() { - var args = arguments, - key = resolver ? resolver.apply(this, args) : args[0], - cache = memoized.cache; - - if (cache.has(key)) { - return cache.get(key); - } - var result = func.apply(this, args); - memoized.cache = cache.set(key, result) || cache; - return result; - }; - memoized.cache = new (memoize.Cache || MapCache); - return memoized; - } - - // Expose `MapCache`. - memoize.Cache = MapCache; - - /** - * Creates a function that negates the result of the predicate `func`. The - * `func` predicate is invoked with the `this` binding and arguments of the - * created function. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Function - * @param {Function} predicate The predicate to negate. - * @returns {Function} Returns the new negated function. - * @example - * - * function isEven(n) { - * return n % 2 == 0; - * } - * - * _.filter([1, 2, 3, 4, 5, 6], _.negate(isEven)); - * // => [1, 3, 5] - */ - function negate(predicate) { - if (typeof predicate != 'function') { - throw new TypeError(FUNC_ERROR_TEXT); - } - return function() { - var args = arguments; - switch (args.length) { - case 0: return !predicate.call(this); - case 1: return !predicate.call(this, args[0]); - case 2: return !predicate.call(this, args[0], args[1]); - case 3: return !predicate.call(this, args[0], args[1], args[2]); - } - return !predicate.apply(this, args); - }; - } - - /** - * Creates a function that is restricted to invoking `func` once. Repeat calls - * to the function return the value of the first invocation. The `func` is - * invoked with the `this` binding and arguments of the created function. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Function - * @param {Function} func The function to restrict. - * @returns {Function} Returns the new restricted function. - * @example - * - * var initialize = _.once(createApplication); - * initialize(); - * initialize(); - * // => `createApplication` is invoked once - */ - function once(func) { - return before(2, func); - } - - /** - * Creates a function that invokes `func` with its arguments transformed. - * - * @static - * @since 4.0.0 - * @memberOf _ - * @category Function - * @param {Function} func The function to wrap. - * @param {...(Function|Function[])} [transforms=[_.identity]] - * The argument transforms. - * @returns {Function} Returns the new function. - * @example - * - * function doubled(n) { - * return n * 2; - * } - * - * function square(n) { - * return n * n; - * } - * - * var func = _.overArgs(function(x, y) { - * return [x, y]; - * }, [square, doubled]); - * - * func(9, 3); - * // => [81, 6] - * - * func(10, 5); - * // => [100, 10] - */ - var overArgs = castRest(function(func, transforms) { - transforms = (transforms.length == 1 && isArray(transforms[0])) - ? arrayMap(transforms[0], baseUnary(getIteratee())) - : arrayMap(baseFlatten(transforms, 1), baseUnary(getIteratee())); - - var funcsLength = transforms.length; - return baseRest(function(args) { - var index = -1, - length = nativeMin(args.length, funcsLength); - - while (++index < length) { - args[index] = transforms[index].call(this, args[index]); - } - return apply(func, this, args); - }); - }); - - /** - * Creates a function that invokes `func` with `partials` prepended to the - * arguments it receives. This method is like `_.bind` except it does **not** - * alter the `this` binding. - * - * The `_.partial.placeholder` value, which defaults to `_` in monolithic - * builds, may be used as a placeholder for partially applied arguments. - * - * **Note:** This method doesn't set the "length" property of partially - * applied functions. - * - * @static - * @memberOf _ - * @since 0.2.0 - * @category Function - * @param {Function} func The function to partially apply arguments to. - * @param {...*} [partials] The arguments to be partially applied. - * @returns {Function} Returns the new partially applied function. - * @example - * - * function greet(greeting, name) { - * return greeting + ' ' + name; - * } - * - * var sayHelloTo = _.partial(greet, 'hello'); - * sayHelloTo('fred'); - * // => 'hello fred' - * - * // Partially applied with placeholders. - * var greetFred = _.partial(greet, _, 'fred'); - * greetFred('hi'); - * // => 'hi fred' - */ - var partial = baseRest(function(func, partials) { - var holders = replaceHolders(partials, getHolder(partial)); - return createWrap(func, WRAP_PARTIAL_FLAG, undefined, partials, holders); - }); - - /** - * This method is like `_.partial` except that partially applied arguments - * are appended to the arguments it receives. - * - * The `_.partialRight.placeholder` value, which defaults to `_` in monolithic - * builds, may be used as a placeholder for partially applied arguments. - * - * **Note:** This method doesn't set the "length" property of partially - * applied functions. - * - * @static - * @memberOf _ - * @since 1.0.0 - * @category Function - * @param {Function} func The function to partially apply arguments to. - * @param {...*} [partials] The arguments to be partially applied. - * @returns {Function} Returns the new partially applied function. - * @example - * - * function greet(greeting, name) { - * return greeting + ' ' + name; - * } - * - * var greetFred = _.partialRight(greet, 'fred'); - * greetFred('hi'); - * // => 'hi fred' - * - * // Partially applied with placeholders. - * var sayHelloTo = _.partialRight(greet, 'hello', _); - * sayHelloTo('fred'); - * // => 'hello fred' - */ - var partialRight = baseRest(function(func, partials) { - var holders = replaceHolders(partials, getHolder(partialRight)); - return createWrap(func, WRAP_PARTIAL_RIGHT_FLAG, undefined, partials, holders); - }); - - /** - * Creates a function that invokes `func` with arguments arranged according - * to the specified `indexes` where the argument value at the first index is - * provided as the first argument, the argument value at the second index is - * provided as the second argument, and so on. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Function - * @param {Function} func The function to rearrange arguments for. - * @param {...(number|number[])} indexes The arranged argument indexes. - * @returns {Function} Returns the new function. - * @example - * - * var rearged = _.rearg(function(a, b, c) { - * return [a, b, c]; - * }, [2, 0, 1]); - * - * rearged('b', 'c', 'a') - * // => ['a', 'b', 'c'] - */ - var rearg = flatRest(function(func, indexes) { - return createWrap(func, WRAP_REARG_FLAG, undefined, undefined, undefined, indexes); - }); - - /** - * Creates a function that invokes `func` with the `this` binding of the - * created function and arguments from `start` and beyond provided as - * an array. - * - * **Note:** This method is based on the - * [rest parameter](https://mdn.io/rest_parameters). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Function - * @param {Function} func The function to apply a rest parameter to. - * @param {number} [start=func.length-1] The start position of the rest parameter. - * @returns {Function} Returns the new function. - * @example - * - * var say = _.rest(function(what, names) { - * return what + ' ' + _.initial(names).join(', ') + - * (_.size(names) > 1 ? ', & ' : '') + _.last(names); - * }); - * - * say('hello', 'fred', 'barney', 'pebbles'); - * // => 'hello fred, barney, & pebbles' - */ - function rest(func, start) { - if (typeof func != 'function') { - throw new TypeError(FUNC_ERROR_TEXT); - } - start = start === undefined ? start : toInteger(start); - return baseRest(func, start); - } - - /** - * Creates a function that invokes `func` with the `this` binding of the - * create function and an array of arguments much like - * [`Function#apply`](http://www.ecma-international.org/ecma-262/7.0/#sec-function.prototype.apply). - * - * **Note:** This method is based on the - * [spread operator](https://mdn.io/spread_operator). - * - * @static - * @memberOf _ - * @since 3.2.0 - * @category Function - * @param {Function} func The function to spread arguments over. - * @param {number} [start=0] The start position of the spread. - * @returns {Function} Returns the new function. - * @example - * - * var say = _.spread(function(who, what) { - * return who + ' says ' + what; - * }); - * - * say(['fred', 'hello']); - * // => 'fred says hello' - * - * var numbers = Promise.all([ - * Promise.resolve(40), - * Promise.resolve(36) - * ]); - * - * numbers.then(_.spread(function(x, y) { - * return x + y; - * })); - * // => a Promise of 76 - */ - function spread(func, start) { - if (typeof func != 'function') { - throw new TypeError(FUNC_ERROR_TEXT); - } - start = start == null ? 0 : nativeMax(toInteger(start), 0); - return baseRest(function(args) { - var array = args[start], - otherArgs = castSlice(args, 0, start); - - if (array) { - arrayPush(otherArgs, array); - } - return apply(func, this, otherArgs); - }); - } - - /** - * Creates a throttled function that only invokes `func` at most once per - * every `wait` milliseconds. The throttled function comes with a `cancel` - * method to cancel delayed `func` invocations and a `flush` method to - * immediately invoke them. Provide `options` to indicate whether `func` - * should be invoked on the leading and/or trailing edge of the `wait` - * timeout. The `func` is invoked with the last arguments provided to the - * throttled function. Subsequent calls to the throttled function return the - * result of the last `func` invocation. - * - * **Note:** If `leading` and `trailing` options are `true`, `func` is - * invoked on the trailing edge of the timeout only if the throttled function - * is invoked more than once during the `wait` timeout. - * - * If `wait` is `0` and `leading` is `false`, `func` invocation is deferred - * until to the next tick, similar to `setTimeout` with a timeout of `0`. - * - * See [David Corbacho's article](https://css-tricks.com/debouncing-throttling-explained-examples/) - * for details over the differences between `_.throttle` and `_.debounce`. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Function - * @param {Function} func The function to throttle. - * @param {number} [wait=0] The number of milliseconds to throttle invocations to. - * @param {Object} [options={}] The options object. - * @param {boolean} [options.leading=true] - * Specify invoking on the leading edge of the timeout. - * @param {boolean} [options.trailing=true] - * Specify invoking on the trailing edge of the timeout. - * @returns {Function} Returns the new throttled function. - * @example - * - * // Avoid excessively updating the position while scrolling. - * jQuery(window).on('scroll', _.throttle(updatePosition, 100)); - * - * // Invoke `renewToken` when the click event is fired, but not more than once every 5 minutes. - * var throttled = _.throttle(renewToken, 300000, { 'trailing': false }); - * jQuery(element).on('click', throttled); - * - * // Cancel the trailing throttled invocation. - * jQuery(window).on('popstate', throttled.cancel); - */ - function throttle(func, wait, options) { - var leading = true, - trailing = true; - - if (typeof func != 'function') { - throw new TypeError(FUNC_ERROR_TEXT); - } - if (isObject(options)) { - leading = 'leading' in options ? !!options.leading : leading; - trailing = 'trailing' in options ? !!options.trailing : trailing; - } - return debounce(func, wait, { - 'leading': leading, - 'maxWait': wait, - 'trailing': trailing - }); - } - - /** - * Creates a function that accepts up to one argument, ignoring any - * additional arguments. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Function - * @param {Function} func The function to cap arguments for. - * @returns {Function} Returns the new capped function. - * @example - * - * _.map(['6', '8', '10'], _.unary(parseInt)); - * // => [6, 8, 10] - */ - function unary(func) { - return ary(func, 1); - } - - /** - * Creates a function that provides `value` to `wrapper` as its first - * argument. Any additional arguments provided to the function are appended - * to those provided to the `wrapper`. The wrapper is invoked with the `this` - * binding of the created function. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Function - * @param {*} value The value to wrap. - * @param {Function} [wrapper=identity] The wrapper function. - * @returns {Function} Returns the new function. - * @example - * - * var p = _.wrap(_.escape, function(func, text) { - * return '

' + func(text) + '

'; - * }); - * - * p('fred, barney, & pebbles'); - * // => '

fred, barney, & pebbles

' - */ - function wrap(value, wrapper) { - return partial(castFunction(wrapper), value); - } - - /*------------------------------------------------------------------------*/ - - /** - * Casts `value` as an array if it's not one. - * - * @static - * @memberOf _ - * @since 4.4.0 - * @category Lang - * @param {*} value The value to inspect. - * @returns {Array} Returns the cast array. - * @example - * - * _.castArray(1); - * // => [1] - * - * _.castArray({ 'a': 1 }); - * // => [{ 'a': 1 }] - * - * _.castArray('abc'); - * // => ['abc'] - * - * _.castArray(null); - * // => [null] - * - * _.castArray(undefined); - * // => [undefined] - * - * _.castArray(); - * // => [] - * - * var array = [1, 2, 3]; - * console.log(_.castArray(array) === array); - * // => true - */ - function castArray() { - if (!arguments.length) { - return []; - } - var value = arguments[0]; - return isArray(value) ? value : [value]; - } - - /** - * Creates a shallow clone of `value`. - * - * **Note:** This method is loosely based on the - * [structured clone algorithm](https://mdn.io/Structured_clone_algorithm) - * and supports cloning arrays, array buffers, booleans, date objects, maps, - * numbers, `Object` objects, regexes, sets, strings, symbols, and typed - * arrays. The own enumerable properties of `arguments` objects are cloned - * as plain objects. An empty object is returned for uncloneable values such - * as error objects, functions, DOM nodes, and WeakMaps. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Lang - * @param {*} value The value to clone. - * @returns {*} Returns the cloned value. - * @see _.cloneDeep - * @example - * - * var objects = [{ 'a': 1 }, { 'b': 2 }]; - * - * var shallow = _.clone(objects); - * console.log(shallow[0] === objects[0]); - * // => true - */ - function clone(value) { - return baseClone(value, CLONE_SYMBOLS_FLAG); - } - - /** - * This method is like `_.clone` except that it accepts `customizer` which - * is invoked to produce the cloned value. If `customizer` returns `undefined`, - * cloning is handled by the method instead. The `customizer` is invoked with - * up to four arguments; (value [, index|key, object, stack]). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to clone. - * @param {Function} [customizer] The function to customize cloning. - * @returns {*} Returns the cloned value. - * @see _.cloneDeepWith - * @example - * - * function customizer(value) { - * if (_.isElement(value)) { - * return value.cloneNode(false); - * } - * } - * - * var el = _.cloneWith(document.body, customizer); - * - * console.log(el === document.body); - * // => false - * console.log(el.nodeName); - * // => 'BODY' - * console.log(el.childNodes.length); - * // => 0 - */ - function cloneWith(value, customizer) { - customizer = typeof customizer == 'function' ? customizer : undefined; - return baseClone(value, CLONE_SYMBOLS_FLAG, customizer); - } - - /** - * This method is like `_.clone` except that it recursively clones `value`. - * - * @static - * @memberOf _ - * @since 1.0.0 - * @category Lang - * @param {*} value The value to recursively clone. - * @returns {*} Returns the deep cloned value. - * @see _.clone - * @example - * - * var objects = [{ 'a': 1 }, { 'b': 2 }]; - * - * var deep = _.cloneDeep(objects); - * console.log(deep[0] === objects[0]); - * // => false - */ - function cloneDeep(value) { - return baseClone(value, CLONE_DEEP_FLAG | CLONE_SYMBOLS_FLAG); - } - - /** - * This method is like `_.cloneWith` except that it recursively clones `value`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to recursively clone. - * @param {Function} [customizer] The function to customize cloning. - * @returns {*} Returns the deep cloned value. - * @see _.cloneWith - * @example - * - * function customizer(value) { - * if (_.isElement(value)) { - * return value.cloneNode(true); - * } - * } - * - * var el = _.cloneDeepWith(document.body, customizer); - * - * console.log(el === document.body); - * // => false - * console.log(el.nodeName); - * // => 'BODY' - * console.log(el.childNodes.length); - * // => 20 - */ - function cloneDeepWith(value, customizer) { - customizer = typeof customizer == 'function' ? customizer : undefined; - return baseClone(value, CLONE_DEEP_FLAG | CLONE_SYMBOLS_FLAG, customizer); - } - - /** - * Checks if `object` conforms to `source` by invoking the predicate - * properties of `source` with the corresponding property values of `object`. - * - * **Note:** This method is equivalent to `_.conforms` when `source` is - * partially applied. - * - * @static - * @memberOf _ - * @since 4.14.0 - * @category Lang - * @param {Object} object The object to inspect. - * @param {Object} source The object of property predicates to conform to. - * @returns {boolean} Returns `true` if `object` conforms, else `false`. - * @example - * - * var object = { 'a': 1, 'b': 2 }; - * - * _.conformsTo(object, { 'b': function(n) { return n > 1; } }); - * // => true - * - * _.conformsTo(object, { 'b': function(n) { return n > 2; } }); - * // => false - */ - function conformsTo(object, source) { - return source == null || baseConformsTo(object, source, keys(source)); - } - - /** - * Performs a - * [`SameValueZero`](http://ecma-international.org/ecma-262/7.0/#sec-samevaluezero) - * comparison between two values to determine if they are equivalent. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to compare. - * @param {*} other The other value to compare. - * @returns {boolean} Returns `true` if the values are equivalent, else `false`. - * @example - * - * var object = { 'a': 1 }; - * var other = { 'a': 1 }; - * - * _.eq(object, object); - * // => true - * - * _.eq(object, other); - * // => false - * - * _.eq('a', 'a'); - * // => true - * - * _.eq('a', Object('a')); - * // => false - * - * _.eq(NaN, NaN); - * // => true - */ - function eq(value, other) { - return value === other || (value !== value && other !== other); - } - - /** - * Checks if `value` is greater than `other`. - * - * @static - * @memberOf _ - * @since 3.9.0 - * @category Lang - * @param {*} value The value to compare. - * @param {*} other The other value to compare. - * @returns {boolean} Returns `true` if `value` is greater than `other`, - * else `false`. - * @see _.lt - * @example - * - * _.gt(3, 1); - * // => true - * - * _.gt(3, 3); - * // => false - * - * _.gt(1, 3); - * // => false - */ - var gt = createRelationalOperation(baseGt); - - /** - * Checks if `value` is greater than or equal to `other`. - * - * @static - * @memberOf _ - * @since 3.9.0 - * @category Lang - * @param {*} value The value to compare. - * @param {*} other The other value to compare. - * @returns {boolean} Returns `true` if `value` is greater than or equal to - * `other`, else `false`. - * @see _.lte - * @example - * - * _.gte(3, 1); - * // => true - * - * _.gte(3, 3); - * // => true - * - * _.gte(1, 3); - * // => false - */ - var gte = createRelationalOperation(function(value, other) { - return value >= other; - }); - - /** - * Checks if `value` is likely an `arguments` object. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is an `arguments` object, - * else `false`. - * @example - * - * _.isArguments(function() { return arguments; }()); - * // => true - * - * _.isArguments([1, 2, 3]); - * // => false - */ - var isArguments = baseIsArguments(function() { return arguments; }()) ? baseIsArguments : function(value) { - return isObjectLike(value) && hasOwnProperty.call(value, 'callee') && - !propertyIsEnumerable.call(value, 'callee'); - }; - - /** - * Checks if `value` is classified as an `Array` object. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is an array, else `false`. - * @example - * - * _.isArray([1, 2, 3]); - * // => true - * - * _.isArray(document.body.children); - * // => false - * - * _.isArray('abc'); - * // => false - * - * _.isArray(_.noop); - * // => false - */ - var isArray = Array.isArray; - - /** - * Checks if `value` is classified as an `ArrayBuffer` object. - * - * @static - * @memberOf _ - * @since 4.3.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is an array buffer, else `false`. - * @example - * - * _.isArrayBuffer(new ArrayBuffer(2)); - * // => true - * - * _.isArrayBuffer(new Array(2)); - * // => false - */ - var isArrayBuffer = nodeIsArrayBuffer ? baseUnary(nodeIsArrayBuffer) : baseIsArrayBuffer; - - /** - * Checks if `value` is array-like. A value is considered array-like if it's - * not a function and has a `value.length` that's an integer greater than or - * equal to `0` and less than or equal to `Number.MAX_SAFE_INTEGER`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is array-like, else `false`. - * @example - * - * _.isArrayLike([1, 2, 3]); - * // => true - * - * _.isArrayLike(document.body.children); - * // => true - * - * _.isArrayLike('abc'); - * // => true - * - * _.isArrayLike(_.noop); - * // => false - */ - function isArrayLike(value) { - return value != null && isLength(value.length) && !isFunction(value); - } - - /** - * This method is like `_.isArrayLike` except that it also checks if `value` - * is an object. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is an array-like object, - * else `false`. - * @example - * - * _.isArrayLikeObject([1, 2, 3]); - * // => true - * - * _.isArrayLikeObject(document.body.children); - * // => true - * - * _.isArrayLikeObject('abc'); - * // => false - * - * _.isArrayLikeObject(_.noop); - * // => false - */ - function isArrayLikeObject(value) { - return isObjectLike(value) && isArrayLike(value); - } - - /** - * Checks if `value` is classified as a boolean primitive or object. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a boolean, else `false`. - * @example - * - * _.isBoolean(false); - * // => true - * - * _.isBoolean(null); - * // => false - */ - function isBoolean(value) { - return value === true || value === false || - (isObjectLike(value) && baseGetTag(value) == boolTag); - } - - /** - * Checks if `value` is a buffer. - * - * @static - * @memberOf _ - * @since 4.3.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a buffer, else `false`. - * @example - * - * _.isBuffer(new Buffer(2)); - * // => true - * - * _.isBuffer(new Uint8Array(2)); - * // => false - */ - var isBuffer = nativeIsBuffer || stubFalse; - - /** - * Checks if `value` is classified as a `Date` object. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a date object, else `false`. - * @example - * - * _.isDate(new Date); - * // => true - * - * _.isDate('Mon April 23 2012'); - * // => false - */ - var isDate = nodeIsDate ? baseUnary(nodeIsDate) : baseIsDate; - - /** - * Checks if `value` is likely a DOM element. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a DOM element, else `false`. - * @example - * - * _.isElement(document.body); - * // => true - * - * _.isElement(''); - * // => false - */ - function isElement(value) { - return isObjectLike(value) && value.nodeType === 1 && !isPlainObject(value); - } - - /** - * Checks if `value` is an empty object, collection, map, or set. - * - * Objects are considered empty if they have no own enumerable string keyed - * properties. - * - * Array-like values such as `arguments` objects, arrays, buffers, strings, or - * jQuery-like collections are considered empty if they have a `length` of `0`. - * Similarly, maps and sets are considered empty if they have a `size` of `0`. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is empty, else `false`. - * @example - * - * _.isEmpty(null); - * // => true - * - * _.isEmpty(true); - * // => true - * - * _.isEmpty(1); - * // => true - * - * _.isEmpty([1, 2, 3]); - * // => false - * - * _.isEmpty({ 'a': 1 }); - * // => false - */ - function isEmpty(value) { - if (value == null) { - return true; - } - if (isArrayLike(value) && - (isArray(value) || typeof value == 'string' || typeof value.splice == 'function' || - isBuffer(value) || isTypedArray(value) || isArguments(value))) { - return !value.length; - } - var tag = getTag(value); - if (tag == mapTag || tag == setTag) { - return !value.size; - } - if (isPrototype(value)) { - return !baseKeys(value).length; - } - for (var key in value) { - if (hasOwnProperty.call(value, key)) { - return false; - } - } - return true; - } - - /** - * Performs a deep comparison between two values to determine if they are - * equivalent. - * - * **Note:** This method supports comparing arrays, array buffers, booleans, - * date objects, error objects, maps, numbers, `Object` objects, regexes, - * sets, strings, symbols, and typed arrays. `Object` objects are compared - * by their own, not inherited, enumerable properties. Functions and DOM - * nodes are compared by strict equality, i.e. `===`. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Lang - * @param {*} value The value to compare. - * @param {*} other The other value to compare. - * @returns {boolean} Returns `true` if the values are equivalent, else `false`. - * @example - * - * var object = { 'a': 1 }; - * var other = { 'a': 1 }; - * - * _.isEqual(object, other); - * // => true - * - * object === other; - * // => false - */ - function isEqual(value, other) { - return baseIsEqual(value, other); - } - - /** - * This method is like `_.isEqual` except that it accepts `customizer` which - * is invoked to compare values. If `customizer` returns `undefined`, comparisons - * are handled by the method instead. The `customizer` is invoked with up to - * six arguments: (objValue, othValue [, index|key, object, other, stack]). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to compare. - * @param {*} other The other value to compare. - * @param {Function} [customizer] The function to customize comparisons. - * @returns {boolean} Returns `true` if the values are equivalent, else `false`. - * @example - * - * function isGreeting(value) { - * return /^h(?:i|ello)$/.test(value); - * } - * - * function customizer(objValue, othValue) { - * if (isGreeting(objValue) && isGreeting(othValue)) { - * return true; - * } - * } - * - * var array = ['hello', 'goodbye']; - * var other = ['hi', 'goodbye']; - * - * _.isEqualWith(array, other, customizer); - * // => true - */ - function isEqualWith(value, other, customizer) { - customizer = typeof customizer == 'function' ? customizer : undefined; - var result = customizer ? customizer(value, other) : undefined; - return result === undefined ? baseIsEqual(value, other, undefined, customizer) : !!result; - } - - /** - * Checks if `value` is an `Error`, `EvalError`, `RangeError`, `ReferenceError`, - * `SyntaxError`, `TypeError`, or `URIError` object. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is an error object, else `false`. - * @example - * - * _.isError(new Error); - * // => true - * - * _.isError(Error); - * // => false - */ - function isError(value) { - if (!isObjectLike(value)) { - return false; - } - var tag = baseGetTag(value); - return tag == errorTag || tag == domExcTag || - (typeof value.message == 'string' && typeof value.name == 'string' && !isPlainObject(value)); - } - - /** - * Checks if `value` is a finite primitive number. - * - * **Note:** This method is based on - * [`Number.isFinite`](https://mdn.io/Number/isFinite). - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a finite number, else `false`. - * @example - * - * _.isFinite(3); - * // => true - * - * _.isFinite(Number.MIN_VALUE); - * // => true - * - * _.isFinite(Infinity); - * // => false - * - * _.isFinite('3'); - * // => false - */ - function isFinite(value) { - return typeof value == 'number' && nativeIsFinite(value); - } - - /** - * Checks if `value` is classified as a `Function` object. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a function, else `false`. - * @example - * - * _.isFunction(_); - * // => true - * - * _.isFunction(/abc/); - * // => false - */ - function isFunction(value) { - if (!isObject(value)) { - return false; - } - // The use of `Object#toString` avoids issues with the `typeof` operator - // in Safari 9 which returns 'object' for typed arrays and other constructors. - var tag = baseGetTag(value); - return tag == funcTag || tag == genTag || tag == asyncTag || tag == proxyTag; - } - - /** - * Checks if `value` is an integer. - * - * **Note:** This method is based on - * [`Number.isInteger`](https://mdn.io/Number/isInteger). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is an integer, else `false`. - * @example - * - * _.isInteger(3); - * // => true - * - * _.isInteger(Number.MIN_VALUE); - * // => false - * - * _.isInteger(Infinity); - * // => false - * - * _.isInteger('3'); - * // => false - */ - function isInteger(value) { - return typeof value == 'number' && value == toInteger(value); - } - - /** - * Checks if `value` is a valid array-like length. - * - * **Note:** This method is loosely based on - * [`ToLength`](http://ecma-international.org/ecma-262/7.0/#sec-tolength). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a valid length, else `false`. - * @example - * - * _.isLength(3); - * // => true - * - * _.isLength(Number.MIN_VALUE); - * // => false - * - * _.isLength(Infinity); - * // => false - * - * _.isLength('3'); - * // => false - */ - function isLength(value) { - return typeof value == 'number' && - value > -1 && value % 1 == 0 && value <= MAX_SAFE_INTEGER; - } - - /** - * Checks if `value` is the - * [language type](http://www.ecma-international.org/ecma-262/7.0/#sec-ecmascript-language-types) - * of `Object`. (e.g. arrays, functions, objects, regexes, `new Number(0)`, and `new String('')`) - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is an object, else `false`. - * @example - * - * _.isObject({}); - * // => true - * - * _.isObject([1, 2, 3]); - * // => true - * - * _.isObject(_.noop); - * // => true - * - * _.isObject(null); - * // => false - */ - function isObject(value) { - var type = typeof value; - return value != null && (type == 'object' || type == 'function'); - } - - /** - * Checks if `value` is object-like. A value is object-like if it's not `null` - * and has a `typeof` result of "object". - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is object-like, else `false`. - * @example - * - * _.isObjectLike({}); - * // => true - * - * _.isObjectLike([1, 2, 3]); - * // => true - * - * _.isObjectLike(_.noop); - * // => false - * - * _.isObjectLike(null); - * // => false - */ - function isObjectLike(value) { - return value != null && typeof value == 'object'; - } - - /** - * Checks if `value` is classified as a `Map` object. - * - * @static - * @memberOf _ - * @since 4.3.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a map, else `false`. - * @example - * - * _.isMap(new Map); - * // => true - * - * _.isMap(new WeakMap); - * // => false - */ - var isMap = nodeIsMap ? baseUnary(nodeIsMap) : baseIsMap; - - /** - * Performs a partial deep comparison between `object` and `source` to - * determine if `object` contains equivalent property values. - * - * **Note:** This method is equivalent to `_.matches` when `source` is - * partially applied. - * - * Partial comparisons will match empty array and empty object `source` - * values against any array or object value, respectively. See `_.isEqual` - * for a list of supported value comparisons. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Lang - * @param {Object} object The object to inspect. - * @param {Object} source The object of property values to match. - * @returns {boolean} Returns `true` if `object` is a match, else `false`. - * @example - * - * var object = { 'a': 1, 'b': 2 }; - * - * _.isMatch(object, { 'b': 2 }); - * // => true - * - * _.isMatch(object, { 'b': 1 }); - * // => false - */ - function isMatch(object, source) { - return object === source || baseIsMatch(object, source, getMatchData(source)); - } - - /** - * This method is like `_.isMatch` except that it accepts `customizer` which - * is invoked to compare values. If `customizer` returns `undefined`, comparisons - * are handled by the method instead. The `customizer` is invoked with five - * arguments: (objValue, srcValue, index|key, object, source). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {Object} object The object to inspect. - * @param {Object} source The object of property values to match. - * @param {Function} [customizer] The function to customize comparisons. - * @returns {boolean} Returns `true` if `object` is a match, else `false`. - * @example - * - * function isGreeting(value) { - * return /^h(?:i|ello)$/.test(value); - * } - * - * function customizer(objValue, srcValue) { - * if (isGreeting(objValue) && isGreeting(srcValue)) { - * return true; - * } - * } - * - * var object = { 'greeting': 'hello' }; - * var source = { 'greeting': 'hi' }; - * - * _.isMatchWith(object, source, customizer); - * // => true - */ - function isMatchWith(object, source, customizer) { - customizer = typeof customizer == 'function' ? customizer : undefined; - return baseIsMatch(object, source, getMatchData(source), customizer); - } - - /** - * Checks if `value` is `NaN`. - * - * **Note:** This method is based on - * [`Number.isNaN`](https://mdn.io/Number/isNaN) and is not the same as - * global [`isNaN`](https://mdn.io/isNaN) which returns `true` for - * `undefined` and other non-number values. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is `NaN`, else `false`. - * @example - * - * _.isNaN(NaN); - * // => true - * - * _.isNaN(new Number(NaN)); - * // => true - * - * isNaN(undefined); - * // => true - * - * _.isNaN(undefined); - * // => false - */ - function isNaN(value) { - // An `NaN` primitive is the only value that is not equal to itself. - // Perform the `toStringTag` check first to avoid errors with some - // ActiveX objects in IE. - return isNumber(value) && value != +value; - } - - /** - * Checks if `value` is a pristine native function. - * - * **Note:** This method can't reliably detect native functions in the presence - * of the core-js package because core-js circumvents this kind of detection. - * Despite multiple requests, the core-js maintainer has made it clear: any - * attempt to fix the detection will be obstructed. As a result, we're left - * with little choice but to throw an error. Unfortunately, this also affects - * packages, like [babel-polyfill](https://www.npmjs.com/package/babel-polyfill), - * which rely on core-js. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a native function, - * else `false`. - * @example - * - * _.isNative(Array.prototype.push); - * // => true - * - * _.isNative(_); - * // => false - */ - function isNative(value) { - if (isMaskable(value)) { - throw new Error(CORE_ERROR_TEXT); - } - return baseIsNative(value); - } - - /** - * Checks if `value` is `null`. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is `null`, else `false`. - * @example - * - * _.isNull(null); - * // => true - * - * _.isNull(void 0); - * // => false - */ - function isNull(value) { - return value === null; - } - - /** - * Checks if `value` is `null` or `undefined`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is nullish, else `false`. - * @example - * - * _.isNil(null); - * // => true - * - * _.isNil(void 0); - * // => true - * - * _.isNil(NaN); - * // => false - */ - function isNil(value) { - return value == null; - } - - /** - * Checks if `value` is classified as a `Number` primitive or object. - * - * **Note:** To exclude `Infinity`, `-Infinity`, and `NaN`, which are - * classified as numbers, use the `_.isFinite` method. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a number, else `false`. - * @example - * - * _.isNumber(3); - * // => true - * - * _.isNumber(Number.MIN_VALUE); - * // => true - * - * _.isNumber(Infinity); - * // => true - * - * _.isNumber('3'); - * // => false - */ - function isNumber(value) { - return typeof value == 'number' || - (isObjectLike(value) && baseGetTag(value) == numberTag); - } - - /** - * Checks if `value` is a plain object, that is, an object created by the - * `Object` constructor or one with a `[[Prototype]]` of `null`. - * - * @static - * @memberOf _ - * @since 0.8.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a plain object, else `false`. - * @example - * - * function Foo() { - * this.a = 1; - * } - * - * _.isPlainObject(new Foo); - * // => false - * - * _.isPlainObject([1, 2, 3]); - * // => false - * - * _.isPlainObject({ 'x': 0, 'y': 0 }); - * // => true - * - * _.isPlainObject(Object.create(null)); - * // => true - */ - function isPlainObject(value) { - if (!isObjectLike(value) || baseGetTag(value) != objectTag) { - return false; - } - var proto = getPrototype(value); - if (proto === null) { - return true; - } - var Ctor = hasOwnProperty.call(proto, 'constructor') && proto.constructor; - return typeof Ctor == 'function' && Ctor instanceof Ctor && - funcToString.call(Ctor) == objectCtorString; - } - - /** - * Checks if `value` is classified as a `RegExp` object. - * - * @static - * @memberOf _ - * @since 0.1.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a regexp, else `false`. - * @example - * - * _.isRegExp(/abc/); - * // => true - * - * _.isRegExp('/abc/'); - * // => false - */ - var isRegExp = nodeIsRegExp ? baseUnary(nodeIsRegExp) : baseIsRegExp; - - /** - * Checks if `value` is a safe integer. An integer is safe if it's an IEEE-754 - * double precision number which isn't the result of a rounded unsafe integer. - * - * **Note:** This method is based on - * [`Number.isSafeInteger`](https://mdn.io/Number/isSafeInteger). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a safe integer, else `false`. - * @example - * - * _.isSafeInteger(3); - * // => true - * - * _.isSafeInteger(Number.MIN_VALUE); - * // => false - * - * _.isSafeInteger(Infinity); - * // => false - * - * _.isSafeInteger('3'); - * // => false - */ - function isSafeInteger(value) { - return isInteger(value) && value >= -MAX_SAFE_INTEGER && value <= MAX_SAFE_INTEGER; - } - - /** - * Checks if `value` is classified as a `Set` object. - * - * @static - * @memberOf _ - * @since 4.3.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a set, else `false`. - * @example - * - * _.isSet(new Set); - * // => true - * - * _.isSet(new WeakSet); - * // => false - */ - var isSet = nodeIsSet ? baseUnary(nodeIsSet) : baseIsSet; - - /** - * Checks if `value` is classified as a `String` primitive or object. - * - * @static - * @since 0.1.0 - * @memberOf _ - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a string, else `false`. - * @example - * - * _.isString('abc'); - * // => true - * - * _.isString(1); - * // => false - */ - function isString(value) { - return typeof value == 'string' || - (!isArray(value) && isObjectLike(value) && baseGetTag(value) == stringTag); - } - - /** - * Checks if `value` is classified as a `Symbol` primitive or object. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a symbol, else `false`. - * @example - * - * _.isSymbol(Symbol.iterator); - * // => true - * - * _.isSymbol('abc'); - * // => false - */ - function isSymbol(value) { - return typeof value == 'symbol' || - (isObjectLike(value) && baseGetTag(value) == symbolTag); - } - - /** - * Checks if `value` is classified as a typed array. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a typed array, else `false`. - * @example - * - * _.isTypedArray(new Uint8Array); - * // => true - * - * _.isTypedArray([]); - * // => false - */ - var isTypedArray = nodeIsTypedArray ? baseUnary(nodeIsTypedArray) : baseIsTypedArray; - - /** - * Checks if `value` is `undefined`. - * - * @static - * @since 0.1.0 - * @memberOf _ - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is `undefined`, else `false`. - * @example - * - * _.isUndefined(void 0); - * // => true - * - * _.isUndefined(null); - * // => false - */ - function isUndefined(value) { - return value === undefined; - } - - /** - * Checks if `value` is classified as a `WeakMap` object. - * - * @static - * @memberOf _ - * @since 4.3.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a weak map, else `false`. - * @example - * - * _.isWeakMap(new WeakMap); - * // => true - * - * _.isWeakMap(new Map); - * // => false - */ - function isWeakMap(value) { - return isObjectLike(value) && getTag(value) == weakMapTag; - } - - /** - * Checks if `value` is classified as a `WeakSet` object. - * - * @static - * @memberOf _ - * @since 4.3.0 - * @category Lang - * @param {*} value The value to check. - * @returns {boolean} Returns `true` if `value` is a weak set, else `false`. - * @example - * - * _.isWeakSet(new WeakSet); - * // => true - * - * _.isWeakSet(new Set); - * // => false - */ - function isWeakSet(value) { - return isObjectLike(value) && baseGetTag(value) == weakSetTag; - } - - /** - * Checks if `value` is less than `other`. - * - * @static - * @memberOf _ - * @since 3.9.0 - * @category Lang - * @param {*} value The value to compare. - * @param {*} other The other value to compare. - * @returns {boolean} Returns `true` if `value` is less than `other`, - * else `false`. - * @see _.gt - * @example - * - * _.lt(1, 3); - * // => true - * - * _.lt(3, 3); - * // => false - * - * _.lt(3, 1); - * // => false - */ - var lt = createRelationalOperation(baseLt); - - /** - * Checks if `value` is less than or equal to `other`. - * - * @static - * @memberOf _ - * @since 3.9.0 - * @category Lang - * @param {*} value The value to compare. - * @param {*} other The other value to compare. - * @returns {boolean} Returns `true` if `value` is less than or equal to - * `other`, else `false`. - * @see _.gte - * @example - * - * _.lte(1, 3); - * // => true - * - * _.lte(3, 3); - * // => true - * - * _.lte(3, 1); - * // => false - */ - var lte = createRelationalOperation(function(value, other) { - return value <= other; - }); - - /** - * Converts `value` to an array. - * - * @static - * @since 0.1.0 - * @memberOf _ - * @category Lang - * @param {*} value The value to convert. - * @returns {Array} Returns the converted array. - * @example - * - * _.toArray({ 'a': 1, 'b': 2 }); - * // => [1, 2] - * - * _.toArray('abc'); - * // => ['a', 'b', 'c'] - * - * _.toArray(1); - * // => [] - * - * _.toArray(null); - * // => [] - */ - function toArray(value) { - if (!value) { - return []; - } - if (isArrayLike(value)) { - return isString(value) ? stringToArray(value) : copyArray(value); - } - if (symIterator && value[symIterator]) { - return iteratorToArray(value[symIterator]()); - } - var tag = getTag(value), - func = tag == mapTag ? mapToArray : (tag == setTag ? setToArray : values); - - return func(value); - } - - /** - * Converts `value` to a finite number. - * - * @static - * @memberOf _ - * @since 4.12.0 - * @category Lang - * @param {*} value The value to convert. - * @returns {number} Returns the converted number. - * @example - * - * _.toFinite(3.2); - * // => 3.2 - * - * _.toFinite(Number.MIN_VALUE); - * // => 5e-324 - * - * _.toFinite(Infinity); - * // => 1.7976931348623157e+308 - * - * _.toFinite('3.2'); - * // => 3.2 - */ - function toFinite(value) { - if (!value) { - return value === 0 ? value : 0; - } - value = toNumber(value); - if (value === INFINITY || value === -INFINITY) { - var sign = (value < 0 ? -1 : 1); - return sign * MAX_INTEGER; - } - return value === value ? value : 0; - } - - /** - * Converts `value` to an integer. - * - * **Note:** This method is loosely based on - * [`ToInteger`](http://www.ecma-international.org/ecma-262/7.0/#sec-tointeger). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to convert. - * @returns {number} Returns the converted integer. - * @example - * - * _.toInteger(3.2); - * // => 3 - * - * _.toInteger(Number.MIN_VALUE); - * // => 0 - * - * _.toInteger(Infinity); - * // => 1.7976931348623157e+308 - * - * _.toInteger('3.2'); - * // => 3 - */ - function toInteger(value) { - var result = toFinite(value), - remainder = result % 1; - - return result === result ? (remainder ? result - remainder : result) : 0; - } - - /** - * Converts `value` to an integer suitable for use as the length of an - * array-like object. - * - * **Note:** This method is based on - * [`ToLength`](http://ecma-international.org/ecma-262/7.0/#sec-tolength). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to convert. - * @returns {number} Returns the converted integer. - * @example - * - * _.toLength(3.2); - * // => 3 - * - * _.toLength(Number.MIN_VALUE); - * // => 0 - * - * _.toLength(Infinity); - * // => 4294967295 - * - * _.toLength('3.2'); - * // => 3 - */ - function toLength(value) { - return value ? baseClamp(toInteger(value), 0, MAX_ARRAY_LENGTH) : 0; - } - - /** - * Converts `value` to a number. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to process. - * @returns {number} Returns the number. - * @example - * - * _.toNumber(3.2); - * // => 3.2 - * - * _.toNumber(Number.MIN_VALUE); - * // => 5e-324 - * - * _.toNumber(Infinity); - * // => Infinity - * - * _.toNumber('3.2'); - * // => 3.2 - */ - function toNumber(value) { - if (typeof value == 'number') { - return value; - } - if (isSymbol(value)) { - return NAN; - } - if (isObject(value)) { - var other = typeof value.valueOf == 'function' ? value.valueOf() : value; - value = isObject(other) ? (other + '') : other; - } - if (typeof value != 'string') { - return value === 0 ? value : +value; - } - value = value.replace(reTrim, ''); - var isBinary = reIsBinary.test(value); - return (isBinary || reIsOctal.test(value)) - ? freeParseInt(value.slice(2), isBinary ? 2 : 8) - : (reIsBadHex.test(value) ? NAN : +value); - } - - /** - * Converts `value` to a plain object flattening inherited enumerable string - * keyed properties of `value` to own properties of the plain object. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Lang - * @param {*} value The value to convert. - * @returns {Object} Returns the converted plain object. - * @example - * - * function Foo() { - * this.b = 2; - * } - * - * Foo.prototype.c = 3; - * - * _.assign({ 'a': 1 }, new Foo); - * // => { 'a': 1, 'b': 2 } - * - * _.assign({ 'a': 1 }, _.toPlainObject(new Foo)); - * // => { 'a': 1, 'b': 2, 'c': 3 } - */ - function toPlainObject(value) { - return copyObject(value, keysIn(value)); - } - - /** - * Converts `value` to a safe integer. A safe integer can be compared and - * represented correctly. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to convert. - * @returns {number} Returns the converted integer. - * @example - * - * _.toSafeInteger(3.2); - * // => 3 - * - * _.toSafeInteger(Number.MIN_VALUE); - * // => 0 - * - * _.toSafeInteger(Infinity); - * // => 9007199254740991 - * - * _.toSafeInteger('3.2'); - * // => 3 - */ - function toSafeInteger(value) { - return value - ? baseClamp(toInteger(value), -MAX_SAFE_INTEGER, MAX_SAFE_INTEGER) - : (value === 0 ? value : 0); - } - - /** - * Converts `value` to a string. An empty string is returned for `null` - * and `undefined` values. The sign of `-0` is preserved. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Lang - * @param {*} value The value to convert. - * @returns {string} Returns the converted string. - * @example - * - * _.toString(null); - * // => '' - * - * _.toString(-0); - * // => '-0' - * - * _.toString([1, 2, 3]); - * // => '1,2,3' - */ - function toString(value) { - return value == null ? '' : baseToString(value); - } - - /*------------------------------------------------------------------------*/ - - /** - * Assigns own enumerable string keyed properties of source objects to the - * destination object. Source objects are applied from left to right. - * Subsequent sources overwrite property assignments of previous sources. - * - * **Note:** This method mutates `object` and is loosely based on - * [`Object.assign`](https://mdn.io/Object/assign). - * - * @static - * @memberOf _ - * @since 0.10.0 - * @category Object - * @param {Object} object The destination object. - * @param {...Object} [sources] The source objects. - * @returns {Object} Returns `object`. - * @see _.assignIn - * @example - * - * function Foo() { - * this.a = 1; - * } - * - * function Bar() { - * this.c = 3; - * } - * - * Foo.prototype.b = 2; - * Bar.prototype.d = 4; - * - * _.assign({ 'a': 0 }, new Foo, new Bar); - * // => { 'a': 1, 'c': 3 } - */ - var assign = createAssigner(function(object, source) { - if (isPrototype(source) || isArrayLike(source)) { - copyObject(source, keys(source), object); - return; - } - for (var key in source) { - if (hasOwnProperty.call(source, key)) { - assignValue(object, key, source[key]); - } - } - }); - - /** - * This method is like `_.assign` except that it iterates over own and - * inherited source properties. - * - * **Note:** This method mutates `object`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @alias extend - * @category Object - * @param {Object} object The destination object. - * @param {...Object} [sources] The source objects. - * @returns {Object} Returns `object`. - * @see _.assign - * @example - * - * function Foo() { - * this.a = 1; - * } - * - * function Bar() { - * this.c = 3; - * } - * - * Foo.prototype.b = 2; - * Bar.prototype.d = 4; - * - * _.assignIn({ 'a': 0 }, new Foo, new Bar); - * // => { 'a': 1, 'b': 2, 'c': 3, 'd': 4 } - */ - var assignIn = createAssigner(function(object, source) { - copyObject(source, keysIn(source), object); - }); - - /** - * This method is like `_.assignIn` except that it accepts `customizer` - * which is invoked to produce the assigned values. If `customizer` returns - * `undefined`, assignment is handled by the method instead. The `customizer` - * is invoked with five arguments: (objValue, srcValue, key, object, source). - * - * **Note:** This method mutates `object`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @alias extendWith - * @category Object - * @param {Object} object The destination object. - * @param {...Object} sources The source objects. - * @param {Function} [customizer] The function to customize assigned values. - * @returns {Object} Returns `object`. - * @see _.assignWith - * @example - * - * function customizer(objValue, srcValue) { - * return _.isUndefined(objValue) ? srcValue : objValue; - * } - * - * var defaults = _.partialRight(_.assignInWith, customizer); - * - * defaults({ 'a': 1 }, { 'b': 2 }, { 'a': 3 }); - * // => { 'a': 1, 'b': 2 } - */ - var assignInWith = createAssigner(function(object, source, srcIndex, customizer) { - copyObject(source, keysIn(source), object, customizer); - }); - - /** - * This method is like `_.assign` except that it accepts `customizer` - * which is invoked to produce the assigned values. If `customizer` returns - * `undefined`, assignment is handled by the method instead. The `customizer` - * is invoked with five arguments: (objValue, srcValue, key, object, source). - * - * **Note:** This method mutates `object`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Object - * @param {Object} object The destination object. - * @param {...Object} sources The source objects. - * @param {Function} [customizer] The function to customize assigned values. - * @returns {Object} Returns `object`. - * @see _.assignInWith - * @example - * - * function customizer(objValue, srcValue) { - * return _.isUndefined(objValue) ? srcValue : objValue; - * } - * - * var defaults = _.partialRight(_.assignWith, customizer); - * - * defaults({ 'a': 1 }, { 'b': 2 }, { 'a': 3 }); - * // => { 'a': 1, 'b': 2 } - */ - var assignWith = createAssigner(function(object, source, srcIndex, customizer) { - copyObject(source, keys(source), object, customizer); - }); - - /** - * Creates an array of values corresponding to `paths` of `object`. - * - * @static - * @memberOf _ - * @since 1.0.0 - * @category Object - * @param {Object} object The object to iterate over. - * @param {...(string|string[])} [paths] The property paths to pick. - * @returns {Array} Returns the picked values. - * @example - * - * var object = { 'a': [{ 'b': { 'c': 3 } }, 4] }; - * - * _.at(object, ['a[0].b.c', 'a[1]']); - * // => [3, 4] - */ - var at = flatRest(baseAt); - - /** - * Creates an object that inherits from the `prototype` object. If a - * `properties` object is given, its own enumerable string keyed properties - * are assigned to the created object. - * - * @static - * @memberOf _ - * @since 2.3.0 - * @category Object - * @param {Object} prototype The object to inherit from. - * @param {Object} [properties] The properties to assign to the object. - * @returns {Object} Returns the new object. - * @example - * - * function Shape() { - * this.x = 0; - * this.y = 0; - * } - * - * function Circle() { - * Shape.call(this); - * } - * - * Circle.prototype = _.create(Shape.prototype, { - * 'constructor': Circle - * }); - * - * var circle = new Circle; - * circle instanceof Circle; - * // => true - * - * circle instanceof Shape; - * // => true - */ - function create(prototype, properties) { - var result = baseCreate(prototype); - return properties == null ? result : baseAssign(result, properties); - } - - /** - * Assigns own and inherited enumerable string keyed properties of source - * objects to the destination object for all destination properties that - * resolve to `undefined`. Source objects are applied from left to right. - * Once a property is set, additional values of the same property are ignored. - * - * **Note:** This method mutates `object`. - * - * @static - * @since 0.1.0 - * @memberOf _ - * @category Object - * @param {Object} object The destination object. - * @param {...Object} [sources] The source objects. - * @returns {Object} Returns `object`. - * @see _.defaultsDeep - * @example - * - * _.defaults({ 'a': 1 }, { 'b': 2 }, { 'a': 3 }); - * // => { 'a': 1, 'b': 2 } - */ - var defaults = baseRest(function(object, sources) { - object = Object(object); - - var index = -1; - var length = sources.length; - var guard = length > 2 ? sources[2] : undefined; - - if (guard && isIterateeCall(sources[0], sources[1], guard)) { - length = 1; - } - - while (++index < length) { - var source = sources[index]; - var props = keysIn(source); - var propsIndex = -1; - var propsLength = props.length; - - while (++propsIndex < propsLength) { - var key = props[propsIndex]; - var value = object[key]; - - if (value === undefined || - (eq(value, objectProto[key]) && !hasOwnProperty.call(object, key))) { - object[key] = source[key]; - } - } - } - - return object; - }); - - /** - * This method is like `_.defaults` except that it recursively assigns - * default properties. - * - * **Note:** This method mutates `object`. - * - * @static - * @memberOf _ - * @since 3.10.0 - * @category Object - * @param {Object} object The destination object. - * @param {...Object} [sources] The source objects. - * @returns {Object} Returns `object`. - * @see _.defaults - * @example - * - * _.defaultsDeep({ 'a': { 'b': 2 } }, { 'a': { 'b': 1, 'c': 3 } }); - * // => { 'a': { 'b': 2, 'c': 3 } } - */ - var defaultsDeep = baseRest(function(args) { - args.push(undefined, customDefaultsMerge); - return apply(mergeWith, undefined, args); - }); - - /** - * This method is like `_.find` except that it returns the key of the first - * element `predicate` returns truthy for instead of the element itself. - * - * @static - * @memberOf _ - * @since 1.1.0 - * @category Object - * @param {Object} object The object to inspect. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @returns {string|undefined} Returns the key of the matched element, - * else `undefined`. - * @example - * - * var users = { - * 'barney': { 'age': 36, 'active': true }, - * 'fred': { 'age': 40, 'active': false }, - * 'pebbles': { 'age': 1, 'active': true } - * }; - * - * _.findKey(users, function(o) { return o.age < 40; }); - * // => 'barney' (iteration order is not guaranteed) - * - * // The `_.matches` iteratee shorthand. - * _.findKey(users, { 'age': 1, 'active': true }); - * // => 'pebbles' - * - * // The `_.matchesProperty` iteratee shorthand. - * _.findKey(users, ['active', false]); - * // => 'fred' - * - * // The `_.property` iteratee shorthand. - * _.findKey(users, 'active'); - * // => 'barney' - */ - function findKey(object, predicate) { - return baseFindKey(object, getIteratee(predicate, 3), baseForOwn); - } - - /** - * This method is like `_.findKey` except that it iterates over elements of - * a collection in the opposite order. - * - * @static - * @memberOf _ - * @since 2.0.0 - * @category Object - * @param {Object} object The object to inspect. - * @param {Function} [predicate=_.identity] The function invoked per iteration. - * @returns {string|undefined} Returns the key of the matched element, - * else `undefined`. - * @example - * - * var users = { - * 'barney': { 'age': 36, 'active': true }, - * 'fred': { 'age': 40, 'active': false }, - * 'pebbles': { 'age': 1, 'active': true } - * }; - * - * _.findLastKey(users, function(o) { return o.age < 40; }); - * // => returns 'pebbles' assuming `_.findKey` returns 'barney' - * - * // The `_.matches` iteratee shorthand. - * _.findLastKey(users, { 'age': 36, 'active': true }); - * // => 'barney' - * - * // The `_.matchesProperty` iteratee shorthand. - * _.findLastKey(users, ['active', false]); - * // => 'fred' - * - * // The `_.property` iteratee shorthand. - * _.findLastKey(users, 'active'); - * // => 'pebbles' - */ - function findLastKey(object, predicate) { - return baseFindKey(object, getIteratee(predicate, 3), baseForOwnRight); - } - - /** - * Iterates over own and inherited enumerable string keyed properties of an - * object and invokes `iteratee` for each property. The iteratee is invoked - * with three arguments: (value, key, object). Iteratee functions may exit - * iteration early by explicitly returning `false`. - * - * @static - * @memberOf _ - * @since 0.3.0 - * @category Object - * @param {Object} object The object to iterate over. - * @param {Function} [iteratee=_.identity] The function invoked per iteration. - * @returns {Object} Returns `object`. - * @see _.forInRight - * @example - * - * function Foo() { - * this.a = 1; - * this.b = 2; - * } - * - * Foo.prototype.c = 3; - * - * _.forIn(new Foo, function(value, key) { - * console.log(key); - * }); - * // => Logs 'a', 'b', then 'c' (iteration order is not guaranteed). - */ - function forIn(object, iteratee) { - return object == null - ? object - : baseFor(object, getIteratee(iteratee, 3), keysIn); - } - - /** - * This method is like `_.forIn` except that it iterates over properties of - * `object` in the opposite order. - * - * @static - * @memberOf _ - * @since 2.0.0 - * @category Object - * @param {Object} object The object to iterate over. - * @param {Function} [iteratee=_.identity] The function invoked per iteration. - * @returns {Object} Returns `object`. - * @see _.forIn - * @example - * - * function Foo() { - * this.a = 1; - * this.b = 2; - * } - * - * Foo.prototype.c = 3; - * - * _.forInRight(new Foo, function(value, key) { - * console.log(key); - * }); - * // => Logs 'c', 'b', then 'a' assuming `_.forIn` logs 'a', 'b', then 'c'. - */ - function forInRight(object, iteratee) { - return object == null - ? object - : baseForRight(object, getIteratee(iteratee, 3), keysIn); - } - - /** - * Iterates over own enumerable string keyed properties of an object and - * invokes `iteratee` for each property. The iteratee is invoked with three - * arguments: (value, key, object). Iteratee functions may exit iteration - * early by explicitly returning `false`. - * - * @static - * @memberOf _ - * @since 0.3.0 - * @category Object - * @param {Object} object The object to iterate over. - * @param {Function} [iteratee=_.identity] The function invoked per iteration. - * @returns {Object} Returns `object`. - * @see _.forOwnRight - * @example - * - * function Foo() { - * this.a = 1; - * this.b = 2; - * } - * - * Foo.prototype.c = 3; - * - * _.forOwn(new Foo, function(value, key) { - * console.log(key); - * }); - * // => Logs 'a' then 'b' (iteration order is not guaranteed). - */ - function forOwn(object, iteratee) { - return object && baseForOwn(object, getIteratee(iteratee, 3)); - } - - /** - * This method is like `_.forOwn` except that it iterates over properties of - * `object` in the opposite order. - * - * @static - * @memberOf _ - * @since 2.0.0 - * @category Object - * @param {Object} object The object to iterate over. - * @param {Function} [iteratee=_.identity] The function invoked per iteration. - * @returns {Object} Returns `object`. - * @see _.forOwn - * @example - * - * function Foo() { - * this.a = 1; - * this.b = 2; - * } - * - * Foo.prototype.c = 3; - * - * _.forOwnRight(new Foo, function(value, key) { - * console.log(key); - * }); - * // => Logs 'b' then 'a' assuming `_.forOwn` logs 'a' then 'b'. - */ - function forOwnRight(object, iteratee) { - return object && baseForOwnRight(object, getIteratee(iteratee, 3)); - } - - /** - * Creates an array of function property names from own enumerable properties - * of `object`. - * - * @static - * @since 0.1.0 - * @memberOf _ - * @category Object - * @param {Object} object The object to inspect. - * @returns {Array} Returns the function names. - * @see _.functionsIn - * @example - * - * function Foo() { - * this.a = _.constant('a'); - * this.b = _.constant('b'); - * } - * - * Foo.prototype.c = _.constant('c'); - * - * _.functions(new Foo); - * // => ['a', 'b'] - */ - function functions(object) { - return object == null ? [] : baseFunctions(object, keys(object)); - } - - /** - * Creates an array of function property names from own and inherited - * enumerable properties of `object`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Object - * @param {Object} object The object to inspect. - * @returns {Array} Returns the function names. - * @see _.functions - * @example - * - * function Foo() { - * this.a = _.constant('a'); - * this.b = _.constant('b'); - * } - * - * Foo.prototype.c = _.constant('c'); - * - * _.functionsIn(new Foo); - * // => ['a', 'b', 'c'] - */ - function functionsIn(object) { - return object == null ? [] : baseFunctions(object, keysIn(object)); - } - - /** - * Gets the value at `path` of `object`. If the resolved value is - * `undefined`, the `defaultValue` is returned in its place. - * - * @static - * @memberOf _ - * @since 3.7.0 - * @category Object - * @param {Object} object The object to query. - * @param {Array|string} path The path of the property to get. - * @param {*} [defaultValue] The value returned for `undefined` resolved values. - * @returns {*} Returns the resolved value. - * @example - * - * var object = { 'a': [{ 'b': { 'c': 3 } }] }; - * - * _.get(object, 'a[0].b.c'); - * // => 3 - * - * _.get(object, ['a', '0', 'b', 'c']); - * // => 3 - * - * _.get(object, 'a.b.c', 'default'); - * // => 'default' - */ - function get(object, path, defaultValue) { - var result = object == null ? undefined : baseGet(object, path); - return result === undefined ? defaultValue : result; - } - - /** - * Checks if `path` is a direct property of `object`. - * - * @static - * @since 0.1.0 - * @memberOf _ - * @category Object - * @param {Object} object The object to query. - * @param {Array|string} path The path to check. - * @returns {boolean} Returns `true` if `path` exists, else `false`. - * @example - * - * var object = { 'a': { 'b': 2 } }; - * var other = _.create({ 'a': _.create({ 'b': 2 }) }); - * - * _.has(object, 'a'); - * // => true - * - * _.has(object, 'a.b'); - * // => true - * - * _.has(object, ['a', 'b']); - * // => true - * - * _.has(other, 'a'); - * // => false - */ - function has(object, path) { - return object != null && hasPath(object, path, baseHas); - } - - /** - * Checks if `path` is a direct or inherited property of `object`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Object - * @param {Object} object The object to query. - * @param {Array|string} path The path to check. - * @returns {boolean} Returns `true` if `path` exists, else `false`. - * @example - * - * var object = _.create({ 'a': _.create({ 'b': 2 }) }); - * - * _.hasIn(object, 'a'); - * // => true - * - * _.hasIn(object, 'a.b'); - * // => true - * - * _.hasIn(object, ['a', 'b']); - * // => true - * - * _.hasIn(object, 'b'); - * // => false - */ - function hasIn(object, path) { - return object != null && hasPath(object, path, baseHasIn); - } - - /** - * Creates an object composed of the inverted keys and values of `object`. - * If `object` contains duplicate values, subsequent values overwrite - * property assignments of previous values. - * - * @static - * @memberOf _ - * @since 0.7.0 - * @category Object - * @param {Object} object The object to invert. - * @returns {Object} Returns the new inverted object. - * @example - * - * var object = { 'a': 1, 'b': 2, 'c': 1 }; - * - * _.invert(object); - * // => { '1': 'c', '2': 'b' } - */ - var invert = createInverter(function(result, value, key) { - if (value != null && - typeof value.toString != 'function') { - value = nativeObjectToString.call(value); - } - - result[value] = key; - }, constant(identity)); - - /** - * This method is like `_.invert` except that the inverted object is generated - * from the results of running each element of `object` thru `iteratee`. The - * corresponding inverted value of each inverted key is an array of keys - * responsible for generating the inverted value. The iteratee is invoked - * with one argument: (value). - * - * @static - * @memberOf _ - * @since 4.1.0 - * @category Object - * @param {Object} object The object to invert. - * @param {Function} [iteratee=_.identity] The iteratee invoked per element. - * @returns {Object} Returns the new inverted object. - * @example - * - * var object = { 'a': 1, 'b': 2, 'c': 1 }; - * - * _.invertBy(object); - * // => { '1': ['a', 'c'], '2': ['b'] } - * - * _.invertBy(object, function(value) { - * return 'group' + value; - * }); - * // => { 'group1': ['a', 'c'], 'group2': ['b'] } - */ - var invertBy = createInverter(function(result, value, key) { - if (value != null && - typeof value.toString != 'function') { - value = nativeObjectToString.call(value); - } - - if (hasOwnProperty.call(result, value)) { - result[value].push(key); - } else { - result[value] = [key]; - } - }, getIteratee); - - /** - * Invokes the method at `path` of `object`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Object - * @param {Object} object The object to query. - * @param {Array|string} path The path of the method to invoke. - * @param {...*} [args] The arguments to invoke the method with. - * @returns {*} Returns the result of the invoked method. - * @example - * - * var object = { 'a': [{ 'b': { 'c': [1, 2, 3, 4] } }] }; - * - * _.invoke(object, 'a[0].b.c.slice', 1, 3); - * // => [2, 3] - */ - var invoke = baseRest(baseInvoke); - - /** - * Creates an array of the own enumerable property names of `object`. - * - * **Note:** Non-object values are coerced to objects. See the - * [ES spec](http://ecma-international.org/ecma-262/7.0/#sec-object.keys) - * for more details. - * - * @static - * @since 0.1.0 - * @memberOf _ - * @category Object - * @param {Object} object The object to query. - * @returns {Array} Returns the array of property names. - * @example - * - * function Foo() { - * this.a = 1; - * this.b = 2; - * } - * - * Foo.prototype.c = 3; - * - * _.keys(new Foo); - * // => ['a', 'b'] (iteration order is not guaranteed) - * - * _.keys('hi'); - * // => ['0', '1'] - */ - function keys(object) { - return isArrayLike(object) ? arrayLikeKeys(object) : baseKeys(object); - } - - /** - * Creates an array of the own and inherited enumerable property names of `object`. - * - * **Note:** Non-object values are coerced to objects. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Object - * @param {Object} object The object to query. - * @returns {Array} Returns the array of property names. - * @example - * - * function Foo() { - * this.a = 1; - * this.b = 2; - * } - * - * Foo.prototype.c = 3; - * - * _.keysIn(new Foo); - * // => ['a', 'b', 'c'] (iteration order is not guaranteed) - */ - function keysIn(object) { - return isArrayLike(object) ? arrayLikeKeys(object, true) : baseKeysIn(object); - } - - /** - * The opposite of `_.mapValues`; this method creates an object with the - * same values as `object` and keys generated by running each own enumerable - * string keyed property of `object` thru `iteratee`. The iteratee is invoked - * with three arguments: (value, key, object). - * - * @static - * @memberOf _ - * @since 3.8.0 - * @category Object - * @param {Object} object The object to iterate over. - * @param {Function} [iteratee=_.identity] The function invoked per iteration. - * @returns {Object} Returns the new mapped object. - * @see _.mapValues - * @example - * - * _.mapKeys({ 'a': 1, 'b': 2 }, function(value, key) { - * return key + value; - * }); - * // => { 'a1': 1, 'b2': 2 } - */ - function mapKeys(object, iteratee) { - var result = {}; - iteratee = getIteratee(iteratee, 3); - - baseForOwn(object, function(value, key, object) { - baseAssignValue(result, iteratee(value, key, object), value); - }); - return result; - } - - /** - * Creates an object with the same keys as `object` and values generated - * by running each own enumerable string keyed property of `object` thru - * `iteratee`. The iteratee is invoked with three arguments: - * (value, key, object). - * - * @static - * @memberOf _ - * @since 2.4.0 - * @category Object - * @param {Object} object The object to iterate over. - * @param {Function} [iteratee=_.identity] The function invoked per iteration. - * @returns {Object} Returns the new mapped object. - * @see _.mapKeys - * @example - * - * var users = { - * 'fred': { 'user': 'fred', 'age': 40 }, - * 'pebbles': { 'user': 'pebbles', 'age': 1 } - * }; - * - * _.mapValues(users, function(o) { return o.age; }); - * // => { 'fred': 40, 'pebbles': 1 } (iteration order is not guaranteed) - * - * // The `_.property` iteratee shorthand. - * _.mapValues(users, 'age'); - * // => { 'fred': 40, 'pebbles': 1 } (iteration order is not guaranteed) - */ - function mapValues(object, iteratee) { - var result = {}; - iteratee = getIteratee(iteratee, 3); - - baseForOwn(object, function(value, key, object) { - baseAssignValue(result, key, iteratee(value, key, object)); - }); - return result; - } - - /** - * This method is like `_.assign` except that it recursively merges own and - * inherited enumerable string keyed properties of source objects into the - * destination object. Source properties that resolve to `undefined` are - * skipped if a destination value exists. Array and plain object properties - * are merged recursively. Other objects and value types are overridden by - * assignment. Source objects are applied from left to right. Subsequent - * sources overwrite property assignments of previous sources. - * - * **Note:** This method mutates `object`. - * - * @static - * @memberOf _ - * @since 0.5.0 - * @category Object - * @param {Object} object The destination object. - * @param {...Object} [sources] The source objects. - * @returns {Object} Returns `object`. - * @example - * - * var object = { - * 'a': [{ 'b': 2 }, { 'd': 4 }] - * }; - * - * var other = { - * 'a': [{ 'c': 3 }, { 'e': 5 }] - * }; - * - * _.merge(object, other); - * // => { 'a': [{ 'b': 2, 'c': 3 }, { 'd': 4, 'e': 5 }] } - */ - var merge = createAssigner(function(object, source, srcIndex) { - baseMerge(object, source, srcIndex); - }); - - /** - * This method is like `_.merge` except that it accepts `customizer` which - * is invoked to produce the merged values of the destination and source - * properties. If `customizer` returns `undefined`, merging is handled by the - * method instead. The `customizer` is invoked with six arguments: - * (objValue, srcValue, key, object, source, stack). - * - * **Note:** This method mutates `object`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Object - * @param {Object} object The destination object. - * @param {...Object} sources The source objects. - * @param {Function} customizer The function to customize assigned values. - * @returns {Object} Returns `object`. - * @example - * - * function customizer(objValue, srcValue) { - * if (_.isArray(objValue)) { - * return objValue.concat(srcValue); - * } - * } - * - * var object = { 'a': [1], 'b': [2] }; - * var other = { 'a': [3], 'b': [4] }; - * - * _.mergeWith(object, other, customizer); - * // => { 'a': [1, 3], 'b': [2, 4] } - */ - var mergeWith = createAssigner(function(object, source, srcIndex, customizer) { - baseMerge(object, source, srcIndex, customizer); - }); - - /** - * The opposite of `_.pick`; this method creates an object composed of the - * own and inherited enumerable property paths of `object` that are not omitted. - * - * **Note:** This method is considerably slower than `_.pick`. - * - * @static - * @since 0.1.0 - * @memberOf _ - * @category Object - * @param {Object} object The source object. - * @param {...(string|string[])} [paths] The property paths to omit. - * @returns {Object} Returns the new object. - * @example - * - * var object = { 'a': 1, 'b': '2', 'c': 3 }; - * - * _.omit(object, ['a', 'c']); - * // => { 'b': '2' } - */ - var omit = flatRest(function(object, paths) { - var result = {}; - if (object == null) { - return result; - } - var isDeep = false; - paths = arrayMap(paths, function(path) { - path = castPath(path, object); - isDeep || (isDeep = path.length > 1); - return path; - }); - copyObject(object, getAllKeysIn(object), result); - if (isDeep) { - result = baseClone(result, CLONE_DEEP_FLAG | CLONE_FLAT_FLAG | CLONE_SYMBOLS_FLAG, customOmitClone); - } - var length = paths.length; - while (length--) { - baseUnset(result, paths[length]); - } - return result; - }); - - /** - * The opposite of `_.pickBy`; this method creates an object composed of - * the own and inherited enumerable string keyed properties of `object` that - * `predicate` doesn't return truthy for. The predicate is invoked with two - * arguments: (value, key). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Object - * @param {Object} object The source object. - * @param {Function} [predicate=_.identity] The function invoked per property. - * @returns {Object} Returns the new object. - * @example - * - * var object = { 'a': 1, 'b': '2', 'c': 3 }; - * - * _.omitBy(object, _.isNumber); - * // => { 'b': '2' } - */ - function omitBy(object, predicate) { - return pickBy(object, negate(getIteratee(predicate))); - } - - /** - * Creates an object composed of the picked `object` properties. - * - * @static - * @since 0.1.0 - * @memberOf _ - * @category Object - * @param {Object} object The source object. - * @param {...(string|string[])} [paths] The property paths to pick. - * @returns {Object} Returns the new object. - * @example - * - * var object = { 'a': 1, 'b': '2', 'c': 3 }; - * - * _.pick(object, ['a', 'c']); - * // => { 'a': 1, 'c': 3 } - */ - var pick = flatRest(function(object, paths) { - return object == null ? {} : basePick(object, paths); - }); - - /** - * Creates an object composed of the `object` properties `predicate` returns - * truthy for. The predicate is invoked with two arguments: (value, key). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Object - * @param {Object} object The source object. - * @param {Function} [predicate=_.identity] The function invoked per property. - * @returns {Object} Returns the new object. - * @example - * - * var object = { 'a': 1, 'b': '2', 'c': 3 }; - * - * _.pickBy(object, _.isNumber); - * // => { 'a': 1, 'c': 3 } - */ - function pickBy(object, predicate) { - if (object == null) { - return {}; - } - var props = arrayMap(getAllKeysIn(object), function(prop) { - return [prop]; - }); - predicate = getIteratee(predicate); - return basePickBy(object, props, function(value, path) { - return predicate(value, path[0]); - }); - } - - /** - * This method is like `_.get` except that if the resolved value is a - * function it's invoked with the `this` binding of its parent object and - * its result is returned. - * - * @static - * @since 0.1.0 - * @memberOf _ - * @category Object - * @param {Object} object The object to query. - * @param {Array|string} path The path of the property to resolve. - * @param {*} [defaultValue] The value returned for `undefined` resolved values. - * @returns {*} Returns the resolved value. - * @example - * - * var object = { 'a': [{ 'b': { 'c1': 3, 'c2': _.constant(4) } }] }; - * - * _.result(object, 'a[0].b.c1'); - * // => 3 - * - * _.result(object, 'a[0].b.c2'); - * // => 4 - * - * _.result(object, 'a[0].b.c3', 'default'); - * // => 'default' - * - * _.result(object, 'a[0].b.c3', _.constant('default')); - * // => 'default' - */ - function result(object, path, defaultValue) { - path = castPath(path, object); - - var index = -1, - length = path.length; - - // Ensure the loop is entered when path is empty. - if (!length) { - length = 1; - object = undefined; - } - while (++index < length) { - var value = object == null ? undefined : object[toKey(path[index])]; - if (value === undefined) { - index = length; - value = defaultValue; - } - object = isFunction(value) ? value.call(object) : value; - } - return object; - } - - /** - * Sets the value at `path` of `object`. If a portion of `path` doesn't exist, - * it's created. Arrays are created for missing index properties while objects - * are created for all other missing properties. Use `_.setWith` to customize - * `path` creation. - * - * **Note:** This method mutates `object`. - * - * @static - * @memberOf _ - * @since 3.7.0 - * @category Object - * @param {Object} object The object to modify. - * @param {Array|string} path The path of the property to set. - * @param {*} value The value to set. - * @returns {Object} Returns `object`. - * @example - * - * var object = { 'a': [{ 'b': { 'c': 3 } }] }; - * - * _.set(object, 'a[0].b.c', 4); - * console.log(object.a[0].b.c); - * // => 4 - * - * _.set(object, ['x', '0', 'y', 'z'], 5); - * console.log(object.x[0].y.z); - * // => 5 - */ - function set(object, path, value) { - return object == null ? object : baseSet(object, path, value); - } - - /** - * This method is like `_.set` except that it accepts `customizer` which is - * invoked to produce the objects of `path`. If `customizer` returns `undefined` - * path creation is handled by the method instead. The `customizer` is invoked - * with three arguments: (nsValue, key, nsObject). - * - * **Note:** This method mutates `object`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Object - * @param {Object} object The object to modify. - * @param {Array|string} path The path of the property to set. - * @param {*} value The value to set. - * @param {Function} [customizer] The function to customize assigned values. - * @returns {Object} Returns `object`. - * @example - * - * var object = {}; - * - * _.setWith(object, '[0][1]', 'a', Object); - * // => { '0': { '1': 'a' } } - */ - function setWith(object, path, value, customizer) { - customizer = typeof customizer == 'function' ? customizer : undefined; - return object == null ? object : baseSet(object, path, value, customizer); - } - - /** - * Creates an array of own enumerable string keyed-value pairs for `object` - * which can be consumed by `_.fromPairs`. If `object` is a map or set, its - * entries are returned. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @alias entries - * @category Object - * @param {Object} object The object to query. - * @returns {Array} Returns the key-value pairs. - * @example - * - * function Foo() { - * this.a = 1; - * this.b = 2; - * } - * - * Foo.prototype.c = 3; - * - * _.toPairs(new Foo); - * // => [['a', 1], ['b', 2]] (iteration order is not guaranteed) - */ - var toPairs = createToPairs(keys); - - /** - * Creates an array of own and inherited enumerable string keyed-value pairs - * for `object` which can be consumed by `_.fromPairs`. If `object` is a map - * or set, its entries are returned. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @alias entriesIn - * @category Object - * @param {Object} object The object to query. - * @returns {Array} Returns the key-value pairs. - * @example - * - * function Foo() { - * this.a = 1; - * this.b = 2; - * } - * - * Foo.prototype.c = 3; - * - * _.toPairsIn(new Foo); - * // => [['a', 1], ['b', 2], ['c', 3]] (iteration order is not guaranteed) - */ - var toPairsIn = createToPairs(keysIn); - - /** - * An alternative to `_.reduce`; this method transforms `object` to a new - * `accumulator` object which is the result of running each of its own - * enumerable string keyed properties thru `iteratee`, with each invocation - * potentially mutating the `accumulator` object. If `accumulator` is not - * provided, a new object with the same `[[Prototype]]` will be used. The - * iteratee is invoked with four arguments: (accumulator, value, key, object). - * Iteratee functions may exit iteration early by explicitly returning `false`. - * - * @static - * @memberOf _ - * @since 1.3.0 - * @category Object - * @param {Object} object The object to iterate over. - * @param {Function} [iteratee=_.identity] The function invoked per iteration. - * @param {*} [accumulator] The custom accumulator value. - * @returns {*} Returns the accumulated value. - * @example - * - * _.transform([2, 3, 4], function(result, n) { - * result.push(n *= n); - * return n % 2 == 0; - * }, []); - * // => [4, 9] - * - * _.transform({ 'a': 1, 'b': 2, 'c': 1 }, function(result, value, key) { - * (result[value] || (result[value] = [])).push(key); - * }, {}); - * // => { '1': ['a', 'c'], '2': ['b'] } - */ - function transform(object, iteratee, accumulator) { - var isArr = isArray(object), - isArrLike = isArr || isBuffer(object) || isTypedArray(object); - - iteratee = getIteratee(iteratee, 4); - if (accumulator == null) { - var Ctor = object && object.constructor; - if (isArrLike) { - accumulator = isArr ? new Ctor : []; - } - else if (isObject(object)) { - accumulator = isFunction(Ctor) ? baseCreate(getPrototype(object)) : {}; - } - else { - accumulator = {}; - } - } - (isArrLike ? arrayEach : baseForOwn)(object, function(value, index, object) { - return iteratee(accumulator, value, index, object); - }); - return accumulator; - } - - /** - * Removes the property at `path` of `object`. - * - * **Note:** This method mutates `object`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Object - * @param {Object} object The object to modify. - * @param {Array|string} path The path of the property to unset. - * @returns {boolean} Returns `true` if the property is deleted, else `false`. - * @example - * - * var object = { 'a': [{ 'b': { 'c': 7 } }] }; - * _.unset(object, 'a[0].b.c'); - * // => true - * - * console.log(object); - * // => { 'a': [{ 'b': {} }] }; - * - * _.unset(object, ['a', '0', 'b', 'c']); - * // => true - * - * console.log(object); - * // => { 'a': [{ 'b': {} }] }; - */ - function unset(object, path) { - return object == null ? true : baseUnset(object, path); - } - - /** - * This method is like `_.set` except that accepts `updater` to produce the - * value to set. Use `_.updateWith` to customize `path` creation. The `updater` - * is invoked with one argument: (value). - * - * **Note:** This method mutates `object`. - * - * @static - * @memberOf _ - * @since 4.6.0 - * @category Object - * @param {Object} object The object to modify. - * @param {Array|string} path The path of the property to set. - * @param {Function} updater The function to produce the updated value. - * @returns {Object} Returns `object`. - * @example - * - * var object = { 'a': [{ 'b': { 'c': 3 } }] }; - * - * _.update(object, 'a[0].b.c', function(n) { return n * n; }); - * console.log(object.a[0].b.c); - * // => 9 - * - * _.update(object, 'x[0].y.z', function(n) { return n ? n + 1 : 0; }); - * console.log(object.x[0].y.z); - * // => 0 - */ - function update(object, path, updater) { - return object == null ? object : baseUpdate(object, path, castFunction(updater)); - } - - /** - * This method is like `_.update` except that it accepts `customizer` which is - * invoked to produce the objects of `path`. If `customizer` returns `undefined` - * path creation is handled by the method instead. The `customizer` is invoked - * with three arguments: (nsValue, key, nsObject). - * - * **Note:** This method mutates `object`. - * - * @static - * @memberOf _ - * @since 4.6.0 - * @category Object - * @param {Object} object The object to modify. - * @param {Array|string} path The path of the property to set. - * @param {Function} updater The function to produce the updated value. - * @param {Function} [customizer] The function to customize assigned values. - * @returns {Object} Returns `object`. - * @example - * - * var object = {}; - * - * _.updateWith(object, '[0][1]', _.constant('a'), Object); - * // => { '0': { '1': 'a' } } - */ - function updateWith(object, path, updater, customizer) { - customizer = typeof customizer == 'function' ? customizer : undefined; - return object == null ? object : baseUpdate(object, path, castFunction(updater), customizer); - } - - /** - * Creates an array of the own enumerable string keyed property values of `object`. - * - * **Note:** Non-object values are coerced to objects. - * - * @static - * @since 0.1.0 - * @memberOf _ - * @category Object - * @param {Object} object The object to query. - * @returns {Array} Returns the array of property values. - * @example - * - * function Foo() { - * this.a = 1; - * this.b = 2; - * } - * - * Foo.prototype.c = 3; - * - * _.values(new Foo); - * // => [1, 2] (iteration order is not guaranteed) - * - * _.values('hi'); - * // => ['h', 'i'] - */ - function values(object) { - return object == null ? [] : baseValues(object, keys(object)); - } - - /** - * Creates an array of the own and inherited enumerable string keyed property - * values of `object`. - * - * **Note:** Non-object values are coerced to objects. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category Object - * @param {Object} object The object to query. - * @returns {Array} Returns the array of property values. - * @example - * - * function Foo() { - * this.a = 1; - * this.b = 2; - * } - * - * Foo.prototype.c = 3; - * - * _.valuesIn(new Foo); - * // => [1, 2, 3] (iteration order is not guaranteed) - */ - function valuesIn(object) { - return object == null ? [] : baseValues(object, keysIn(object)); - } - - /*------------------------------------------------------------------------*/ - - /** - * Clamps `number` within the inclusive `lower` and `upper` bounds. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category Number - * @param {number} number The number to clamp. - * @param {number} [lower] The lower bound. - * @param {number} upper The upper bound. - * @returns {number} Returns the clamped number. - * @example - * - * _.clamp(-10, -5, 5); - * // => -5 - * - * _.clamp(10, -5, 5); - * // => 5 - */ - function clamp(number, lower, upper) { - if (upper === undefined) { - upper = lower; - lower = undefined; - } - if (upper !== undefined) { - upper = toNumber(upper); - upper = upper === upper ? upper : 0; - } - if (lower !== undefined) { - lower = toNumber(lower); - lower = lower === lower ? lower : 0; - } - return baseClamp(toNumber(number), lower, upper); - } - - /** - * Checks if `n` is between `start` and up to, but not including, `end`. If - * `end` is not specified, it's set to `start` with `start` then set to `0`. - * If `start` is greater than `end` the params are swapped to support - * negative ranges. - * - * @static - * @memberOf _ - * @since 3.3.0 - * @category Number - * @param {number} number The number to check. - * @param {number} [start=0] The start of the range. - * @param {number} end The end of the range. - * @returns {boolean} Returns `true` if `number` is in the range, else `false`. - * @see _.range, _.rangeRight - * @example - * - * _.inRange(3, 2, 4); - * // => true - * - * _.inRange(4, 8); - * // => true - * - * _.inRange(4, 2); - * // => false - * - * _.inRange(2, 2); - * // => false - * - * _.inRange(1.2, 2); - * // => true - * - * _.inRange(5.2, 4); - * // => false - * - * _.inRange(-3, -2, -6); - * // => true - */ - function inRange(number, start, end) { - start = toFinite(start); - if (end === undefined) { - end = start; - start = 0; - } else { - end = toFinite(end); - } - number = toNumber(number); - return baseInRange(number, start, end); - } - - /** - * Produces a random number between the inclusive `lower` and `upper` bounds. - * If only one argument is provided a number between `0` and the given number - * is returned. If `floating` is `true`, or either `lower` or `upper` are - * floats, a floating-point number is returned instead of an integer. - * - * **Note:** JavaScript follows the IEEE-754 standard for resolving - * floating-point values which can produce unexpected results. - * - * @static - * @memberOf _ - * @since 0.7.0 - * @category Number - * @param {number} [lower=0] The lower bound. - * @param {number} [upper=1] The upper bound. - * @param {boolean} [floating] Specify returning a floating-point number. - * @returns {number} Returns the random number. - * @example - * - * _.random(0, 5); - * // => an integer between 0 and 5 - * - * _.random(5); - * // => also an integer between 0 and 5 - * - * _.random(5, true); - * // => a floating-point number between 0 and 5 - * - * _.random(1.2, 5.2); - * // => a floating-point number between 1.2 and 5.2 - */ - function random(lower, upper, floating) { - if (floating && typeof floating != 'boolean' && isIterateeCall(lower, upper, floating)) { - upper = floating = undefined; - } - if (floating === undefined) { - if (typeof upper == 'boolean') { - floating = upper; - upper = undefined; - } - else if (typeof lower == 'boolean') { - floating = lower; - lower = undefined; - } - } - if (lower === undefined && upper === undefined) { - lower = 0; - upper = 1; - } - else { - lower = toFinite(lower); - if (upper === undefined) { - upper = lower; - lower = 0; - } else { - upper = toFinite(upper); - } - } - if (lower > upper) { - var temp = lower; - lower = upper; - upper = temp; - } - if (floating || lower % 1 || upper % 1) { - var rand = nativeRandom(); - return nativeMin(lower + (rand * (upper - lower + freeParseFloat('1e-' + ((rand + '').length - 1)))), upper); - } - return baseRandom(lower, upper); - } - - /*------------------------------------------------------------------------*/ - - /** - * Converts `string` to [camel case](https://en.wikipedia.org/wiki/CamelCase). - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category String - * @param {string} [string=''] The string to convert. - * @returns {string} Returns the camel cased string. - * @example - * - * _.camelCase('Foo Bar'); - * // => 'fooBar' - * - * _.camelCase('--foo-bar--'); - * // => 'fooBar' - * - * _.camelCase('__FOO_BAR__'); - * // => 'fooBar' - */ - var camelCase = createCompounder(function(result, word, index) { - word = word.toLowerCase(); - return result + (index ? capitalize(word) : word); - }); - - /** - * Converts the first character of `string` to upper case and the remaining - * to lower case. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category String - * @param {string} [string=''] The string to capitalize. - * @returns {string} Returns the capitalized string. - * @example - * - * _.capitalize('FRED'); - * // => 'Fred' - */ - function capitalize(string) { - return upperFirst(toString(string).toLowerCase()); - } - - /** - * Deburrs `string` by converting - * [Latin-1 Supplement](https://en.wikipedia.org/wiki/Latin-1_Supplement_(Unicode_block)#Character_table) - * and [Latin Extended-A](https://en.wikipedia.org/wiki/Latin_Extended-A) - * letters to basic Latin letters and removing - * [combining diacritical marks](https://en.wikipedia.org/wiki/Combining_Diacritical_Marks). - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category String - * @param {string} [string=''] The string to deburr. - * @returns {string} Returns the deburred string. - * @example - * - * _.deburr('déjà vu'); - * // => 'deja vu' - */ - function deburr(string) { - string = toString(string); - return string && string.replace(reLatin, deburrLetter).replace(reComboMark, ''); - } - - /** - * Checks if `string` ends with the given target string. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category String - * @param {string} [string=''] The string to inspect. - * @param {string} [target] The string to search for. - * @param {number} [position=string.length] The position to search up to. - * @returns {boolean} Returns `true` if `string` ends with `target`, - * else `false`. - * @example - * - * _.endsWith('abc', 'c'); - * // => true - * - * _.endsWith('abc', 'b'); - * // => false - * - * _.endsWith('abc', 'b', 2); - * // => true - */ - function endsWith(string, target, position) { - string = toString(string); - target = baseToString(target); - - var length = string.length; - position = position === undefined - ? length - : baseClamp(toInteger(position), 0, length); - - var end = position; - position -= target.length; - return position >= 0 && string.slice(position, end) == target; - } - - /** - * Converts the characters "&", "<", ">", '"', and "'" in `string` to their - * corresponding HTML entities. - * - * **Note:** No other characters are escaped. To escape additional - * characters use a third-party library like [_he_](https://mths.be/he). - * - * Though the ">" character is escaped for symmetry, characters like - * ">" and "/" don't need escaping in HTML and have no special meaning - * unless they're part of a tag or unquoted attribute value. See - * [Mathias Bynens's article](https://mathiasbynens.be/notes/ambiguous-ampersands) - * (under "semi-related fun fact") for more details. - * - * When working with HTML you should always - * [quote attribute values](http://wonko.com/post/html-escaping) to reduce - * XSS vectors. - * - * @static - * @since 0.1.0 - * @memberOf _ - * @category String - * @param {string} [string=''] The string to escape. - * @returns {string} Returns the escaped string. - * @example - * - * _.escape('fred, barney, & pebbles'); - * // => 'fred, barney, & pebbles' - */ - function escape(string) { - string = toString(string); - return (string && reHasUnescapedHtml.test(string)) - ? string.replace(reUnescapedHtml, escapeHtmlChar) - : string; - } - - /** - * Escapes the `RegExp` special characters "^", "$", "\", ".", "*", "+", - * "?", "(", ")", "[", "]", "{", "}", and "|" in `string`. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category String - * @param {string} [string=''] The string to escape. - * @returns {string} Returns the escaped string. - * @example - * - * _.escapeRegExp('[lodash](https://lodash.com/)'); - * // => '\[lodash\]\(https://lodash\.com/\)' - */ - function escapeRegExp(string) { - string = toString(string); - return (string && reHasRegExpChar.test(string)) - ? string.replace(reRegExpChar, '\\$&') - : string; - } - - /** - * Converts `string` to - * [kebab case](https://en.wikipedia.org/wiki/Letter_case#Special_case_styles). - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category String - * @param {string} [string=''] The string to convert. - * @returns {string} Returns the kebab cased string. - * @example - * - * _.kebabCase('Foo Bar'); - * // => 'foo-bar' - * - * _.kebabCase('fooBar'); - * // => 'foo-bar' - * - * _.kebabCase('__FOO_BAR__'); - * // => 'foo-bar' - */ - var kebabCase = createCompounder(function(result, word, index) { - return result + (index ? '-' : '') + word.toLowerCase(); - }); - - /** - * Converts `string`, as space separated words, to lower case. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category String - * @param {string} [string=''] The string to convert. - * @returns {string} Returns the lower cased string. - * @example - * - * _.lowerCase('--Foo-Bar--'); - * // => 'foo bar' - * - * _.lowerCase('fooBar'); - * // => 'foo bar' - * - * _.lowerCase('__FOO_BAR__'); - * // => 'foo bar' - */ - var lowerCase = createCompounder(function(result, word, index) { - return result + (index ? ' ' : '') + word.toLowerCase(); - }); - - /** - * Converts the first character of `string` to lower case. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category String - * @param {string} [string=''] The string to convert. - * @returns {string} Returns the converted string. - * @example - * - * _.lowerFirst('Fred'); - * // => 'fred' - * - * _.lowerFirst('FRED'); - * // => 'fRED' - */ - var lowerFirst = createCaseFirst('toLowerCase'); - - /** - * Pads `string` on the left and right sides if it's shorter than `length`. - * Padding characters are truncated if they can't be evenly divided by `length`. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category String - * @param {string} [string=''] The string to pad. - * @param {number} [length=0] The padding length. - * @param {string} [chars=' '] The string used as padding. - * @returns {string} Returns the padded string. - * @example - * - * _.pad('abc', 8); - * // => ' abc ' - * - * _.pad('abc', 8, '_-'); - * // => '_-abc_-_' - * - * _.pad('abc', 3); - * // => 'abc' - */ - function pad(string, length, chars) { - string = toString(string); - length = toInteger(length); - - var strLength = length ? stringSize(string) : 0; - if (!length || strLength >= length) { - return string; - } - var mid = (length - strLength) / 2; - return ( - createPadding(nativeFloor(mid), chars) + - string + - createPadding(nativeCeil(mid), chars) - ); - } - - /** - * Pads `string` on the right side if it's shorter than `length`. Padding - * characters are truncated if they exceed `length`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category String - * @param {string} [string=''] The string to pad. - * @param {number} [length=0] The padding length. - * @param {string} [chars=' '] The string used as padding. - * @returns {string} Returns the padded string. - * @example - * - * _.padEnd('abc', 6); - * // => 'abc ' - * - * _.padEnd('abc', 6, '_-'); - * // => 'abc_-_' - * - * _.padEnd('abc', 3); - * // => 'abc' - */ - function padEnd(string, length, chars) { - string = toString(string); - length = toInteger(length); - - var strLength = length ? stringSize(string) : 0; - return (length && strLength < length) - ? (string + createPadding(length - strLength, chars)) - : string; - } - - /** - * Pads `string` on the left side if it's shorter than `length`. Padding - * characters are truncated if they exceed `length`. - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category String - * @param {string} [string=''] The string to pad. - * @param {number} [length=0] The padding length. - * @param {string} [chars=' '] The string used as padding. - * @returns {string} Returns the padded string. - * @example - * - * _.padStart('abc', 6); - * // => ' abc' - * - * _.padStart('abc', 6, '_-'); - * // => '_-_abc' - * - * _.padStart('abc', 3); - * // => 'abc' - */ - function padStart(string, length, chars) { - string = toString(string); - length = toInteger(length); - - var strLength = length ? stringSize(string) : 0; - return (length && strLength < length) - ? (createPadding(length - strLength, chars) + string) - : string; - } - - /** - * Converts `string` to an integer of the specified radix. If `radix` is - * `undefined` or `0`, a `radix` of `10` is used unless `value` is a - * hexadecimal, in which case a `radix` of `16` is used. - * - * **Note:** This method aligns with the - * [ES5 implementation](https://es5.github.io/#x15.1.2.2) of `parseInt`. - * - * @static - * @memberOf _ - * @since 1.1.0 - * @category String - * @param {string} string The string to convert. - * @param {number} [radix=10] The radix to interpret `value` by. - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.map`. - * @returns {number} Returns the converted integer. - * @example - * - * _.parseInt('08'); - * // => 8 - * - * _.map(['6', '08', '10'], _.parseInt); - * // => [6, 8, 10] - */ - function parseInt(string, radix, guard) { - if (guard || radix == null) { - radix = 0; - } else if (radix) { - radix = +radix; - } - return nativeParseInt(toString(string).replace(reTrimStart, ''), radix || 0); - } - - /** - * Repeats the given string `n` times. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category String - * @param {string} [string=''] The string to repeat. - * @param {number} [n=1] The number of times to repeat the string. - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.map`. - * @returns {string} Returns the repeated string. - * @example - * - * _.repeat('*', 3); - * // => '***' - * - * _.repeat('abc', 2); - * // => 'abcabc' - * - * _.repeat('abc', 0); - * // => '' - */ - function repeat(string, n, guard) { - if ((guard ? isIterateeCall(string, n, guard) : n === undefined)) { - n = 1; - } else { - n = toInteger(n); - } - return baseRepeat(toString(string), n); - } - - /** - * Replaces matches for `pattern` in `string` with `replacement`. - * - * **Note:** This method is based on - * [`String#replace`](https://mdn.io/String/replace). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category String - * @param {string} [string=''] The string to modify. - * @param {RegExp|string} pattern The pattern to replace. - * @param {Function|string} replacement The match replacement. - * @returns {string} Returns the modified string. - * @example - * - * _.replace('Hi Fred', 'Fred', 'Barney'); - * // => 'Hi Barney' - */ - function replace() { - var args = arguments, - string = toString(args[0]); - - return args.length < 3 ? string : string.replace(args[1], args[2]); - } - - /** - * Converts `string` to - * [snake case](https://en.wikipedia.org/wiki/Snake_case). - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category String - * @param {string} [string=''] The string to convert. - * @returns {string} Returns the snake cased string. - * @example - * - * _.snakeCase('Foo Bar'); - * // => 'foo_bar' - * - * _.snakeCase('fooBar'); - * // => 'foo_bar' - * - * _.snakeCase('--FOO-BAR--'); - * // => 'foo_bar' - */ - var snakeCase = createCompounder(function(result, word, index) { - return result + (index ? '_' : '') + word.toLowerCase(); - }); - - /** - * Splits `string` by `separator`. - * - * **Note:** This method is based on - * [`String#split`](https://mdn.io/String/split). - * - * @static - * @memberOf _ - * @since 4.0.0 - * @category String - * @param {string} [string=''] The string to split. - * @param {RegExp|string} separator The separator pattern to split by. - * @param {number} [limit] The length to truncate results to. - * @returns {Array} Returns the string segments. - * @example - * - * _.split('a-b-c', '-', 2); - * // => ['a', 'b'] - */ - function split(string, separator, limit) { - if (limit && typeof limit != 'number' && isIterateeCall(string, separator, limit)) { - separator = limit = undefined; - } - limit = limit === undefined ? MAX_ARRAY_LENGTH : limit >>> 0; - if (!limit) { - return []; - } - string = toString(string); - if (string && ( - typeof separator == 'string' || - (separator != null && !isRegExp(separator)) - )) { - separator = baseToString(separator); - if (!separator && hasUnicode(string)) { - return castSlice(stringToArray(string), 0, limit); - } - } - return string.split(separator, limit); - } - - /** - * Converts `string` to - * [start case](https://en.wikipedia.org/wiki/Letter_case#Stylistic_or_specialised_usage). - * - * @static - * @memberOf _ - * @since 3.1.0 - * @category String - * @param {string} [string=''] The string to convert. - * @returns {string} Returns the start cased string. - * @example - * - * _.startCase('--foo-bar--'); - * // => 'Foo Bar' - * - * _.startCase('fooBar'); - * // => 'Foo Bar' - * - * _.startCase('__FOO_BAR__'); - * // => 'FOO BAR' - */ - var startCase = createCompounder(function(result, word, index) { - return result + (index ? ' ' : '') + upperFirst(word); - }); - - /** - * Checks if `string` starts with the given target string. - * - * @static - * @memberOf _ - * @since 3.0.0 - * @category String - * @param {string} [string=''] The string to inspect. - * @param {string} [target] The string to search for. - * @param {number} [position=0] The position to search from. - * @returns {boolean} Returns `true` if `string` starts with `target`, - * else `false`. - * @example - * - * _.startsWith('abc', 'a'); - * // => true - * - * _.startsWith('abc', 'b'); - * // => false - * - * _.startsWith('abc', 'b', 1); - * // => true - */ - function startsWith(string, target, position) { - string = toString(string); - position = position == null - ? 0 - : baseClamp(toInteger(position), 0, string.length); - - target = baseToString(target); - return string.slice(position, position + target.length) == target; - } - - /** - * Creates a compiled template function that can interpolate data properties - * in "interpolate" delimiters, HTML-escape interpolated data properties in - * "escape" delimiters, and execute JavaScript in "evaluate" delimiters. Data - * properties may be accessed as free variables in the template. If a setting - * object is given, it takes precedence over `_.templateSettings` values. - * - * **Note:** In the development build `_.template` utilizes - * [sourceURLs](http://www.html5rocks.com/en/tutorials/developertools/sourcemaps/#toc-sourceurl) - * for easier debugging. - * - * For more information on precompiling templates see - * [lodash's custom builds documentation](https://lodash.com/custom-builds). - * - * For more information on Chrome extension sandboxes see - * [Chrome's extensions documentation](https://developer.chrome.com/extensions/sandboxingEval). - * - * @static - * @since 0.1.0 - * @memberOf _ - * @category String - * @param {string} [string=''] The template string. - * @param {Object} [options={}] The options object. - * @param {RegExp} [options.escape=_.templateSettings.escape] - * The HTML "escape" delimiter. - * @param {RegExp} [options.evaluate=_.templateSettings.evaluate] - * The "evaluate" delimiter. - * @param {Object} [options.imports=_.templateSettings.imports] - * An object to import into the template as free variables. - * @param {RegExp} [options.interpolate=_.templateSettings.interpolate] - * The "interpolate" delimiter. - * @param {string} [options.sourceURL='lodash.templateSources[n]'] - * The sourceURL of the compiled template. - * @param {string} [options.variable='obj'] - * The data object variable name. - * @param- {Object} [guard] Enables use as an iteratee for methods like `_.map`. - * @returns {Function} Returns the compiled template function. - * @example - * - * // Use the "interpolate" delimiter to create a compiled template. - * var compiled = _.template('hello <%= user %>!'); - * compiled({ 'user': 'fred' }); - * // => 'hello fred!' - * - * // Use the HTML "escape" delimiter to escape data property values. - * var compiled = _.template('<%- value %>'); - * compiled({ 'value': '