Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove Annotated MAF before Import #958

Merged
merged 2 commits into from
Jan 19, 2024
Merged

Remove Annotated MAF before Import #958

merged 2 commits into from
Jan 19, 2024

Conversation

averyniceday
Copy link
Collaborator

think this is related to the file renaming effort from a while back where we merge all data_mutations* into one profile - data_mutations_annotated wasn't getting imported but now it's getting merged into data_mutations_extended.

To fix this we just make sure to remove the mutations_annotated file from the dir as part of processing

@averyniceday averyniceday changed the title remove annotated MAF to prevent duplicate Remove Annotated MAF before Import May 17, 2022
@averyniceday averyniceday requested a review from sheridancbio May 17, 2022 16:18
@averyniceday averyniceday mentioned this pull request May 17, 2022
Closed
sheridancbio
sheridancbio requested changes May 20, 2022
View reviewed changes
Copy link
Contributor

@sheridancbio sheridancbio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rather than duplicating file contents and then removing the old file, I think we can just use the equivalent of unix "mv"

import-scripts/subset_and_merge_crdb_pdx_studies.py Outdated
@@ -581,6 +581,7 @@ def annotate_maf(destination_to_source_mapping, root_directory, annotator_jar):
# otherwise remove annotated maf if exists in destination study directory
if annotator_status == 0:
shutil.copy(annot_maf, orig_maf)
os.remove(annot_maf)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better/faster/simpler to change:
shutil.copy(anot_maf, orig_maf)
os.remove(annot_maf)
to:
os.replace(annot_maf, orig_maf)

@averyniceday averyniceday merged commit 9774ed9 into knowledgesystems:master Jan 19, 2024
@averyniceday averyniceday deleted the fixpdx branch January 19, 2024 21:07
sheridancbio pushed a commit to sheridancbio/cmo-pipelines that referenced this pull request Feb 5, 2024
@averyniceday @sheridancbio
Remove Annotated MAF before Import (knowledgesystems#958)
ee7b0b4 
* remove annotated MAF to prevent duplicate

* Update subset_and_merge_crdb_pdx_studies.py

---------

Co-authored-by: Avery Wang <[email protected]>
sheridancbio pushed a commit to sheridancbio/cmo-pipelines that referenced this pull request Feb 5, 2024
@averyniceday @sheridancbio
Remove Annotated MAF before Import (knowledgesystems#958)
0dab8ac 
* remove annotated MAF to prevent duplicate

* Update subset_and_merge_crdb_pdx_studies.py

---------

Co-authored-by: Avery Wang <[email protected]>
sheridancbio pushed a commit to sheridancbio/cmo-pipelines that referenced this pull request Feb 9, 2024
@averyniceday @sheridancbio
Remove Annotated MAF before Import (knowledgesystems#958)
16b4b42 
* remove annotated MAF to prevent duplicate

* Update subset_and_merge_crdb_pdx_studies.py

---------

Co-authored-by: Avery Wang <[email protected]>
callachennault pushed a commit to callachennault/cmo-pipelines that referenced this pull request Feb 12, 2024
@averyniceday @callachennault
Remove Annotated MAF before Import (knowledgesystems#958)
2b15bcc 
* remove annotated MAF to prevent duplicate

* Update subset_and_merge_crdb_pdx_studies.py

---------

Co-authored-by: Avery Wang <[email protected]>
callachennault pushed a commit to averyniceday/cmo-pipelines that referenced this pull request Feb 20, 2024
@averyniceday @callachennault
Remove Annotated MAF before Import (knowledgesystems#958)
ce77680 
* remove annotated MAF to prevent duplicate

* Update subset_and_merge_crdb_pdx_studies.py

---------

Co-authored-by: Avery Wang <[email protected]>
callachennault pushed a commit to averyniceday/cmo-pipelines that referenced this pull request Feb 20, 2024
@averyniceday @callachennault
Remove Annotated MAF before Import (knowledgesystems#958)
c61c14e 
* remove annotated MAF to prevent duplicate

* Update subset_and_merge_crdb_pdx_studies.py

---------

Co-authored-by: Avery Wang <[email protected]>
sheridancbio pushed a commit to mandawilson/cmo-pipelines that referenced this pull request Mar 27, 2024
@mandawilson @sheridancbio
parent 556e9b9
173f341 
author Manda Wilson <[email protected]> 1703199176 -0500
committer Robert Sheridan <[email protected]> 1711560265 -0400

upgrade to java 21

switch to genome-nexus-annotation-pipeline that uses new maf repo

updated to spring 6, spring batch 5, spring boot 3 to match cbioportal

fix typos

Updates to AZ-MSKIMPACT to integrate with CDM (knowledgesystems#1098)

Fix bug in checking for duplicate Mutation Records (knowledgesystems#1099)

* Check if mutationRecord is duplicated before annotating

* Populate mutationMap in loadMutationRecordsFromJson

* add addRecordToMap

* Remove comments, add local vars for debugging

* Remove duplicate MAF variants for AZ

* Fix remove-duplicate-maf-variants call

* revert whitespace change

updates for migrating darwin and crdb to java11 (knowledgesystems#1080)

pom changes for pulling moved dependencies
changes to java args to silence warnings

Co-authored-by: cbioportal import user <[email protected]>

Remove Annotated MAF before Import (knowledgesystems#958)

* remove annotated MAF to prevent duplicate

* Update subset_and_merge_crdb_pdx_studies.py

---------

Co-authored-by: Avery Wang <[email protected]>

Script to combine arbitrary files (knowledgesystems#1104)

* Script to combine arbitrary files

* Modify unit tests to work with script changes

* Remove unnecessary column specifier

* Fix syntax bug

Add sophia script (knowledgesystems#1105)

* Add sophia script

* rename transpose_cna file

* Add filter-clinical-arg-functions script

* Add az var to correct automation environment

* Add correct path to transpose_cna script

* Call seq_date function

* Add seq_date before filtering columns

* syntax fix

* Fix call to filter out clinical attribute columns

* Fix nonsigned out file path

* Automate folder name

* directory fixes

* remove quotes?

* change date formatting

* output filepath for duplicate variants script

* use az_msk_impact_data_home var

* move sophia_data_home to automation environment

* Add comments

* Change dir structures in sophia script to match new repo structure

* Add git operations

* Remove test file

* Fix dirs for sophia zip command

* remove quotes

* Zip files before cleanup

* move zip step before git push

Add script for merging Dremio/SMILE into cmo-access (knowledgesystems#1102)

- adds cfdna clinical and timeline data from dremio/SMILE
- converts patient identifiers using "dmp over cmo" identifier logic from dremio
- dremio patient id mapping table export code called to produce mapping table
- main script then calls update_cfdna_clinical_sample_patient_ids_via_dremio.sh
- merge.py used to combine clinical data from dremio with clinical data from cmo-access
- metadata headers added using new script : merge_clinical_metadata_headers_py3.py
- other import process flow (similar to other import scripts) followed
- error detection step added after debugging for sporadic data loss in results

Co-authored-by: Manda Wilson <[email protected]>

Modify preconsume script to work on one cohort at a time (knowledgesystems#1107)

Call correct function name

add options for logging in for different accounts

Preconsume archer-solid-cv4 and add fetch loop (knowledgesystems#1129)

* Handle archer-solid-cv4 samples
* Add loop
* move each cohort to its own dir and fix filename

switch to genome-nexus-annotation-pipeline that uses new maf repo

use updated genome-nexus-annotation-pipeline

update version of cmo-pipelines to 1.0.0

Convert BatchConfiguration to new Spring Batch format

drop unneeded dependency from redcap

removed gdd, updated crdb and ddp batch configs to spring batch 5

removed commons-lang

start of converting cvr to spring batch 5

fix cvr fetcher BatchConfiguration

fixed redcap pipeline spring batch 5 configuration

make spring-batch-integration match batch version

Co-authored-by: Manda Wilson <[email protected]>

drop darwin fetcher (and docs/scripts)
mandawilson pushed a commit to mandawilson/cmo-pipelines that referenced this pull request Mar 27, 2024
@averyniceday @mandawilson
Remove Annotated MAF before Import (knowledgesystems#958)
e32366d 
* remove annotated MAF to prevent duplicate

* Update subset_and_merge_crdb_pdx_studies.py

---------

Co-authored-by: Avery Wang <[email protected]>
mandawilson pushed a commit to mandawilson/cmo-pipelines that referenced this pull request Mar 27, 2024
@averyniceday @mandawilson
Remove Annotated MAF before Import (knowledgesystems#958)
8128c92 
* remove annotated MAF to prevent duplicate

* Update subset_and_merge_crdb_pdx_studies.py

---------

Co-authored-by: Avery Wang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sheridancbio sheridancbio sheridancbio requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@averyniceday @sheridancbio