Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BIOMAGE-1774] - Check sample file sizes #773

Merged
merged 18 commits into from
Jul 21, 2022
Merged

[BIOMAGE-1774] - Check sample file sizes #773

merged 18 commits into from
Jul 21, 2022

Conversation

aerlaut
Copy link
Contributor

@aerlaut aerlaut commented Jul 15, 2022

Description

Add a check for sample files. Check if samples have the correct barcodes, features and matrix files.

To test, you'll need to modify the some of your samples to trigger the errors:

  • Invalid barcodes: barcodes has more/less lines than record in matrix. Edit a barcodes.tsv and increase/delete lines.
  • Invalid features: features has more/less lines than recorded in matrix. Edit a features.tsv and increase/delete lines.
  • Transposed matrix: happens if the number of expected features and barcode files are swapped. Edit a matrix.mtx file and swap the 1st and 2nd entry of the 3rd line (e.g. 100 200 3000 -> 200 100 3000).

If you need help in testing, ask Agi.

Details

URL to issue

https://biomage.atlassian.net/browse/BIOMAGE-1774

Link to staging deployment URL (or set N/A)

https://ui-agi-lumpy-flounder.scp-staging.biomage.net/

Links to any PRs or resources related to this PR

N/A

Integration test branch

master

Merge checklist

Your changes will be ready for merging after all of the steps below have been completed.

Code updates

Have best practices and ongoing refactors being observed in this PR

  • Migrated any selector / reducer used to the new format.

Manual/unit testing

  • Tested changes using InfraMock locally or no tests required for change, e.g. Kubernetes chart updates.
  • Validated that current unit tests for code work as expected and are sufficient for code coverage or no unit tests required for change, e.g. documentation update.
  • Unit tests written or no unit tests required for change, e.g. documentation update.

Integration testing

You must check the box below to run integration tests on the latest commit on your PR branch.
Integration tests have to pass before the PR can be merged. Without checking the box, your PR
will not pass the required status checks for merging.

  • Started end-to-end tests on the latest commit.

Documentation updates

  • Relevant Github READMEs updated or no GitHub README updates required.
  • Relevant Wiki pages created/updated or no Wiki updates required.

Optional

  • Staging environment is unstaged before merging.
  • Photo of a cute animal attached to this PR.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 15, 2022

📦 Next.js Bundle Analysis

This analysis was generated by the next.js bundle analysis action 🤖

⚠️ Global Bundle Size Increased

Page Size (compressed)
global 522.38 KB (🟡 +462 B)
Details

The global bundle is the javascript bundle that loads alongside every page. It is in its own category because its impact is much higher - an increase to its size means that every page on your website loads slower, and a decrease means every page loads faster.

Any third party scripts you have added directly to your app using the <script> tag are not accounted for in this analysis

If you want further insight into what is behind the changes, give @next/bundle-analyzer a try!

Two Pages Changed Size

The following pages changed size from the code in this PR compared to its base branch:

Page Size (compressed) First Load
/data-management 248.56 KB (🟡 +1.79 KB) 770.94 KB
/experiments/[experimentId]/data-processing 497.14 KB (-1 B) 1019.51 KB
Details

Only the gzipped size is provided here based on an expert tip.

First Load is the size of the global bundle plus the bundle for the individual page. If a user were to show up to your website and land on a given page, the first load size represents the amount of javascript that user would need to download. If next/link is used, subsequent page loads would only need to download that page's bundle (the number in the "Size" column), since the global bundle has already been downloaded.

Any third party scripts you have added directly to your app using the <script> tag are not accounted for in this analysis

Next to the size is how much the size has increased or decreased compared with the base branch of this PR. If this percentage has increased by 20% or more, there will be a red status indicator applied, indicating that special attention should be given to this.

@codecov
Copy link

codecov bot commented Jul 15, 2022

Codecov Report

Merging #773 (8ed149c) into master (1fb19d7) will increase coverage by 0.09%.
The diff coverage is 87.67%.

@@            Coverage Diff             @@
##           master     #773      +/-   ##
==========================================
+ Coverage   83.74%   83.83%   +0.09%     
==========================================
  Files         471      472       +1     
  Lines        7934     8000      +66     
  Branches     1531     1551      +20     
==========================================
+ Hits         6644     6707      +63     
+ Misses       1238     1235       -3     
- Partials       52       58       +6     
Impacted Files Coverage Δ
src/redux/actions/samples/loadSamples.js 96.55% <ø> (ø)
src/utils/pushNotificationMessage.js 4.76% <14.28%> (ø)
src/utils/upload/sampleValidator.js 95.08% <95.08%> (ø)
src/utils/upload/processUpload.js 82.35% <100.00%> (+1.40%) ⬆️

src/utils/upload/sampleInspector.js Outdated Show resolved Hide resolved
src/utils/upload/processUpload.js Outdated Show resolved Hide resolved
@cosa65
Copy link
Member

cosa65 commented Jul 18, 2022

with the (valid) WT1 sample I am getting that the features file is invalid.

I also tried with some bigger experiments and I nothing seems to happen (I don't get any invalid error but the samples don't show up either)

@kafkasl kafkasl force-pushed the check-file-sizes branch from afb774a to 36dd920 Compare July 20, 2022 15:07
Comment on lines +86 to +92
const [
expectedNumFeatures,
expectedNumBarcodes,
] = await extractSampleSizes(matrix);

const numBarcodesFound = await getNumLines(barcodes);
const numFeaturesFound = await getNumLines(features);
Copy link
Contributor Author

@aerlaut aerlaut Jul 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are error messages for invalid matrix file size, but I don't see it getting checked here. Are we checking the matrix file size? @kafkasl

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, I removed it

@kafkasl kafkasl merged commit 17963e6 into master Jul 21, 2022
@kafkasl kafkasl deleted the check-file-sizes branch July 21, 2022 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants