Plate replicate rollups and AbDisc assay re-import

[labkey-klum]

Rationale

Introduces support for the calculation of standard deviation and mean values for replicate well groups for protocols that support plate metadata in LKB folders. Replicate aggregates are computed for numerical measures in the results domain and are automatically joined to the results views.

In support of LKB AB discovery scenarios (and potentially others), this also introduces the notion of the addition of assay run data over time through the existing assay run re-import feature. Plate set plate data can be added to a run over multiple imports. The server will attempt to combine / merge prior data with new data on a plate boundary within a single plate set during each subsequent import / re-import.

Changes

• Create a protocol schema domain : AssayPlateReplicateStats to store the replicate standard deviation and mean values. One row per replicate well group per run will be created.
• When the assay results domain changes, create fields in the AssayPlateReplicateStats domain to track any numeric measures.
• On insert, compute and save replicate stat values to the protocol schema table.
• Join the assay results table to the replicate stats table.
• AssayPlateTriggerFactory to handle updates and deletes on the results table. Depending on whether the updates occur on replicate well group rows, recompute and update replicate stats. Unfortunately, assays handle inserts and updates via different code paths, the former happens in
• AssayRunUploadContext.ReImportOption enum to control the server side re-import behavior. The parameter can be passed into the ImportRunApiAction and defaults to the current behavior of REPLACE.
• AssayPlateMetadataServiceImpl.mergeReRunData handles the details of merging current and previous results data. If data is merged, the uploaded data is replaced with the tsv tabular representation of the combined data and the data output ExpData is updated to reflect the change. This is always the case regardless of whether previous or current data is in the graphical format.
• Hit selections are now cleared from the previous run regardless of whether this is a straight re-import or the ABDisc re-import where plate set data can potentially be merged.
• Hit selections are propagated in the merge case on a plate by plate basis if the new data is not replacing any existing plate data. The implementation was a little disjointed so here are the details:
  • Hit re-selection needs to be done after new assay results have been persisted because we need the result row IDs. This happens in PlateMetadataImportHelper.afterBatchInsert.
  • We only know about the incoming versus existing plate data during the parse phase of the new data file. This happens in AssayPlateMetadataServiceImpl.mergeReRunData. We end up removing all hit selections that we do not plan to carry forward based on our plate merge policy. Any remaining hits are carried forward in the ImportHelper.

Debug Notes

• Until the client dev is in place, the assay run re-import behavior can be controlled by changing the default reImportOption in the ImportRunApiForm around ImportRunApiAction:356

implementation Details

Statistical values should appear automatically in the default view of the results table. Stats fields should be aligned close to their source.

After the re-import case if data is combined with a previous run

Related Pull Requests

LabKey/commonAssays#813

[labkey-chrisj]

One thing we may want to do is apply a standard format for decimal/float-based columns; with the default format we end up not rounding numbers we probably ought to round in our mean/stddev calculation column: